1. Arent Janszoon Ernststraat 595-H NL-1082 LD Amsterdam info@sig.nl www.sig.nl December 12th, 2008 Static Estimation of Test Coverage Benevol’08 - Session 5 : Analysis Tiago Alves & Joost Visser

2. I 262 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Software Improvement Group Characterization Based in Amsterdam, The Netherlands Spin-off from the Centre of Mathematics and Information Technology (CWI) Fact-based IT consulting Services Software Risk Assessment Exhaustive research of software quality and risks Answers specific research questions One time execution Software Monitor Automated quality measurements executed frequently (daily / weekly) Information presented in a web-portal DocGen Automated generation of technical quality

3. I 263 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Our Customers Financial and Insurance Companies IT Logistical Other Government

4. I 264 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Software testing Production code Test code Method under test Unit test Test execution Test failure Test coverage

5. I 265 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Measuring test coverage Pros: Indicator for test quality Indicator for quality of the software under test Higher coverage => better software quality (in principle) Cons: Full installation required (sources + libraries) Instrumentation of source/byte code Problematic in embedded systems Execution (Hardware and time constrains) Not appropriate to compute in the context of software quality assessment!!

6. I 266 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Research Challenge Motivation 13th Testdag, Delft, November 2007 I. Heitlager, T. Kuipers, J. Visser “Observing unit test maturity in the wild” Research questions Is it possible to estimate test coverage without running tests? What trade-offs can be made between sophistication and accuracy? Requirements Use only static analysis Scale to large systems Robust against incomplete systems

7. I 267 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Where do we stand? Solution sketch Sources of imprecision Dealing with imprecision Specification Implementation Validation

8. I 268 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Solution sketch 1.Extract Extract structural and call information Determine set of test classes 2.Slice (modified) Slice graph starting from test methods Set of methods reached from test code Take into account class initializer calls 3.Count (per class) Determine number of defined methods Determine number of covered methods 4.Estimate Class coverage Package coverage System coverage

9. I 269 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Modified slicing specification

10. I 26 Code coverage formulas 10 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Defined methods: Covered methods:

11. I 2611 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves What can go wrong? (Sources of imprecision) Java language Control flow Dynamic dispatch (inheritance) Overloading General issues Frameworks / Libraries call backs Identification of test code ///CLOVER:OFF flags

12. I 2612 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Dealing with imprecision Pessimistic approach Report only what can be determined to be true False negatives Estimates lower bound for coverage Optimistic approach Report everything that might be true False positives Estimates upper bound for coverage Pessimistic vs. Optimistic (software assessment context) Pessimistic will always report low coverage Optimistic will be sensitive to lack of coverage Optimistic will not take into account library calls

13. I 2613 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Where do we stand? Code query technologies Definition of abstractions Querying the results Specification Implementation Validation

14. I 26 Code query technologies 14 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves

15. I 2615 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Definition of abstractions Class extension class AnalyzedClass extends Class { AnalyzedClass() { this.fromSource() and... } class TestClass extends AnalyzedClass { TestClass() { isJUnitClassTest(this) } } class CodeClass extends AnalyzedClass { CodeClass() { not isJUnitClassTest(this) }... int numberOfDefinedMethods() { result = count( NonAbstractCallable m | this.containsCallable(m)) } int numberOfCoveredMethods() { result = count( NonAbstractCallable m | this.containsCallable(m) and m.isTestCovered()) }

16. I 2616 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Querying the results Class-level query from CodeClass c select c.getQualifiedName() as ClassName, c.numberOfCoveredMethods() as NumberOfCoveredMethods, c.numberOfDefinedMethods() as NumberOfDefinedMethods order by ClassName

17. I 26 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Querying the results Class-level query results for JPacMan Code query Code query results Total # methods # Covered methods -

18. I 2618 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Where do we stand? Experimental design Data set characterization Comparison of results Specification Implementation Validation

19. I 2619 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Experimental design Data set selection and characterization Open-source and proprietary Java systems Different application domains Different sizes Execution of experiment Clover execution (configuring clover + ant, running tests) XML Clover extraction (XSLT transformations for CSV generation) SemmleCode execution (text file export + scripts for CSV generation) Custom built java tool to read CSV files and generate Excel XLS

20. I 26 Statistical analysis Distributions Histogram of the coverage estimation Histogram of the real (clover) coverage Correlation Spearman (rank-correlation) Estimation different Histogram of the differences Dispersion Inter-quartile ranges (dispersion) 20 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves

21. I 2621 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Data set characterization SystemVersionAuthorLOC # Packages # Classes # Methods JPacMan3.0.4 Arie van Deursen 2.5k346335 Certification 2008073 1 SIG3.8k1499413 G System 2008021 4 C Company6.4k17126789 Dom4j1.6.1MetaStuff24.3k142713,606 Utils1.61SIG37.7k375064,533 JGap3.3.3Klaus Meffert42.9k274514,995 Collections3.2.1Apache55.4k127146,974 PMD5.0b6340 Xavier Le Vourch 62.8k1108946,856 R System 2008021 4 C Company82.3k6697611,095 JFreeChart1.0.10JFree 127.7 k 6087510,680 DocGenr40981SIG 127.7 k 1121,78614,909 Analysis1.39SIG 267.5 k 2843,19922,315

22. I 2622 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Statistical analysis (Class and package coverage comparison) System name SpearmanMedianInterquartile range ClassPackageClassPackageClassPackage JPacMan0.467*10-0.1300.037- Certification0.368**0.5200000.015 G System0.774**0.694**0000.045 Dom4j0.584**0.620**0.1670.1180.3330.220 Utils0.825**0.778**00.01400.100 JGap0.733**0.786**000.4330.125 Collections0.549**0.776**00.0490.0270.062 PMD0.638**0.655**00.0580.0970.166 R System0.727**0.723**0-0.0790.0430.162 JFreeChart0.632**0.694**00.0480.1750.172 DocGen0.397**0.459**00.1000.4000.386 Analysis0.391**0.486**0-0.0160.3330.316

23. I 2623 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Statistical analysis (System-level coverage comparison) SystemStaticCloverDiff JPacMan 88.06%93.53%-5.47% Certification 92.82%90.09%2.73% G System 89.61%94.81%-5.19% Dom4j 57.40%39.37%18.03% Utils 74.95%70.47%4.48% JGap 70.51%50.99%19.52% Collections 82.62%78.39%4.23% PMD 80.10%70.76%9.34% R System 65.10%72.65%-7.55% JFreeChart 69.88%61.55%8.33% DocGen 79.92%69.08%10.84% Analysis 71.74%88.23%-16.49% Correlation: 0.769

24. I 2624 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Utils revisions

25. I 2625 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Conclusion Is it possible to determine test coverage without running tests? Yes we can!!! Spearman: high correlation between static and clover coverage In general static coverage identifies the same values as Clover What trade-offs can be made between sophistication and accuracy? Average absolute difference: 9% Class and Package coverage needs further improvement Implementation SemmleCode: 92 LOC Java SIG Analysis: 256 LOC

26. I 2626 IPA Herfstdagen, Nunspeet, November 24-28, 2008 Static Estimation of Test Coverage Tiago Alves Thank you! Questions? Tiago Alves t.alves@sig.nl Joost Visser j.visser@sig.nl