1. EMI INFSO-RI-261611 Software Metric Definitions, Reports and Analysis in EMI Authors: Eamonn Kenny (TCD), Gianni Pucciani (CERN) Date: Tuesday 12 th April 2011 1

2. EMI INFSO-RI-261611 Measure quality/quantity of the software. Encourage conformance between the 4 middleware groups. Highlight previously unlooked at problems (e.g: using static analysers) 2 Why do we need metrics?

3. EMI INFSO-RI-261611 To measure: – Performance: e.g: How long to fix a bug? – Complexity: e.g: How many fixed bugs? – Support: e.g: How complex is a component? E.g: Cyclomatic Complexity and SLOC EMI is funded to provide quality software and to measure its quality With metrics one can see trends and understand what needs to be improved and how many defects a component has. 3 Why do we need metrics?

4. EMI INFSO-RI-261611 Metrics Role in EMI SA2 4 Area Leaders Product Teams (PT) SA2.3 Metrics SA2.2 QA Plan SA2.4 Tools SA2.5 QA Review Top 3 Metrics?Existing Middleware Metrics? Tool Definition Verification SA1 Quality Control Evaluation Definitions & Analysis EMT Weekly Meetings SA2 SA1 JRA1 Report Generation SA2.1 Coordination PEB link Survey Weekly report Define & Structure

5. EMI INFSO-RI-261611 5 Model for Metric Collection Practical metrics, not a theoretical set! Governed by the end goals of the project End-Goals Suggests a Question Leads to a Metric

6. EMI INFSO-RI-261611 6 Metric Template Metrics Id E.g: PriorityBugs Name Description Measurement Calculation Mathematical formula Input(s) & Units E.g: time range in days Output(s) & Units E.g: Average time in hours Scope Thresholds/Target Value Tools Availability Per middleware availability Goals Quality factor Follows the McCall factors Risks Special Notes

7. EMI INFSO-RI-261611 Metrics Description Example 7

8. EMI INFSO-RI-261611 Areas of CoverageTypes of Metrics Process Management (Bug-tracking related) Relates to priority, severity, open/closed bugs, state changes in bugs, improving turnaround times External:Quality in Use (Bug-tracking/EGI) 3 rd level GGUS related metrics (KPIs for SA1) Product Team software - Static Analysers Language specific analysers, SLOC, CLOC, Cyclomatic Complexity, etc. Product Team software - Testing, platform, etc. Unit tests, supported platforms, bug density Optional Add-on ToolsValgrind/Helgrind (Memory leak & thread checking) 8 Metric Categories

9. EMI INFSO-RI-261611 Average Time To Close a Bug (PriorityBugs, BugSeverityDistribution) Time To Fix a Bug (SA1-QC) Bug-tracking Process 9 Open Priority Bugs Open Is Accepted? YES Accepted Fixed Can I test it? Test successful? Not Tested Closed Rejected NO YES NO YES Suggested Metrics Untouched Open Bugs (DSA 1.1 requirement)

10. EMI INFSO-RI-261611 Exposed XML to ETICS (run anytime) Middleware Bug- Mapping (needs periodic intervention) Middleware Bug- tracking (Snapshot taken daily) 10 Handling Multiple Bug-Trackers ARC Bugzilla dCache Request Tracker (RT) gLite Savannah UNICORE SourceForge Multiple bug-trackers One common interface

11. EMI INFSO-RI-261611 Metric NameDesignedImplementedRecipient Open Priority Bugs (High/Immediate) EMT/Reports Successful Builds metric Open Untouched Bugs (> 14 days) Fixed Bugs SA1-QC Priority Bugs (High/Immediate) Periodic Reports/ Deliverables/ Product Team Reports Bug Severity Distribution Backlog Management Index Integration test effectiveness metric Delay on release schedule metric Up-to-date documentation metric 11 SQAP/Bug-tracking Metrics

12. EMI INFSO-RI-261611 12 Fixed Bugs (SA1-QC) Required for assessing the number of regression tests produced in association with each fixed bug.

13. EMI INFSO-RI-261611 13 Open Untouched Bugs (weekly EMT) Report from 10 th March 2011 Inter-quartile ranges used in visualization

14. EMI INFSO-RI-261611 14 Successful Builds Metric (for EMT) Highlight: product teams having failures due to their own internal issues Highlight: product teams causing other product teams to fail

15. EMI INFSO-RI-261611 15 Backlog Management Index Track whether product teams have an increasing backlog or decreasing backlog of bugs or defects?

16. EMI INFSO-RI-261611 Metric NameDesignedImplementedRecipient Unit test coverage metric Reports/ Deliverables/ Periodic PT reports Number of supported Platforms Total bug density Bug density per release Cyclomatic complexity C/C++ metrics – CCCC, cppcheck Java metrics - FindBugs, PMD, Checkstyle Python metrics - pylint Code commenting metrics 16 Product related Metrics

17. EMI INFSO-RI-261611 17 Static Analysers: Java – FindBugs (PT) Project Overview - Results per Product team Finer Grained Reporting for Product Teams

18. EMI INFSO-RI-261611 18 Source Lines of Code Source lines of code is particularly important when assessing the Bug Density Distribution (i.e: Bugs open per lines of codes)

19. EMI INFSO-RI-261611 More Static Analysers 19

20. EMI INFSO-RI-261611 The metrics are useful and we defined them for the whole of EMI: – There is now a common framework for producing reports for the EMT, deliverables and Product Teams (PTs). – The reporting structure is interpretable at the project level, activity level & Product Teams level. – Static analysers produce reports highlighting problems previously not seen by developers. 20 Current Status

21. EMI INFSO-RI-261611 We do have an objective way to measure performance/complexity/support. EMI can provide quality software and measure its quality. With the current metric reports one can see trends and understand what needs to be improved and how many defects a component has. 21 Conclusions

22. EMI INFSO-RI-261611 A few important metrics are currently being implemented. The metrics will be reassessed and evaluated after the EMI-1 release. 22 Future Work