1. The Quality Assurance Metric Infrastructure in the EMI Project
Eamonn Kenny, TCD
Manchester,

2. EMI Quality Assurance Activities
Quality Assurance Policies
definition and monitoring
Metrics and KPIs
definition and collection
Quality Control
verification and reporting
Build/Test Tools and Repositories
selection, maintenance and integration
Certification and Integration Testbeds
setup, maintenance and deployment
29/03/2012
TCD

3. What’s different in EMI?
Description
Multiple Institutions
Company/Large Organisation
Programming Languages
Many
Few or one
Bug Trackers
One
Feature Trackers
One (may be the bug-tracker)
Unit testing
Variety of analysers
Discrete, well defined,
run over full code base
Build systems
You cannot apply all of SQuaRE ISO to a working well established system
29/03/2012
TCD

4. Trackers  Format  Metrics
Savannah
Metrics RT
29/03/2012
TCD

5. Request for Change (RfC) lifecycle
Open
Accept?
YES
Accepted
Fixed
Test?
Successful?
Not Tested
Tested
Closed
Rejected NO
Won’t Fix
Duplicate
Invalid
Integration Candidate
Fixed
Resolved
Rejected
Hiding and managing
the complexity
29/03/2012
TCD

6. Implemented Framework
Validation
emi-rfc.cern.ch QA
Policy
Tabulated Formats
Dashboard
Pull
CSV
/XSL
emiqa.web.cern.ch
Chart Generator (Java Framework)
Manual Validation
Tracker to
Product
Mappings
Quality
Control
Release
Manager
29/03/2012
TCD

7. Dashboard for Validation/Reports
29/03/2012
TCD

8. Framework Actors 29/03/2012 TCD Other formats dashboard Common Format
Dump/Export of RfC Tracker
Bug/Feature Trackers
Exported files in Common Format
Common formats
Tabulated Formats
Product Teams
QA Metrics
Metric Calculation & Visualisation
Per Product Calculations
Trend Graphs
XY Bar/Stacked Charts Stacked, Trend Graphs, Distributions
Outputs
QA Metrics & Tools
Release Manager,
QA, QC
Common Format
Dated
Builds Results with Static Analysers
Multiple Builds
Verification & Tests
Releases Dashboard
Product information
Product Teams
QA Tools
QA Policy & Testing
+ Release Manager
29/03/2012
TCD

9. Java Framework Implementation
Data
Build System XML
RfC/Tracker XML
Vertification/Testing XML
Metrics
Associated Tests per Update
Associated Tests per Release + Updates
SLOC per Update (single and trend)
Defects per SLOC
Development task
Deployment KPIs
RPMlint/Lintian tables
Plotting Tools
Bar Charts
Stacked Bar Charts
Trend Graphs
Line plots
29/03/2012
TCD

10. Sample Results
29/03/2012
TCD

11. EMI Year 3 work summary New dashboard layout and fixes
Combined static analyser reporting
Density plots for quality control
GitHub/EMI product team integration
Ongoing release/update metrics
29/03/2012
TCD

12. Questions?
Any Questions?