1. EMI is partially funded by the European Commission under Grant Agreement RI-261611 SA2 – Quality Assurance Alberto AIMAR (CERN) SA2 Leader EMI Second EC Review 12 June 2012, Brussels

2. EMI INFSO-RI-261611 Context Objectives and Recommendations Impact on EMI Quality Build and Test (ETICS) Integration Infrastructure (EMI Testbed) Metrics Quality Control Conclusions Outline 12/06/2012EMI 2 nd EC Review2 https://bit.ly/emisa2

3. EMI INFSO-RI-261611 Quality Assurance in EMI (SA2) 12/06/2012EMI 2nd EC Review3

4. EMI INFSO-RI-261611 DoW Objectives 12/06/2012EMI 2nd EC Review4 DoW SO 1.4 Common software quality assurance policies and metrics for all engineering activities Monitoring metrics and trends quality control activities value trends, reviews and related tests, providing support and consultancy in QA matters. Allow EMI to pass customer acceptance criteria continually improve software quality and process itself DoW SO 1.5 Continuous building, integration and testing process by selecting and maintaining tools and resources for building and testing either within the project or in collaboration with external resource providers DoW SO 1.4 Common software quality assurance policies and metrics for all engineering activities Monitoring metrics and trends quality control activities value trends, reviews and related tests, providing support and consultancy in QA matters. Allow EMI to pass customer acceptance criteria continually improve software quality and process itself DoW SO 1.5 Continuous building, integration and testing process by selecting and maintaining tools and resources for building and testing either within the project or in collaboration with external resource providers

5. EMI INFSO-RI-261611 SA2 Quality Assurance 12/06/2012EMI 2nd EC Review5 QA Policies Build and Test Tools Integration Infrastructure QA Metrics Quality Control Reports & Dashboards Reports & Dashboards Development Trackers NA2JRA1 SA1 JRA1 SA1 SA2 Quality Assurance

6. EMI INFSO-RI-261611 EMI merged 4 established Middleware projects Each had its own QA practices, tools, reports, etc. Working with a 3-years vision Y1: explain, define and implement Y2: review and automate Y3: consolidate and optimize Benefitted from existing QA practices Used existing QA tools, resources and expertise Involved PTs, SA1, JRA1 and also explained QA constraints Associated policies with tools to support them for build, test, release and automated reports SA2 QA Strategy 12/06/2012EMI 2nd EC Review6

7. EMI INFSO-RI-261611 1. Resubmit SQAP deliverable Submitted in October 2011. Reorganized as a plan for the 3 years. 2. Reduce correlated deliverables Merged QA and QC deliverable (DSA2.3.3) 3.Integrate Quality Control from SA1 and JRA1 into SA2 Completed by June 2011 Solid QC integration with policies, metrics and reports 4. Foster sustainable solutions after EMI Added standard tools (Mock, Pbuilder) Compel compliance to established distributions (EPEL, Debian) Distribution of buildable sources in addition to binaries Recommendations from 1 st Review 12/06/2012EMI 2nd EC Review7

8. EMI INFSO-RI-261611 All SA2 deliverables submitted DSA2.1 - Quality Assurance Plan rewritten and resubmitted taking into account the review of Y1 DSA2.2.3 - QA Tools Documentation update of previous deliverables DSA2.2.1 and DSA2.2.2 DSA2.3.3 - Periodic QA and QC Reports includes DJRA1.7.3 - Software Development QC Report SA2 Deliverables 12/06/2012EMI 2nd EC Review8

9. EMI INFSO-RI-261611 Full support Fedora (SL5, SL6), Debian 6 Standard build tools (Mock and PBuilder) Standard packaging/repositories (EPEL and APT) Two ways of building Use ETICS for build and packaging Give sources to build with Mock/Pbuilder All EMI software builds for EMI platforms Single project configuration Used by PTs and for global Nightly Builds Better workload management Maven mirror, VM images for developers Elastic build infrastructure EMI Tools and ETICS in Y2 EMI 2nd EC Review9 12/06/2012 Montlhy ETICS Builds DoW SO 1.5 : Continuous integration, building and testing process Build and Test Tools

10. EMI INFSO-RI-261611 Fundamental for releases 3 platforms x Releases x RCs x 32/64 bits >200 installations of pre-EMI, EMI-1, EMI-2 products CERN, CESNET, CNAF INFN, DESY, JUELICH, KOSICE, NIIF Everything has been deployed and tested on the QA Testbed 15 EMI 1 Updates + EMI 2 Release Provided feedback on documentation, and deployment issues to PTs Fundamental for testing, training, demos, monitoring Adopted common EMI deployment tests, integration tests across products Used training and demos, supported “EGI TF 2011 best demo” award Collaboration with EGI is operational and effective Releases EMI project regularly pass the EGI validation  you will see later Deployed SAM-NAGIOS for automated testing and monitoring Testing Infrastructure in Y2 12/06/201210 Integration Infrastructure DoW SO 1.5 : Continuous integration, building and testing process

11. EMI INFSO-RI-261611 Metrics on code, process, docs Reaction to RfCs, delays in releases Compliance to standards and templates Automated reports and live dashboards Focus on metrics immediately useful RfC management, certification and testing Total control of Development Process Status of the products, tests, trackers Used by QC and Product Teams Faster and compliant product releases Submit all QA issues to PTs for prompt action Products ready for passing the QC tests QA Metrics and Reports in Y2 12/06/2012EMI 2nd EC Review11 DoW SO 1.4: Common software quality assurance policies and metrics QA Metrics Reports & Dashboards Reports & Dashboards

12. EMI INFSO-RI-261611 EMI RfC Dashboard 12/06/2012EMI 2nd EC Review12 Data from >30 trackers of all EMI Product Teams, status of EMI development Access to all the RfC in each tracker, search and select, export to Excel http://emi-rfc.cern.ch

13. EMI INFSO-RI-261611 EMI Verification Dashboard 12/06/2012EMI 2nd EC Review13 Access to all reports of each product, complete view of the products http://emi-verification.cern.ch

14. EMI INFSO-RI-261611 Software Quality Assurance Plan updated Reorganised to match your recommendations Added milestones and targets that were in other documents All Product comply with the platforms supported Multiple platforms support (release, configuration, packaging, doc, etc.) Quality Control monitors compliance to policies Uses the QA dashboards to view and extract the desired metrics Metrics automatically extracted to tabular data (e.g. for SA2 and SA1) Weekly reports and plots from the metrics and constant follow-up Quality Control has impact on work of Product Teams Products passing are deployed, product not passing are discussed with the PTs RfCs are issued and the execute the changes QA Policies and Control in Y2 12/06/2012EMI 2nd EC Review14 QA Policies Quality Control DoW SO 1.4: Common software quality assurance policies and metrics

15. EMI INFSO-RI-261611 QC of SA2 Services (KPIs) 12/06/2012EMI 2nd EC Review15 KPITargetP1Q5Q6Q7Q8P2 KSA2.2 - Services Availability ETICS 97% 97.5 %98.0 %95.8 %96.8 %97.9 %97.1 % Testbed 97% 98.3 %99.0 %98.9 %99.1 %99.3 %99.1 % KSA2.3 - Testbed Size 200 CPUs7397112154204 KSA2.6 - No of Requests ETICS -23033364746162 Testbed -3514811538 KSA2.7 - Average Response Time ETICS -6.4 h3.2 h2.45 h0.95 h0.73 h1.68 h Testbed -5.2 h8.1 h2.65 h4.7 h1.30 h5.0 h KSA2.8 - Average Solution Time ETICS -21.9 h96.6 h133.0 h84.2 h57.9 h90.0 h Testbed -20.3 h45.5 h46.6 h18.5 h39.6 h37.1 h

16. EMI INFSO-RI-261611 QC – Testing Improvements 12/06/2012EMI 2nd EC Review16 EMI 1 Updates 1-7 Updates 8-14 Quality Control

17. EMI INFSO-RI-261611 QC – Regression Testing 12/06/2012EMI 2nd EC Review17 Report of Regression Tests for High/Immediate RfCs Data not collected before update 5 Policy enforced by Release Manager after update 7 Quality Control

18. EMI INFSO-RI-261611 Convergence to standards of EMI software QC – Compliance to Fedora/EPEL 12/06/2012EMI 2nd EC Review18 Quality Control 90 % 60 %

19. EMI INFSO-RI-261611 QC – Defect per Product 12/06/2012EMI 2nd EC Review19 Quality Control Number of Defects per Products per Quarter

20. EMI INFSO-RI-261611 From EGI TCB 11 – April 2012 20 DoW SO 1.4: Allow EMI to pass customer acceptance criteria 80% 70% 90% 100%

21. EMI INFSO-RI-261611 Main Achievements New platforms successfully supported Convergence to standards, QA has an impact on the actual quality Faster testing and successful deployment of products Very high success rate in the acceptance of our releases to EGI Excellent collaboration and results across all EMI activities SA1 for releases, updates and deployment. NA2 for training, documentation. JRA1 quality control and dashboards Quality Control merge in SA2 a very good decision Quicker feedback to EMT and within SA2 (metrics, reports, reviews) Someone external to PTs checks products (deployment, compliance) Y3: Consolidate and Optimize Continue to support/improve current tools, testbed, dashboard and reports QC on sustainability of middleware products beyond the EMI Y3 Conclusions 12/06/2012EMI 2nd EC Review21

22. EMI INFSO-RI-261611 Thank You 22