1. EMI INFSO-RI-261611 SA2: Quality Assurance Status Report Alberto Aimar(SA2) SA2 Leader EMI First EC Review 22 June 2011, Brussels

2. EMI INFSO-RI-261611 Outline Objectives Achievements KPIs Lessons Learned Outlook SA2 – Alberto Aimar - EMI First EC Review 22 June 2011 2

3. EMI INFSO-RI-261611 WP Overview SA2 – Alberto Aimar - EMI First EC Review 22 June 2011 3 Insert WP Pie Charts (2) – From PO Insert WP Effort Table – From PO

4. EMI INFSO-RI-261611 SA2 Objectives and Achievements Common software quality assurance process and metrics for all activities  EMI QA Policy and Metrics Continuous integration and testing process Providing tools and resources for building and testing Within the project or with external resource providers  EMI Tools, EMI Integration Infrastructure Pass customer acceptance criteria, improve software quality Monitoring metrics trends, reviewing quality and tests Support and consultancy in QA matters  EGI Acceptance Criteria  Metrics Reports, QA Reviews  QA Support Units, Training Material 4

5. EMI INFSO-RI-261611 Objective: Common QA Process Software Quality Assurance Plan defined very early SQA factors, SQA management, Roles and Procedures, Standard Practices and conventions Additional detailed policies were needed Release management Change management Configuration and Integration Packaging Documentation Certification Testing  Achievement: The EMI software follows the QA Policies. Defined and verified (by SA2 and SA1) SA2 – Alberto Aimar - EMI First EC Review - 22 June 2011 5

6. EMI INFSO-RI-261611 SQAP Development Policies CHEP 2010, Taipei DRAFT - EMI QA Activities - A.Aimar (CERN) - DRAFT 6 https://twiki.cern.ch/twiki/bin/view/EMI/SA2#EMI_Policy_Documents

7. EMI INFSO-RI-261611 Objective: Common QA Metrics Metrics needed to quantify and qualify status of software Use as much as possible automated numerical metrics Started with a selection of applicable metrics – Metrics on code, process, support, documentation – Internal and external metrics (code and process) – Language dependent (Java, C++, Python) Examples of metrics – Reaction time to critical bugs, delays in releases – Complexity, bug density, test coverage QA standards references – ISO /IEC 9126-1,-2,-3,-4 and ISO/IEC 25000  Achievement : Metrics collected across all EMI. QA Reports are generated and used for QA Reviews SA2 – Alberto Aimar - EMI First EC Review - 22 June 2011 7

8. EMI INFSO-RI-261611 Metric NameDesignedImplementedRecipient Open Priority Bugs (High/Immediate) EMT/Reports Successful Builds metric Open Untouched Bugs (> 14 days) Fixed Bugs SA1-QC Priority Bugs (High/Immediate) Periodic Reports/ Deliverables/ Product Team Reports Bug Severity Distribution Backlog Management Index Integration test effectiveness metric - Delay on release schedule metric Up-to-date documentation metric SQAP/Bug-tracking Metrics 8

9. EMI INFSO-RI-261611 9 Objective: Provide EMI QA Tools Initial situation very heterogeneous Different cultures, environments, requirements and priorities Approach Uniform the build environment to provide a common software engineering workflow without affect developer efficiency Provide a comprehensive infrastructure able to satisfy the totality of needs Introduce testing, QA metrics generation and reporting in the same tool-chain Support everybody on the new system asap Single tool-chain and workflow Build, integration, Packaging, Testing, QA Uniform and compliant packaging, QA metrics created in builds  Achievement: All EMI is BUILD, PACKAGED and DISTRIBUTED all in identical way with the EMI SA2 Tools

10. EMI INFSO-RI-261611 LanguagesRelease Integration Dependency Management PackagingTestingBug Tracker Progress SA2 – Alberto Aimar - EMI First EC Review - 22 June 2011 10

11. EMI INFSO-RI-261611 Objective: Provide Common QA Infrastructure SA2 installs all the EMI Services (production + release candidate) Testbed on EMI integration and testing process A single Testing Infrastructure within Quality Assurance Product Team (PT) – centric model for the individual certification Product Inter-Component Testing Infrastructure Provide common infrastructure for inter-component tests, i.e. functionality tests focusing on the interaction among different client and components provided from other Product Teams Interaction across PTs components Large Scale testing Provide an infrastructure for large scale acceptance, scalability and interoperability testing of EMI components. The focus is on stressing the service reproducing real production environment.  Achievement: All EMI Software (54 packages, 5 sites) installed on EMI testbed. Successfully used inter-component testing and verification. Large scale testing is difficult so early 11

12. EMI INFSO-RI-261611 12 Testbed Status SA2 – Alberto Aimar - EMI First EC Review - 22 June 2011 Monitoring: Nagios, service availability, service probes not for all services Testers VO: testers.eu-emi.eu Support: GGUS, with EMI-Testbed Support Unit Documentation: https://twiki.cern.ch/twiki/bin/view/EMI/TestBedhttps://twiki.cern.ch/twiki/bin/view/EMI/TestBed

13. EMI INFSO-RI-261611 SA2 KPIs (1/2) Code/KPIDescriptionEstimated Target Q1Q2Q3Q4 KSA2.1 Services Reliability % uptime dependent only on the SA2 services 99%---- KSA2.1.1 - ETICS 99%100%98%100% KSA2.1.2 - CERN (21 hosts) -99.5%100% KSA2.1.3 - ARC (9 hosts) -100% 99% KSA2.1.4 - INFN (8 hosts) -100% 99% KSA2.1.5 – UNICORE (4 hosts) -100% 99% KSA2.1.6 - CESNET (1 host) -100% KSA2.2 Services Availability % uptime dependent only on the SA2 services 97%---- KSA2.2.1 - ETICS 98%99.7%96%99% KSA2.2.2 - CERN (21 hosts) -99.5%97%100% KSA2.2.3 - ARC (9 hosts) -90%100%99% KSA2.2.4 - INFN (8 hosts) -100% 99% KSA2.2.5 - UNICORE (4 hosts) -100%99% KSA2.2.6 - CESNET (1 host) -100% SA2 – Alberto Aimar - EMI First EC Review - 22 June 2011 13

14. EMI INFSO-RI-261611 SA2 KPIs (2/2) SA2 – Alberto Aimar - EMI First EC Review - 22 June 2011 14 Code/KPIDescriptionEstimated Target Q1Q2Q3Q4 KSA2.3 Distributed Testbed Size Number of CPUs available for distributed testing Year 1: 50 CPUs Year 2: 200 CPUs Year 3: 500 CPUs -706090 KSA2.6 Number of Support Requests Number of user request/tickets per quarter for the SA2 services Within QA Plan and agreed Operational level Agreements with the other WPs KSA2.6.1 - ETICS -392775 KSA2.6.2 - EMI Testbed --822 KSA2.7 Average Support Response Time Average time to the first reply to the user Within QA Plan and agreed OLA with the other WPs time in working hours KSA2.7.1 - ETICS -8.25.25.9 KSA2.7.2 - EMI Testbed --6.04.3 KSA2.8 Average Support Request Life Time Average life time of a request/ticket: time from start to end of a Within QA Plan and agreed OLA with the other WPs time in working hours KSA2.8.1 - ETICS 19.817.410.837.6 KSA2.8.2 - EMI Testbed --11.229.5

15. EMI INFSO-RI-261611 Lessons Learned Merging four middleware, tools, habits is complicated but can be done with support from the partners and give very good results Task forces involving the PTs and whoever wanted to participate was important to explain constraints and get quickly to shared solutions Reviewing and adapting keeping in mind the QA goals of the project vs. single developer habits Objective metrics, same for all, explain metrics details Policies and rules can only work if one provides early tools and testbed infrastructure with useful services (“the carrot”) SA2 – Alberto Aimar - EMI First EC Review - 22 June 2011 15

16. EMI INFSO-RI-261611 SA2 Activities in Year 2 SA2 had a fast start and now is going to move also to review and automation mode SA2 Y1: definition &implementation, Y2: review & automation, Y3: consolidation & future Activities from Y1 will continue in Y2 Reviewing our QA activity in Y1 for policies, tools, metrics and infrastructure Add new platforms (SL6 and Debian 6) to the SA2 services (ETICS and test beds) Identify the key improvements factors/metrics and work with the PTs to apply them SA2 – Alberto Aimar - EMI First EC Review - 22 June 2011 16

17. EMI INFSO-RI-261611 SA2 – Alberto Aimar - EMI First EC Review 22 June 2011 17 Thank you EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611