1. Wojciech Sliwinski Wojciech.Sliwinski@cern.ch Beams Department, Controls Group CERN

2. Outline  Defining Software Quality  Continuous Improvement  Context – Accelerator Controls Codebase  Applying QA & CI for the Controls Codebase  Conclusions 2Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN24th October 2011

3. Outline  Defining Software Quality  Continuous Improvement  Context – Accelerator Controls Codebase  Applying QA & CI for the Controls Codebase  Conclusions 3Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN24th October 2011

4. Software Quality (SWQ)…  We know it’s “a good thing” ™  People think… “I know it when I see it” “I know the value of it”  But what are the actual characteristics of SWQ?  Do we have a common understanding of SWQ?  The lack of it normally appears as consequences… 424th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

5. 1985: Therac-25 – Software issues…  Radiation Deaths linked to Software Errors Killed 2 people Injured dozens of others http://www.ccnr.org/fatal_dose.html 524th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

6. 1996: Ariane 5 – floating point conversion error 624th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

7. 2007: Airbus A340 – Software issues … 724th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

8. 80s – The Kano Model (Marketing) Customer Satisfaction Product Features Basic Needs Met Linear Performance Delight, Surprise, Innovate 24th October 20118Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN Satisfied Dissatisfied Needs well fulfilled Needs not fulfilled

9. Early and Other Quality Definitions  Zen and the art of motorcycle maintenance, Robert Pirsig, 1974: “If quality exists in an object, then you must explain why scientific instruments are unable to detect it” “On the other hand, if quality is subjective, [existing only in the eye of the observer] then this Quality is just a fancy name for whatever you'd like.” “Quality is not objective. It doesn't reside in the material world [...] Quality is not subjective. It doesn't reside merely in the mind.”  McCall et al, 1977 and Boehm et al, 1978 First elaborate studies on the notion of 'software quality’ Quality factors – only indirectly measurable like “reliability” Quality criteria – objective aspects that support the factors, like “uptime”. 924th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

10. IEEE SWQ Standards  IEEE Glossary of Software Engineering Terminology defines it as “the degree to which a system, component, or process meets customer or user needs or expectations”  IEEE Std 1061-1992 IEEE Standard for a Software Quality Metrics Methodology Does not actually prescribe any metrics! Defines method to evaluate metrics that could be applicable to a particular case  Plenty more – eg. IEEE Std 730 and friends 1024th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

11. ISO Quality Standards  ISO 9000 series of standards for quality management systems: Process based “the degree to which a set of inherent characteristics fulfils the requirements”  ISO 9126 Software Quality Characteristics and Metrics, 2001 Comprehensive but … large & complex 6 key quality attributes: ○ Functionality, Reliability, Usability ○ Efficiency, Maintainability, Portability 1124th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

12. SWQ – Huge Set of Characteristics 12 Only a subset applies to all projects Choose those that suit the project goals 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

13. McCalls’ categories of SWQ factors Source: J.A. McCall, P.K. Richards and G.F. Walters, Factors in Software Quality, RADC-TR-77-369, 1977, US Dep. of Commerce. 13 Operation: CorrectnessDoes it do what I want? ReliabilityDoes it do it accurately all of the time? EfficiencyWill it run on my hardware as well as it can? IntegrityIs it secure? UsabilityCan I run it? Revision: MaintainabilityCan I fix it? TestabilityCan I test it? FlexibilityCan I change it? Transition: PortabilityWill I be able to use it on another machine? ReusabilityWill I be able to reuse some of the software? InteroperabilityWill I be able to interface it with another system? 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

14. Internal & External Quality Attributes 14 Important Primarily to UsersImportant Primarily to Developers AvailabilityMaintainability EfficiencyPortability FlexibilityReusability IntegrityTestability Interoperability Reliability Robustness Usability 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

15. The SW Quality “Iceberg” 1524th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

16. The Maintainability Index (MI) ParameterNameMeasures aveVAverage Halstead complexityComputational density aveV (g′)Average extended cyclomatic complexityLogical complexity aveLOCAverage count of lines of codeCode size PerCMAverage percent of lines of commentsHuman insight 16 MI = 171 – 5.2 x ln(aveV) - 0.23 x aveV(g’) - 16.2 x ln(aveLOC) + 50 x sin √2.4PerCM 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

17. MI applied to FreeBSD codebase 1724th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN Kernel CodebaseUser Programs Modules with MI < 40 are rejected !

18. Applying Quality  Computers don’t care about quality!  Metrics are tests for quality characteristics  Quality “clusters”  Testing is ensuring a quality of conformance Bugs occur in clusters too!  Like testing, if quality is low, builds should fail  Quality characteristics must be part of the whole development lifecycle Quality products only come from quality requirements, quality design, quality code, … 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN18

19. Applying Quality (Assurance) Software Quality consists of: 1. Engineering Methods (proper analysis, design, …) 2. Standards and Procedures (common to all) 3. Technical Reviews (eg. code reviews) 4. Configuration management (repositories, versioning) 5. Measurement (code, metrics) 6. Testing (unit, system, integration, acceptance) 7. Improvement (of the items above) 1924th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN Quality is a way of life If you think Quality is expensive, try an accident…

20. Capability Maturity Model (CMM) Levels * 1. Initial (chaotic, ad hoc, individual heroics) ○ the starting point for use of a new process. 2. Repeatable ○ some processes are repeatable, possibly with consistent results 3. Defined ○ the process is defined/confirmed as a standard business process, and decomposed to levels 1 and 2 4. Managed (quantatively) ○ using process metrics, management can identify ways to adjust and adapt the process to particular projects 5. Optimized ○ process management includes deliberate process optimization/improvement. * Capability Maturity Model from SE Institute. 2024th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

21. Outline  Defining Software Quality  Continuous Improvement  Context – Accelerator Controls Codebase  Applying QA & CI for the Controls Codebase  Conclusions 21Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN24th October 2011

22. Continuous Improvement (CI) – Origins  Deming’s Plan-Do-Check-Act (PDCA)  The Toyota Production System “Stop the line” quality control  About eliminating “waste” (obstacles)  A learning, non-judgmental whole team approach  Adopted by the Agile movement for SWD  Caveat: Development is an “empirical process”  Introduced when an organization recognizes it has arrived at some impasse 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN22

23. CI - How it works  “People (at all levels) freely discuss the information, issues, ideas, evaluate them, choose, plan and execute improvements”  Focuses on how things get done: Standardized (and ing) common tasks Only fixing possible issues e.g.: repeated mistakes, time sinks, quality holes  Very suitable where tools, technology, external outcomes and requirements change often 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN23

24. CI - What it relies on  Objective information: To evaluate current situation and processes Assess applied improvements Judge outcome as worthwhile, or retry Demonstrate added value of CI against overhead  Improvement culture: “Free speech” – everyone’s ideas are welcome A real desire to try it and improve Accepting there will not be total agreement Replacing skeptical inaction with real experimentation 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN24

25. CI – It can fail!  No one “owns” responsibility for implementation  No charter to make changes  No time allocated for improvement  No follow up by team or sponsors 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN25

26. Improvement Practices  Change Agents AKA: Coaches, Monitors, Consultants, Champions One or more floating people searching for improvements  Kaizen time aka “renovation days” Team plans some improvements All actions executed by all at same time  Reflection workshops Regular meetings to decide on improvements Apply one or more ideas Keep what works 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN26

27. CI - Summary  Team driven incremental changes  Team evolves and improves together  Drives increase in quality, efficiency and satisfaction  Not a tool or technique – it is an attitude  Not leader oriented, instead peer based consensus  Keeps us ahead in best practice and professionalism 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN27

28. Bibliography  Software Requirements K. Wiegers, Microsoft Press, 2003  Code Complete: A Practical Handbook of Software Construction S. McConnell, Microsoft Press, 2004  Software Engineering: Principles and Practice H. van Vliet, John Wiley & Sons, 2008  Wikipedia page on SWQ ISO standard 9126. 2824th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

29. Bibliography  The Toyota Way, J. K. Liker, 2004  Agile and Iterative Development, C. Larman, 2004  Sustainable Software Development, Kevin Tate, 2006  Implementing Lean Software Development, M. & T. Poppendick, 2007  Rapid Development, S. McConnell, 1996  The Enterprise and Scrum, K. Schwaber, 2007  Collaboration Explained, J. Tabaka, 2006  Peopleware Papers: The Human side of Software, L. Constantine, 2001 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN29

30. Outline  Defining Software Quality  Continuous Improvement  Context – Accelerator Controls Codebase  Applying QA & CI for the Controls Codebase  Conclusions 30Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN24th October 2011

31. Context - CERN Accelerator Complex 24th October 201131Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

32. Context - The CERN Control Centre 24th October 201132Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

33. Controls Software Infrastructure Client tier Server tier Resource tier CTRL DB Business Layer Device Layer Presentation Layer CMW A 3-tier architecture Presentation Layer – Graphical interactive applications – Fixed Displays Business Layer – General purpose and specific Application servers Device Layer – Front End Device servers Communication to the equipment goes through Controls MiddleWare CMW 24th October 201133Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

34. Controls Software Codebase - Situation  Complex Accelerator domain  ~6 million LOC  Java/J2EE ~200 developers (CERN + other labs) ~1000 projects/modules  C/C++ ~35 developers (CERN + other labs) +20 major projects  Oracle database  Eclipse IDE + plugins  SVN + Maven + CommonBuild(  Atlassian Suite (JIRA, Fisheye, Crucible, etc.)  Nowadays encourage Scrum  Focused on Software Quality 34Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN24th October 2011

35. Controls Software Codebase - Situation  Lots of Code ~6 million lines, 20’000+ classes. … and still growing 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN35

36. Situation – Self Assessment and Outlook  Unknown quality Quality unchecked No mentoring or guidance? Limited standards  High Staff turnover Students, Project Associates, Fellows …  More projects coming (PS renovation, …)  More service provision (looking after running services)  Will this cause future problems like maintenance? 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN36

37. Outline  Defining Software Quality  Continuous Improvement  Context – Accelerator Controls Codebase  Applying QA & CI for the Controls Codebase  Conclusions 37Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN24th October 2011

38. Objectives – improve Software Quality  Introduce quality improvement as an integral part of the everyday development work Leverage tools to automate the process as much as possible Establish guidelines and metrics to measure progress Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN3824th October 2011

39. Quality in the Development Cycle  For each stage Agreed mandatory activities and deliverables Tools identified and integrated with IDE (Eclipse), giving immediate feedback Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN39 Design Implement, Test, Document Deploy, Maintain 24th October 2011

40. Design Reviews  Design meetings with external members  To verify the soundness of design in an early stage  To identify overlapping functionality  Promote a set of common components and libraries Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN40  At the level of sub- components, main classes and design patterns  UML: class and sequence diagrams Design Implement, Test, Document Deploy, Maintain 24th October 2011

41. Code Reviews & Code Analysis  Goal: Identify defects Ensure maintainable code Verify conventions are followed  Static code analysis tools to identify common mistakes and bug patterns: FindBugs Checkstyle Eclipse warnings Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN41  Person-to-person time consuming Only for critical code Mentoring of junior developers  Light-weight person-to- person code reviews with FishEye + Crucible Design Implement, Test, Document Deploy, Maintain 24th October 2011

42. Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN42 Files being reviewed Author and reviewers Comments inline 24th October 2011

43. Reviewing Code (Crucible) 43

44. Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN44 A list of ‘bugs’ The ‘bug’ line indicated ‘Bug’ explained 24th October 2011

45. Testing & Continuous Integration  General agreement on benefits of testing and CI  To increase level of testing, unit tests mandatory deliverable of each project >30% test coverage for non-trivial classes, measured with Clover Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN45  To discover inter-project issues early, Continuous Builds with Bamboo: CI server from Atlassian Compiles and runs unit tests Builds fail on: ○ Low test coverage ○ Basic PMD rules Design Implement, Test, Document Deploy, Maintain 24th October 2011

46. Continuous Integration and Test (Bamboo) 46 Test coverage for project Code metrics Classes with highest risk

47. Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN47 Red = not covered Green= covered 24th October 2011

48. Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN48 Triggered by changes in a dependency accsoft- commons- core accsoft- common s-util accsoft- commons- concentration accsoft- commons- io Top/flop list generated (Work in progress) 24th October 2011

49. Integration and System Testing with CO Testbed  CO Testbed - mini-accelerator lab Hardware and Software Core components from the Accelerators Controls duplicated into this Test Environment  System and Integration tests are executed by Bamboo CI server Automatic execution of tests with “real” system Automatic deployment of Release Candidates 4924th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

50. TIMING FEC03 FEC01 FEC02 FEC04FEC05 SERVER06 SERVER07 5024th October 2011 Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

51. Integration Tests done through Client APIs 51 Bamboo CI Server CMW Proxy CMW Directory Svc RBAC A1 Service Config DB Client APIs (JAPC, CMW, RBAC) Timing CMW, RBAC A2, FESA PPC/LynxOSLinux/i386 TESTS Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN JMS 24th October 2011

52. CO Testbed controlled by Bamboo CI server 24th October 201152Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

53. 53 1. 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

54. 54 1. 2. 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

55. 5524th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

56. Challenges  Think of and organize us as one big team, not many small ones  The code belongs to the group, not to a project or an individual developer  The guidelines and standards established and agreed by everyone Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN56 Challenges Agree on standards and configurationsAgree on standards and configurations 24th October 2011

57. Encourage developers to invest time on quality  Quality-related tasks part of the yearly objectives for a project  Progress measured against the level a project adheres to guidelines – top/flop lists  Focus the effort where the effect is highest, rely on tools as much as possible  ‘CI days’ – common days with a common goal Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN57 Challenges 24th October 2011

58. Conclusions  Integrated quality improvement in the development cycle  Introduced guidelines/standards and supporting tools  Increased developer motivation through Immediate feedback when developing Official tracking of progress and top/flop lists  Increased awareness of benefits, application in individual projects and confidence when making changes  Agile approach  reflect, identify ”waste” and improve Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN5824th October 2011