Software Quality and Team Development Practices based on CERN Experience Rostislav Titov, GS-AIS-EB Section Leader, CERN

CERN AIS The reality today Failure 31.1% of IT projects will be canceled before they ever get completed 52.7% of projects will cost 189% of their original estimate More than 50% of software projects fail today. 2009: highest failure rate in over a decade! Success Only 16.2% software projects that are completed on time and on budget In the large companies only 9% of their projects come in on time and on budget. Catastrophes –Ariane 5 (7bn dev, 500 million) numerical conversion error

CERN AIS Typical Failures User requirements not met Software unreliable It works great for me, now deploy it for 10,000 users… Too late (You took so long that our requirements have changed…) You did what I asked. but I didn’t say what I meant…

CERN AIS Reasons Unrealistic Schedules –Race through the requirements, produce a superficial design and rush into coding. Changing Requirements During Development “Writing software from requirements is like walking on water – its easier when frozen” Inappropriate Staffing Poor-Quality Work There's a saying about software quality: “If it doesn't have to work, we can build it really fast.” Believing in magic –Pitfalls of commercial products Cheap Good Fast Choose two

CERN AIS Project Success factors Project Success Factors and % of Responses 1User Involvement15.90% 2Executive Management Support13.90% 3Clear Statement of Requirements13.00% 4Proper Planning9.60% 5Realistic Expectations8.20% 6Smaller Project Milestones % 7Competent Staff7.20% 8Ownership5.30% 9Clear Vision & Objectives2.90% 10Hardworking, Focused Staff2.40% Other13.90% Project Challenged Factors and % of Responses 1Lack of User Inputs12.80% 2Incomplete Requirements & Specifications12.30% 3Changing Requirements & Specifications11.80% 4Lack of Executive Support7.50% 5Technology Incompetence7.00% 6Lack of Resources6.40% 7Unrealistic Expectations5.90% 8Unclear Objectives5.30% Other23.00%

CERN AIS Analysis State Goal “send a man to the moon before end of the decade & return him safely to Earth”, JF Kennedy Specify the problem not the solution Classification M Must o S Should C Could 0 W Would Concept of Operations document “This is my understanding of what you want” Beware of requirements ‘gold plating’ Verifiable : use-cases Projects completed by the largest American companies have only approximately 42% of the originally proposed features and functions.

CERN AIS Good Design Principles Consider alternative approaches Not tunnel vision Traceable to the requirements Correct & complete Not reinvent the wheel Use a pattern Adaptability Accommodate Change Maximize Cohesion Minimize Coupling Understandabilty A design must be understandable if it is to support modification. “A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away” - Antoine de Saint Exupéry Mars Polar Lander ($165 million) – design error

CERN AIS Good & Bad design Good Design Change in one part of the system doesn't always require a change in another part of the system. Every piece of logic has one and one home. The logic is near the data it operates on. System can be extended with changes in only one place. Simplicity Bad Design One conceptual change requires changes to many parts of the system. Logic has to be duplicated. Cost of a bad design becomes overwhelming. Can't remember where all the implicitly linked changes have to take place. Can't add a new function without breaking an existing function. Complexity

CERN AIS Coding “It doesn’t matter if I write poor code… the compiler will tell me if there is problem” “It doesn’t matter if I make a mistake… it will come out in testing” Mars Climate Orbiter ($125 million) metric conversion error

CERN AIS Bugs Experienced software engineers inject one defect in about every 10 lines of code. The programmers aren't incompetent or lazy - they're just human. All humans make mistakes, but in software, these mistakes result in defects. This means that a modest-size program of 100,000 lines of code typically would start with about 10,000 defects. Examples : INTEL Pentium : no more than Bugs Cell Phone ( loc) up to 600 errors. Windows-95: 10 Mill. loc: up to errors.

CERN AIS The Cost of Change Cost CERN AIS Time

CERN AIS Code Reviews - guidelines Form –Product is guilty until proven innocent –Producer is innocent because he/she is not on trial –More likely to find bugs if you assume they are there –Evaluate product not producer –Emphasize "review" aspect; do not "fix it here". –Raise problems. Do not discuss solutions

CERN AIS Code Reviews - guidelines Management Involvement –NONE! –Not a manager's status meeting –Management is not represented during inspections –Inspections must not be used as a tool to evaluate workers –… Not a committee, not a working group, not a status report & not an appraisal instrument …

CERN AIS Benefits Primary objective –remove defects as early as possible in the development process Other benefits : –Early Testing –Project Tracking –Educational – share best practices –Training of new/junior programmers –Improved Communication –Improved Individual Quality –Cross-training –Process-improvement –Shared Responsibility – no individual blame

CERN AIS The “Yes, buts...” I don’t have time for this... Good programmer’s code doesn’t need reviewing Its only a ‘minor’ piece of code Code Changes, then what? –2 nd pair eyes rule –Pair programming

CERN AIS Coding Standards – why? Why ? 80% of the lifetime cost of a piece of software goes to maintenance. Hardly any software is maintained for its whole life by the original author. Code conventions improve the readability of the software, allowing engineers to understand new code more quickly and thoroughly. Cannot review unless you have standards... endless debate – was driving too fast? Cannot answer without defined speed limits Recommend best practices, avoid bad practices Maintainable & reliable software is key Produces Common Code Ownership

CERN AIS Now imagine you have A multi-domain, multi-lingual horizontal software application supporting over 16’000 users in 42 countries composed of : 1.5 millionLines of Code 6,000 Java classes 10,000HTML templates many other files Welcome to EDH!

CERN AIS EDH: Good Old Days ( ) “Imagination rules the world” Mac or PC or Unix? C or C++ or ? University atmosphere Freedom & Individualism Choice, choice, choice...

CERN AIS Results Healthy outside, but unhealthy inside Evolution from freedom to Chaos! Development Platform : Mac, PC & Unix Code : C,C++,Python, Prolog, ProC, PL/SQL, Perl... Comments : Spanish, Italian, French & English... Bugs : “Y2K don’t care” Obvious code never reviewed : Why would you show your code to anybody? Never did at university... Results count! Consequence : Maintenance became the primary resource-consuming activity

CERN AIS From University to Industry Individual Development Practices Team Development Practices Freedom of Choice for Development Environment Uniform development environment Common set of tools Choice of language & technology Single technology choice Free selection of tools Individual Code Responsibility (& blame) Common Code Ownership (& learning)... driven by the members of the team (not management imposed) Quality of the Product Quality of the Process

CERN AIS Coding: Tools Atlassian JIRA –Issue tracking and project tracking –EDH: every change must have a JIRA Process should be as lightweight as possible Atlassian GreenHopper –Agile project management (Scrum) –EDH: 2-week sprints Atlassian Confluence –Documentation (WIKI style)

CERN AIS Coding: Tools (2) Atlassian Crucible –Online code reviews –EDH: Every production line of code must be reviewed Atlassian FishEye –Browse version control repository (CVS, SVN) –Real-time notifications of code changes –Web-based reporting, visualisation and code sharing Atlassian Bamboo –Continuous integration Atlassian Clover –Java code coverage metrics

CERN AIS Coding: Tools (3)

CERN AIS Testing Bugs Standard Software: 25 bugs per 1000 lines of program. Good Software: 2 errors per 1000 lines. Space Shuttle Software: < 1 errors per lines. Example Handy (Cellular Phone): lines of program: up to 600 errors. Windows-95: 10 Mill. lines: up to errors. Sept – Jpeg buffer overrun bug in MS windows “an attacker who successfully exploited this vulnerability could take complete control of an affected system, including installing programs; viewing, changing, or deleting data; or creating new accounts with full privileges” Defects Testing ≠ Debugging You may have zero bugs, but s/w may not meet requirements, scale, respond-in time…

CERN AIS Testing Software testing is an art : requires a tester's creativity, experience and intuition, together with proper techniques. Most of the testing methods and practices are not very different from 20 years ago. Testing is more than just debugging. Testing is expensive. Automation helps. Complete testing is infeasible. Tradeoff Testing, while essential, may not be the most effective method to improve software quality.

CERN AIS What do you test? Correctness testing Security testing Reliability testing Stress testing Scalability testing Performance testing Usability testing

CERN AIS And after that? Software Maintenance 80% of lifetime of software The Legacy Crisis The relative cost for maintaining software and managing its evolution now represents more than 90% of its total cost. Example costs –Annual software maintenance cost in USA has been estimated to be more than $70 billion –Y2K : $8.38 billion US dollar, $90 million for Nokia 50% of their time is spent in the process of understanding the code!!!

CERN AIS Types of Maintenance Corrective Maintenance (21%) –A process that includes diagnosis and correction of errors. Adaptive Maintenance (25%) –Activity that modifies software to properly interface with a changing environment (hardware and software). Perfective Maintenance (50%) –Activity for adding new capabilities, modifying existing functions and making general enhancements. –This accounts for the majority of all effort expended on maintenance. Preventive Maintenance (4%) –Activity which changes software to improve future maintainability or reliability or to provide a better basis for future enhancements. –Still relatively rare.

CERN AIS Your challenge ! Come in the 9% of projects on time & in budget Engineer your software (design, review & maintenance in mind) Control the spiraling IT costs & improve the reputation of the industry

CERN AIS Thank You For More Information