1. CERN – European Organization for Nuclear Research Information Technology – Administrative Information Services Software Team Development Practices based on Java Development Teams at CERN Derek Mathieson James Purvis IT-AIS

2. CERN AIS The reality today Failure 31.1% of IT projects will be canceled before they ever get completed 52.7% of projects will cost 189% of their original estimate More than 50% of software projects fail today. Success Only 16.2% software projects that are completed on time and on budget In the large companies only 9% of their projects come in on time and on budget. Catastrophe –Ariane 5 (17bn dev, 500 million) – numerical conversion error –Mars Climate Orbiter ($125 million) – metric conversion error –Mars Polar Lander ($165 million) – design error

3. CERN AIS The answer Q :What is the key to running a successful software project? A: Don’t believe in Magic! There is no silver bullet …But there are some step that can be carried out at each step in the process…

4. CERN AIS Project Success factors Project Success Factors and % of Responses 1User Involvement15.90% 2Executive Management Support13.90% 3Clear Statement of Requirements13.00% 4Proper Planning9.60% 5Realistic Expectations8.20% 6Smaller Project Milestones0 - 7.7% 7Competent Staff7.20% 8Ownership5.30% 9Clear Vision & Objectives2.90% 10Hardworking, Focused Staff2.40% Other13.90% Project Challenged Factors and % of Responses 1Lack of User Inputs12.80% 2Incomplete Requirements & Specifications12.30% 3Changing Requirements & Specifications11.80% 4Lack of Executive Support7.50% 5Technology Incompetence7.00% 6Lack of Resources6.40% 7Unrealistic Expectations5.90% 8Unclear Objectives5.30% Other23.00%

5. CERN AIS Factors CostQuality Speed Risk Four interdependent factors Not possible to have best of all four Cannot have cheap, high quality, little risk & built quickly You can achieve 2 successfully and other 2 have to be managed Risk and quality are most important: The system must work and successfully meet user requirements. This leaves speed (time) and cost (money) to be adjusted accordingly

6. CERN AIS Software Development Cycle Analysis “Failing to build the right thing” Design “We didn't have time to do a design." Implementation “Do you want it on time or documented?” Test “It's not a bug, it's a feature”

7. CERN AIS Requirements Analysis “Writing software from Requirements is like walking on water – its easier when frozen”

8. CERN AIS Typical Failures User requirements not Met Software unreliable Too late (You took so long that our requirements have changed…) It works great for me, now deploy it for 10,000 users… You did what I asked. but I didn’t say what I meant… Projects completed by the largest American companies have only approximately 42% of the originally proposed features and functions.

9. “Many projects fail because developers fail to build the right thing” —Grady Booch CERN AIS

10. CERN AIS Reasons Unrealistic Schedules –When faced with an unrealistic schedule, engineering teams often behave irrationally. They race through the requirements, produce a superficial design and rush into coding. Inappropriate Staffing Changing Requirements During Development Poor-Quality Work –There's a saying about software quality: “If it doesn't have to work, we can build it really fast.” Believing in Magic

11. CERN AIS Analysis State Goal “send a man to the moon before end of the decade & return him safely to Earth”, JF Kennedy Specify the problem not the solution Classification M Must o S Should C Could 0 W Would Concept of Operations document “This is my understanding of what you want” Beware of requirements ‘gold plating’ 80/20 rule Verifiable : use-cases

12. CERN AIS Techniques used at CERN IEEE (Classical SRS) Interviews Mockups CASE XP & JITRA Storyboarding Separate process from s/w Have person from business perform & write the Analysis

13. CERN AIS Design One of the things that tools can do is to help bad designers create ghastly designs much more quickly than they ever could in the past

14. CERN AIS Example Failures Ariane 5 (half billion dollars) British destroyer H.M.S. Sheffield sunk in the Falkland Islands Y2K bug F-16 & equator Mars Climate Orbiter Mars Polar Lander

15. CERN AIS Reasons Too complex “We didn't have time to do a design." Jumping into the coding Architect: Someone who knows the difference between that which could be done and that which should be done. One of the things that tools can do is to help bad designers create ghastly designs much more quickly than they ever could in the past

16. CERN AIS Design Evolutionary process “Modular programming” “Structured Programming” “Data Flow” “Data Structures” “Object Oriented” All have common characteristics

17. CERN AIS Good Design Principles Consider alternative approaches Not tunnel vision Traceable to the requirements Correct & complete Not reinvent the wheel Use a pattern Adaptability Accommodate Change Maximize Cohesion Minimize Coupling Understandabilty A design must be understandable if it is to support modification.

18. CERN AIS Good & Bad design Good Design Change in one part of the system doesn't always require a change in another part of the system. Every piece of logic has one and one home. The logic is near the data it operates on. System can be extended with changes in only one place. Simplicity Bad Design One conceptual change requires changes to many parts of the system. Logic has to be duplicated. Cost of a bad design becomes overwhelming. Can't remember where all the implicitly linked changes have to take place. Can't add a new function without breaking an existing function. Complexity

19. CERN AIS The Cost of Change Cost CERN AIS Time

20. CERN AIS Design First Design team (>1) –Design requires different skills –Collaborative Informal design reviews –Standards based –Patterns Maxims –Storyboard –Metaphor – vision –Class Responsibility

21. CERN AIS Lessons Learned It’s not always right first time –Refactoring is OK Keep the vision simple Automate repetitive tasks TEST the design –By modifying the problem

22. CERN AIS Design Techniques used at CERN Oracle Designer (ER diagrams) UML (little) Design Patterns (use not abuse) CRC Informal reviews

23. CERN AIS Coding “It doesn’t matter if I write poor code… the compiler will tell me if there is problem” “It doesn’t matter if I make a mistake… it will come out in testing”

24. CERN AIS Code Review Example Can you understand the following? public class ba { public static final String cur = "FF"; public void dep (int i) { bal -=i; } public void wit (int i) { bal +=i;} public String get () { return Integer.toString (bal) + " " + cur } private int bal; } And identify the defect?

25. CERN AIS The same code public class BankAccount { public static final String CURRENCY = " FF " ; private int m_balance; public void deposit (int amount) { balance = balance - amount; } public void withdraw (int amount) { balance = balance + amount; } public String getBalance () { return Integer.toString (bal) + " " + CURRENCY; }

26. CERN AIS Now imagine you have A multi-domain, multi-lingual horizontal software application supporting 10,500 users in 42 countries composed of : 1.2 millionLines of Code 6,000 Java classes 10,000HTML templates many other files Welcome to EDH!

27. CERN AIS Bugs Experienced software engineers inject one defect in about every 10 lines of code. The programmers aren't incompetent or lazy - they're just human. All humans make mistakes, but in software, these mistakes result in defects. This means that a modest-size program of 100,000 lines of code typically would start with about 10,000 defects. Examples : INTEL Pentium : no more than 80-90 Bugs Cell Phone (200 000 loc) up to 600 errors. Windows-95: 10 Mill. loc: up to 200 000 errors.

28. CERN AIS Good Old Days (1991..) “Imagination rules the world” Mac or PC or Unix? C or C++ or ? University atmosphere Freedom & Individualism Choice, choice, choice...

29. CERN AIS Results Healthy outside, but unhealthy inside Evolution from freedom to Chaos! Development Platform : Mac, PC & Unix Code : C,C++,Python, Prolog, ProC PL/SQL, Perl... Comments : Spanish, Italian, French & English... Bugs : “Y2K don’t care” Obvious code never reviewed : Why would you show your code to anybody? Never did at university... Results count! Consequence : Maintenance became the primary resource-consuming activity

30. CERN AIS The “authoritarian” new days (1998...) Autocratic –You will use a PC –You will use the chosen IDE –You will use Version Control –You will develop in Java –You will adhere to coding standards –You will show your code to others Production Environment is Sacred –Scheduled Deploys –Team decision, No individual actions –No unreviewed code goes live Doesn’t sound very motivating? Ironically it is more motivating!

31. CERN AIS From University to Industry Individual Development Practices Team Development Practices Freedom of Choice for Development Environment Uniform development environment Common set of tools Choice of language & technology Single technology choice Free selection of tools Individual Code Responsibility (& blame) Common Code Ownership (& learning)... driven by the members of the team (not management imposed) Quality of the Product Quality of the Process

32. CERN AIS Requires Concrete Practices Code Reviews Coding Standards Design Reviews Mentoring Uniform development environment Coherent set of development & deployment tools Single language & Technology Test procedures, unit testing & usability studies

33. CERN AIS Code Reviews What are they? Need Standards!

34. CERN AIS Coding Standards – why? Why ? 80% of the lifetime cost of a piece of software goes to maintenance. Hardly any software is maintained for its whole life by the original author. Code conventions improve the readability of the software, allowing engineers to understand new code more quickly and thoroughly. Cannot review unless you have standards... endless debate – was driving too fast? Cannot answer without defined speed limits Recommend best practices, avoid bad practices Maintainable & reliable software is key Produces Common Code Ownership (familiar – leave system ex.)

35. CERN AIS The essence of code reviews Now we have rules, we can check conformance & look for defects... Q: How many Code reviewers it take to change a lightbulb?

36. CERN AIS The essence of code reviews Now we have rules, we can check conformance & look for defects... Q: How many Code reviewers it take to change a lightbulb? A: Code reviewers don't change anything....They just report that it's dark.

37. CERN AIS Code Reviews - guidelines Form –Product is guilty until proven innocent –Producer is innocent because he/she is not on trial –More likely to find bugs if you assume they are there –Evaluate product not producer –Emphasize "review" aspect; do not "fix it here". –Raise problems. Do not discuss solutions

38. CERN AIS Code Reviews - guidelines Form Format –Three people minimum, seven people maximum –Roles : Author Moderator Scribe Reviewer(s) –Preparation –Short & businesslike (2 hrs max) –Not off-site. No telephones or interruptions –Donuts or other home goodies –Dismiss the guru who wants to demonstrate that their way is better

39. CERN AIS Code Reviews - guidelines Form Format Management Involvement –NONE! –Not a manager's status meeting –Management is not Presence during inspections –Inspections must not be use as a tool to evaluate workers –.. Not a committee, not a working group, not a status report & not a MAPS (appraisal)...

40. CERN AIS Benefits Primary objective remove defects as early as possible in the development process Other benefits : Early Testing Project Tracking Educational – share best practices Training of new/junior programmers Improved Communication Improved Individual Quality Cross-training Process-improvement Shared Responsibility – no individual blame Industry recognition

41. CERN AIS The “Yes, buts...” I don’t have time for this... Good programmer’s code doesn’t need reviewing Its only a ‘minor’ piece of code Code Changes, then what? –2 nd pair eyes rule –Pair programming

42. CERN AIS Configuration Management … from Chaos to repeatable…

43. CERN AIS Configuration Management Software Configuration Management is a discipline for controlling the evolution of software systems. Standard definition[1][2] for CM taken from IEEE standard 729- 1983[SCM] includes: –Identification: identifies structure of the product, its components and its type, making it unique (CI*). Addresses “What version of the file is this?” –Control: Manages release of a product and changes done throughout the life cycle. Uses baseline** product. Addresses “What changes went into the latest version of this product ?” –Status Accounting: Recording and reporting the status of components & change requests. Addresses “How many files were affected by fixing this bug?” –Audit and Review: Maintaining consistency, validating completeness of components that form a product. Addresses “Are all the correct versions of files used in this current release?”

44. CERN AIS Typical Development Cycle CVS Server Staging Server Build and Test Locally Check-out Commit Deploy

45. CERN AIS JDeveloper Integration Deploy to test server...

46. CERN AIS Staged Deployment CVS Server Staging Server Test against Staging Server

47. CERN AIS Staged Deployment CVS Server Staging Server Test against Staging Server Production Server

48. CERN AIS Staged Deployment CVS Server Staging Server Deploy to Production Server Production Server Begin ‘Grace Period’ Continue to Build and Test Locally

49. CERN AIS Staged Deployment CVS Server Staging Server Production Server

50. CERN AIS Testing Bugs Standard Software: 25 bugs per 1000 lines of program. Good Software: 2 errors per 1000 lines. Space Shuttle Software: < 1 errors per 10000 lines. Example Handy (Cellular Phone): 200 000 lines of program: up to 600 errors. Windows-95: 10 Mill. lines: up to 200 000 errors. Sept 24 2004 – Jpeg buffer overrun bug in MS windows “an attacker who successfully exploited this vulnerability could take complete control of an affected system, including installing programs; viewing, changing, or deleting data; or creating new accounts with full privileges” Defects Testing ≠ Debugging You may have zero bugs, but s/w may not meet requirements, scale, respond-in time…

51. CERN AIS What do you test? Correctness testing –Black Box –White Box Security testing Reliability testing Stress testing Scalability testing Performance testing Usability testing Automation

52. CERN AIS Testing Software testing is an art : requires a tester's creativity, experience and intuition, together with proper techniques. Most of the testing methods and practices are not very different from 20 years ago. Testing is more than just debugging. Testing is expensive. Automation helps. Complete testing is infeasible. Tradeoff Testing, while essential, may not be the most effective method to improve software quality.

53. CERN AIS And after that? Sofware Maintenance 80% of lifetime of software The Legacy Crisis The relative cost for maintaining software and managing its evolution now represents more than 90% of its total cost. Example costs –Annual software maintenance cost in USA has been estimated to be more than $70 billion –Y2K : $8.38 billion US dollar, $90 million for Nokia 50% of their time is spent in the process of understanding the code!!!

54. CERN AIS Legacy Code An average Fortune 100 company maintains 35 million lines of code These companies add in average 10% each year only in enhancements As a result, the amount of code maintained doubles in size every 7 years E.g. 70% or more of the still active business applications are written in COBOL (at least 200 billion lines of COBOL-code still existing in mainframe computers alone)

55. CERN AIS Types of Maintenance Corrective Maintenance (21%) –A process that includes diagnosis and correction of errors. Adaptive Maintenance (25%) –Activity that modifies software to properly interface with a changing environment (hardware and software). Perfective Maintenance (50%) –Activity for adding new capabilities, modifying existing functions and making general enhancements. –This accounts for the majority of all effort expended on maintenance. Preventive Maintenance (4%) –Activity which changes software to improve future maintainability or reliability or to provide a better basis for future enhancements. –Still relatively rare.

56. CERN AIS Process Unstructured Maintenance only available element is the source code painstaking evaluation of the code, complicated by poor internal documentation, things such as program structure, global data structures, system interfaces, performance and/or design constraints are difficult to ascertain and frequently misinterpreted, ramifications of changes to the code are difficult to assess, regression tests are impossible to conduct because no record of testing exists. Structured Maintenance Begins with an evaluation of the design documentation. The impact of required modifications or corrections is assessed and an approach is planned. The design is modified and reviewed, new source code is developed and regression tests are conducted. Emergency bug fixes cost more than routine, scheduled corrections. performed under pressure, disrupt the orderly process of delivering new releases introduce new errors.

57. CERN AIS Results Maintenance Costs Typical software organizations spend anywhere from 40 to 70 percent of all funds conducting maintenance. Maintenance-bound organizations result : loss or postponement of development opportunities. Customer dissatisfaction when requests cannot be addressed. Reduction in overall software quality as a result of changes that introduce latent errors in the maintained software.

58. CERN AIS Your challenge ! Come in the 9% of projects on time & in budget Engineer your software (design, review &maintenance in mind) Are you an artist, scientist or engineer? Art or Science? Control the spiraling IT costs & improve the reputation of the industry

59. CERN AIS