1. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 1 Forrest Shull &Fraunhofer Center for Experimental Software Patricia LarsenEngineering - Maryland Mike Stark,GSFC State-of-the-Art Software Inspections at NASA

2. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 2 Outline Introduction to this Initiative Summary of year 1 work: Assessing NASA lessons learned Current (year 2) work –Updating training course –Metrics collection plan –Controlled experiment Preview of year 3 work and request for participation

3. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 3 Project Information This project –a collaboration between JPL, GSFC, CSC, and FC-MD –funded by the Office of Safety and Mission Assurance (OSMA) Software Assurance Research Program Major objectives include: –A lessons learned report about current state-of-the-practice in inspections at NASA –Providing training in effective inspection techniques –Running of studies to provide support for new inspection techniques and measure their effectiveness 

4. Why Inspections? Long-term benefits GSFC SEL: Steady decrease in error rates. 85% improvement over 15 years. GSFC SEL: Steady decrease in error rates. 85% improvement over 15 years. 0 2 4 6 8 10 12 14 16 Development Error Rates (1976 -1995) Errors per KDLOC 19961976197819801982198419861988199019921994 Project Midpoint Ada Projects FORTRAN Projects

5. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 5 Quick Overview Individual Preparation Team Meeting Author Follow-up Support

6. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 6 Types of Inspections Software inspections can be used to review: System Requirements System Design Subsystem Requirements Subsystem Design Software Requirements Architectural Design Detailed Design Source Code Test Plans Test Procedures & Functions Operator’s Manual, Standards, Plans, Etc. (Source: JPL Formal Inspections training)

7. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 7 Year 1: Lessons Learned Report Based on interviews with a range of NASA projects from several Centers –17 participants, across 5 Centers (and 7 sites) –Generally highly experienced personnel ( >7 years at current Center, > 10 inspections) Investigated why people use (or decided not to use) inspections; what they inspect; how to introduce inspections effectively Analyzed important issues; perceived benefits; problems and difficulties

8. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 8 Year 1: Lessons Learned Report Summary of results –Even informal inspection practices have benefits –More formal practices have more benefits, but require management support and resources –Developers rarely question the utility after their first successful inspection State of the practice: –Fewer groups can afford full formality –Requirements and design reviews are most important and most frequently used –Used to achieve:  Communication (team cohesion, technical understanding)  Training (team building, cross-training)  Defect reduction

9. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 9 Year 2 Work Act on the lessons learned report by: –Creating updated training course to address issues –Developing a metrics collection plan to demonstrate ROI in context –Running controlled experiments to evaluate state-of-the-art practices that can be introduced at NASA

10. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 10 Year 2: Updated training JPL Formal Inspection training identified as highly beneficial from LLR Working with GSFC’s FSB to adapt JPL materials to address current project constraints: –Modularization  Tailored to roles: moderators, inspectors, metrics collection –Shortened lecture, incorporating actual project work –Tailoring –Less JPL-specificity –Suggested agendas for various project constraints

11. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 11 Year 2: Updated training Incorporating lessons learned from synergistic research activities –UMCP: investigating learning curve of inspection practices  Measures of effectiveness vs. time  Applying “observational” empirical methods  Changes in process conformance over time –NSF: level of detail issues related to subject experience  Measures of effectiveness for different procedures  Interaction of detailed procedures with previous expertise

12. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 12 Year 2: Updated training –CeBASE/NSF: collection, abstraction, and refinement of industrial data across domains, e.g.:  “Cost savings rule”: Cost to find & fix software defects is about 100x more expensive after delivery than in early lifecycle phases, for certain types of defects.  IBM: 117:1 between code and use  Toshiba: 137:1 between pre- and post-shipment  Data Analysis Center for Software: 100:1  “Inspection effectiveness rule”: Reviews and inspections find more than 50% of the defects in an artifact, regardless of the lifecycle phase applied.  50-70% across many companies (Laitenberger)  64% on large projects at Harris GCSD (Elliott)  60% in PSP design/code reviews (Roy)  50-95%, rising with increased discipline (O’Neill)  … many others Source: F. Shull et al., “What We Have Learned about Fighting Defects,” Proceedings of 8 th International Metrics Software Metrics Symposium, Ottawa, Canada, June 2002.

13. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 13 Year 2: Metrics plan Worked with ST5 project to define feasible metrics set answering relevant questions Application to FSB projects Example Goal: Analyze software inspections for the purpose of evaluation with respect to number of defects reaching test phase. –Example Questions: Does introducing inspections in early phases…  …reduce the defect density of software at testing?  …reduce the overall testing effort?  …save more effort than it requires?  …result in less time to find and fix (types of) defects than if they were found through testing?

14. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 14 Year 2: Controlled experiment Running of short-term studies at JPL to evaluate candidate state-of-art practices –Controlled experiment between two teams with different approaches –As part of training class: before/after reviews –Re-inspection of artifact with defect history –Case study on live project

15. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 15 Year 3: Tech transfer Next year: transfer and dissemination of results –Delivery of train-the-trainers material –Measurement and data collection –Planning to turn over materials to  Projects at GSFC FSB, JPL, IVV directly  JPL SEPG  GSFC SEPG

16. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 16 Request for Participation We are looking for: –NASA projects & project leads to directly participate –Civil servants who are overseeing contracts to allow/encourage contractors to participate Controlled experiments: Participants spend 1-2 days for: –Receiving training in state-of-the-art techniques that can be taken away and used on real projects. –Receiving feedback on types of defects detected and effectiveness of the training, and some comparison to their usual approach Pilot studies: Participants work with us on actual projects and: –Receive training in state-of-the-art techniques tailored for their environment & project –Receive extended support for inspections including  data collection  consultation  analysis & feedback

17. Contents 1 Description of 1 Description of Initiative Initiative 3 Year 2: Updated 3 Year 2: Updated Training/Metrics Training/Metrics 2 Year 1: NASA 2 Year 1: NASA Lessons Learned Lessons Learned 4 Year 3: Tech 4 Year 3: Tech Transfer Transfer 5 Request for 5 Request for Participation Participation Slide 17 Contact Info Contact –Forrest (lead researcher)  fshull@fc-md.umd.edu  301-403-8970 –Mike (current government POC for contract)  michael.e.stark@gsfc.nasa.gov  301-286-5048