Contents 1 Description of 1 Description of Initiative Initiative 3 Results: 3 Results: Characterization Characterization 2 Description of 2 Description of Lessons Learned Lessons Learned Study Study 4 Results: Lessons 4 Results: Lessons Learned Learned 5 Implications & 5 Implications & Next Steps Next Steps Slide 1 Forrest Shull,Fraunhofer Center for Experimental Software Engineering - Maryland Judith Bachman,Computer Sciences Corporation John Van Voorhis,Fraunhofer Center for Experimental Software Engineering – Maryland Mike Stark,GSFC John Kelly,JPL Software Inspections at NASA: Lessons Learned Report on State of the Practice

Contents 1 Description of 1 Description of Initiative Initiative 3 Results: 3 Results: Characterization Characterization 2 Description of 2 Description of Lessons Learned Lessons Learned Study Study 4 Results: Lessons 4 Results: Lessons Learned Learned 5 Implications & 5 Implications & Next Steps Next Steps Slide 2 Project Information This initiative –a collaboration between JPL, GSFC, CSC, and FC-MD –funded by the Office of Safety and Mission Assurance (OSMA) Software Assurance Research Program Major objectives include: –A lessons learned report about current state-of-the- practice in inspections at NASA –Running of experiments to investigate tailoring and training of new inspection techniques –Running and support of pilot studies to provide long-term support for piloting new inspection techniques

Contents 1 Description of 1 Description of Initiative Initiative 3 Results: 3 Results: Characterization Characterization 2 Description of 2 Description of Lessons Learned Lessons Learned Study Study 4 Results: Lessons 4 Results: Lessons Learned Learned 5 Implications & 5 Implications & Next Steps Next Steps Slide 3 Lessons Learned: Process “Inspection” defined broadly: Any technical examination process during which a product is examined for defect detection by more than just the author. 1.Analyzed existing process recommendations. 2.Contacted key personnel at centers for potential participants, then participants directly. 3.Distributed characterization questionnaire. 4.Performed semi-directed interview - elicit processes, attitudes, experiences. 5.Analysis of qualitative data for lessons learned.

Contents 1 Description of 1 Description of Initiative Initiative 3 Results: 3 Results: Characterization Characterization 2 Description of 2 Description of Lessons Learned Lessons Learned Study Study 4 Results: Lessons 4 Results: Lessons Learned Learned 5 Implications & 5 Implications & Next Steps Next Steps Slide 4 Lessons Learned: Subjects 17 subjects –9 GSFC & Wallops; 4 JPL; 2 GRC; 1 JSC; 1 LRC –7 years on average at current center –Majority participated in >10 inspections at that center –Projects:  Flight SW; data capture; control centers; mission planning  Team size mostly <10 people; 2/3 of projects last longer than 1 year. –¾ of respondents currently doing inspections. Not a comprehensive or random sample… But experienced people and representative environments.

Contents 1 Description of 1 Description of Initiative Initiative 3 Results: 3 Results: Characterization Characterization 2 Description of 2 Description of Lessons Learned Lessons Learned Study Study 4 Results: Lessons 4 Results: Lessons Learned Learned 5 Implications & 5 Implications & Next Steps Next Steps Slide 5 Characterization Results (1) What do people inspect? –Most often: requirements, high-level design, low-level design (14/17) –Also test plans (12/17) and code (13/17)  Only 1 picked code inspections as most beneficial: took over management in mid-development.  Other respondents who inspect code  develop many small projects with little mission criticality  don’t have chance to influence requirements. –Teams tend to start inspections in the first phase where:  They have input;  The document is sufficiently stable and complicated

Characterization of Development Errors

Contents 1 Description of 1 Description of Initiative Initiative 3 Results: 3 Results: Characterization Characterization 2 Description of 2 Description of Lessons Learned Lessons Learned Study Study 4 Results: Lessons 4 Results: Lessons Learned Learned 5 Implications & 5 Implications & Next Steps Next Steps Slide 7 Characterization Results (2) Why do people inspect? –“Changing/missing/misinterpreted requirements”: on average most important, most often mentioned as source of errors.  “Unstable requirements” also a major development problem. –“Coding errors” were mentioned by everyone but consistently ranked least important. –Most important thing to look for in any phase is consistency with the previous documents. How do people inspect? –Almost all knew of or trained on a process. (JPL, SSDM, RA) –9 used a process. –Very few use checklists. –Only 8 ever collected metrics.  No correlation with size/mission criticality.  Those who still do use relatively simple metrics.  Simple tools for those who do: websites, Excel, Access…

Contents 1 Description of 1 Description of Initiative Initiative 3 Results: 3 Results: Characterization Characterization 2 Description of 2 Description of Lessons Learned Lessons Learned Study Study 4 Results: Lessons 4 Results: Lessons Learned Learned 5 Implications & 5 Implications & Next Steps Next Steps Slide 8 Characterization Results (3) Do people want to inspect? –Not at first. –Many were convinced by being involved in effective inspections. No failing projects with inspections Some used inspections to repair projects Several subjects felt that all successful development required inspections.

Contents 1 Description of 1 Description of Initiative Initiative 3 Results: 3 Results: Characterization Characterization 2 Description of 2 Description of Lessons Learned Lessons Learned Study Study 4 Results: Lessons 4 Results: Lessons Learned Learned 5 Implications & 5 Implications & Next Steps Next Steps Slide 9 Lessons Learned Results (1) Communication Benefit - Most important –Team cohesion: Understanding who to go to with a question. –Communication to the customer  Metrics  “White box reviews” / adaptability –Getting the right technical info to the right people. –Can survive without inspections if communication happens anyway –Effective practice: combine inspections with other meetings Training Benefit –Many projects required significant time for learning, had staffing issues –Team building: Getting on board –Cross-training, novice training Defect Reduction Benefit –It happens - mainly by more communication and better trained staff.

Contents 1 Description of 1 Description of Initiative Initiative 3 Results: 3 Results: Characterization Characterization 2 Description of 2 Description of Lessons Learned Lessons Learned Study Study 4 Results: Lessons 4 Results: Lessons Learned Learned 5 Implications & 5 Implications & Next Steps Next Steps Slide 10 Lessons Learned Results (2) Using Perspectives during Review –Perspective: Point-of-view and existing expertise against which a reviewer inspects a software product. –[Mars Climate Orbiter, Mishap Investigation Board Phase I Report: had inspections but critical staff were missing] –Everybody who had a choice looked to optimize the set of perspectives.  Even teams with low formality otherwise. –Some would reschedule if a critical perspective was missing –Typically, no checklists but different perspectives implicitly assumed to look for different things. –When checklists are used:  “Don’t use them as a replacement for thinking.”

Contents 1 Description of 1 Description of Initiative Initiative 3 Results: 3 Results: Characterization Characterization 2 Description of 2 Description of Lessons Learned Lessons Learned Study Study 4 Results: Lessons 4 Results: Lessons Learned Learned 5 Implications & 5 Implications & Next Steps Next Steps Slide 11 Lessons Learned Results (3) Staffing / Mgmt. –Rare that initial plans include process/metrics support.  “Inadequate staffing” second-largest development problem.  Understaffed projects, even if they want inspection help, have more pressing concerns. –Functional support lacking, especially for metrics collection. –Losing follow-up: surest way to demotivate people.  Chief benefit of a more formal process is action item tracking with suitable rigor. –Support is necessary… –… and worth paying for.

Contents 1 Description of 1 Description of Initiative Initiative 3 Results: 3 Results: Characterization Characterization 2 Description of 2 Description of Lessons Learned Lessons Learned Study Study 4 Results: Lessons 4 Results: Lessons Learned Learned 5 Implications & 5 Implications & Next Steps Next Steps Slide 12 Implications and Next Steps Candidates for further study: –Perspective-Based Reading  Focus on consistency in requirements/design; individual review  Communication/Training: Novice and cross-training; explicit responsibilities and expertise  Perspectives: Make important perspectives explicit; go beyond checklists –JPL Formal Inspections  Positive feedback; often formed the basis for processes  Communication: Formal assignment of roles  Staffing/Mgmt: Very strong on follow-up tracking –DaimlerChrysler: inspection “sampling” techniques  Staffing/Mgmt: Seems well-suited to software acquisition and projects on tight schedule

Contents 1 Description of 1 Description of Initiative Initiative 3 Results: 3 Results: Characterization Characterization 2 Description of 2 Description of Lessons Learned Lessons Learned Study Study 4 Results: Lessons 4 Results: Lessons Learned Learned 5 Implications & 5 Implications & Next Steps Next Steps Slide 13 Request for Participation We are looking for: –Civil servants to directly participate –Civil servants who are overseeing contracts to allow/encourage contractors to participate Controlled experiments: Participants spend 1 day for: –Receiving training in state-of-the-art techniques that can be taken away and used on real projects. –Receiving feedback on types of defects detected and effectiveness of the training, and some comparison to their usual approach Pilot studies: Participants work with us on actual projects and: –Receive training in state-of-the-art techniques tailored for their environment & project –Receive extended support for inspections including  data collection  consultation  analysis & feedback

Contents 1 Description of 1 Description of Initiative Initiative 3 Results: 3 Results: Characterization Characterization 2 Description of 2 Description of Lessons Learned Lessons Learned Study Study 4 Results: Lessons 4 Results: Lessons Learned Learned 5 Implications & 5 Implications & Next Steps Next Steps Slide 14 Contact Info Forrest Shull