1. Formal Technical Reviews Matt Graham 18 November 2004 EECS 814 University of Kansas

2. Outline  Introduction  Generic Inspection and Variations  Inspection throughout Development Cycle  FTR Metrics  Tools  Conclusion

3. FTR: What it is...  A method involving a structured encounter in which a group of technical personnel analyzes an artifact according to a well-defined process. ::Output::  A structured artifact that assesses or improves the quality of the artifact as well as the quality of the method. ::Input::

4. FTR: What it is... A structured FTR is a forum for:  Finding defect information for the Author.  Educating peers about product.  Providing fault likelihood data for testers.  Providing detailed status report for managers.  Allowing process improvement group a test to measure.

5. FTR: What it isn’t...  Walk-through  No Measurements  Often informal, impromptu  Main purpose is developer training

6. Why Review?  Improves overall product quality  Early detection of defects  Reduces rework and its associated costs  Educate the participants and provide training  Setting standards of excellence / maintain process improvement momentum  Improve schedule performance  IBM reported 83% and AT&T 92% defect detection through inspections

7. Inspection Process Overview Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  6 Steps  Team  2 - 6 members  Experienced and involved in product development  Distinct and important roles

8. Inspection Process Overview Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  Roles:  Organizer – plans the inspection activities  Moderator – ensures that procedures are followed and moderates the meetings.  Inspectors – responsible for detecting defects in the product  Presenter – presents the product in logical fashion paraphrased at a suitable rate  Author – author of the developed product.  Recorder – records the defects during the meeting  Collector – collects the defects if there is no meeting.

9. Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  Objective – organize inspection  Check to see if work products pass the entry criteria  Select inspection participants and assign roles  Schedule inspection meeting  Distribute inspection material  Prepare inspection documentation  Ensure product inspection readiness

10. Objective Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  Objective – educate participants on the product being inspected  Author explains the inspection materials  Can be beneficial when:  The inspection artifact is complex  If the artifact is part of large system  If new members participate in inspection  Could be rolled into previous or next phase if product is simple enough  Communicate inspection goals!

11. Defect Detection Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  Objective – to find defects  Can be divided into 2 phases:  Preparation (individual) phase.  Meeting (group) phases.  Individual preparation with the goal of detecting defects can help inspectors to be more prepared.

12. Defect Collection Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  Objective – collectively agree and document the defects (triage)  Decide on further inspection needed or not  Decisions made subjectively as a group.

13. Defect Collection Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  Objective – correct the defects collected in the previous phases  The author makes the detected changes and sends it for follow-up

14. Follow Up Planning Overview Defect Detection Defect Collection Defect Correction Follow-Up  Objective – to make sure all defects are resolved  Only one person involved and he/she verifies the defect resolution  Verifier should “return and report” to inspection team

15. Formal Inspection Methods  Fagan Inspection  Asynchronous Inspection  Phased Inspections  N-Fold Inspections

16. Fagan Inspection Overview Preparation Inspection Rework Follow-Up  4 Roles  Author  Reader  Moderator  Scribe  Meeting centric – cost of scheduling and time  Meetings add little to defect detection

17. Asynchronous Software Inspection Initialization Inspection Review Inspection Compile Final Defect List  Team  Author not involved  Moderator and Inspectors Re-Work

18. Asynchronous Software Inspection Initialization Inspection Review Inspection Compile Final Defect List  No meetings  Easy to assess participation  Process improvement from documenting all correspondences  Allows parallel communication  Can be distributed in space and time  Eliminates group approval Re-Work

19. Asynchronous Software Inspection Initialization Inspection Review Inspection Compile Final Defect List  Moderator sends out material  Initial Individual Review – create list of defects  Circulate the copy of defect list to all inspectors and discuss via email  Individual Review – update defect list and send to Moderator  Moderator compiles final defect list, send it to author and follow up – eliminates group approval Re-Work

20. Phased Inspections  Consists of several coordinated partial inspections called phases  Each phase inspects for a specific property or small set of related properties  Each phase responsible for thorough checking of the properties  Some phases have single inspectors and others have multiple inspectors  Reference documents are provided to inspectors at the beginning of each phase and each inspects using checklists  Domain specific checklists  Application specific checklists

21. N-fold Inspection  N independent teams inspect the same product using traditional inspection method  Collective effort of multiple teams more faults than a single team  Moderator collects faults from independent teams and composes the final defect list

22. Requirements Inspection  Reading Techniques are techniques to analyze work product during defect detection  Ad-hoc  Checklist based reading  Scenario based readings  Perspective based reading  Defect based reading

23. Perspective based Reading  Inspectors stand in for specific stakeholders in the document to verify the quality of requirements.  Designer – detail for creating system components  Tester – detail for constructing test plans  User – correct functionality  For each perspective, reviewer creates high level work products from the requirements document

24. Code Inspections  Cognitive models for program comprehension  Bottom-up  Top-Down  Integrated  Systematic and as-needed comprehension

25. FTR Metrics  Helps you measure the effectiveness of the inspection  Aids in continuous process improvement  Provides feedback to management  A unit of work product varies often:  Usually 1000 lines of source code (not including comments)  Page counts and line counts for text

26. Some Generic Metrics  Average preparation effort and preparation rate per unit of material  Average examination effort per unit of material  Average explanation rate per unit of material  Average number of defects and major defects found per unit of material  Average hours per defect and per major defect

27. Nine Key Metrics by AT&T for Code Inspections  Total Non Comment lines of source code inspected, in KLOC  Average lines of code inspected  Average preparation rate  Average inspection rate  Average effort per KLOC  Average effort per fault detected.  Average faults detected per KLOC  Percentage of re-inspections  Defect-removal efficiency

28. Tools  Various tools for different inspection methods.  ICICLE – for inspection of C & C++ programs  Scrutiny & InspeQ for specific inspection processes  ASSIST –supports generic inspection process

29. Choosing a tool...  Threads of discussion  Sharing of Information  Train of thought  Visual Cues  Reaching a consensus  Coordination

30. The Future of FTR  FTR adoption is slow:  Managers see added cost of inspections instead of the benefits of greatly reduced defect leakage  Practitioners already have tight schedules and reluctant to take on additional responsibilities  Companies postpone adoption since peer reviews are part of CMM level 3.

31. FTR Barriers  Insufficient preparation  Moderator domination  Incorrect review rate  Ego-involvement and personality conflict  Issue resolution and meeting digression  Recording difficulties and clerical overhead

32. Conclusion  Organization’s size, culture and industry should be considered in deciding on the FTR method to use.  Review does not replace testing but can make it easier  Success of inspection lies in:  Process Adherence  Systematic Reading technique  Optimization

33. FTR: References ● Philip Johnson: http://www2.ics.hawaii.edu/~johnson/FTR/ ● Tom Gilb: http://www.result-planning.com/ ● IEEE STD 1024-1197