1. Andy Moyer

2. Cleanroom Software Engineering  What is it?  Goals  Properties of Cleanroom  Cleanroom Technologies  Case Studies  Critiques

3. Cleanroom Software Engineering: What is it?  Set of principles and practices for the specification, development, and certification of software-intensive systems.  Based on the work of Harlan D. Mills who was an employee of IBM in early 1980s. Influenced by Dijkstras structured programming, Nicholas Wirth on stepwise refinement, and David Parnas on modular design  Methods based on two principles: Programs are rules for mathematical functions Software testing is sampling

4. Where the Idea Came From  The cleanroom model follows the ideas of manufacturing semiconductors in a “cleanroom” environment.

5. Goals of the Cleanroom Process  Main goal: Achieve or approach zero defects  Other goals: Prevent defects Create correct, verifiable software Incremental development

6. Properties of Cleanroom  Spend a lot of time and money "up-front" preventing defects  Use statistical methods to ensure quality  Formally state and “prove” requirement needs  Specification, development, and certification may be done by separate teams depending on the size of the project

7. Cleanroom Technologies 1. Incremental development under Statistical Process Control 2. Function-Based Specification, Design, and Verification 3. Correctness Verification 4. Statistical Testing and Software Verification

8. Incremental Development Under Statistical Process Control  Used to reduce the risk associated with managing large systems by focusing on smaller, manageable subsystems of increments.  Each increment is developed and tested separate from the whole system before integrating into the whole system.  As increments are added, the whole system is tested.  Some amount of end-user function used to obtain customer confirmation or clarification

9. Incremental Development Under Statistical Process Control  Increment should have these properties: Externally executable Contain all functionality of prior increments  Benefits: Specification, design, and certification activities may be performed often Timely feedback can be obtained from customer Intellectual control over a system

10. Cleanroom Technologies 1. Incremental development under Statistical Process Control 2. Function-Based Specification, Design, and Verification 3. Correctness verification 4. Statistical Testing and Software Verification

11. Function-Based Specification, Design, and Verification  Objectives: Expand the specification into implementation via small steps Maintain intellectual control by designing data and control structures at appropriate levels of abstraction  The Three Boxes: Black Box State Box Clear Box

12. Function-Based Specification, Design, and Verification  The Box Structures Begins with an external view – Black Box Transformed into a state machine – State Box Fully developed into a procedure – Clear Box

13. The Black Box  A view of an object that hides data implementation and process implementation. It describes how a system responds to stimuli, usually in a formal specification language.  No internal state is looked at  Our good friend – Z (Zed)

14. The State Box  A view of an object that shows data implementation but hides process implementation. It describes how "state" information is transformed.  Derived from the black box – first step to implementation  This is represented as a finite state machine

15. The Clear Box  A view of an object that shows both data implementation and process implementation. The goal is to stepwise refine procedures and to prove them correct.  Derived from the state box  May introduce new black boxes defining major operations  Creates the required responses (outputs) to stimuli (inputs)

16. Stepwise Refinement of Specifications to Code  Steps: 1. Expand clear box into lower level designs 2. Verify the correctness of the expanded designs 3. Expand low level designs into code

17. Cleanroom Technologies 1. Incremental development under Statistical Process Control 2. Function-Based Specification, Design, and Verification 3. Correctness verification 4. Statistical Testing and Software Verification

18. Correctness Verification  The primary debugging process for Cleanroom  Each box structure subject to correctness verification  Objectives: Determine that the box structures correctly implements design Remove any errors that were introduced during development Completely review the code for completeness, consistency, and correctness

19. Correctness Verification  So what is verified? If the box structure behaves as defined Correctness of each refinement Response mapping defined in one step is preserved

20. Cleanroom Technologies 1. Incremental development process model 2. Stepwise refinement of specifications to code 3. Correctness verification of developed code 4. Statistical Testing and Software Verification

21. Statistical Testing and Software Verification  Each increment of compilable functionality is statistically tested to: Ensure that it meets the quality standards defined by the development organization Certify a level of reliability that the product will deliver in the field Provide feedback for quality control and process improvement

22. Statistical Testing and Software Verification  Steps taken: 1. Plan the test ○ Stratification plan which describes each layer to be developed ○ The sampling plan ○ A description of which physical testing resources are to be devoted to each increment and each stratum 2. Develop usage models required by the test plan 3. Develop and validate the usage distribution

23. Example Usage Model

24. NASA Case Study  In March of 1990 NASA preformed a case study of SEL Cleanroom versus their typical process  Each group applied the process to the same project – a Coarse/Fine Attitude Determination Subsystem of the Upper Atmosphere Research Satellite.  Completed system contained about 34,000 Source Lines of Code (SLOC) of primarily FORTRAN

25. NASA Case Study-SEL Cleanroom Lifecycle

26. NASA Case Study-Experience Comparison

27. NASA Case Study  Testers and developers are on completely separate teams  Developers have no access to the mainframe computer for compilation and testing  Developers rely on code reading instead of unit testing to verify correctness  Testers use a statistical testing approach

28. NASA Case Study - Results

29.  Failure Rate: Standard SEL process – 6 errors/KSLOC Cleanroom SEL procees – 3.3 errors/KSLOC  The inexperience of the cleanroom team had no effect on the outcome  Impact of unstable requirements lessened by concentrated effort in team reviews

30. DoD Case Study  The STARS program is a US Department of Defense (DoD) reasearch and development program  Emphasizes: Process driven Re-use based Integrated software engineering environment

31. DoD Case Study  Typical Process Productivity measured at 121 loc/person month Failure rate: 23~27 failures/KLOC  Cleanroom Process Productivity measured at 559 loc/person month Failure rate: 1 failure/KLOC

32. DoD Case Study  Important benefits: Improved Productivity Improved quality High staff morale

33. Critiques  In 1997, Beizer argued the elimination of unit testing is contradicting “known testing theory and common sense”  Also, is it possible to find bugs without compiling/running code?

34. References  1. Cleanroom Software Engineering. Retrieved from University of Texas at Arlington website: http://www.uta.edu/cse/levine/fall99/cse5324/c r/clean/page1.html 2. DACS, The Data and Analysis Center for Software. Retrieved January 5, 2009 Found at: www.dacs.dtic.mil/databases/url/key.php? keycode=64 http://www.uta.edu/cse/levine/fall99/cse5324/c r/clean/page1.htmlwww.dacs.dtic.mil/databases/url/key.php? keycode=64

35. References  3. Green, Scott et al. (1990). "The Cleanroom Case Study in the Software Engineering Laboratory: Project Description and Early Analysis". NASA. Retrieved from http://ntrs.nasa.gov/archive/nasa/casi.ntrs. nasa.gov/19910008271_1991008271.pdf 4. Prowell, Stacy et al. (1993). "Cleanroom Software Engineering: Technology and Process". Reading, MA: Addison Wesley Longman, Inc. http://ntrs.nasa.gov/archive/nasa/casi.ntrs. nasa.gov/19910008271_1991008271.pdf