Copyright © 2003 Software Quality Research Laboratory Software Production Essentials Seeing Past the Buzz Words

Copyright © 2003 Software Quality Research Laboratory Why Process? Quality Quality Maximize Customer Satisfaction Maximize Customer Satisfaction Minimize Rework and Repair Minimize Rework and Repair Productivity Productivity Optimize Production Cost Optimize Production Cost Shorten Time to Market Shorten Time to Market This is not a tradeoff. Quality is Free.

Copyright © 2003 Software Quality Research Laboratory Cost of Correcting an Error 1X in Requirements Analysis 1X in Requirements Analysis 5X in Specification and Design 5X in Specification and Design 10X in Coding 10X in Coding 30X in Integration 30X in Integration 70X in Acceptance Test 70X in Acceptance Test 300X in Field 300X in Field Typical Range: $50 - $14,000

Copyright © 2003 Software Quality Research Laboratory Basic Process Elements Requirements Definition Requirements Definition Coding Coding Release Release Everybody does it, but the order may vary.

Copyright © 2003 Software Quality Research Laboratory How Do We Fill the Gaps? Requirements Definition Requirements Definition Specification?/Design?/Verification?/? Specification?/Design?/Verification?/? Coding Coding Test?/ Inspection?/? Test?/ Inspection?/? Release Release No shortage of lifecycle/process candidates.

Copyright © 2003 Software Quality Research Laboratory Software Production is Incremental An efficient, repeatable process is necessary for expanding, changing requirements. ? ? ?

Copyright © 2003 Software Quality Research Laboratory Software Production Keys (1) A seamless, traceable transformation from requirements to code specification. (2) Mapping specified code to a robust, maintainable architecture. (4) Quantitative reliability assessment via model-based statistical testing. (3) Peer review of key work products including 100% code inspection.

Copyright © 2003 Software Quality Research Laboratory Specification Process Overview

Copyright © 2003 Software Quality Research Laboratory Rigorous Specification Process Establish system boundary. Establish system boundary. Define human/software/hardware interfaces. Define human/software/hardware interfaces. Itemize stimuli. Itemize stimuli. Itemize responses. Itemize responses. Perform enumeration. Perform enumeration. Perform canonical sequence analysis. Perform canonical sequence analysis. Generate state machine specification. Generate state machine specification. Transform to formal notation (e.g., ACL2). Transform to formal notation (e.g., ACL2).

Copyright © 2003 Software Quality Research Laboratory Map Specified Code to Architecture

Copyright © 2003 Software Quality Research Laboratory Independent Review Domain Expert Review Domain Expert Review Initial Requirements Initial Requirements Black Box Specification Black Box Specification Usage Model Usage Model Development Team Peer Review Development Team Peer Review Architecture Specification Architecture Specification Test Plan Test Plan Code Inspection Code Inspection Automated Enforcement of Coding Standards Automated Enforcement of Coding Standards Manual Verification of Functional Correctness Manual Verification of Functional Correctness

Copyright © 2003 Software Quality Research Laboratory Testing is Always Sampling Population (All Uses) Sample (Tests) What to test: a statistically appropriate sample How much to test: to an acceptable reliability estimate

Copyright © 2003 Software Quality Research Laboratory Statistical Usage Testing Statistical Usage Testing involves Modeling the population of uses Modeling the population of uses Selecting a sample for testing Selecting a sample for testing Estimating field performance from test sample results Estimating field performance from test sample results Making a release decision Making a release decision

Copyright © 2003 Software Quality Research Laboratory Model-Based Statistical Testing

Copyright © 2003 Software Quality Research Laboratory Usage Model: All Possible Uses and their Likelihood Begin Beta End Start A B Quit STATE MACHINE Nodes are states-of-use Arcs are possible stimuli Probabilities define expected usage Test case is path from initial to terminal state A B B A p=1 p=.9 p=.1 p=.01 p=.05 p=.85 Idle Alpha p=.89

Copyright © 2003 Software Quality Research Laboratory A Usage Model is a Finite-State Markov Chain Well-understood formalism Well-understood formalism Rich body of analytical results Rich body of analytical results Engineering basis for testing Engineering basis for testing Objectivity in test planning and management Objectivity in test planning and management Simulates “use” of product and not the product itself Simulates “use” of product and not the product itself

Copyright © 2003 Software Quality Research Laboratory Statistical Testing based on a Usage Model affords options Coverage testing…all states and arcs Coverage testing…all states and arcs Importance testing…most probable paths Importance testing…most probable paths Random testing…representative usage Random testing…representative usage Testing to reliability goal…0.99, 0.999, … Testing to reliability goal…0.99, 0.999, … Testing to level of confidence…95%, 99%… Testing to level of confidence…95%, 99%… Testing to budget with measured reliability Testing to budget with measured reliability

Copyright © 2003 Software Quality Research Laboratory Test Automation

Copyright © 2003 Software Quality Research Laboratory Results Analysis / Release Decision Record all failures by test case number and transition where failure occurs, then: Compute estimated reliability based on testing experience Compute estimated reliability based on testing experience Evaluate stopping criteria Evaluate stopping criteria Release or fix and retest based on semi-quantitative risk assessment Release or fix and retest based on semi-quantitative risk assessment

Copyright © 2003 Software Quality Research Laboratory Benefits of Model Based Testing Better Product Better Product Clearer requirements, improved specification Clearer requirements, improved specification Better Use of Resources Better Use of Resources Quantitative optimization of testing strategy Quantitative optimization of testing strategy Reusable assets: model, test framework, oracle Reusable assets: model, test framework, oracle Shorter Life Cycle Shorter Life Cycle Test planning in parallel with development Test planning in parallel with development Easier test automation Easier test automation Better Management Better Management Quantitative support for management decisions Quantitative support for management decisions Quantification of expected field reliability Quantification of expected field reliability Clear path to continuous process improvement Clear path to continuous process improvement

Copyright © 2003 Software Quality Research Laboratory Essential Process Elements Requirements Definition Requirements Definition Robust Architecture Definition Robust Architecture Definition Sequence-Based Specification Sequence-Based Specification Domain Expert and Peer Review Domain Expert and Peer Review Coding Coding 100% Code Inspection 100% Code Inspection Model-Based Testing/Certification Model-Based Testing/Certification Incremental Release Incremental Release Fill the gaps with rigorous engineering practices.