Thomas L. Gilchrist Testing Basics Set 4: Strategies & Metrics By Thomas L. Gilchrist, 2009

Thomas L. Gilchrist What are the Tradeoffs …changes in one require compensation in others...

Thomas L. Gilchrist Tradeoffs Schedule Cost/Resources Scope Technology Quality / Critical Success Factors

Thomas L. Gilchrist How much testing? …risk!...

Thomas L. Gilchrist Testing and Risk The degree of risk increases The degree of testing

Thomas L. Gilchrist High Low High Low Where are you now? Level of testing Confidence in process TREND Testing and Risk

Thomas L. Gilchrist Testing Goals Testing is about finding errors. If you don’t find any errors, testing is a waste of time.

Thomas L. Gilchrist Testing and Risk Maturity of the development process The cost to find an error/bug

Thomas L. Gilchrist Definitions Requirement –A condition or capability needed by a user to solve a problem or achieve an objective –A condition or capability that must be met or possessed by a system …to satisfy a contract, standard, specification or other formally imposed document.

Thomas L. Gilchrist Test Terms Test Categories –Unit Testing –Integration Testing –System Testing –Acceptance Testing –Regression Testing Static Testing Dynamic Testing

Thomas L. Gilchrist The Test Process

Thomas L. Gilchrist The Testing Process Step 1 - Set Test Objectives Step 2 - Develop Test Plan Step 3 - Execute Tests Step 4 - Summarize and Report Results

Thomas L. Gilchrist General Workbench Input Source Materials Process Product ENTRYENTRY EXITEXIT Standards, templates, rules and checklists Do Work Rework Testing

Thomas L. Gilchrist Testing Workbench Test Process Products for Test Problems Tested Products ENTRYENTRY EXITEXIT Test Toolbox Test QC Test Measures

Thomas L. Gilchrist The Software Test Process Step 1 - Develop Test Objectives Task 1 - Organize Test Team Task 2 - Perform Risk Assessment Task 3 - Set Test Objectives Task 4 - Quality Control Step 2 - Develop Test Plan Task 1 - Define Business Functions Task 2 - Define Structural Functions Task 3 - Define Tests Task 4 - Create Function/Test Matrix Task 5 - Admin Test Requirements Task 6 - Formalize System Test Plan Task 7 - System Test Plan QC Task 8 - Develop Unit Test Plan Task 9 - Unit Test Plan QC Step 3 - Execute Tests Task 1 - Select Test Tools Task 2 - Develop Test Cases Task 3 - Execute Tests Task 4 - Quality Control Step 4 - Summarize and Report Results Task 1 - Record Defects Task 2 - Perform Data Reduction Task 3 - Develop Findings Task 4 - Formalize Report Task 5 - Quality Control

Thomas L. Gilchrist Test Process “V” Diagram Business Need Define Requirements Design System Build System Verify Acceptance Test System Test Integration Test Unit Test Verify Validate Validates Test Planning Test Objectives Test Execution (Static) Test Execution (Dynamic) & Test Reporting

Thomas L. Gilchrist The SEI CMM

Thomas L. Gilchrist Immature Vs Mature Organization ImmatureMature Heroic Efforts of Dedicated Team Repeating Proven Methods of an Organization Not ManagedManaged People Management Technology Process Not Fully Utilized Organization al Focus No "Memory", Improvised Repeatable, Followed, Predictable From: SEI CMM/Bill Curtis, et al.

Thomas L. Gilchrist CMM Architecture LevelKey Process Areas 1 Initial Heroes* 5 Optimizing System CQI Defect Prevention Technology Innovation Process Change Management 4 Managed System Measurement Process Measurement and Analysis Quality Management 3 Defined Engineering Technical Practices Organizational Process Focus Organizational Process Definition Peer Reviews Training Program Intergroup Coordination Software Product Engineering Integrated Software Management 2 Repeatable Project Management Software Project Planning Software Project Tracking Software Subcontract Management Software Quality Assurance Software Configuration Management Requirements Management Process Focus From: SEI CMM/Bill Curtis, et al.

Thomas L. Gilchrist Taking Action Based On Data

Thomas L. Gilchrist Measurement What’s you least favorite measurement? What’s your favorite measurement?

Thomas L. Gilchrist Designing a Metric Program GQMGQM * From "A Methodology for Collecting Valid Software Engineering Data" by Victor Basili and David Weiss. IEEE Transactions of Software Engineering, Vol. SE-10, No. 4, Nov. 1984

Thomas L. Gilchrist Designing a Metric Program Goal Q M * From "A Methodology for Collecting Valid Software Engineering Data" by Victor Basili and David Weiss. IEEE Transactions of Software Engineering, Vol. SE-10, No. 4, Nov. 1984

Thomas L. Gilchrist Designing a Metric Program Goal Question M * From "A Methodology for Collecting Valid Software Engineering Data" by Victor Basili and David Weiss. IEEE Transactions of Software Engineering, Vol. SE-10, No. 4, Nov. 1984

Thomas L. Gilchrist Designing a Metric Program Goal Question Metric * From "A Methodology for Collecting Valid Software Engineering Data" by Victor Basili and David Weiss. IEEE Transactions of Software Engineering, Vol. SE-10, No. 4, Nov. 1984

Thomas L. Gilchrist Designing a Metric Program Goal Question Metric * From "A Methodology for Collecting Valid Software Engineering Data" by Victor Basili and David Weiss. IEEE Transactions of Software Engineering, Vol. SE-10, No. 4, Nov. 1984

Thomas L. Gilchrist Understanding Metric Data Data is not information! DataInformation From: “Understanding Variation”, D.J. Wheeler

Thomas L. Gilchrist Understanding Metric Data Data is not information! DataInformationAnalysis From: “Understanding Variation”, D.J. Wheeler

Thomas L. Gilchrist Key Variation Concepts Variation exists everywhere Two types –Common causes (noise) –Special case (signal) Variation, waste, & predictability. 94% of all problems in an organization are due to common causes and only management can fix them. (Deming, “Out of Crisis”)

Thomas L. Gilchrist Run Charts Metric Time

Thomas L. Gilchrist Specification Targets Upper Specification Limit Lower Specification Limit Metric Time Voice of the customer

Thomas L. Gilchrist Shewhart’s Control Charts Every data set has noise. “Noise” confuses and clouds single value comparisons. Some data sets contain signal (special causes). Control charts separate potential signals from noise.

Thomas L. Gilchrist Control Chart Metric Time Voice of the process

Thomas L. Gilchrist Control Chart Metric Time Voice of the process Upper Control Limit Lower Control Limit

Thomas L. Gilchrist Signal and Noise Upper Control Limit Lower Control Limit Metric Time Voice of the process "Special Cause" "Common Cause" Region Data excursions

Thomas L. Gilchrist Serial Correlation Metric Time Voice of the process Upper Control Limit Lower Control Limit

Thomas L. Gilchrist Recommendations Improve measurement systems –Commit to measurement, not guessing –Devise and use common measurement tools and definitions –Train data collectors on metric definitions Reduce process variation Improve process results

Thomas L. Gilchrist Variation In Software In software, when one process has wide variation, more than likely, they all have have wide variation. High precision in software measurement is unnecessary at this time.

Thomas L. Gilchrist Suboptimization Suboptimization occurs when one doesn’t think of the total system...it is about dependencies.

Thomas L. Gilchrist X+Y=1 Suboptimization X+Y=1 Suboptimization occurs when one doesn’t think of the total system...it is about dependencies.

Thomas L. Gilchrist Suboptimization X is big and Y is big = goodness How do we allocate resources between x and Y? What are some examples? Suboptimization occurs when one doesn’t think of the total system...it is about dependencies. X+Y=1