1. Software Defect Prevention Using Orthogonal Defect Classification
Twin-SPIN
January 6, 2005
Presented by:
Megan Graham, ASQ CSQE

2. Overview
Defect prevention is a process used to improve software quality
Orthogonal Defect Classification is a tool that characterizes defect data used in defect analysis

3. Agenda
About Defect Prevention
ODC Concepts
Applying ODC to the DP Process
Summary

4. Agenda
About Defect Prevention
ODC Concepts
Applying ODC to the DP Process
Summary

5. Defining Defect Prevention
What does it mean to you?
Defect prevention is a process whose purpose is to:
identify the common causes of defects, and
change the relevant process(es) to prevent that type of defect from recurring. (SEI)
Take what we already know and apply it to what we think we know to produce quality software.
Key points:
It is a process in and of itself that focuses on the system of software development
What we already know -> DP uses historical data – either ancient history or recent history, but what’s done is done and we learn from our mistakes
Apply it: This feeds an intentional and specific action or set of actions to eliminate the conditions that allowed the defect.
What we think we know: We think that the current process will be similar enough to predict future actions.
Quality software: Quality depends on business needs and goals. Sufficiently good software, mission critical, etc.

6. A Bug’s Life Requirements Design Code Test
This simple software lifecycle shows many stages at which point bugs may be injected and/or detected.
Injected: Any activity whose purpose is to create software or analyze acceptable behavior (requirements, coding, debugging, testing).
Detection: Any activity whose purpose is to find defects (testing, review, inspections, etc.)
Bugs are injected into the process at different stages
Click: We have strategies in place to detect and kill the bugs
Click: Optimally, we detect bugs as close to the point of injection as possible – this is called defect containment
Test

7. A Bird’s Eye View of DP Defect Prevention Historical Project
Decrease
Defects
Injected
Increase
Detected
Defect
Prevention
Historical
Project
Current Project
Optimally, defect prevention uses defect data throughout the software lifecycle to improve both defect injection and defect detection processes for the next phase. This will ultimately result in higher quality software, and most likely cheaper.

8. Applying Defect Prevention
A defect in the software is also a defect in the process (injection and/or detection)
For DP to work, we need to turn software defects into actionable process defects
Software
defects
Process DP
In order for DP to work, you need to understand how the software defect manifested itself.
How do you do that?

9. And then there was ODC… Orthogonal Defect Classification
Developed at IBM in the 1990s by Ram Chillarege
Methodology to characterize software defects and translate into process defects

10. Agenda
About Defect Prevention
ODC Concepts
Applying ODC to the DP Process
Summary

11. Simple ODC Classification Scheme
Type
Defect
ODC creates defect profiles and other defect data that can be used for various metrics
Trigger

12. Defect Types
FUNCTION: Affects significant capability, end-user interfaces, product interfaces, interface with hardware architecture or global data structure(s).
LOGIC: Affects the functionality of a code module and can be fixed by re-implementing an algorithm or local data structure without a need for requesting a high level design change.
INTERFACE: Affects the interaction of components via macros, call statements and/or parameters.
CHECKING: Affects program logic that would properly validate data and values before they are stored or used in computation.
ASSIGNMENT: Requires change in a few lines of code, such as initialization of control blocks or data structures.
TIMING/SERIALIZATION: Affects the proper management of shared and real-time resources.
BUILD/PACKAGE/MERGE: Affects product version or configuration; requires a formal changes to reconfigure or rebuild the product.

13. Defect Triggers Fault Failure Trigger 1 Trigger 2 Trigger n
A trigger is any event that allowed a fault to become a failure.
Multiple triggers may uncover the same fault.
The same trigger may be used at any point in the software lifecycle.
There are 3 families of triggers: review/inspection, function test, and system test.

14. Review/Inspection Triggers
DESIGN CONFORMANCE: Comparing the implemented design against a reference –design document, pattern, or guideline.
LOGIC/FLOW: Checking for correctness or flaws using knowledge of the practice.
BACKWARD COMPATIBILITY: Examining compatibility with prior version of the product.
LATERAL COMPATIBILITY: Examining for compatibility with other products and platforms that need to work with this release.
CONCURRENCY: Serialization, shared resources, multi-threaded tasks, timing, etc.
INTERNAL DOCUMENT: Inconsistencies in prologs, and sections in the same work product.
LANGUAGE DEPENDENCY: Programming standards, specific implementation considerations, environment restrictions, execution modes, etc.
SIDE EFFECTS: Usage behavior beyond design, but relevant in context. Do A; B happens.
RARE SITUATION: Unusual issues related to idiosyncrasy of environment, hardware, or software.

15. Function Test Triggers
SIMPLE PATH: White box – Straightforward usage of code path and branches.
COMPLEX PATH: White box – Exercising conditionals, and circuitous coverage.
COVERAGE: Black box – Straightforward usage of function or single parameterized execution.
VARIATION: Black box – Straightforward like coverage, but with a variety of parameters.
SEQUENCING: Black box – Multiple functions, with a specific sequence (order is important).
INTERACTION: Black box – When two or more bodies of code are involved (order is not important).

16. System Test Triggers
WORKLOAD STRESS: Pushing the limits of performance, resources, users, queues, traffic, etc.
RECOVERY: Invoke exception handling, recovery, termination, error percolation, etc.
STARTUP/RESTART: Major events of turning on, off, or changing the degree of service availability.
HARDWARE CONFIGURATION: Issues surfaced as a consequence of changes in hardware setup.
SOFTWARE CONFIGURATION: Issues surfaced as a consequence of changes in software setup.
BLOCKED TEST/NORMAL MODE: Test could not run during System Test, or customer found nonspecific trigger. Look for additional trigger.

17. ODC v5.11 Classification Scheme
Trigger
Activity
Impact
Opener
Closer
Defect
Age
Target
IBM uses more fields, with their own field options
This scheme can be customized to fit your own organization’s process and metrics needs
Type
Source
Qualifier
Source: ODC v5.11, IBM Center for Software Engineering,

18. Agenda
About Defect Prevention
ODC Concepts
Applying ODC to the DP Process
Summary

19. When to Apply DP
DP can be applied to one or more phases of the software lifecycle
Depends on maturity, goals, etc.
RQTS
DSGN
CODE
TEST
Defect Prevention
Field
Analysis
Action

20. Defect Analysis Orthogonal Defect Classification
What are the attributes of the defects?
Defect Causal Analysis
When are defects being injected?
Defect Trigger Analysis
How are defects being detected?
Two types of analyses
The more data you collect, the more you can analyze (but be careful to avoid collecting data just to collect data – use it or lose it)
ODC provides a structure for the defect data

21. Defect Causal Analysis
Purpose: Characterize process issues that lead to injection of defects.
Step 1: ID a set of defects.
Step 2: Identify Defect Types with team of experts.
Step 3: Plot the distribution of Defect Types.
Step 4: Map Defect Types back to development activity.
Step 5: Develop action plan to address process deficiencies.
Step 6: Monitor process to ensure changes were effective.
If you are performing the typing in hindsight, best to get a team of engineers that were involved with the code
If you add the field to the bug reports, then you just need to plot periodically

22. Defect Type Mapped to Phase
H/L Level Design
Interface
Configuration Mgmt
Build/Package/Merge
Implementation
Timing/Serialization
Assignment
LLD/Implementation
Checking
Low Level Design
Logic
Architecture/HLD
Function

23. Sample Distribution
Plot number or percentage, then nice to plot biggest to smallest; work on biggest first

24. Defect Trigger Analysis
Purpose: Characterize process issues that allowed defects to escape to later phases.
Step 1: ID a set of defects.
Step 2: Identify Defect Trigger with team of experts.
Step 3: Plot distributions: Defect Trigger by Family, Review Triggers, Function Test Triggers, System Test Triggers.
Step 4: Map Defect Trigger family back to detection activity(ies).
Step 5: Develop action plan to increase missed Defect Triggers.
Step 6: Monitor process to ensure changes were effective.
DTA can be used in two ways; one, for process improvement, and 2, for status

25. Defect Trigger Family Mapped to Phase
Testing in real-world scenarios or extreme scenarios
System Test
Testing of specified scenarios
Function Test
Peer Reviews or Inspections
Review/Inspection

26. Sample Distributions

27. Additional Use for DTA Purpose: Determine quality status.
Step 1: Plot distribution of Defect Triggers after each detection activity.
Step 2: Compare to historical defect profile to determine if profile is as expected.
Step 3: Develop action plan, if necessary, to get back on track.
If your project thinks it’s one phase, but the distribution is of an earlier phase, then you’re really in the earlier phase!

28. Agenda
About Defect Prevention
ODC Concepts
Applying ODC to the DP Process
Summary

29. The Big Picture: DP & ODC
Mission: Improve software quality by using readily available data to decrease defects injected and increase defects detected.
Defect Prevention
Analysis
Action
RQTS
DSGN
CODE
TEST
Field
ODC Classification of Software Defects

30. Important Points about ODC
A defect in the software is a defect in the process.
Implementing ODC is very cost-effective
Enhances data already collected (software defects)
Adding fields that are completed real-time make data collection virtually free!
Tooled to quickly identify process defects (mapping)
ODC can be implemented in stages
Start with field defects, then move to in-process analysis
Utilize defect profiling in-process to predict quality and project status
Fields can be tailored to your own organization

31. References
Handbook of Software Reliability Engineering (Michael R. Lyu, Editor; IEEE Computer Society Press/McGraw-Hill)
Ram Chillarege (
IBM Research Center for Software Engineering (
Soheil Khajenoori, SIAGroup