Six Sigma and CMMI : Showdown in Software Process Engineering Jonathan Beckham

Process Improvement Goals Process Goals –Improve schedule and budget predictability –Improve cycle time / time to market –Increase productivity –Improve quality (as measured by defects) –Increase customer satisfaction –Improve employee morale –Decrease cost of quality

Process Improvement Goals Companies want to increase Return On Investment

What affects software?

Six Sigma: What is it? According to David Card, (IEEE Software 2000), Six sigma is “A generic quantitative approach to improvement that applies to any process.” “Six Sigma is a disciplined, data-driven approach and methodology for eliminating defects (driving towards six standard deviations between the mean and the nearest specification limit) in any process -- from manufacturing to transactional and from product to service.” –isixsigma.com

What’s the missing piece? In essence, six sigma is an approach to finding the cause of business problems and solving them making an impact on the bottom line of a corporation and decreasing variation for products. Six Sigma is more of a methodology or a philosophy.

Six Sigma Levels Six Sigma can be identified at three levels: –Metric –Methodology –Philosophy

Metric 3.4 Defects Per Million Opportunities (DPMO) DPMO allows you take complexity of product/process into account Refers to statistics. The +-6 standard deviations (sigmas) away from the standard mean Most companies are within 3 standard deviations of the mean

Metric The higher the sigma value, the better the process is capable of producing defect free results. To be in this six sigma range you must have a quality product % of the time.

Philosophy Reduce variation in your business and make customer-focused, data driven decisions Translate customer needs into operational measurements

Methodology Structured problem solving techniques and roadmap Two primary sub-methodologies in Six Sigma –DMAIC –DMADV

DMAIC Define-Measure-Analyze-Improve-Control Tool for incremental process improvement of existing processes within an organization that fall below required specification

DMAIC cont. Define –Define the problem or process to improve upon related to the customer and goals Measure –How can you measure this process in a systematic way? Analyze –Analyze the process or problem and identify the way in which it can be improved. What are the root causes of problems within the process? Improve –Once you know the causes of the problems, present solutions for them and implement them Control –Utilize Statistical Process Control to continuously measure your results and ensure you are improving –Several Software Packages available to assist in measuring yield, defects per million opportunities, etc.

DMADV Define-Measure-Analyze-Design-Verify Methodology for producing new processes that meet the Six Sigma Quality levels desired Similar to DMAIC, however, we have a design stage here DFSS –Design-For-Six-Sigma –Using models or prototypes to create designs and ensure they are effective in meeting goals

DMAIC VS. DMADV

Six Sigma Personnel Champion –Manager/Director/Executive who makes sure the resources are in place for a Six Sigma project Master Black Belts –Quality experts in an organization –Responsible for strategic implementation –Teach/Mentor other Black and Green Belts Black Belts –Six Sigma team leaders responsible for implementing process improvement projects within the business Green Belts –Employee of an organization that has some training in Six Sigma and may lead a Six Sigma project, but only as part of their job

CMM: What is it? “A Capability Maturity Model (CMM) is a reference model of mature practices in a specified discipline, used to improve and appraise a group’s capability to perform that discipline” -SEI CMM contains a list of key process areas that includes software engineering, project management, and process improvement divided into different levels

CMMI CMM sunset –No more SEI CMM training since 2003 Some vendors may still offer it –CMM Appraisals will end this year SEI states other vendors may offer this but questions value Version 1.1 CMMI released in December 2001 –Adds requirements development, validation –Identifies strategy for verification and validation or products (testing) –Adds relevant stakeholders and level of involvement (Identify key people) –Less focus on “documented procedures” (Only when critical to a process)

CMMI Models Capability Maturity Model Integration brings together four disciplines –Systems Engineering (SE) –Software Engineering (SW) –Integrated Product and Process Development (IPPD) (Level 3 add on) – Supplier Sourcing (SS) or Acquisition (Level 3 add on)

CMMI Maturity Levels

Maturity Levels of Organizations 9/28/04

SW-CMMI Strengths Engineering focused which assists in integration and interaction among engineering departments Recognized Internationally DOD contracts require certain levels for bids Utilizes best practices in the industry

Six Sigma Strengths Generic Process Improvement Strategy can be applied to any process Trained personnel working on process improvements for organization on continuous basis

SW-CMMI Weaknesses Statistical methods often not used at levels 4 and 5 when they could ease data analysis –Six Sigma embraces statistical methods –Improvement in CMMI CMMI goals are not always aligned with business and customer goals –Generally Software Specific, but improving with CMMI CMMI assessors are certified, but implementers are not so they may not understand underlying concepts (statistical process control, etc) –Six Sigma has competence certifications for relevant subject matter

Six Sigma Weaknesses Derived from manufacturing which has a real tangible process that is easy to measure and apparent, as opposed to software engineering No formal connection to ISO 9000 –ISO 9000 and CMM have milestones and certifications Six Sigma relies on training personnel (black belts) that look for areas that need improvement which may not be the best way to go about this –Costs also involved in training for Six Sigma Focuses on Internal process experience and doesn’t consider external technology or best practices

CMMI Case Studies Lockheed Martin M&DS –SW CMM ML 2 (1993) –SW CMM ML 3 (1996) –CMMI SW ML5 (2002) –Increased software productivity by 30% –Decreased unit software cost by 20% –Decreased defect find and fix costs by 15%

CMMI Case Studies Bosch Gasoline Systems –CMM Level 4 –Predictability – internal on-time delivery improved by 15% –Less Rework – first pass yield improved by 10% –Product Quality – reduction in error cases in company by one order of magnitude

Six Sigma Case Study >$3 billion energy company Customer dissatisfaction with software deliverables Customer satisfaction improved by 2 points on their “report card” Product quality was improved by 30% Reduction of rework by 50% was achieved Project savings of $150 K was achieved

So which one is better? “The is no silver bullet” – Fred Brooks Neither one is necessarily “better” Both have weaknesses Both have multiple case studies and large corporations that have realized real savings and process improvements utilizing their techniques Utilizing both is possible

CMM and Six Sigma Similarities Both emphasize reducing defects as their main process improvement goal Both focus on quantitative decision making that relies on measurement –Six Sigma and CMM level 4 and 5 Both utilize similar basic statistical process control techniques –As mentioned, Six Sigma has a better foundation in this area

Using Six Sigma and CMM Six sigma provides a foundation to define, measure, analyze, improve, and control the process –Utilize process mapping Can work in conjunction with CMM to map out a process and identify gaps within software processes –Failure Modes and Effects Analysis Detects how a process can fail and assigns severity to it to prioritize actions

Using Six Sigma and CMM CMM level 4 and 5 and Six Sigma are often seen as similar: –Six Sigma training can help to deal with CMM’s Quantitative Process Management and Defect Prevention requirements –CMM can clarify how to use Six Sigma techniques for software process –Important to note that organizations can achieve level 4 and 5 without using Six Sigma techniques Many utilize their own statistical methods and analysis Sometimes ad-hoc, as mentioned before CMM does not tell you necessarily how to accomplish a level, just what need to be accomplished

Benefits from SW CMM and Six Sigma Study of utilizing them together: –Tata Consultancy Service development center reduced cost of process failures from 5 to 1 percent, thus reducing total quality cost –Improved cycle time and productivity –Scheduled slippage was reduced from a range of +-20 percent to +-2 percent saving the company roughly $700k over 3 years –Amazingly, at the time of the article, clients had not reported any defects in their software products!

Summary Six Sigma is a process methodology that focuses on quantitatively measuring a process in order to control and improve upon it Six Sigma strives for 3.4 Defects Per Million Opportunities (DPMO) so that it is within six standard deviations of the mean SW CMMI is a process improvement model for software engineering that is divided into levels and contains a list of key process areas

Summary Six Sigma and CMMI can be utilized together to improve the software engineering process Six Sigma’s more advanced statistical methods can be applied to assist in quantitatively analyzing data within CMMI’s level 4 and 5 Six Sigma can also complement any individual process within CMMI by utilizing techniques such as process mapping

Conclusion Lastly, the effects of implementing Six Sigma and CMMI can highly improve the quality of the product developed and individual processes which can directly impact a business's bottom line and customer satisfaction

References David N. Card, “Sorting out Six Sigma and the CMM”, IEEE Software, May June 2000, pp Jack Horgan, “Six Sigma and CMM”, September , 2004, M. Murugappan and G. Keeni, “Blending CMM and Six Sigma to Meet Business Goals”, IEEE Software, March/April 2003 “IT Perspective: Balancing Six Sigma and the Capability Maturity Model (CMM ®/CMMI SM )”, rticles/2002_10/six_sig.jsp rticles/2002_10/six_sig.jsp SEI Website, Six Sigma, Thomas Pyzdek, "DMAIC and DMADV", Pyzdek Consulting, Inc. Six Sigma Handbook,

Questions Or Comments?