1 Convener: Houman Younessi Convener: Houman Younessi Software Engineering Management Software Engineering Management Course # CISH-6050 Lecture 7: Software Process Assessment 07/09/2012

2 CISH Software Engineering Management AGENDA … Process Assessment -SEI SW-CMM Improvement Approach -Do organizational assessments pay off? -Whistleblowers -Reference: SPICE Assessment Documents Process Assessment -SEI SW-CMM Improvement Approach -Do organizational assessments pay off? -Whistleblowers -Reference: SPICE Assessment Documents

3 CISH Software Engineering Management SW-CMM Process Assessment SEI SW-CMM Improvement Approach: IDEAL -Initiating -Diagnosing -Establishing -Acting -Leveraging Diagnose process, establish priorities, & act on them SEI SW-CMM Improvement Approach: IDEAL -Initiating -Diagnosing -Establishing -Acting -Leveraging Diagnose process, establish priorities, & act on them

4 CISH Software Engineering Management SW-CMM: IDEAL Initiating: -Stimulus for improvement -Set context & sponsorship -Establish improvement infrastructure Diagnosing: -Appraise & characterize current practice -Develop recommendations -Document phase results Initiating: -Stimulus for improvement -Set context & sponsorship -Establish improvement infrastructure Diagnosing: -Appraise & characterize current practice -Develop recommendations -Document phase results

5 CISH Software Engineering Management SW-CMM: IDEAL … Establishing: -Set strategy & priorities -Establish process action items -Plan actions Acting: -Define processes and measures -Plan & execute pilots -Plan, execute, & track installation Establishing: -Set strategy & priorities -Establish process action items -Plan actions Acting: -Define processes and measures -Plan & execute pilots -Plan, execute, & track installation

6 CISH Software Engineering Management SW-CMM: IDEAL … Leveraging: -Document & analyze lessons -Revise organizational approach Leveraging: -Document & analyze lessons -Revise organizational approach

7 CISH Software Engineering Management SW-CMM Process Assessment General Classes of SW-CMM Appraisal 1.Software Process Assessment  Determine state of organization’s software process 2.Software Capability Evaluations  Identify contractors qualified to perform software work General Classes of SW-CMM Appraisal 1.Software Process Assessment  Determine state of organization’s software process 2.Software Capability Evaluations  Identify contractors qualified to perform software work

8 CISH Software Engineering Management SW-CMM Process Assessment … Software Process Assessment: -Identify improvement priorities within organization -Assessment team uses CMM to guide identifying & prioritizing findings -Findings & KPA guidance used to plan improvement strategy for organization Software Process Assessment: -Identify improvement priorities within organization -Assessment team uses CMM to guide identifying & prioritizing findings -Findings & KPA guidance used to plan improvement strategy for organization

9 CISH Software Engineering Management SW-CMM Process Assessment … Software Capability Evaluations: -Identify risks associated with a project or contract to build high quality on schedule & budget -During acquisition process, capability evaluation may be performed on bidders -Findings of an evaluation may be used to identify risk with using a contractor -Performed on existing contracts to monitor process performance Software Capability Evaluations: -Identify risks associated with a project or contract to build high quality on schedule & budget -During acquisition process, capability evaluation may be performed on bidders -Findings of an evaluation may be used to identify risk with using a contractor -Performed on existing contracts to monitor process performance

10 CISH Software Engineering Management Software Process Assessment & Capability Evaluation Steps:

11 CISH Software Engineering Management SW-CMM Process Assessment … Common Steps: -Team Selection  Select team trained in CMM  Knowledgeable in SE & mgmt -Maturity Questionnaire  Site reps complete questionnaire -Response Analysis  Analyze results of questionnaire  Investigation areas = KPAs Common Steps: -Team Selection  Select team trained in CMM  Knowledgeable in SE & mgmt -Maturity Questionnaire  Site reps complete questionnaire -Response Analysis  Analyze results of questionnaire  Investigation areas = KPAs

12 CISH Software Engineering Management SW-CMM Process Assessment … Common Steps … -On-site Visit  Using results analysis, conduct on- site visit to view process areas  Using KPAs as guide, question, listen, review & synthesize info  Apply professional judgment  Document rationale for situations where KPAs not met Common Steps … -On-site Visit  Using results analysis, conduct on- site visit to view process areas  Using KPAs as guide, question, listen, review & synthesize info  Apply professional judgment  Document rationale for situations where KPAs not met

13 CISH Software Engineering Management SW-CMM Process Assessment … Common Steps … -Findings  At end of on-site period, team produces list of findings  Identifies strengths & weaknesses of org’s software processes  Software Process Assessment -> Basis for PI recommendations  Software Capability Evaluation -> Findings part of risk analysis Common Steps … -Findings  At end of on-site period, team produces list of findings  Identifies strengths & weaknesses of org’s software processes  Software Process Assessment -> Basis for PI recommendations  Software Capability Evaluation -> Findings part of risk analysis

14 CISH Software Engineering Management SW-CMM Process Assessment … Common Steps … -KPA Profile  Team prepares KPA profile, showing where KPAs satisfied / not satisfied by organization  KPA can be satisfied and still have associated findings, as long as findings don’t identify major problems achieving goals of KPA Common Steps … -KPA Profile  Team prepares KPA profile, showing where KPAs satisfied / not satisfied by organization  KPA can be satisfied and still have associated findings, as long as findings don’t identify major problems achieving goals of KPA

15 CISH Software Engineering Management SW-CMM Process Assessment … Differences: Process Assessments & Capability Evaluation -Results of process assessment or capability evaluation may differ -Assessment / evaluation scope may vary:  Different definitions of ‘Organization’  Org may be based on senior management, geo location, common app, profit/loss center, etc.  Sample of selected projects Differences: Process Assessments & Capability Evaluation -Results of process assessment or capability evaluation may differ -Assessment / evaluation scope may vary:  Different definitions of ‘Organization’  Org may be based on senior management, geo location, common app, profit/loss center, etc.  Sample of selected projects

16 CISH Software Engineering Management SW-CMM Process Assessment … Differences … -Motivation, objective, outcome & results ownership differ  These factors lead to differences in dynamics of interviews, scope of inquiry, info collected, & results  Assessment & evaluation methods are different  Assessment training doesn’t prepare team to do evaluation, vice versa Differences … -Motivation, objective, outcome & results ownership differ  These factors lead to differences in dynamics of interviews, scope of inquiry, info collected, & results  Assessment & evaluation methods are different  Assessment training doesn’t prepare team to do evaluation, vice versa

17 CISH Software Engineering Management SW-CMM Process Assessment … Differences … -Process Assessment – performed in open, collaborative environment  Commitment from mgmt & staff to do process improvement  Objective: surface problems & help improve organization  Emphasis on interviews as tool for understanding organization’s software process Differences … -Process Assessment – performed in open, collaborative environment  Commitment from mgmt & staff to do process improvement  Objective: surface problems & help improve organization  Emphasis on interviews as tool for understanding organization’s software process

18 CISH Software Engineering Management SW-CMM Process Assessment … Differences … -Capability Evaluation – performed in audit-oriented environment  Objective tied to monetary considerations  Emphasis on documented audit trail that reveals software process actually implemented by organization Differences … -Capability Evaluation – performed in audit-oriented environment  Objective tied to monetary considerations  Emphasis on documented audit trail that reveals software process actually implemented by organization

19 CISH Software Engineering Management AGENDA … Process Assessment -SEI SW-CMM Improvement Approach -Do organizational assessments pay off? -Whistleblowers -Reference: SPICE Assessment Documents Process Assessment -SEI SW-CMM Improvement Approach -Do organizational assessments pay off? -Whistleblowers -Reference: SPICE Assessment Documents

20 CISH Software Engineering Management Software Process Assessments Why do process assessment? -“Why Do Organizations Have Assessments? Do They Pay Off?”, CMU SEI Technical Report, July, Panel discussion from SEPG ’99 in Atlanta, GA on March 10, Each panelist is one of most active lead assessor for SEI for CMM-Based Appraisals for Internal Process Improvement (CBA IPI) Why do process assessment? -“Why Do Organizations Have Assessments? Do They Pay Off?”, CMU SEI Technical Report, July, Panel discussion from SEPG ’99 in Atlanta, GA on March 10, Each panelist is one of most active lead assessor for SEI for CMM-Based Appraisals for Internal Process Improvement (CBA IPI)

21 CISH Software Engineering Management Software Process Assessments … Electronic Data Systems -Why do assessments?  Driven by customer: higher maturity to keep customers; extra bidding points for higher maturity; DoD requirements -Do assessments pay off?  Yes!  Need exec sponsorship, funding  Everyone learns, helps org culture Electronic Data Systems -Why do assessments?  Driven by customer: higher maturity to keep customers; extra bidding points for higher maturity; DoD requirements -Do assessments pay off?  Yes!  Need exec sponsorship, funding  Everyone learns, helps org culture

22 CISH Software Engineering Management Software Process Assessments … Reuters -Why do assessments?  Framework for continuous process improvement; maturity levels build on previous level  Raises awareness in group -Do assessments pay off?  Clear view of cost, effort, resource  Measure key performance items  Method for introducing global change Reuters -Why do assessments?  Framework for continuous process improvement; maturity levels build on previous level  Raises awareness in group -Do assessments pay off?  Clear view of cost, effort, resource  Measure key performance items  Method for introducing global change

23 CISH Software Engineering Management Software Process Assessments … U. S. Navy -Why do assessments?  Early SEI affiliate  Beta test group for Maturity Questionnaire  Continued with SEI assessments -Do assessments pay off?  Invaluable to organization  In 1997 determined 4.1 to 1 ROI  Education for new team members U. S. Navy -Why do assessments?  Early SEI affiliate  Beta test group for Maturity Questionnaire  Continued with SEI assessments -Do assessments pay off?  Invaluable to organization  In 1997 determined 4.1 to 1 ROI  Education for new team members

24 CISH Software Engineering Management Software Process Assessments … Panel Q&A -Why pay cost of doing CBA IPI? -How often do you do full CBA IPI? -How small of organization to do CBA IPI? -How to reduce impact/stress on org when doing CBA IPI? Panel Q&A -Why pay cost of doing CBA IPI? -How often do you do full CBA IPI? -How small of organization to do CBA IPI? -How to reduce impact/stress on org when doing CBA IPI?

25 CISH Software Engineering Management AGENDA … Process Assessment -SEI SW-CMM Improvement Approach -Do organizational assessments pay off? -Whistleblowers -Reference: SPICE Assessment Documents Process Assessment -SEI SW-CMM Improvement Approach -Do organizational assessments pay off? -Whistleblowers -Reference: SPICE Assessment Documents

26 CISH Software Engineering Management Software Process Assessments … Blowing the Whistle on Troubled Software Projects -Article by Mark Keil & Daniel Robey -Controversy behind going forward to management when auditors find problems with projects -Case study of IS Auditors  1999 Study  75 Auditors Blowing the Whistle on Troubled Software Projects -Article by Mark Keil & Daniel Robey -Controversy behind going forward to management when auditors find problems with projects -Case study of IS Auditors  1999 Study  75 Auditors

27 CISH Software Engineering Management Software Process Assessments … Key Points -Auditor obligation vs. risk -Reluctance to blow the whistle -Reluctance to hear the whistle -Conditions that encourage or influence whistle blowing Key Points -Auditor obligation vs. risk -Reluctance to blow the whistle -Reluctance to hear the whistle -Conditions that encourage or influence whistle blowing

28 CISH Software Engineering Management AGENDA … Process Assessment -SEI SW-CMM Improvement Approach -Do organizational assessments pay off? -Whistleblowers -Reference: SPICE Assessment Documents Process Assessment -SEI SW-CMM Improvement Approach -Do organizational assessments pay off? -Whistleblowers -Reference: SPICE Assessment Documents

29 CISH Software Engineering Management SPICE Assessment: Reference SPICE Software Process Assessment Information -SPICE documentation available free of charge from Software Quality Institute web site: wnload.html SPICE Software Process Assessment Information -SPICE documentation available free of charge from Software Quality Institute web site: wnload.html

30 CISH Software Engineering Management SPICE Assessment: Reference … SPICE Part 3 – Rating Process -Section 4.2: Defining Assessment Input -Section 4.4: Assessing and Rating Process -Section 4.5: Recording the Assessment Output SPICE Part 3 – Rating Process -Section 4.2: Defining Assessment Input -Section 4.4: Assessing and Rating Process -Section 4.5: Recording the Assessment Output

31 CISH Software Engineering Management SPICE Assessment: Reference … SPICE Part 4 – Guide to Conducting Assessment -Section 4.2: Process Rating Scheme -Section : Team-based Assessment -Section 4.4: Assessment Stages (8) -Section 4.5: Success Factors for Process Assessment -Section 5.1: Reviewing Assessment Inputs SPICE Part 4 – Guide to Conducting Assessment -Section 4.2: Process Rating Scheme -Section : Team-based Assessment -Section 4.4: Assessment Stages (8) -Section 4.5: Success Factors for Process Assessment -Section 5.1: Reviewing Assessment Inputs

32 CISH Software Engineering Management SPICE Assessment: Reference … SPICE Part 4 … -Section 5.2: Selecting Process Instances -Section 5.3: Preparing for Team-based Assessment -Section 5.4: Collecting and Verifying Information -Section 5.5: Determine Actual Ratings for Process Instances -Section 5.6: Determining Derived Ratings -Section 5.7: Validating the Ratings -Section 5.8: Presenting Assessment Output SPICE Part 4 … -Section 5.2: Selecting Process Instances -Section 5.3: Preparing for Team-based Assessment -Section 5.4: Collecting and Verifying Information -Section 5.5: Determine Actual Ratings for Process Instances -Section 5.6: Determining Derived Ratings -Section 5.7: Validating the Ratings -Section 5.8: Presenting Assessment Output

33 CISH Software Engineering Management SPICE Assessment: Reference … SPICE Part 5 – Construction, Selection, & Use of Assessment Tools -Section 4.1: Form and Purpose of an Assessment Instrument -Section 4.2 (and Annexes A, B, C, D): Implementation of Standard Indicators -Section 4.3: Tailoring of Indicators Contained in Assessment Instrument -Section 4.4: Modular Assessment Instruments SPICE Part 5 – Construction, Selection, & Use of Assessment Tools -Section 4.1: Form and Purpose of an Assessment Instrument -Section 4.2 (and Annexes A, B, C, D): Implementation of Standard Indicators -Section 4.3: Tailoring of Indicators Contained in Assessment Instrument -Section 4.4: Modular Assessment Instruments

34 CISH Software Engineering Management SPICE Assessment: Reference … SPICE Part 5 … -Section 4.5: Capturing and Processing Assessment Data -Section 4.6: Using an Assessment Instrument SPICE Part 6 – Qualifications & Training of Assessors -Section 4.1: Role of the Assessor -Section 5: Assessor Competence SPICE Part 5 … -Section 4.5: Capturing and Processing Assessment Data -Section 4.6: Using an Assessment Instrument SPICE Part 6 – Qualifications & Training of Assessors -Section 4.1: Role of the Assessor -Section 5: Assessor Competence

35 CISH Software Engineering Management References M. Paulk, B. Curtis, M. B. Chrissis, C. V. Weber, "Capability Maturity Model for Software", Version 1.1, Software Engineering Institute, Carnegie Mellon University, February, Available at W. S. Humphrey, Managing the Software Process, Addison-Wesley, Reading, MA, 1989 M. Paulk, C. V. Weber, B. Curtis, M. B. Chrissis, The Capability Maturity Model: Guidelines for Improving the Software Process, Addison-Wesley, Boston, MA, 1995 “Process Maturity Profile of the Software Community 2002 Mid-Year Update”, Software Engineering Institute, Carnegie Mellon University, August, Available at CMM/2002aug.pdf M. Paulk, B. Curtis, M. B. Chrissis, C. V. Weber, "Capability Maturity Model for Software", Version 1.1, Software Engineering Institute, Carnegie Mellon University, February, Available at W. S. Humphrey, Managing the Software Process, Addison-Wesley, Reading, MA, 1989 M. Paulk, C. V. Weber, B. Curtis, M. B. Chrissis, The Capability Maturity Model: Guidelines for Improving the Software Process, Addison-Wesley, Boston, MA, 1995 “Process Maturity Profile of the Software Community 2002 Mid-Year Update”, Software Engineering Institute, Carnegie Mellon University, August, Available at CMM/2002aug.pdf

36 CISH Software Engineering Management References … D. Dunaway, R. Berggren, G. des Rochettes, P. Iredale, I. Lavi, G. Taylor, “Why Do Organizations Have Assessments? Do They Pay Off?”, Software Engineering Institute, Carnegie Mellon University, July, Available at 12abstract.html M. Keil, D. Robey, “Blowing the Whistle on Troubled Software Projects”, Communications of the ACM, Vol. 44, No. 4, April 2001, pp PDF file (ACMp87-Keil.pdf) available online via ACM Digital Library when accessed from the RPI at Hartford Library ISO/IEC TR 15504: Software Process Assessment (SPICE) documentation suite. Available at D. Dunaway, R. Berggren, G. des Rochettes, P. Iredale, I. Lavi, G. Taylor, “Why Do Organizations Have Assessments? Do They Pay Off?”, Software Engineering Institute, Carnegie Mellon University, July, Available at 12abstract.html M. Keil, D. Robey, “Blowing the Whistle on Troubled Software Projects”, Communications of the ACM, Vol. 44, No. 4, April 2001, pp PDF file (ACMp87-Keil.pdf) available online via ACM Digital Library when accessed from the RPI at Hartford Library ISO/IEC TR 15504: Software Process Assessment (SPICE) documentation suite. Available at