1. CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved CSE7315M37 Slide 1 January 10, 2004 SMU CSE 7315 / NTU SE 584-N Planning and Managing a Software Project Module 37 Process Appraisal and Assessment

2. CSE7315M37 Slide # 2 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Objective of This Module To examine methods of appraising and assessing process maturity

3. CSE7315M37 Slide # 3 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Process Appraisal “If you don’t know where you are, a map won’t help” (1) An appraisal is a way to determine where you are (1) Humphrey (textbook)

4. CSE7315M37 Slide # 4 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Why Do Appraisals? Most organizations suffer because they cannot see their problems, not because they cannot solve them. Most organizations are focused on defining solutions, not on defining problems. Most organizations tend to solve symptoms, rather than the fundamental, underlying problems. Why not just solve the problems?

5. CSE7315M37 Slide # 5 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Responding to Symptoms Problem Symptom Response Longer Term Cure Response Temporary Cure

6. CSE7315M37 Slide # 6 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Types of “Appraisals” Reviews Audits Capability Evaluations Self Assessments Other Appraisals

7. CSE7315M37 Slide # 7 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Possible Exam Question  Compare and contrast the different forms of appraisal with respect to: – Cost – Length – Who Performs – Objectives – Advantages – Drawbacks – etc.

8. CSE7315M37 Slide # 8 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Reviews Purpose: To learn the status of the project Performed by: Managers Method: Practitioners report on the status and plans of their projects, following specified formats and reporting on specific metrics

9. CSE7315M37 Slide # 9 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Reviews (continued) Typical Duration: A few hours to several days Achieves: Uncovers problems (or, at least, symptoms) Drawbacks: Does not identify solutions May motivate hiding of problems Advantages: Relatively inexpensive Tends to get everyone back on the same track

10. CSE7315M37 Slide # 10 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Audits Purpose: To study a project in detail and find problems. Also to keep things on track Performed by: Independent technical experts, often outsiders Method: Experts question practitioners and examine artifacts of their process to determine what is happening Typical Duration: Several days to several weeks

11. CSE7315M37 Slide # 11 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Audits (continued) Achieves: Uncovers real problems more often Informs staff that management cares about the results Advantages: Tends to uncover real problems Tends to confirm or disprove suspected problems

12. CSE7315M37 Slide # 12 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Audits (continued) Drawbacks: More expensive than a review Does not identify solutions May motivate hiding of problems Can generate hostility and lack of cooperation Can de-motivate Outsiders often do not understand and even if they do, they are not believed You don’t trust us!

13. CSE7315M37 Slide # 13 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Capability Evaluations Purpose: To determine if an organization is capable of doing a job well Performed by: Outside experts in all affected areas Method: -- Many methods have been used for years -- For example, each of the armed services has ways of doing contractor capability evaluations before selecting contractors -- SEI has defined the SCE method, based on the CMM

14. CSE7315M37 Slide # 14 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved SEI Definition Typical duration: several days A software capability evaluation is an appraisal by a trained team of professionals to identify contractors who are qualified to perform the software work or to monitor the state of the software process used on an existing software effort.

15. CSE7315M37 Slide # 15 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Capability Evaluations (continued) The SEI SCE (Software Capability Evaluation) is like an organized audit where the CMM is the model against which the program is evaluated CMM 1.1

16. CSE7315M37 Slide # 16 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved SEI Capability Evaluations What They Achieve They are based on well established best practices for software development So the results can be seen in comparison with accepted best practices They uncover problems (or, at least, symptoms) They note lack of problems They note capabilities

17. CSE7315M37 Slide # 17 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved SEI Capability Evaluations Drawbacks They do not identify solutions They may motivate hiding of problems They are relatively expensive Inexperienced evaluators can misinterpret the findings, resulting in incorrect or unfair results – Example: focusing on practices rather than goals and commitments

18. CSE7315M37 Slide # 18 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved SEI Capability Evaluations Drawbacks (continued) Inexperienced or immature managers may not understand the conclusions, interpret them correctly, or act appropriately – Example: instituting metrics when the problem is communication

19. CSE7315M37 Slide # 19 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved “Self” Assessments Purpose: To help an organization learn where it is, how to improve, and what to do next Performed by: A team of inside and/or outside technical and management experts Method: There are many methods of self assessment SEI has defined the two methods of software capability assessment based on the CMM There have been several versions of the SEI method, which can cause confusion

20. CSE7315M37 Slide # 20 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved SEI Definition (from Paulk) “A software process assessment is an appraisal by a trained team of software professionals to determine the state of an organization's current software process, to determine the high-priority software process-related issues facing an organization, and to obtain the organizational support for software process improvement.”

21. CSE7315M37 Slide # 21 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved “Self” Assessments Achieve They uncover problems They identify recommended solutions They an be used to motivate – People “buy in” when they participate in the assessment

22. CSE7315M37 Slide # 22 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved “Self” Assessment Drawbacks Can mislead if too many internal participants Can be misused as a sort of audit or proof of capability Can be very expensive Can do more harm than good if practitioners are motivated to change and management does not support the changes

23. CSE7315M37 Slide # 23 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Assessment vs. Evaluation CMM 1.0 Evaluations Find Problems Evaluate Capability For External Use Assessments Find Problems Find Solutions Motivate Improvement For Internal Use CMM 1.1 CMM 1.0 Humphrey 5-Level Model

24. CSE7315M37 Slide # 24 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved SEI Assessment Methods SPA – Software Process Assessment CBA-IPI – CMM Based Appraisal for Internal Process Improvement With both methods, SEI receives results and records confidentially. Results are published in aggregate form, quarterly.

25. CSE7315M37 Slide # 25 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved SEI Publishes a Maturity Profile http://www.sei.cmu.edu/sema/profile.html

26. CSE7315M37 Slide # 26 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Maturity Profile as of March, 2001

27. CSE7315M37 Slide # 27 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved SPA Method Version 1 was based on a questionnaire or survey, derived from Watts Humphrey’s model Typical duration: 1 week Several intermediate versions have been based on intermediate versions of the CMM The most recent and current version is based on CMM 1.1 In each case there have also been several degrees of “certification” of assessors, leading to many misunderstandings and unverifiable claims

28. CSE7315M37 Slide # 28 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Training for SEI Assessments Initial Assessment Method (1988-93) – Assessors were trained by SEI – Later, assessors were given “certification” by SEI – An assessment was deemed “more reliable” if one or more trained and/or certified assessors were involved – See Humphrey, chapter 3, for more details Later Assessment Methods – Individuals and/or organizations certified as assessment team leaders and/or as trainers

29. CSE7315M37 Slide # 29 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Confidentiality SEI does not release results of any specific assessment or confirm or deny the validity of any assessment They only release composite results to show overall industry characteristics and trends

30. CSE7315M37 Slide # 30 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved CBA-IPI Method Typical duration: 1-2 weeks More comprehensive than SPA Places increased emphasis on CMM, when compared with the SPA method Intended to foster improvement for an organization, not just focused on software May be conducted by internal or external personnel Must be “led” by an “SEI Certified Lead Appraiser”

31. CSE7315M37 Slide # 31 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved CMM-based Appraisals Maturity Levels Key Process Areas Common Features Key Practices Goals Implementation Infrastructure or Activities Process Capability Address Describe Achieve Contain Organized by Contain Indicate Questions

32. CSE7315M37 Slide # 32 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved SPA and CBA-IPI Issues These appraisals are intended to help an organization evaluate itself – But too often, organizations will publicize their appraisal results in the form of an achieved SEI maturity level – SEI states that “when the focus is primarily on achieving a maturity level, it can distort the purpose of process improvement by diverting attention from genuine process improvement activities”

33. CSE7315M37 Slide # 33 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved SPA and CBA-IPI Issues (continued) Contracting agencies, especially in the government, may encourage incorrect use of assessments by insisting on specific SEI maturity levels for contract eligibility SEI specifically states that “finding a maturity level is optional” for SPA and CBA-IPI and “not intended or appropriate” for an SCE

34. CSE7315M37 Slide # 34 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved More SPA and CBA-IPI Issues Assessment expense has generated many lesser approaches – Typical appraisal cost: $80,000 (large org) Assessment team (external): $45,000 Preparation $15,000 Time for participants$20,000 – Alternatives are OK for individual use, but the less extensive the less reliable the results – Lack of external participation, while saving money, can also result in too-harsh or too- lenient appraisals

35. CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved CSE7315M37 Slide 35 January 10, 2004 Low Cost Assessment Approaches

36. CSE7315M37 Slide # 36 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved “Temperature Taker” Select one or two random questions pertaining to key practices of a given KPA (a total of 6-12 questions for level 2, for example) Ask these questions of each project and discuss answers with participants Tally results at organization level to find strengths and weaknesses

37. CSE7315M37 Slide # 37 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Characteristics of “Temperature Taker” Costs about 2 hours per project to collect data plus a few hours to tally Spots significant issues And communicates the organization’s concern about maturity But may miss significant issues

38. CSE7315M37 Slide # 38 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved “Continuous Record” Organization selects key practices to focus on within each KPA Each project maintains a list of key practices and a chart of whether they do each practice Display a graph in the work area and report monthly to the organization Tally at the organization level

39. CSE7315M37 Slide # 39 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Characteristics of “Continuous Record” Costs less than 1 hour per project per reporting period Spots issues and trends And communicates the organization’s concern about maturity But relies on project self-assessment, so may still miss significant issues

40. CSE7315M37 Slide # 40 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved “Internal Assessment” Organization defines an assessment team of internal individuals Periodically, this team does an assessment Results are reported as in a full assessment

41. CSE7315M37 Slide # 41 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Characteristics of “Internal Assessment” Costs less than a full assessment – No external participants – Assessors have a smaller “learning curve” Spots issues and trends And communicates the organization’s concern about maturity But relies on internal participants, so may still miss significant issues or blow issues out of proportion

42. CSE7315M37 Slide # 42 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Module Summary Appraisals of various kinds use the CMM or other models to evaluate an organization’s maturity The more effective appraisal methods identify solutions as well as problems and foster buy-in and acceptance at all levels in the organization

43. CSE7315M37 Slide # 43 January 10, 2004 CSE 7315 - SW Project Management / Module 37 - Process Appraisal and Assessment Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved END OF MODULE 37