1. © John Durbin Slide 1May 2004 At the Chicago SPIN Metrics - Creating a Balanced Scorecard aligned to Corporate Objectives and Goals John Durbin Johndurbin@ieee.org Chicago - SPIN

2. © John Durbin Slide 2May 2004 At the Chicago SPIN Balanced Scorecards at the Operational Level u How to:  Align Metrics to organizational scorecards  Align Bottom-up Balances Scorecard to the next higher level u Do you have points of special interest? u Balanced Scorecard used is just an example for bottom up alignment to “organizational goals”

3. © John Durbin Slide 3May 2004 At the Chicago SPIN Topics NOT covered… u Why Balanced Scorecards were devised? u Good and Bad about Balanced Scorecards u Metrics vs. Performance indicators u Well defined metrics u Successful metrics deployment u Negative Behavioral impact of metrics u Life cycle for metrics for changing behavior u Etc.

4. © John Durbin Slide 4May 2004 At the Chicago SPIN What is a metric? u A Quantitative result from one or more measurements u And is Verifiable from a defined process Review: for Common terminology

5. © John Durbin Slide 5May 2004 At the Chicago SPIN How are metrics used? u Support Decisions or u Drive change or u Diagnose problems Review: for Common terminology

6. © John Durbin Slide 6May 2004 At the Chicago SPIN Performance Indicators - Flavors Planning Execution Lessons Learned Performance Indicators Measurements / Metrics Assessments Surveys Performance Indicators maybe used for: Review: for Common terminology

7. © John Durbin Slide 7May 2004 Where to start? u Mission (Values – Interests)  To help our clients create the metrics-driven processes and systems needed to change to be more successful. u Strategic Imperatives u Critical Success Factors (capabilities) u Key Performance Indicators (KPIs) > Goals u Align Management and Decision-Making Processes

8. © John Durbin Slide 8May 2004 Performance measures provide actionable data and use targets for achieving operational excellence in critical processes to focus on proper allocation of resources and appropriate incentive for improvement. Process Strategy Performance measures provide an ongoing mechanism for measuring an organization’s success in channeling its resources toward achieving its near-and long-term strategic goals. People Performance measures align the objectives of the individual with the overall organizational strategy ensuring an overall commitment toward common goals and objectives. Balanced Scorecard Purpose

9. © John Durbin Slide 9May 2004 At the Chicago SPIN Balanced Scorecard (Norton) Financial Internal Business Processes Learning & Growth Customer First use at Analog Devices in 1987 Objectives Measures Targets Initiatives

10. © John Durbin Slide 10May 2004 Strategic ImperativesCritical Success Factors Key Performance Indicators Or Goals Be a Full Service Sales Expertise Provider Integration w/other Business Units Build Brand Aggressive Marketing Image Project Management Expertise Increase Value Increase Service Perception Turnaround Fix Infrastructure Train People Develop Systems Win-Loss Ratio % Cross-BU Sales Milestone Completion # New Customers On-time Project Completion # New Distributors Market Share Service Response Time Customer Satisfaction Inventory Turns Training Effectiveness Milestone Completion Define Key Indicators

11. © John Durbin Slide 11May 2004 At the Chicago SPIN Need for Balanced Metrics for Performance u Avoid missing key issue u Long term focus u Do the Right Things u Avoid disasters (1-Digital Equip.> Selling incentives on $ of computers; not profits, 2- Tech. Co. > $ ship product on-time: no $ for/of repair)

12. © John Durbin Slide 12May 2004 At the Chicago SPIN A - Balanced Scorecard u Customers give us top two scores of five- 70% u Customers recommend our product - 80% u Met or exceed profit targets - $$. u Demonstrate SEI CMM level 4 by… ;-) u … u Employees more than satisfied > 75% u “Grow” 300 new Six Sigma Black Belts

13. © John Durbin Slide 13May 2004 At the Chicago SPIN A Fed. Gov. Balanced Scorecard Customer Perspective Customer Satisfaction - % of customers satisfied with timeliness - % of customers satisfied with quality Effective Service Partnership - % of customers satisfied with the responsiveness, cooperation, and communication skills of the acquisition office Internal Business Processes Perspective Acquisition Excellence: Effective Quality Control System - Ratio of protests sustained by General Accounting Office and Court of Federal Claims Acquisition Excellence: Effective use of Alternative Procurement Practices - Number of actions using Electronic Commerce Fulfilling Public Policy Objectives - % achievement of socio-economic goals - % competitive procurement of total procurements Learning and Growth Perspective Information Availability for Strategic Decision- making - The extent of reliable management information Quality Workforce - % of employees meeting mandatory qualification standards Employee Satisfaction: Quality Work Environment - % of employees satisfied with the work environment Employee Satisfaction: Executive Leadership - % of employees satisfied with the professionalism, culture, values and empowerment Financial Perspective Minimizing Administrative Costs - Cost to spend ratio Maximizing Contract Cost Avoidance - Cost avoidance through use of purchase cards - % of prompt payment interest paid of total $ disbursed

14. © John Durbin Slide 14May 2004 At the Chicago SPIN Balanced Scorecard For Operational Group u Support Organizational Goals & Results u Aligns to Top Scorecard (s) u Consistent focus and perspective u How do we get a Balanced Scorecard defined to align upward?  GE’s (conceptual) Formula view or  Pyramid conceptual model

15. © John Durbin Slide 15May 2004 At the Chicago SPIN Cook Book Recipe: Two Dimensions to Build u First, VERTICALLY Build alignment to Goals and Metrics above you. u Second, HORIZONTALLY Derive a balanced set of goals and metrics across top level perspective. u Last, Select and Prune Metrics

16. © John Durbin Slide 16May 2004 At the Chicago SPIN GE’s Big Y, little y & x Results that matter Which Major activities will enable you to achieve those results? Y = f (y 1, y 2, y 3... ) On-going Sponsorship and Review Management Stewardship

17. © John Durbin Slide 17May 2004 At the Chicago SPIN Y = f (y i ) and y i = f (x ij ) yiyi Enabling Results that matter Which Operating activities will enable you to achieve those results? = f (x i1, x i2, x i3... ) Process owner’s Review Results/ Process Focus

18. © John Durbin Slide 18May 2004 At the Chicago SPIN Project Manag’t Team or Work Cells Functional Management Stakeholders Organization Operating Units Drivers: Top- down Enablers: Bottom-up Metrics Drive Improvements Few Metrics Operating unit Scorecards Different Processes Enterprise Balanced Scorecard: Financial, Customer, Internal Business, and Innovation and learning. Metrics helps you do what you are doing while you are doing it! Conceptual Model for Your Balanced Scorecards Across different initiatives Deliverable and milestone focused Process, diagnostic and Supportive Enterprise Milestone status - Product results Key Performance Indicators Process support

19. © John Durbin Slide 19May 2004 At the Chicago SPIN Project Manag’t Team or Work Cells Functional Management Stakeholders Customers Organization other layers Drivers: Top- down Enablers: Bottom-up Metrics Drive Vital Few Metrics Customer Survey Cost/Benefits Plan Analyze Design Build Operate RUP, Agile, Waterfall Hardware Dev. Marketing - Logistics Customer Satisfaction Employee Satisfaction, ROI Profits: $, DM, FF, etc. Improvements Helps you do what you are doing while you are doing it! Align Metrics for Consistency of Purpose Across projects and Methods Deliverable and Phase focused Process, Diagnostic and task focused Market Business Unit Phase status - Exit criteria Key Process Indicators Process support

20. © John Durbin Slide 20May 2004 At the Chicago SPIN Performance Indicators Metrics else Indicators Your Process -Development -Enhancement -Support InputsOutputs Time & Cost Of resources Product Size Defects What? Why? Attributes People Tools Techniques Physical Environment What?

21. © John Durbin Slide 21May 2004 At the Chicago SPIN A Project Process’ Metrics Model Input Process Output Project Metrics Productivity Rate Product Test Defect Density Duration Variance Percentage Cost per unit code Attributes People Environment Tools Techniques sTime sCost s Size sDefects Resources & People Deliverables Time - Schedule PlanAnalyzeDesignIntegrateDeployOperateBuild Risk?

22. © John Durbin Slide 22May 2004 At the Chicago SPIN Balanced Scorecard (Norton) Financial Internal Business Processes Learning & Growth Customer First use at Analog Devices in 1987 Objectives Measures Targets Initiatives

23. © John Durbin Slide 23May 2004 At the Chicago SPIN Balanced Scorecard Horizontally u Financial Perspective: (ROI)  How do we look to our shareholders? u Internal Business Perspective: (Process)  What Business Processes must we EXCEL at? u Customer Perspective: (Vision)  How should we appear to our customers? u Innovation & Learning Perspective: (Change)  How do we continue to improve and add value?

24. © John Durbin Slide 24May 2004 At the Chicago SPIN Balanced Scorecard Vertically – A Drill Down u Improve Customer Satisfaction:  Reduce delivered software defects –Increase software Verification t Better Requirements Traceability to …. t Improve Inspections effectiveness to…. –Improve software Validation  Faster feature enhancements –Use Agile paradigm to continually verify …

25. © John Durbin Slide 25May 2004 At the Chicago SPIN Scorecard trials u Improve Customer Satisfaction  Reduce delivered software defects –Better customer acceptance coverage <??? –Greater storage for test cases <??? –New Configuration Management tools <??? –Use Agile Methods <???  What is common to the above? (Internal vs goal focus)

26. © John Durbin Slide 26May 2004 At the Chicago SPIN XYZ Retail Goods Co. u L1- Products in every Wal-Mart  L2 Timely and Accurate market research L3 Improve link to data sources l4 Improve extract timing and accuracy of market data.  L2 Lower Costs –L3 Reduce delays of misplaced goods l4 Correct tracking system for Goods availability u L1- 20% annual sales growth

27. © John Durbin Slide 27May 2004 At the Chicago SPIN Critical Success Factors for Metric Definitions u Anticipate “What’s In It For Me?”  For Sponsors - Owners  For Participants u Seek common and clearly defined terminology (currency for Knowledge) u Must have owner committed not just involved

28. © John Durbin Slide 28May 2004 Select and Prune Metrics “Not everything that can be counted counts. Not everything that counts can be counted.” Albert Einstein

29. © John Durbin Slide 29May 2004 At the Chicago SPIN For a small group or persons - Minimize number of Metrics u Too many metrics to “focus on” causes “thrashing” or dysfunction behavior. u “Foreground metrics” - people may be measured by << less the 5 (maybe 7) u “Background metrics” – people must not be measured on >> are for heads up and MUST be trusted as non performance measurement – these may be prior metrics that were always under “control”

30. © John Durbin Slide 30May 2004 At the Chicago SPIN Examples of Balanced Set for Software Development u Vital Few Metrics (Like for internal delivery) u Testing Metrics (Like a stand alone group) u SQERT (Like for Contracts)

31. © John Durbin Slide 31May 2004 At the Chicago SPIN The Vital Few Metrics are a Balanced Set u Quality  Process Quality  Product Quality u Performance  Productivity Rate  Delivery Rate u Predictability  Duration Variance Percentage  Effort Variance Percentage

32. © John Durbin Slide 32May 2004 At the Chicago SPIN Example Metrics for Measuring Systems Building Productivity Rate = (speed of the process) Defect Density = Delivery Rate= (elapsed time) Cost Per output unit= Product Size Effort No. of Defects Found in Product Test Product Size Elapsed Time Project Cost Product Size Copyright 1996

33. © John Durbin Slide 33May 2004 At the Chicago SPIN 1. Fault Density 6. Effort Variance Percentage 3. Productivity Rate 4. Delivery Rate 2. Defect Density Metrics # of Faults (for 3 month period) # of Faults (for 3 month period) Actual Effort Planned Effort # of Defects found During Product Test # of Project Function Points Actual Start Date (of analysis) Actual Start Date (of analysis) Actual End Date (of first roll-out) Actual End Date (of first roll-out) Planned Start Date (of analysis) Planned Start Date (of analysis) Planned End Date (of first roll-out) Planned End Date (of first roll-out) Measurement Data Minimize Source Data 5. Duration Variance Percentage

34. © John Durbin Slide 34May 2004 At the Chicago SPIN A Testing Metrics are a Balanced Set u Productivity  Test Execution Rate  Problem Rate  Problem Fix Rate u Process Quality  Repair Effectiveness Percentage  Repair Effort Percentage u Product Quality  Number of problems by module u What could be added?

35. © John Durbin Slide 35May 2004 At the Chicago SPIN S.Q.E.R.T. Balanced Set u Size u Time u Cost u Defects u Scope u Quality u Effort u Risk u Timeline

36. © John Durbin Slide 36May 2004 At the Chicago SPIN References… u Cox, B.G. and Chinnappa, B. N. (1995). "Unique Features of Business Surveys." Ch. 1 in Business Survey Methods. Wiley. NY. u Deming, W. Edwards. (1986). Out of the Crisis. MIT Center for Advanced Engineering Study. Cambridge, MA. u Hibbard, J. (1997). "Knowing What We Know". Information Week. [Oct. 20]. pp. 46-64. u Juran, J.M. (1989).Juran on Leadership for Quality. Free Press. New York. u Gabor, A. (1990). The Man Who Discovered Quality. Penguin Books. New York. u Kaplan, R.S., and Norton, D.P. (1996) The Balanced Scorecard: Translating Strategy into Action. Harvard Business School Press. Boston. u Shewhart, W. (1939). Statistical Method from the Viewpoint of Quality Control. in Deming, W.E., ed., Graduate School of the Department of Agriculture. Washington, DC.

37. © John Durbin Slide 37May 2004 At the Chicago SPIN Other sources u Me>> Johndurbin@ieee.orgJohndurbin@ieee.org u http://www.balancedscorecard.org/ http://www.balancedscorecard.org/ u http://oamweb.osec.doc.gov/bsc/guide.htm http://oamweb.osec.doc.gov/bsc/guide.htm u For more than you wanted to know: http://harvardbusinessonline.hbsp.harvard.ed u/b02/en/home/index.jhtml?_requestid=20215 http://harvardbusinessonline.hbsp.harvard.ed u/b02/en/home/index.jhtml?_requestid=20215 u http://www.sixsigmabenchmarking.com/ http://www.sixsigmabenchmarking.com/ u http://www.som.cranfield.ac.uk/som/cbp/pma/ http://www.som.cranfield.ac.uk/som/cbp/pma/

38. © John Durbin Slide 38May 2004 At the Chicago SPIN End of presentation u Remaining slides are FYI extras  Goal Question Metric  Deployment Critical Success factors

39. © John Durbin Slide 39May 2004 At the Chicago SPIN Goal-Question-Metric Paradigm u Top-down selection process u Linked to organizational goals u Ensures traceable and purposeful metrics u G-Q-M process  Identify your KPI or goals  What questions can you ask to determine whether or not your goal was met?  What metrics answer your question?

40. © John Durbin Slide 40May 2004 At the Chicago SPIN Applying G-Q-M u Must decide responsibilities for these actions  Defining the Metrics  Communicating  Educating  Gathering Data  Reporting Data  Applying Metrics Decision, Change, Diagnosis Victor R. Basili – Univ. Maryland

41. © John Durbin Slide 41May 2004 At the Chicago SPIN G-Q-M Relationship

42. © John Durbin Slide 42May 2004 At the Chicago SPIN G-Q-M Example Goal:Maximize customer satisfaction Question:How long does it take to fix a problem? Metric:Mean time to acknowledge problem. Metric:Mean time to deliver solution. Metric:Schedule versus actual delivery. Question:How many problems are affecting customers? Metric:Incoming defect rate. Metric:Open critical and serious defects.

43. © John Durbin Slide 43May 2004 At the Chicago SPIN G-Q-M Example Goal: Question: Metric: Improve Inspection Results Is time well focused?Are participants prepared? - Review time per inspection (< 2 hours target) - # Interruptions (zero target) - Preparation per inspection time (1 to 2 hours) - Materials Available (y,n)

44. © John Durbin Slide 44May 2004 At the Chicago SPIN Process Maturity for Metrics Effectiveness Use over TIME Technology transfer curve or process maturity curve “Doing it” Metrics to support transition & learning two or less metrics to increase quality Several metrics to increase both quality and productivity Reduce to none or One Process Metrics to sustain quality or productivity Check other processes for metrics conflicts Idealistic curve

45. © John Durbin Slide 45May 2004 At the Chicago SPIN Metrics Definition Template Œ Œ Ž ‘    ’ ’ Why? Who Owns? Defined? When? Suppliers? Baseline & Target? Related Metrics?

46. © John Durbin Slide 46May 2004 At the Chicago SPIN Deployment - Critical Success Factors? u Definition u Deployment u Management u Support u Reinforcing Communications

47. © John Durbin Slide 47May 2004 At the Chicago SPIN Checklist for Better Behavior Metrics should: u Support goals linked to business objectives u Support goals or higher level metrics u Assess products and process (not people) u Be manageable alongside the other metrics u Be meaningful to people reporting u Be understood by all affected u Encourage correct positive behavior u Be a positive (+) indicator (when possible) u Be visibly acted upon (decisions and diagnostics) u Be collected from within the target process u Be about something that the measured group can control