Presentation on theme: "Performance Measurement 101 PART 1"— Presentation transcript:
1 Performance Measurement 101 PART 1 Oregon Public Performance Measure Association (OPPMA) Annual Meeting8:45 – 11:45 Sessionwith Rita Conrad, Ken Smith and Laura WipperWillamette University, Salem, ORRead the “OPPMA Welcome” document. Be sure to thank the many people that helped to put this together and the participants for their contributions to improve the performance of their organizations.
2 Today’s Session Part 1: The basics (~60 min) Intro & Brief HistoryCriteriaLogic modelsPart 2: Small groups (~75 min)PracticeReport-outPart 3: Making it happen (~60 min)Three approachesDialogueBriefly introduce the session – with an outline of what we plan to do. Also, that they may move to the Advanced class over at the business building at any time – in order to accomplish their own learning objectives.Also briefly introduce the three presenters. I have an “academic” perspective – Rita comes from the “Policy” viewpoint – and Laura is the “Organization” expert. We have many decades among us – and yet the total experiences in the entire room are much greater than just our own – even in an introductory session. Please, Please, Please – participate!!State how important it is that we accomplish the individual learning objectives of the participants as much as possible.2jtAGAWilsonville04-04
3 Late 1800’s-1930’s New York Municipal Research Board Audit of Portland, OregonInternational City Management Association (ICMA)Herbert Simon, Nobel Prize winnerThree “Take-Aways” for this brief history:Very long history of performance measurement/management in governmentOregon has been a leader throughout much of this historyMany successes by Oregon and others – but numerous challenges remain
4 1970’s Productivity Optimistic Views Negative or Cautionary Views Harry Hatry (1972; 1978)Negative or Cautionary ViewsQuinn (1978);Burkhead and Hennigan (1978)In my book chapter with Lee Schiffel – we talk about two “Symposia” that appeared in the leading journal Public Administration Review.The first Symposia occurred in the beginning of the 1970’s and the second in the latter 1970’s.Both included those who were optimistic and favorable (such as Harry Hatry) as well as those who noted concerns and were skeptical. This balance of proponents and opponents in the Academic as well as Professional communities continues today.
5 1980’s – Why Accountants?Governmental Accounting Standards Board (GASB)Government Finance Officers Association (GFOA)Association of Government Accountants (AGA)Oregon is the best state at performance reporting – by a long shot (Smith, Cheng, Smith, Schiffel 2006 – available online)It may be a surprise that accountants are one of the professional bodies that have played a significant role in the development of performance measurement and management initiatives.The GASB did a series of research reports in the late 1980’s and conclude that “Its Time has Come” for state and local government organizations to engage in external performance reporting. GASB chose a perhaps obscure phrase to describe what most just call “performance reporting” – they call it “Service Efforts and Accomplishments” or SEA Reporting.The GFOA is a professional organization composed of finance directors or CFO’s of governments. Finance and accounting have many similarities – but there are fundamental differences where accountants are often primarily focused on how to measure and record past events. Finance individuals tend to focus primarily on the present and future decisions and steps that need to be made in order to have adequate cash and resources for the organization to operate.Accountants generally prefer highly standardized language and structures both within their organization as well as across other organizations – in order to facilitate comparisons. Finance professionals are generally concerned with their own unique problems.The AGA has been a supporter of government accountability and external performance reporting – aligning its Certificate of Achievement in SEA Reporting in direct connection with the categories promoted by GASB. My research with Schiffel, Cheng and another Smith show that very few states are excellent at performance reporting – and Oregon is by far the best.
6 Original Oregon Model @ ODOT Roots in work of Glenn FelixProductivity matrixBaseline – or average – levels of performanceHistorical best or optimum goalsRelative weights, performance levels and performance indexEfficiency and EffectivenessCost vs. QualityGainshare pilot at ODOT proves value is in the measures and the conversation
7 Striving for Balance Effectiveness Efficiency Cost per product, service or resultLabor per product, service or resultDirect hoursIndirect hoursAdministrative hoursOutcome/GoalWhat’s Important Along the WayCustomer SatisfactionWork Life Quality
8 Work Life Quality At ODOT Regularly gauged the capacity of the organizationSick leaveSafetyTurnoverEmployee SurveyEqual Opportunity EmploymentTraining
9 Development of Measures Series of workshopsInvolved almost all staffAligned activities to purpose, outcomes, goalsMeasures were customized by staffAlignment through Key Result AreasSome were “roll-ups”
10 Statewide RolloutStatewide measurement initiative began in 1991 with 16 agencies – Phase 1More phases followed – 100+ agenciesAgency-level measurement facilitated by mentors from previous phasesExpanded upon ODOT modelEfficiency and effectivenessTied goal-orientation to Oregon Benchmarks
11 Some 90’s Success Stories DMVPositive experience with a negative indexWays and Means liked the candid conversationAdult and Family ServicesMoved focus from application processing to family self-sufficiencyLead the nation in welfare reformVocational RehabilitationDecided federal goal of two months back on the job not high enough to justify costSet goal at 18 monthsDepartment of CorrectionsSharper focus on direct recipient of program efforts180 degree swing on recidivism
12 Original Oregon Model Adopted across the US Implemented nationwide by US Army Corps of EngineersOne of first two agencies to act following federal Government Performance Results Act of 1993At one point, 70 Federal agencies were implementing the “Oregon” model following success of Corps
13 Elements of Oregon’s Key Performance Measure System Tiered, linked system for external reportingCommon languageLogic modelsCriteriaBudget
14 Linking state performance to benchmarks Progress Board prepares KPM-Benchmark Crosswalks for W&M subcommitteesMeasuring Up by Jonathan Walters 98Progress Board prepares Performance Measure Guidelines for State Agency Budget InstructionsOregon laws required agency performance measures 930908939495969798990001020304050607SB555 gives Progress Board first time evaluation functionHB 3358 gives performance measurement role to Progress BoardNational Performance Measurement MovementLFO and BAM assume more performance measurement and management functions
15 Benchmarks.oregon.govOregon Benchmarks and Performance Measures
17 Why measure performance? To get results. It’s at the core of results-based managementFosters internal learning and improvement Is the ship running well?Provides greater external accountability Is the ship on course?Informs policy-making Should the ships change course?17
18 CRITERIA What makes a good performance measure? Common languageAlignment (line-of-sight)Accurate and reliable dataA few key measuresTime-framed targetsSomeone in chargeComparisonsCustomer satisfaction18
19 Criterion #1. Common language Altitude30,000 ft.OUTCOME, high-level = Societal resultOUTCOME, intermediate = Entity resultOUTPUT = Product or serviceINPUT = Time or moneyEFFICIENCY =Input per output or Input per outcome20,000 ft.10,000 ft.5,000 ft.19
20 Practical application 3. High-Level Outcomes (HLO) – Is the world you are attempting to affect changing?Oregon Benchmark #65, Juvenile Recidivism – % of juveniles referred w/in 12 months …2. Intermediate Outcomes – Is the strategy producing the desired result? % of treated youth with reduced risk factors1. Outputs – Is the strategy getting done? # of youth completing treatment20
21 HLO - Mission or Benchmark Key Performance Measures Criterion #2. Alignment - line-of-sight to goals and high-level outcomesHLO - Mission or BenchmarkIntermediateOutcomes3C2ImpactOutputs3BGoal1Strategies to achieve the goal3AKey Performance Measures321
22 Criterion #3. Accurate and reliable data Trustworthy data is essential. Example: verifiable records trump estimatesPer measure - at least one data point, preferably severalData should match the measure.22jtAGAWilsonville04-04
23 Goal: Prevent Juvenile Crime How high up do you want to go? Criterion #4. Few key measures – different measures for different purposesMore agency influenceGoal: Prevent Juvenile CrimeJuvenile recidivism DECREASES.High-Level Outcome% of served youth with mitigated risk factors INCREASES.Intermediate Outcome“So That”More public interest% of high risk youth completing program INCREASES.Intermediate Outcome“So That”% of grantees trained INCREASES.Output“So That”How high up do you want to go?23
24 Criterion #5. Time-framed targets 2003 Actual: 20TARGET = Desired level at a given point in timeAmbitious but realisticSet targets using:trend datacomparisonsexpert opinionExample: hours of travel delay per capita per year2007 Target: 1524jtAGAWilsonville04-04
25 Criterion #6. Someone in charge Management, staff and stakeholders need to know where the buck stopsCritical for improving performance25
26 Criterion #7. Measures of customer satisfaction This is a required Key Performance Measure (KPM) for Oregon state agencies.Rationale: Those who pay for their government should be satisfied with their government.26
27 Criterion #8. Comparisons “We resolved complaints within three weeks on average.”Is this good or bad?How does actual performance compare to industry standards?Rationale: to better understand whether action is needed.Difficult but not impossible!Agency suggestion: “peer” agencies develop common measures and comparator27
28 LOGIC MODEL - a tool for thinking it through Getting startedIMPORTANT: Engage those whose performance will be measured.Have a clear statement of goals.“Look up” to mission & high-level outcomes (benchmarks)“Look down” to what you are doing and how you are measuring performanceAgency Goals1Performance Measures32High-level outcomes28
29 Logic model example #1 – Citizen involvement in land use planning HLO: % of cities with neighborhood organizations.2INTERMEDIATE OUTCOMES % participating citizens with improved understandingCustomer satisfaction ratings3CImpactGOAL: Citizen involvement (C.I.) in land use planning1OUTPUTS # citizens trained. # C.I. guidelines distributed.3BAgency Performance Measures3STRATEGYJointly sponsor, with cities, regional educational events for private citizens every quarter.3A29
30 Logic model example #2 – Juvenile crime HLOJuvenile Arrests (OBM#61)2INTERMEDIATE OUTCOMES% of juveniles in JCP programs with significantly mitigated risk factors.3CImpactGOALReduce juvenile crime.1OUTPUTS # grants awarded by county# days of TA delivered by county3BAgency Performance Measures3STRATEGY Award grants to local contractors to conduct “best practice” juvenile crime prevention programs (JCP).3A30
31 Logic model example #3 – Healthy, thriving children HLO: % of kindergarteners ready to learn (Benchmark #18)2INTERMEDIATE OUTCOMES% of children from participating (trained) families entering school ready to learn3CImpactGOAL Healthy, thriving children.1OUTPUTS # grants awarded by county3BAgency Performance Measures3STRATEGY Award grants to local contractors to design/deliver “best practice” parent education classes3A31
32 PART 2: Small group exercise – 20 min. Pick an organization from within your groupPick a goal from that organizationWork through the logic model as a teamSelect two key measures – the most important – for box #3Select a spokesperson to report out32
33 Performance Measurement 101: Part 3 Oregon Public Performance Measure Association (OPPMA) Annual Meeting8:45 – 11:45 Sessionwith Rita Conrad, Ken Smith and Laura WipperWillamette University, Salem, OR33
34 Part 3 – Making it Happen Performance Management Integrated across many systems/disciplinesMultiple Steps that usually include:Strategy Reports with data Discuss/Argue Learn/Change Revisit StrategyNo single dominant view – but common tools:Balanced Scorecard, Strategy Maps, LEAN, Six Sigma, TQM, Variance Analysis, Organization Alignment/Incentives, etc.Numerous barriers, challenges and gaps
36 BackgroundLaura has worked in ODOT and other agencies for almost 20 yearsKen has studied performance reporting by cities and states for over 17 years
37 Why create a “model”Our hope is to help participants create their own mental models of:Where they and their organization are atWhat are the next steps/models they can useWhat choices and priorities they should consider as they move forwardThe models are not intended to be judgmental – where one level is always better or more appropriate….the costs/benefits for each approach need to be carefully considered….not every model works for every person
38 Measuring to Managing Measures Integrated Operating Unit Approach SystemIntegratedMacro SystemUnconnectedMeasures:Quantitative orQualitative OnlyIntegrated Measures:Goal-BasedLeading & LaggingMacro & MicroInter-ConnectedMeasures:Organization-BasedProgram-Based
39 Operating Unit “system” Integrated MACRO System Three Models OverviewSingle MeasuresOperating Unit “system”Integrated MACRO SystemDescriptionLists one or many measures – does NOT show logic model how the measures connect in a logical fashionIllustrates or describes how various measures or explanatory information impacts the desired outcome for a single entity or systemIllustrates or describes how various measures or explanatory information impacts the desired outcome for a several interacting entities/systemsExamplesSome of the Oregon APPR’sBalanced ScorecardOregon Benchmarks “actual”Carl DeMaio’s “Perf Logic Model”GASB “ideal” with explanatory infoOregon Benchmarks “ideal”City of Coral Springs – case studyStrengthsEasy – can be great place to startCollecting and sharing raw data often starts good conversationsLogic model connections internal to the system/entityExhaustive modelWeaknessesLack of logic model limits usage – and allows different interpretationsNo connection to explanatory factors outside the entity such as other orgs & socio-economic environmentVery complicated – especially if using statistical tools like regressionEquationIf we measure – performance will improveOutcome = org measure 1 org measure 2 org measure 3Outcome = Org measures + Outside Orgs + Environment
40 A Model for Integrated Measures DV = IV + IV + IVOrganization'sMeasuresSocietalOutcomesEnvironmentalMeasuresOther Orgs'MeasuresDV: Dependent VariablesIV: Independent Variables
41 An Example using Education MeasuresApproachOperating UnitSystemIntegratedMacro System% At Grade LevelAdequate ProgressAfter School ProgramsDemographics & BehaviorTest ScoresAttendenceParent Volunteers% At Grade LevelLiteracy RatesLiteracy Strands=++
47 DV = IV + IV + IVOrganization'sMeasuresOther Orgs'EnvironmentalSocietalOutcomes=++
48 “5 Principles” by Bob Paladino Similar to many other models – I like his since he uses both government and business examples – and book available on Amazon $25I also like his model since it focuses explicitly on the need for:a Chief Performance Officer (CPO)Using multiples tools (BSC, LEAN, Maps, etc)p.s. I also like him because he is a CPA
49 Five Key PrinciplesEstablish CPM (Corporate Performance Management) Office and OfficerRefresh and Communicate StrategyCascade and Manage StrategyImprove PerformanceManage and Leverage Knowledge
51 Challenges/Barriers“9 out of 10 companies fail to implement their business strategies” Paladino p. 11Hey there government folks – the so called “business sector” struggles with putting these good ideas into action just like we do
52 Paladino’s Four Barriers to Implementation VisionOnly 5% of the workforce understands the strategyManagement85% of executive teams spend less than one hour per month discussing strategyResource60% of organizations don’t link budgets to strategyPeopleOnly 25% of managers have incentives linked to strategy
53 Bob Behn - Harvard lecturer (Has a nice monthly newsletter) Why Performance Management is NOT sweeping the world? (2002)Practical reasons: It just does not workPolitical reasons: Does not win electionsManagerial reasons: It is da?n hardPsychological reasons:Creates legit fear andRequires organization changesBehn argues that performance management has NOT swept the world – that there are many instances where a deep investment and activity in performance management has NOT occurred – though he certainly acknowledges that many people talk about it and that there do appear to be isolated instances of quality application.Behn suggests four plausible reasons why not everyone has “jumped on the performance bandwagon” – 4 of these have been written about or discussed by other authors (Practical, Political, Managerial and Psych fear). He adds his background in organization psychology to discuss additional barriers from the ORGANIZATION CHANGE view – that performance management is a huge paradigm shift and difficult for people and their organizations to quickly shift ot the a new mentality…..new mentalities include preferring learning over success – willingness to try new things – being transparent with information and progress in a timely fashion – among others…..The bottom line from this article is there are valid reasons to be skeptical about performance management – and to honestly look for the real reasons why it is hard (and worthwhile) in order to overcome certain hurdles and avoid others altogether.
54 Building Your PlanThree different tools – same basic idea to self-evaluate using a Gap AnalysisActiveStrategy online “Strategy Activator”19 questionsPaladino Chapter 9 – “3 Step Self-Diagnostic”35 best practices (Score yourself – Gap – Action)Weidner “Accountability and Transparency”17 questionsWe will illustrate each of these approaches.The online quiz is certainly intended to generate consulting business – and I have had no prior experience with this particular consulting firm. The thing I like about their approach is they seem to appreciate the importance of:1) Making the strategy explicitly visual via cascaded scorecards2) Common data elements using tools like XML3) Need for lots of training in both technical and leadership/buy-in topicsPaladino’s self-assessment is useful in that it ties directly to his Five Key Principles. His model is a comprehensive one – certainly close to what we call the integrated approach – and is similar to Active Strategy in the importance of top level buy-in, the need for explicit visual strategy maps, and the importance of taking action. Paladino focuses a bit more on organizational learning while Active Strategy focuses on technical data issues – both are often underappreciated and underinvested.Weidner is interesting in that it comes from someone that used the GASB approach during his early work in the city/county government of Nashville/Davidson County Tennessee.
55 Additional Case Studies: (time permitting) City of Coral Springs, FloridaNorth Carolina Benchmarking ProjectExPeRT Willamette
56 A few links to useful sources BOOKSBalanced Measures for Strategic Planning, by Kathleen Monahan (2000)Balanced Scorecard, by Robert Kaplan and David Norton (1st book in 1992 – several additional books and articles)Five Key Principles of Corporate Performance Management, by Bob Paladino (2007)BLOGSGoverning.com is very good – 10 or so writers like Miller, Metzenbaum, etc.COMPUTER CONSULTANTSActive Strategy – 9 phase modelORGANIZATIONSNational Performance Management Advisory CommissionThere are undoubtedly many other links in each category. These are just a few that are particularly relevant to the presentation given today – or to ones that I have directly communicated with. The themes within each are very stable across other books, consultants and professional organizations.