Presentation is loading. Please wait.

Presentation is loading. Please wait.

Performance Measurement 101 PART 1

Similar presentations

Presentation on theme: "Performance Measurement 101 PART 1"— Presentation transcript:

1 Performance Measurement 101 PART 1
Oregon Public Performance Measure Association (OPPMA) Annual Meeting 8:45 – 11:45 Session with Rita Conrad, Ken Smith and Laura Wipper Willamette University, Salem, OR Read the “OPPMA Welcome” document. Be sure to thank the many people that helped to put this together and the participants for their contributions to improve the performance of their organizations.

2 Today’s Session Part 1: The basics (~60 min)
Intro & Brief History Criteria Logic models Part 2: Small groups (~75 min) Practice Report-out Part 3: Making it happen (~60 min) Three approaches Dialogue Briefly introduce the session – with an outline of what we plan to do. Also, that they may move to the Advanced class over at the business building at any time – in order to accomplish their own learning objectives. Also briefly introduce the three presenters. I have an “academic” perspective – Rita comes from the “Policy” viewpoint – and Laura is the “Organization” expert. We have many decades among us – and yet the total experiences in the entire room are much greater than just our own – even in an introductory session. Please, Please, Please – participate!! State how important it is that we accomplish the individual learning objectives of the participants as much as possible. 2 jtAGAWilsonville04-04

3 Late 1800’s-1930’s New York Municipal Research Board
Audit of Portland, Oregon International City Management Association (ICMA) Herbert Simon, Nobel Prize winner Three “Take-Aways” for this brief history: Very long history of performance measurement/management in government Oregon has been a leader throughout much of this history Many successes by Oregon and others – but numerous challenges remain

4 1970’s Productivity Optimistic Views Negative or Cautionary Views
Harry Hatry (1972; 1978) Negative or Cautionary Views Quinn (1978); Burkhead and Hennigan (1978) In my book chapter with Lee Schiffel – we talk about two “Symposia” that appeared in the leading journal Public Administration Review. The first Symposia occurred in the beginning of the 1970’s and the second in the latter 1970’s. Both included those who were optimistic and favorable (such as Harry Hatry) as well as those who noted concerns and were skeptical. This balance of proponents and opponents in the Academic as well as Professional communities continues today.

5 1980’s – Why Accountants? Governmental Accounting Standards Board (GASB) Government Finance Officers Association (GFOA) Association of Government Accountants (AGA) Oregon is the best state at performance reporting – by a long shot (Smith, Cheng, Smith, Schiffel 2006 – available online) It may be a surprise that accountants are one of the professional bodies that have played a significant role in the development of performance measurement and management initiatives. The GASB did a series of research reports in the late 1980’s and conclude that “Its Time has Come” for state and local government organizations to engage in external performance reporting. GASB chose a perhaps obscure phrase to describe what most just call “performance reporting” – they call it “Service Efforts and Accomplishments” or SEA Reporting. The GFOA is a professional organization composed of finance directors or CFO’s of governments. Finance and accounting have many similarities – but there are fundamental differences where accountants are often primarily focused on how to measure and record past events. Finance individuals tend to focus primarily on the present and future decisions and steps that need to be made in order to have adequate cash and resources for the organization to operate. Accountants generally prefer highly standardized language and structures both within their organization as well as across other organizations – in order to facilitate comparisons. Finance professionals are generally concerned with their own unique problems. The AGA has been a supporter of government accountability and external performance reporting – aligning its Certificate of Achievement in SEA Reporting in direct connection with the categories promoted by GASB. My research with Schiffel, Cheng and another Smith show that very few states are excellent at performance reporting – and Oregon is by far the best.

6 Original Oregon Model @ ODOT
Roots in work of Glenn Felix Productivity matrix Baseline – or average – levels of performance Historical best or optimum goals Relative weights, performance levels and performance index Efficiency and Effectiveness Cost vs. Quality Gainshare pilot at ODOT proves value is in the measures and the conversation

7 Striving for Balance Effectiveness Efficiency
Cost per product, service or result Labor per product, service or result Direct hours Indirect hours Administrative hours Outcome/Goal What’s Important Along the Way Customer Satisfaction Work Life Quality

8 Work Life Quality At ODOT
Regularly gauged the capacity of the organization Sick leave Safety Turnover Employee Survey Equal Opportunity Employment Training

9 Development of Measures
Series of workshops Involved almost all staff Aligned activities to purpose, outcomes, goals Measures were customized by staff Alignment through Key Result Areas Some were “roll-ups”

10 Statewide Rollout Statewide measurement initiative began in 1991 with 16 agencies – Phase 1 More phases followed – 100+ agencies Agency-level measurement facilitated by mentors from previous phases Expanded upon ODOT model Efficiency and effectiveness Tied goal-orientation to Oregon Benchmarks

11 Some 90’s Success Stories
DMV Positive experience with a negative index Ways and Means liked the candid conversation Adult and Family Services Moved focus from application processing to family self-sufficiency Lead the nation in welfare reform Vocational Rehabilitation Decided federal goal of two months back on the job not high enough to justify cost Set goal at 18 months Department of Corrections Sharper focus on direct recipient of program efforts 180 degree swing on recidivism

12 Original Oregon Model Adopted across the US
Implemented nationwide by US Army Corps of Engineers One of first two agencies to act following federal Government Performance Results Act of 1993 At one point, 70 Federal agencies were implementing the “Oregon” model following success of Corps

13 Elements of Oregon’s Key Performance Measure System
Tiered, linked system for external reporting Common language Logic models Criteria Budget

14 Linking state performance to benchmarks
Progress Board prepares KPM-Benchmark Crosswalks for W&M subcommittees Measuring Up by Jonathan Walters 98 Progress Board prepares Performance Measure Guidelines for State Agency Budget Instructions Oregon laws required agency performance measures 93 09 08 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 SB555 gives Progress Board first time evaluation function HB 3358 gives performance measurement role to Progress Board National Performance Measurement Movement LFO and BAM assume more performance measurement and management functions

15 Oregon Benchmarks and Performance Measures

16 Activity #1 Getting to Know You

17 Why measure performance? To get results.
It’s at the core of results-based management Fosters internal learning and improvement Is the ship running well? Provides greater external accountability Is the ship on course? Informs policy-making Should the ships change course? 17

18 CRITERIA What makes a good performance measure?
Common language Alignment (line-of-sight) Accurate and reliable data A few key measures Time-framed targets Someone in charge Comparisons Customer satisfaction 18

19 Criterion #1. Common language
Altitude 30,000 ft. OUTCOME, high-level = Societal result OUTCOME, intermediate = Entity result OUTPUT = Product or service INPUT = Time or money EFFICIENCY =Input per output or Input per outcome 20,000 ft. 10,000 ft. 5,000 ft. 19

20 Practical application
3. High-Level Outcomes (HLO) – Is the world you are attempting to affect changing? Oregon Benchmark #65, Juvenile Recidivism – % of juveniles referred w/in 12 months … 2. Intermediate Outcomes – Is the strategy producing the desired result? % of treated youth with reduced risk factors 1. Outputs – Is the strategy getting done? # of youth completing treatment 20

21 HLO - Mission or Benchmark Key Performance Measures
Criterion #2. Alignment - line-of-sight to goals and high-level outcomes HLO - Mission or Benchmark Intermediate Outcomes 3C 2 Impact Outputs 3B Goal 1 Strategies to achieve the goal 3A Key Performance Measures 3 21

22 Criterion #3. Accurate and reliable data
Trustworthy data is essential. Example: verifiable records trump estimates Per measure - at least one data point, preferably several Data should match the measure. 22 jtAGAWilsonville04-04

23 Goal: Prevent Juvenile Crime How high up do you want to go?
Criterion #4. Few key measures – different measures for different purposes More agency influence Goal: Prevent Juvenile Crime Juvenile recidivism DECREASES. High-Level Outcome % of served youth with mitigated risk factors INCREASES. Intermediate Outcome “So That” More public interest % of high risk youth completing program INCREASES. Intermediate Outcome “So That” % of grantees trained INCREASES. Output “So That” How high up do you want to go? 23

24 Criterion #5. Time-framed targets
2003 Actual: 20 TARGET = Desired level at a given point in time Ambitious but realistic Set targets using: trend data comparisons expert opinion Example: hours of travel delay per capita per year 2007 Target: 15 24 jtAGAWilsonville04-04

25 Criterion #6. Someone in charge
Management, staff and stakeholders need to know where the buck stops Critical for improving performance 25

26 Criterion #7. Measures of customer satisfaction
This is a required Key Performance Measure (KPM) for Oregon state agencies. Rationale: Those who pay for their government should be satisfied with their government. 26

27 Criterion #8. Comparisons
“We resolved complaints within three weeks on average.” Is this good or bad? How does actual performance compare to industry standards? Rationale: to better understand whether action is needed. Difficult but not impossible! Agency suggestion: “peer” agencies develop common measures and comparator 27

28 LOGIC MODEL - a tool for thinking it through
Getting started IMPORTANT: Engage those whose performance will be measured. Have a clear statement of goals. “Look up” to mission & high-level outcomes (benchmarks) “Look down” to what you are doing and how you are measuring performance Agency Goals 1 Performance Measures 3 2 High-level outcomes 28

29 Logic model example #1 – Citizen involvement in land use planning
HLO: % of cities with neighborhood organizations. 2 INTERMEDIATE OUTCOMES % participating citizens with improved understanding Customer satisfaction ratings 3C Impact GOAL: Citizen involvement (C.I.) in land use planning 1 OUTPUTS # citizens trained. # C.I. guidelines distributed. 3B Agency Performance Measures 3 STRATEGY Jointly sponsor, with cities, regional educational events for private citizens every quarter. 3A 29

30 Logic model example #2 – Juvenile crime
HLO Juvenile Arrests (OBM#61) 2 INTERMEDIATE OUTCOMES % of juveniles in JCP programs with significantly mitigated risk factors. 3C Impact GOAL Reduce juvenile crime. 1 OUTPUTS # grants awarded by county # days of TA delivered by county 3B Agency Performance Measures 3 STRATEGY Award grants to local contractors to conduct “best practice” juvenile crime prevention programs (JCP). 3A 30

31 Logic model example #3 – Healthy, thriving children
HLO: % of kindergarteners ready to learn (Benchmark #18) 2 INTERMEDIATE OUTCOMES % of children from participating (trained) families entering school ready to learn 3C Impact GOAL Healthy, thriving children. 1 OUTPUTS # grants awarded by county 3B Agency Performance Measures 3 STRATEGY Award grants to local contractors to design/deliver “best practice” parent education classes 3A 31

32 PART 2: Small group exercise – 20 min.
Pick an organization from within your group Pick a goal from that organization Work through the logic model as a team Select two key measures – the most important – for box #3 Select a spokesperson to report out 32

33 Performance Measurement 101: Part 3
Oregon Public Performance Measure Association (OPPMA) Annual Meeting 8:45 – 11:45 Session with Rita Conrad, Ken Smith and Laura Wipper Willamette University, Salem, OR 33

34 Part 3 – Making it Happen Performance Management
Integrated across many systems/disciplines Multiple Steps that usually include: Strategy  Reports with data  Discuss/Argue  Learn/Change  Revisit Strategy No single dominant view – but common tools: Balanced Scorecard, Strategy Maps, LEAN, Six Sigma, TQM, Variance Analysis, Organization Alignment/Incentives, etc. Numerous barriers, challenges and gaps

35 Some Models for Measurement
Cell Organ Person

36 Background Laura has worked in ODOT and other agencies for almost 20 years Ken has studied performance reporting by cities and states for over 17 years

37 Why create a “model” Our hope is to help participants create their own mental models of: Where they and their organization are at What are the next steps/models they can use What choices and priorities they should consider as they move forward The models are not intended to be judgmental – where one level is always better or more appropriate….the costs/benefits for each approach need to be carefully considered….not every model works for every person

38 Measuring to Managing Measures Integrated Operating Unit Approach
System Integrated Macro System Unconnected Measures: Quantitative or Qualitative Only Integrated Measures: Goal-Based Leading & Lagging Macro & Micro Inter-Connected Measures: Organization-Based Program-Based

39 Operating Unit “system” Integrated MACRO System
Three Models Overview Single Measures Operating Unit “system” Integrated MACRO System Description Lists one or many measures – does NOT show logic model how the measures connect in a logical fashion Illustrates or describes how various measures or explanatory information impacts the desired outcome for a single entity or system Illustrates or describes how various measures or explanatory information impacts the desired outcome for a several interacting entities/systems Examples Some of the Oregon APPR’s Balanced Scorecard Oregon Benchmarks “actual” Carl DeMaio’s “Perf Logic Model” GASB “ideal” with explanatory info Oregon Benchmarks “ideal” City of Coral Springs – case study Strengths Easy – can be great place to start Collecting and sharing raw data often starts good conversations Logic model connections internal to the system/entity Exhaustive model Weaknesses Lack of logic model limits usage – and allows different interpretations No connection to explanatory factors outside the entity such as other orgs & socio-economic environment Very complicated – especially if using statistical tools like regression Equation If we measure – performance will improve Outcome = org measure 1  org measure 2  org measure 3 Outcome = Org measures + Outside Orgs + Environment

40 A Model for Integrated Measures
DV = IV + IV + IV Organization's Measures Societal Outcomes Environmental Measures Other Orgs' Measures DV: Dependent Variables IV: Independent Variables

41 An Example using Education
Measures Approach Operating Unit System Integrated Macro System % At Grade Level Adequate Progress After School Programs Demographics & Behavior Test Scores Attendence Parent Volunteers % At Grade Level Literacy Rates Literacy Strands = + +

42 An Example using ODOT





47 DV = IV + IV + IV Organization's Measures Other Orgs' Environmental Societal Outcomes = + +

48 “5 Principles” by Bob Paladino
Similar to many other models – I like his since he uses both government and business examples – and book available on Amazon $25 I also like his model since it focuses explicitly on the need for: a Chief Performance Officer (CPO) Using multiples tools (BSC, LEAN, Maps, etc) p.s. I also like him because he is a CPA

49 Five Key Principles Establish CPM (Corporate Performance Management) Office and Officer Refresh and Communicate Strategy Cascade and Manage Strategy Improve Performance Manage and Leverage Knowledge

50 Illustration of “5 Key Principles”

51 Challenges/Barriers “9 out of 10 companies fail to implement their business strategies” Paladino p. 11 Hey there government folks – the so called “business sector” struggles with putting these good ideas into action just like we do

52 Paladino’s Four Barriers to Implementation
Vision Only 5% of the workforce understands the strategy Management 85% of executive teams spend less than one hour per month discussing strategy Resource 60% of organizations don’t link budgets to strategy People Only 25% of managers have incentives linked to strategy

53 Bob Behn - Harvard lecturer (Has a nice monthly newsletter)
Why Performance Management is NOT sweeping the world? (2002) Practical reasons: It just does not work Political reasons: Does not win elections Managerial reasons: It is da?n hard Psychological reasons: Creates legit fear and Requires organization changes Behn argues that performance management has NOT swept the world – that there are many instances where a deep investment and activity in performance management has NOT occurred – though he certainly acknowledges that many people talk about it and that there do appear to be isolated instances of quality application. Behn suggests four plausible reasons why not everyone has “jumped on the performance bandwagon” – 4 of these have been written about or discussed by other authors (Practical, Political, Managerial and Psych fear). He adds his background in organization psychology to discuss additional barriers from the ORGANIZATION CHANGE view – that performance management is a huge paradigm shift and difficult for people and their organizations to quickly shift ot the a new mentality… mentalities include preferring learning over success – willingness to try new things – being transparent with information and progress in a timely fashion – among others….. The bottom line from this article is there are valid reasons to be skeptical about performance management – and to honestly look for the real reasons why it is hard (and worthwhile) in order to overcome certain hurdles and avoid others altogether.

54 Building Your Plan Three different tools – same basic idea to self-evaluate using a Gap Analysis ActiveStrategy online “Strategy Activator” 19 questions Paladino Chapter 9 – “3 Step Self-Diagnostic” 35 best practices (Score yourself – Gap – Action) Weidner “Accountability and Transparency” 17 questions We will illustrate each of these approaches. The online quiz is certainly intended to generate consulting business – and I have had no prior experience with this particular consulting firm. The thing I like about their approach is they seem to appreciate the importance of: 1) Making the strategy explicitly visual via cascaded scorecards 2) Common data elements using tools like XML 3) Need for lots of training in both technical and leadership/buy-in topics Paladino’s self-assessment is useful in that it ties directly to his Five Key Principles. His model is a comprehensive one – certainly close to what we call the integrated approach – and is similar to Active Strategy in the importance of top level buy-in, the need for explicit visual strategy maps, and the importance of taking action. Paladino focuses a bit more on organizational learning while Active Strategy focuses on technical data issues – both are often underappreciated and underinvested. Weidner is interesting in that it comes from someone that used the GASB approach during his early work in the city/county government of Nashville/Davidson County Tennessee.

55 Additional Case Studies: (time permitting)
City of Coral Springs, Florida North Carolina Benchmarking Project ExPeRT Willamette

56 A few links to useful sources
BOOKS Balanced Measures for Strategic Planning, by Kathleen Monahan (2000) Balanced Scorecard, by Robert Kaplan and David Norton (1st book in 1992 – several additional books and articles) Five Key Principles of Corporate Performance Management, by Bob Paladino (2007) BLOGS is very good – 10 or so writers like Miller, Metzenbaum, etc. COMPUTER CONSULTANTS Active Strategy – 9 phase model ORGANIZATIONS National Performance Management Advisory Commission There are undoubtedly many other links in each category. These are just a few that are particularly relevant to the presentation given today – or to ones that I have directly communicated with. The themes within each are very stable across other books, consultants and professional organizations.

57 Contact Information Rita Conrad Ken Smith Laura Wipper

Download ppt "Performance Measurement 101 PART 1"

Similar presentations

Ads by Google