2. Evaluation and Performance Measurement Use in Government: Use This Use David J. Bernstein, Ph.D., Senior Study Director, Westat 2012 American Evaluation Association Conference Evaluation Use and Government Evaluation Topical Interest Groups Panel Session 757, 211 C/D October 27, 2012, 8:00 to 9:30 a.m.

3. Outline of the Presentation 1. Compare and contrast Patton's evaluation- utilization concepts with Epstein’s performance measurement (PM) use model. 2. Apply Patton's evaluation-utilization premises in a government performance measurement framework, AKA “utilization-focused performance measurement.” 3. Provide examples of the divergent and diverse evaluation and PM needs of sponsors (funders) and evaluation/program stakeholders. 2

4. Patton and Epstein: Use These Uses 3 Patton Three Primary Uses of Evaluation: 1. Judging Merit and Worth 2. Improving Programs 3. Generating Knowledge Epstein Three Uses of Performance Measures 1 1. Improve Public Accountability/Communication 2. Improving Service Performance 3. Improve Decision Making Four uses of evaluation processes: 1. Enhancing shared understandings 2. Reinforcing interventions 3. Supporting participant engagement 4. Developing programs and organizations Source: Patton 1997, Fundamental Premises of Utilization-Focused Evaluation #6, p. 382. Four uses of performance measures 2 1. Improve Public Accountability/Communication 2. Improve Service Performance, Decision Making 3. All 3 of Epstein’s PM uses. 4. All 3 of Epstein’s PM uses. Source: 1 Epstein 1988, p. 5. 2 Bernstein extrapolation of Epstein.

5. Patton’s 3 Primary Uses of Evaluation Findings and Complementary PM Uses A. Evaluation usesB. Patton’s Examples 1. Judge merit or worthSummative evaluation Accountability Audits Accreditation/licensing Quality control Cost-benefit decisions Decide a program’s future 2. Improve programsFormative evaluations I.D. strengths/weaknesses Continuous improvement Quality enhancement Learning organization 3. Generate knowledgeGeneralizations about effectiveness Theory building Synthesize patterns Policy making 4 Source:Column A, B: Patton 1997, p. 76 Column C: Bernstein 2000 C. Complementary PM Uses Outcome monitoring Reporting results Performance, other audits PM audits Customer service measures Performance-based budgets Cost-benefit analysis Performance benchmarking Program/process monitoring I.D. strengths/weaknesses Continuous improvement Baldrige Performance Excellence Don’t do it! Evaluation uses PMs, but PM is not evaluation! Theories of change/Logic modeling Trend analysis Policy monitoring

6. Intended/Unintended Users and Uses Unintended UsesIntended Uses Unintended Users Unintended Trouble Unintended Success Intended Users Unintended Need for More Technical Assistance *MQP* 5

7. Government Examples: Premise 1, 3 P1.Commitment to intended use by intended users should be the driving force in an evaluation. At every decision point…the evaluator asks intended users, ‘How would that affect your use of this evaluation?’ P3. The personal factor significantly contributes to use…evaluations should be specifically user oriented. Examples: Montgomery County Finance Strategic Planning and Performance Measurement Development. Evaluation of the Helen Keller National Center (Rehabilitation Services Administration, U.S. Department of Education) 6

8. Government Examples: Premise 4-7 P4.Careful and thoughtful stakeholder analysis should inform identification of primary intended users…[Multiple stakeholders] have an interest in evaluation, but the degree and nature of their interests will vary. Political sensitivity and ethical judgments are involved in identifying primary intended users and uses [emphasis added]. P5.Evaluations must be focused in some way; focusing on intended use by intended users is the most useful way…Because no evaluation can serve all potential stakeholders’ interests equally well, stakeholders representing various constituencies should come together to negotiate what issues and questions deserve priority. P6.Focusing on intended use requires making deliberate and thoughtful choices. P7.Useful evaluations must be designed and adapted situationally. Example: Montgomery County Central [Arrest] Processing Pilot Program. 7

9. Government Examples: Premise 13 P13.Use is different from reporting and dissemination. Reporting and dissemination may be means to facilitate use, but they should not be confused with such intended uses as making decisions, improving programs, changing thinking, empowering participants, and generating knowledge (see premise 6). Example: Governmental Accountability Standards Board Research on Concept Statement #2: Service Efforts and Accomplishments. Led to changes published in Governmental Accountability Standards Board Concept Statement #5, including removal of a section of Concepts Statement 2, titled, “Developing Reporting Standards for SEA Information.” 8 Source: NEWS RELEASE 12/15/08: GASB Issues Concepts Statement No. 5, Service Efforts and Accomplishments Reporting (an amendment of GASB Concepts Statement No. 2). http://www.gasb.org/cs/ContentServer?c=GASBContent_C&pagename=GASB%2FGASBContent_C%2FGASBN ewsPage&cid=1176156721790. http://www.gasb.org/cs/ContentServer?c=GASBContent_C&pagename=GASB%2FGASBContent_C%2FGASBN ewsPage&cid=1176156721790

10. Government Examples: Premise 11 P11.Evaluators have a rightful stake in an evaluation in that their credibility and integrity are always at risk, thus the mandate for evaluators to be active-reactive-adaptive…Evaluators are active in presenting to intended users their own best judgments about appropriate evaluation focus and methods…Evaluators’ credibility and integrity are factors affecting use as well as the foundation of the profession. In this regard, evaluators should be guided by the profession’s standards and principles. AEA Guiding Principles for Evaluators A. Systematic Inquiry. B. Competence. C. Integrity/Honesty. D. Respect for People. E. Responsibilities for General and Public Welfare. Examples: Most evaluations on which I’ve worked since July 2004. 9 Source: American Evaluation Association 2004. http://www.eval.org/Publications/GuidingPrinciplesPrintable.asp.http://www.eval.org/Publications/GuidingPrinciplesPrintable.asp

11. References American Evaluation Association. Guiding Principles for Evaluators. Downloaded from http://www.eval.org/Publications/GuidingPrinciples.asp. http://www.eval.org/Publications/GuidingPrinciples.asp Bernstein, D.J. (2000). Local government performance measurement use: Assessing system quality and effects. George Washington University, Washington, DC. Available from UMI-ProQuest, https://order.proquest.com/OA_HTML/pqdtibeCCtpItmDspRte.jsp?item=9964679&site x=10020:22372:US&track=DxWeb&dlnow=1&rpath=http%3A//dissexpress.umi.com/dx web%3F%26query%3DAU%2528david%2520joseph%2520bernstein%2529%26page %3D1. https://order.proquest.com/OA_HTML/pqdtibeCCtpItmDspRte.jsp?item=9964679&site x=10020:22372:US&track=DxWeb&dlnow=1&rpath=http%3A//dissexpress.umi.com/dx web%3F%26query%3DAU%2528david%2520joseph%2520bernstein%2529%26page %3D1 Bromberg, D. (2011). Use Me. Westchester, PA: Appleseed Recordings. Epstein, P. (1988). Using Performance Measurement in Government: A Guide to Improving Decisions, Performance, and Accountability. New York, NY: National Civic League Press. Patton, MQ. (1997). Utilization-Focused Evaluation: The New Century Text, Edition 3. Thousand Oaks, CA: Sage Publications. (Note: a 4 th Edition, 2008), is available. 10

12. Contact Information 11 For a copy of this PowerPoint or other information, contact: David J. Bernstein, Ph.D. Senior Study Director, Westat 1600 Research Blvd, RA1292 Rockville, MD 20850 davidbernstein@westat.com