Extending Your Evaluation's Short Half-life So It Is Longer Than a Day: A Personal Perspective David J. Bernstein, Ph.D. Senior Study Director, Westat 2011 American Evaluation Association Conference Government Evaluation Topical Interest Group Business Meeting, Keynote, and Discussion November 4, 2011

Disclaimers 2 The opinions expressed are those of the author, and do not reflect the position of Westat, its clients, AEA, the Government Evaluation TIG, or my parents. I am definitely not Joe Wholey, who gave the first Government Evaluation TIG Keynote last year. As you’ll see, I’m also not Michael Quinn Patton. I’ve learned a lot from both, consider them mentors, and am honored to know them.

Presentation Outline 1. Introduction 2. Evaluation Use as Praxis: Personal Observations 3. What Is Evaluation Use? 4. Evaluation Is a Photo, Not a Video 5. Government Evaluation Use Strategies 6. Random Thoughts 7. Contact Information 3

Evaluation Use As Praxis Praxis* is: 1. Action or practice, as: a. Exercise or practice of an art, science, or skill. b. Customary practice or conduct. 2. Practical application of a theory. My comments will be based on my practical application of a “theory” of evaluation use as espoused by Michael Quinn Patton in the first edition of his book Utilization-Focused Evaluation, and maybe the 2 nd, 3 rd, and 4th. * Source: 4

Michael Quinn Patton To paraphrase Senator Lloyd Bentsen during a 1988 Vice Presidential debate: I served with Michael Quinn Patton (in AEA). I know Michael Quinn Patton (from AEA). Michael Quinn Patton is a friend of mine (on Facebook and I think in real life). I am no Michael Quinn Patton.* * However, I sure do like the way he thinks/writes. 5

What is “Evaluation Use” 6 1.Go read Utilization-focused Evaluation (see prior slide). 2.Paul Epstein (1988*) conception of three functional areas of use for performance measures: Program monitoring. Decision-making. Accountability. Epstein, P.D. (1988). Using performance measurement in local government: A guide to improving decisions, performance, and accountability. New York, NY: National Civic League Press.

Evaluation is Typically a Photo… 7 …not a video. An evaluation typically results in a report. One day or one administration later the evaluation report is all but forgotten. If an evaluation was a video it would be a bit like performance measurement/monitoring system, but it is not. Like the still photograph, the evaluation may become a distant memory for program staff, funders, and stakeholders unless…

Government Evaluation Use Strategies 8 An evaluation MAY get used if it: Addresses the research questions that  were asked (evaluation questions),  should have been asked (desired but not put in writing), couldn’t be asked (unanticipated consequences), and  the answers to which can be put into action. Captures the "spirit" and "likeness" of the program. Acknowledges temporal issues (the status of program implementation) but finds a way to identify timeless findings and recommendations. Is “useful” to a variety of audiences and stakeholders. Captures the interest of the key evaluation stakeholder, the evaluation sponsor.

Random Observations on Evaluation Use 9 It’s a fact: an evaluation that does not get done cannot be used (see next point). OMB review requires a minimum of 3 months, usually much more. Call your Congressman. Ask him or her to have the Paperwork Reduction Act modified to allow reasonable data collection from Federally-funded grantee, and focus OMB review on burden hours not the number of individuals from whom data is collected. Both clients and stakeholders have been known to use evaluations. Pragmatism and objectivity are not (always) at odds.

More Random Observations 10 Using the right methods appropriately to address the research questions of interest helps ensure usability. If an evaluation report falls in the forest and nobody hears it, it becomes “shelfware” (software that is never used).* Everything happens by chance, but very little happens by accident (synchronicity). If you have to ask to have your evaluation used, you probably shouldn’t. *Source :

Conclusion 11 Sometimes you have to adjust to driving conditions.

Contact Information 12 For more information or a copy of this presentation please contact: David J. Bernstein, Ph.D. Senior Study Director Westat 1600 Research Boulevard, RA1292 Rockville, MD (301)