Presentation on theme: "February 2009 Measuring Value: Using Program Evaluation to Understand What’s Working -- Or Isn’t Juliana M. Blome, Ph.D., MPH Office of Program Analysis."— Presentation transcript:
February 2009 Measuring Value: Using Program Evaluation to Understand What’s Working -- Or Isn’t Juliana M. Blome, Ph.D., MPH Office of Program Analysis and Evaluation National Institute of General Medical Sciences MORE Program Directors Meeting Colorado Springs, Colorado June 12, 2009
NIGMS February 2009 “Program Evaluation…!?”
NIGMS February 2009 Program Evaluation: What is it? Program evaluations are individual, systematic studies that use objective measurement and analysis to answer specific questions about how well a program is working. - GAO/GGD Program Evaluation
NIGMS February 2009 Evaluation Answers Questions Such As…. Does it work? How well does it work? Does it do what we want it to? Does it work for the reasons we think it does? Is it cost effective? Are the benefits worth it? What are the unintended consequences?
NIGMS February 2009 Research vs. Program Evaluation Evaluation Judges merit or worth Policy & program interests of stakeholders paramount Provides information for decision-making on specific program Conducted within setting of changing actors, priorities, resources, & timelines Research Produces generalizable knowledge Scientific inquiry based on intellectual curiosity Advances broad knowledge and theory Controlled setting
NIGMS February 2009 Why bother? To gain insight about a program and its operations To improve practice - to modify or adapt practices to enhance the likelihood of success To assess effects – to determine if we’re meeting our goals and provide evidence of effectiveness
NIGMS February 2009 Guidelines for Conducting Successful Evaluations Invest heavily in planning early on Integrate evaluation into ongoing program activities Use knowledgeable, experienced evaluators
NIGMS February 2009 Evaluator Skills Evaluation theory and methods Research methods (design, planning, statistics, qualitative and quantitative methods) Data collection, analysis and interpretation Communication and Interpersonal skills Content area skills Project management Ethics At universities and colleges, this type of expertise is found in the social and behavioral sciences departments!
NIGMS February 2009 Evaluation Costs The National Science Foundation's “rule of thumb” about evaluation budgets is 10% of the total grant amount.
NIGMS February 2009 Types of Evaluations Needs Assessment What is nature & extent of the issues program should address? Planning phase Feasibility Study Is evaluation appropriate and/or affordable? Maturity/timeliness issue? Process or outcome evaluation produced Process Evaluation Is program is being conducted & producing output as planned? How can process can be improved? Outcome Evaluation To what extent have a program’s goals have been met?
NIGMS February 2009 Standards Utility Feasibility Propriety Accuracy Engage Stakeholders Focus Evaluation Design Describe the program Gather credible evidence Justify conclusions Use and share lessons learned CDC Framework for Program Evaluation Steps Milstein et al, Health Promotion Practice, July 2000, Vol 1(3): For more info:
NIGMS February 2009 Evaluation Standards Utility Evaluations should serve the practical information needs of a given audience Feasibility Evaluations take place in the field and should be realistic, prudent, diplomatic and frugal Propriety The rights of individuals affected by evaluations should be protected Accuracy Evaluations should produce and convey accurate information about a program’s merit and/or worth Guiding Principles for Evaluators, American Evaluation Association,
NIGMS February 2009 CDC Framework: Key Steps in Evaluation 1. Engage stakeholders 2. Describe the program 3. Focus the evaluation design 4. Gather credible evidence 5. Justify conclusions 6. Ensure use and share lessons
NIGMS February 2009 Step 1- Engage Stakeholders Who are the stakeholders? Those involved in program operations, those affected by the program operations, and users of evaluation results
NIGMS February 2009 What are the goals and specific aims of the program? What problem or need is it designed to address? What are the measurable objectives? What are the strategies to achieve the objectives? What are the expected effects? What are the resources and activities? How is the program supposed to work? Step 2 - Describe the Program
NIGMS February 2009 “I think you should be more explicit here in Step Two.” By Sidney Harris, Copyright 2007, The New Yorker
NIGMS February 2009 Step 3 - Focus the evaluation design What do you want to know? Consider the purpose, uses, questions, methods, roles, budgets, deliverables etc. An evaluation cannot answer all questions for all stakeholders.
NIGMS February 2009 Step 4 - Gather credible evidence Evidence must be believable, trustworthy, and relevant Information scope, sources, quality, logistics Methodology & data collection Who is studied and when
NIGMS February 2009 Step 5 - “Justify” Conclusions Consider data: Analysis and synthesis - determine findings Interpretation - what do findings mean? Judgments - what is the value of findings based on accepted standards? Recommendations – - what claims can be made? - what are the limitations of your design?
NIGMS February 2009 Step 6 - Use and share results Share lessons learned with stakeholders! Provide feedback, offer briefings. disseminate findings
NIGMS February 2009 Are you overwhelmed? Next Session – Moving from the abstract to the concrete