Overview of Program Evaluation Program Evaluation Basics Webinar Series Mary E. Arnold, Ph.D. Associate Professor and Youth Development Specialist Oregon State University 4-H Professional Development Webinar October 11, 2012

Overview of Program Evaluation Let’s start at the very beginning- a very good place to start Goals for Today’s Webinar 1.Learn a basic history of youth program evaluation 2.Learn reasons for conducting evaluations 3.Learn the different levels of evaluation 4.Learn Bennett’s Hierarchy- an well known Extension evaluation model 5.Learn different evaluation purposes 6.Learn the steps of the evaluation process 7.Review expectations for evaluation for OSU Extension 8.Review expectations for compliance with OSU’s Institutional Review Board (IRB)

Arnold, M. E. & Cater, M. (2011). From then to now: Emerging directions for youth program evaluation. Journal of Youth Development, 6(3). On-line: Article NO:110603FA005. A Brief History of Youth Program Evaluation Publications related to youth program evaluation have flourished in the past 10 years; 20 years ago there was very little to be found The evaluation field as a whole is only now approaching 40 years 1960s saw rise to federally mandated (and funded) evaluations Evaluation results were poor, and had little effect on program expansions or reductions This gave rise to methodological critiques Focus on rigor- experimental design to measure outcomes The importance of qualitative methods to understand the “human” side of social programs And, add in the developmental influences at play in youth programs

*** Carnegie Council on Youth Development (1989). Turning points: Preparing American youth for the 21 st century. Washington, DC: Author. Youth Development Program Evaluation Began in the late 1800s Focused on success stories – it was the right thing to do! 1980s dawn of the age of accountability Late 1980s youth development programs emerged as different from intervention programs, with different outcomes*** for youth: Intellectually reflective (we might call this critical thinking today) Enroute to a life of meaningful work Good citizens Caring and ethical Healthy Even so, first efforts were focused on “reach” and participant satisfaction Finally, emerged an emphasis on demonstrating impact Poll # 1: What has been the primary focus of evaluations you have done?

Why Do We Evaluate? Help others understand the program (stakeholders) Understand the need for a program Improve the program Improve teaching Understand the program’s impact Determine if the program is progressing as planned Determine if the program is worth the cost Meet grant reporting criteria Meet administrative requirements Poll # 2: Why do you evaluate your programs?

Tiers of Program Evaluation Adapted From:Weiss, H. B. & Jacobs, F. H. (1988). Evaluating Family Programs. New York: Aldine de Gruyter.

“Bennett’s Hierarchy” Bennett, C. (1975). Up the hierarchy. Journal of Extension. PDF available at: 2-a1.pdfhttp:// 2-a1.pdf

Types of Evaluation Planning Evaluation (Needs Assessment) Process Evaluation Evaluability Evaluation Outcome Evaluation Efficiency Evaluation Garber, P.A., & Grinnell, R. M. (1994). Evaluation and Quality Improvement. Needham Heights, MA: Allyn & Bacon. Poll # 3: What types of evaluations have you done?

The Steps of a Program Evaluation Forming an Evaluation Team Developing Evaluation Questions Gathering Data Conducting Data Analysis Interpreting Evaluation Results Sharing Results Taking Action

The Steps of a Program Evaluation Forming an Evaluation Team Developing Evaluation Questions Gathering Data Conducting Data Analysis Interpreting Evaluation Results Sharing Results Taking Action

The Steps of a Program Evaluation Forming an Evaluation Team Developing Evaluation Questions Gathering Data Conducting Data Analysis Interpreting Evaluation Results Sharing Results Taking Action

The Steps of a Program Evaluation Forming an Evaluation Team Developing Evaluation Questions Gathering Data Conducting Data Analysis Interpreting Evaluation Results Sharing Results Taking Action

The Steps of a Program Evaluation Forming an Evaluation Team Developing Evaluation Questions Gathering Data Conducting Data Analysis Interpreting Evaluation Results Sharing Results Taking Action

The Steps of a Program Evaluation Forming an Evaluation Team Developing Evaluation Questions Gathering Data Conducting Data Analysis Interpreting Evaluation Results Sharing Results Taking Action

The Steps of a Program Evaluation Forming an Evaluation Team Developing Evaluation Questions Gathering Data Conducting Data Analysis Interpreting Evaluation Results Sharing Results Taking Action

The Steps of a Program Evaluation Forming an Evaluation Team Developing Evaluation Questions Gathering Data Conducting Data Analysis Interpreting Evaluation Results Sharing Results Taking Action

Program Evaluation Expectations ALL Extension faculty members must: Document measurable program outcomes on at lest one priority educational effort annually through an impact statement in SOARS This expectation takes into account the faculty member’s length of service, rank and experience in current assignment For those who have scholarship in their position description, evaluation efforts often lead to scholarly production (conference presentations, journal articles, etc.) Poll # 4: What is the primary reason you have conducted program evaluations in the past 3 years?

Institutional Review Board (IRB) AKA… Do I have to do that? Research is determined by three qualities: 1.Systematic inquiry into a phenomena 2.That is designed to develop or contribute 3.Generalizable knowledge Human Subjects Are: Living individuals about whom an investigator conducting research obtains: 1.Data through an intervention or interaction with the individual, or 2.Identifiable private information

Institutional Review Board (IRB) Okay, I have to do it… now what? Go to the IRB Website at: 1.Complete the “Does Your Study Require IRB Review? Form 2.Complete online ethics training modules 3.Complete the steps listing under “Preparing and Initial Submission” at initial-submission

That’s all for now! Join in next month for: Program Planning for Effective Program Evaluation That’s all for now! Join in next month for: Program Planning for Effective Program Evaluation Don’t forget to complete an evaluation of today’s webinar at: