How’s it Working? Evaluating Your Program MAAPS Conference, 7 May 2010 Debra Smith & Judah Leblang Program Evaluation & Research Group School of Education, Lesley University

PERG Founded 1976 Over 600 program evaluation and research studies in various educational settings Also offers professional development and consultation

Session participants will: Be introduced to the basics of program evaluation through an example Define a question or questions about their own program Identify methods for collecting data that would help to answer their question/s Discuss next steps

What is program evaluation? A type of applied research focused on systematically collecting and analyzing data to help answer questions about a program, or some aspect of a program, in order to make decisions about it.

Purposes Accountability Program development Generating knowledge

Formative vs Summative Formative evaluation offers feedback along the way to improve programs Summative evaluations “sum up” the results of a program at the end of a period of development or implementation.

Audiences Funders Program leaders Program participants Organizational partners Others

Evaluation process 1. Goals/ logic model 2. Questions 3. Evaluation plan 4. Data collection 5. Data analysis 6. Reporting PROGRAM

An example: Evolutions After school program begun in 2005, connected with Peabody Museum of Natural History at Yale University—initially involved approximately 40 low SES/ minority students

Evolutions program goals To provide opportunities for students to: Prepare for post-secondary (college) education; Learn about scientific—and other careers; Expand their knowledge of and interest in science (science literacy); Develop transferable skills for the future; and learn about the Peabody Museum/museum careers.

Logic models Map a coherent chain of connections between goals, resources, activities and what you expect (short term), want (over an intermediate period) and hope (in the long term) to happen. They also reflect your assumptions and theory of action or change.

Logic Model Key Concepts CategoryResources or Inputs Activities — OutputsShort-term outcomes Long-term outcomes General information Staff, funds, materials, space, etc What we plan to do/who we will do it for The results of our program —direct outputs Outcomes (changes) at completion of the project year or soon after Outcomes (changes) several years beyond completion of the project

And EVO example CategoryResources or Inputs Activities—OutputsShort-term outcomes Long-term outcomes General information Staff, funds, materials, space, etc What we plan to do/who we will do it for The results of our program— direct outputs Outcomes (changes) at completion of the project year or soon after Outcomes (changes) several years beyond completion of the project EVO examples Full time project director funds from Peabody Museum and other funders, classroom space, etc. In-depth exploration of: science topics tours of Peabody collections Yale scientist labs Students will meet: at least 6 scientists students will visit no less than 2 natural history museums Students will: learn skills associated with producing a museum exhibition Understand key science themes Students will: understand different types of careers within disciplines understand the college application process be inspired to pursue a career in the sciences

Goal Rationale Assumptions Resources ActivitiesOutputs Short-term outcomes Mid-term outcomes Long-term outcomes Logic models may look different..

Develop a logic model for your own program/ project

Evaluation process 1. Goals/ logic model 2. Questions 3. Evaluation plan 4. Data collection 5. Data analysis 6. Reporting PROGRAM

Questions: Think Goldilocks Specific but not too detailed Important but not too broad in scope

Key Questions: Part One How does EVO prepare students for college or high school? How are EVO students involved in developing an exhibit at the museum? Do students develop increased “science literacy,” as defined by EVO staff?

Key Questions: Part Two How (if at all) do students express more confidence about and interest in doing science? Are students more aware of careers in science? How (if at all) do students demonstrate increased knowledge of the college application process, and develop criteria for choosing a college that meets their needs?

What questions do you want to answer about your program?

Evaluation process 1. Goals/ logic model 2. Questions 3. Evaluation plan 4. Data collection 5. Data analysis 6. Reporting PROGRAM

Data collection methods Observation Interviews/ focus groups Surveys Document/artifact review

PERG Evaluation Matrix Evolutions Data collection activities>> EVALUATION QUESTIONS: Observe Evo students Student focus groups Interview project director Review project docs and artifacts Examine pre-post survey Student prep for college/academic planning √√√√√ Student involvement in museum exhibit √√√ Students' development of science literacy √√√√ Student learning√√√√√ Students' interest in science/environment √√√√√ Students' confidence in doing science √√√ Students' interest in/knowledge of science careers √√√√

Technical considerations: Validity Will the data answer the questions? Are we asking the right questions?

Triangulation Is there adequate triangulation (use of multiple methods and/or data sources) to ensure validity?

Drafting your own matrix: What data will help you answer your questions?

Evaluation process 1. Goals/ logic model 2. Questions 3. Evaluation plan 4. Data collection 5. Data analysis 6. Reporting PROGRAM

Collecting data Make sure your plan is doable given time and resources available. Design instruments to focus your data collection, ensure consistency and avoid bias. Be organized: take notes, develop a system for tracking/ filing your data.

Collecting data Communicate clearly about what you are doing, why and how the findings will be shared and used. Be mindful of human subjects protections. Does your organization have an institutional review board (IRB)?

The First Year: site visit On-site data collection Focus groups with students Interviews with director, project staff Observation of end of year event Parent interviews

Evaluation process 1. Goals/ logic model 2. Questions 3. Evaluation plan 4. Data collection 5. Data analysis 6. Reporting PROGRAM

Analyzing data What stands out? What are the patterns? What are the similarities? What are the differences? Is more information needed?

Reliability Are the patterns in the data, or judgments about the data, consistent?

Validity, again Is the data helping you answer the questions? Is the data credible?

Evaluation process 1. Goals/ logic model 2. Questions 3. Evaluation plan 4. Data collection 5. Data analysis 6. Reporting PROGRAM

Reporting Consider purpose and audience/s Reporting relevant findings, questions/ recommendations Engaging stakeholders in discussion Using findings to inform next steps

Results of the first-year evaluation The impact of the evaluation on EVO— more focused program, clearer objectives, suggestions for sustainability. Evidence of program success: Retention, student engagement, positive changes in students’ view of doing science and scientists.

The Ongoing Evaluation-- shaping the program: Implementation of evaluator suggestions—examples: informational interviewing, developing a smaller exhibit, refining requirements for students

EVO: 2006-Today Continued development and expansion of EVO—2006 until today: Expansion of the program from approximately 40 to more than 80 students, introduction of internships and Sci Corps. – Different areas of science focus— environmental awareness, geoscience, depending on funding sources.

Evaluation resources W.K. Kellogg Foundation Evaluation Handbook Kellogg Logic Model Development Guide Basic Guide to Program Evaluation

Evaluation resources Program Evaluation & Research Group Lesley University 29 Everett St. Cambridge, MA