1. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 Measuring Program Impact M. Jean Young, Ph.D. Evaluation, Instructional Design & Development MJ Young & Associates Tucson, AZ

2. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 Measuring Program Impact Outline Goals and Objectives Interventions and Outcomes Evaluation vs. Research

3. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 What We Know Goals are where you want to end up. Goals include statements e.g.:... more scientifically literate students Objectives are how you get there.

4. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 Goals and Objectives Objectives are what you want those students to know and be able to do:... as they learn to be more scientifically literate. How they will behave Objectives are best assessed if written as measurable (Start with an action verb).

5. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 Objectives – Action Verbs The measurable action verb is the behavior a teacher/student will exhibit: Explain, describe, compare, infer, predict... Higher order – evaluate, interpret, find patterns, create a model... Express: appreciation (rate aspects of the program highly), greater confidence, motivation to take some action

6. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 Where We’re Headed According to the U.S. National Research Council, students who are proficient in science: Know, use and interpret scientific explanations of the natural world. Generate and evaluate scientific evidence and explanations. Understand the nature and development of scientific knowledge. Participate productively in scientific practices and discourse. from Taking Science to School, 2007

7. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 What We Know Different intervention levels lead to different outcomes. Students learn best if they are involved in their own learning.

8. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 Intervention & Outcomes* Awareness One to two days: Address one idea and/or skill. Creates interest Provides orientation Example: International Masterclasses *Assumes no or little prior knowledge

9. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 Intervention & Outcomes* Knowledge One week: Address one or two major concepts/principles/procedures. Participants will “know” about and be able to explain one to three major foci. Teachers can teach it to their students. Example: CERN National Teacher Programs *Assumes no or little prior knowledge

10. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 Intervention & Outcomes* Knowledge Two to three weeks: Address a major topic in depth that students will “know” or teachers can teach. Example: HST It takes at least four weeks for teachers or students to understand scientific research when working in a team with scientists. Example: QuarkNet Research Internships *Assumes no or little prior knowledge

11. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 Evaluation and Research Do you want to evaluate or research an aspect of your program? Does it matter? Evaluation involves: Finding out–how many, impacts/outcomes (expected and unexpected), why, how. Quantitative and qualitative measures.

12. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 Evaluation and Research Do you want to evaluate or research an aspect of your program? Does it matter? Research addresses a basic question such as how students learn from an intervention. Methodology is usually more limited than evaluation but depends on the question.

13. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 Evaluation and Research Which is evaluation and which is research? Questions to ask: Do you have several objectives, including an “express” objective, you want to achieve? Do you want to know if teachers are taking what they learn back to the classroom? Do you want to know if students know and/or are able to do what they learned several months after they leave the program?

14. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 Evaluation and Research Questions to ask: Do you want to know if one kind of intervention is better than another? Do you want to know what impact the program has had on teachers? on students?

15. Helping Develop America’s Technological Workforce MJ Young, IPPOG, November 2015 Program Impact Measuring the impact of your program means evaluating your program. Begin with measurable objectives aligned to an appropriate level of intervention.