Developing an Effective Evaluation to Check for Understanding Susan E. Schultz, Ph.D. Evaluation Consultant PARK Teachers

Overview Part 1: Components of Program Development Part 2: Goals and Types of Evaluation Part 3: Creating an Effective Evaluation Plan Developing an Effective Evaluation Plan

PARK Teachers PART 3: Creating an Effective Evaluation Plan

PARK Teachers Developing an Effective Evaluation Plan Identify Program Goals Create an Evaluation Matrix Use variety of Assessment Techniques Select Assessment Strategies Pilot and Revise Assessments

PARK Teachers “What are the three key ideas or messages that I want all participants to know when they complete the program?” “What overarching question or statement brings all of your goals together?” Identify the program goals and essential question

PARK Teachers The PARK program goals focused on pre-service teachers being able to: Improve their geology content knowledge, Engage in quality instructional practices, Experience national park as a research education site, Learn about educational resources in national parks, Transfer their knowledge to their local community. The essential question was, “How do you recognize evidence of geologic change in your environment?” Identify the program goals and essential question

PARK Teachers Check for Understanding Task 3: Outcomes Part 1: Using your group’s essential question and program goals, write your group’s outcomes on a piece of poster paper. Part 2: Share your group’s outcomes with the whole group.

PARK Teachers Create an evaluation matrix based on program goals An evaluation matrix is an outline of the evaluation plan. The matrix includes the: Program Outcomes, Task to achieve the outcomes, and Assessment strategies to measure the outcomes. Let’s look at some specific examples from the PARK program on the next two slides.

PARK Teachers Educational OutcomeTasksMeasurement Develop a model for creating a partnership to focus on pre-service education -Develop observation protocol and procedures for tracking decision- making process. -Make observations and record decisions made at the partnership meetings. Document the formation and continuation of the partnership between constituents. Design a pre-service teacher graduate course module on geology Evaluating content of curriculum using a specific set of criteria such as: -Identify explicitly stated goals for curriculum -Alignment of goals with best practice in scientific community on topic of geology -Identify key teaching strategies to teach geology with an inquiry focus. -Examine transitions within and between topics in the curriculum -Examine assessment strategies -Identify connections to home communities Evaluate each component of the geology curriculum based on the criteria and make suggestions for revision to curriculum developers. Sample of PARK Program Evaluation Matrix

PARK Teachers Educational OutcomeTasksMeasurement Enhance teachers’ knowledge of geology content related to the natural setting provided by GGNRA - Create instruments to assess geology content - Pilot instruments & revise assessments - Administer tests - Analyze data / Write report Pre and post content tests to determine change in pre- service teachers’ knowledge of geology. - Develop teachers’ awareness of informal science education resources, particularly at National Parks, available to supplement and enhance science instruction - Develop teachers’ ability to use National Parks as labs for innovative, relevant place-based science instruction, including collaborating with informal educators - Create survey instruments to assess NPS informal science resources - Pilot survey instruments & revise - Administer surveys - Analyze data / Write report - Pre and post surveys to determine whether participants’ learned about informal science resources and how they might incorporate them into their teaching. -Interviews with sub samples of pre-service teachers about how they might use NP as a part of their curriculum in the future. (depends on funding) Sample of PARK Program Evaluation Matrix

PARK Teachers Check for Understanding Task 4: Evaluation Matrix Part 1: Complete your group’s evaluation matrix for your project A. Break down each program outcome and generate a list of tasks that need to be completed to achieve the specific outcome. B: Identify how you plan to measure or evaluate if you have achieved the outcome. Part 2: Present your evaluation matrix to the whole group.

PARK Teachers Select the Appropriate Assessment Strategies It is imperative that you select the appropriate assessment strategy when developing your evaluation plan. The type of assessment used must match the: program outcomes and type of knowledge being evaluated.

13 PARK Teachers Assessing Different Types of Knowledge Essential to select the appropriate assessment instrument to evaluate different types of knowledge. Most programs focus on a combination of content and performance-based knowledge. Content Knowledge Performance or Skill Knowledge

PARK Teachers Part 1: Select the Appropriate Assessments A. Identify how you plan to measure or evaluate if you have achieved the outcome. B: Think about the desired outcome and the type of knowledge being evaluated when selecting the assessment instrument. Part 2: Present your assessment selection to the whole group. Check for Understanding Task 5: Selecting Assessments

PARK Teachers Pilot and Revise Assessment Instruments Piloting and Revising the Assessment Instruments is Essential! The revision process helps you to: clarify what you are trying to assess, make revisions to other program components (content, instructional activities, or course planning)