1. Ronald Head Achieving the Dream Data Coach
Formalizing Evaluation Strategies: The Development of an Evaluation Plan to Gauge Progress (Formative and Summative)
Welcome to Formalizing Evaluation Strategies: The Development of an Evaluation Plan to Gauge Progress (Formative and Summative). This presentation is a follow-up to the presentation I gave last spring at the KCCD district office on data evaluation and repeats some of the information I provided in that presentation.
Ronald Head
Achieving the Dream Data Coach

2. Implement interventions
Evaluate
Implement interventions
Develop interventions
Identify gaps
Analyze data
Collect data
First, a little background. I’ve shown this slide before but it’s a nice depiction of where evaluation occurs in the process. At this point in your Achieving the Dream journey, you’ve collected data, analyzed data, identified achievement gaps, and developed specific interventions. You are now in the second year of Achieving the Dream and in the process of implementing your interventions. To ensure that they succeed, you need to develop a written evaluation plan, and then evaluate the interventions.

3. What IS an evaluation plan?
For our purposes, we can define an evaluation plan as a document that describes how you will monitor and evaluate your student success interventions and how you will use the results to determine success and to modify the interventions as appropriate.
How you will monitor and evaluate your student success interventions and how you will use the results to determine success and to modify the interventions as appropriate

4. What IS included in an evaluation plan?
Planning model
Interventions
Etc.
WHAT?
Logic Model
Evaluation plan
HOW?
An evaluation plan tells you how to answer the WHAT?, in this case, your interventions, and the HOW?, or the process by which you will measure the interventions (logic model, etc.).

5. What IS an evaluation plan?
Who is responsible for what?
WH0?
Timeline for evaluating interventions
WHEN?
It will also clarify the WHO? (who is responsible for what?), and the WHEN? (the timeline for evaluating the interventions)

6. Why develop an evaluation plan?
Provides direction
Let’s us know whether our interventions are successful
Allows for mid-course corrections
Creates a shared sense of responsibility
Why should you develop a written evaluation plan? An evaluation plan
Provides direction
Let’s us know whether our interventions are successful
Allows for mid-course corrections
Creates a shared sense of responsibility

7. How do you develop an evaluation plan?
1. Describe your intervention
2. Identify evaluation questions
3. Complete the evaluation plan
4. Monitor the evaluation plan
5. Use results for improvement
We can develop an evaluation plan by following the five steps in this slide.
SOURCE: Rincones-Gómez, R. J. (2009). Evaluating Student Success Interventions: Principles and Practices of Student Success. Silver Springs, MD: Achieving the Dream.

8. Logic Model Inputs Activities Outputs Outcomes
Resources needed: human, financial, organiza-tional, etc.
Activities
Tasks that use your resources in order to produce an output
Outputs
Results or products generated by the completion of your activities
Outcomes
Effects or changes the intervention makes on participants
A logic model is one tool for describing our interventions. Here are the typical elements of a logic model. Regardless of whether you use a logic model or not, you should consider all elements in this slide as you develop your evaluation plan.
SOURCE: Rincones-Gómez, R. J. (2009). Evaluating Student Success Interventions: Principles and Practices of Student Success. Silver Springs, MD: Achieving the Dream.

9. Help developing an evaluation plan
Rigoberto Rincones-Gómez, an Achieving the Dream data coach, has written extensively about evaluation and logic models. You will find his Evaluating Student Success Interventions helpful and it is available at the link listed in this slide.

10. 3. Complete an evaluation plan
Evaluation Questions
Expected Outcomes
Tasks
Personnel
Timeline
Data source(s)
Analysis
Reporting
Here is another way to look at an evaluation plan. These are typical elements included in an evaluation.

11. Intervention #1 Intervention: Improve the intake process for students.
Success Indicator: Number of student completing matriculation components.
Baseline Data: Number and % of students completing matriculation during previous terms.
Target: An increased number and/or % of students completing matriculation.
Success = meeting target
Here is one way to think of assessing your first intervention.

12. Evaluating Data Elements Baseline (% using services)
Target
Improved matriculation services
50%
60% 
 52%
 56%
So, the figures might look something like this (these figures are fabricated).

13. 5 AtD KPI’s KPI Baseline Target Actual 2007-08 - 2009/10 (by 2015-16)
1. Developmental Completion (English)
20.60%
25%
Developmental Completion (Math)
18.90%
2. Gateway Course Completion
62.20%
70%
3. Successful Course Completion
60.20%
65%
4. Persistence (Fall-to-Spring)
72.90%
80%
Persistence (Fall-to-Fall)
55.60%
60%
5. Certificate/Degree/Transfer
14.70%
18%
You will also need to assess your progress toward the AtD 5 KPI’s. Here is an example of setting targets.

14. Summative assessment = conducted at the end to determine the success of an intervention.
Formative assessment = used mid-stream to improve processes, products, etc.
Defining success based on target data, as shown in the previous slides, is an example of summative assessment. You will also want to conduct formative assessment throughout the evaluation process to make mid-stream corrections.

15. Examples of Formative Assessment: Focus groups Questionnaires
Interviews
Etc.
Examples of formative assessment include focus groups, surveys and questionnaires, interviews, etc. Focus groups are particularly effective when dealing with AtD interventions. I have material on focus groups that I can and Michael Carley can train moderators and others in conducting focus groups.

16. 5. Use results for improvement
Learn
Share
Communicate
Improve student success
Use the results of the evaluation to learn, to share, to communicate, and above all, to improve student success.

17. Some final thoughts … Evaluation plans come in many
different shapes and forms …
Evaluation plans come in many different shapes and forms so don’t feel you have to strictly follow what has been presented here. Feel free to be creative!
Feel free to be creative!

18. Questions & Discussion
Questions and Discussion
Are there any questions or items you wish to discuss?
Questions & Discussion