1. Process Evaluation the implementation phase

2. Were the magic bullets used?
How do you know when teachers put a new program into action?
What kind of follow-up is needed to insure that a staff development workshop actually makes it to the classroom?

3. Were the magic bullets used?
Ongoing Monitoring
Ongoing Technical Assistance
Mentoring / Modeling / Model Classrooms
Resources / Materials
Incorporate the new program into administrative expectations / supervision / performance evaluation

4. CIPPI Model Context Input Process Product Impact
The goal is to make INFORMED judgments about Program merit or value.

5. The Nature of Evaluation Tasks
The goal is to make INFORMED judgments about Program merit or value.
Making judgments requires a thorough understanding of the implementation of the treatment.
What was delivered to the target audience?

6. Process Evaluation
The main function of a process evaluation, or the process evaluation phase of a comprehensive evaluation, is to monitor the implementation of the Program.
You can’t make judgments of program merit before you know if the program reached the target audience.

7. Evaluation Strategies
Observations
Interviews
Implementation Checklists
Contact / Activity logs
Records Review

8. The Implementation Plan
Has the plan for Program implementation, reported in the Input evaluation phase, been put into action?
Has the Program made any changes to the original implementation plan?

9. The Implementation Plan
Has the Implementation plan been successfully communicated to the frontline service delivery staff in a thorough manner?
Has the frontline service delivery staff been trained in a thorough enough manner to lead to complete implementation?

10. Stakeholder Expectations
What do the stakeholders (especially funding agencies) expect to happen?
What level of accountability for Program implementation do they expect?

11. Implementation Standards
Do Program implementation standards exist?
Is there an Implementation Fidelity Checklist available? If not, one may need to be developed with the input of stakeholders and Program personnel.

12. Process Evaluation Plan
Create a plan for documenting the implementation of the Program.
Attendance data, activity logs, contacts records, descriptions of the activities or interactions, descriptions of the content of the services delivered, materials disseminated, etc.
Make a timeline for who collects what and when.

13. Dose-Response
It may be helpful to develop an Intensity of Intervention Scale. Such scales can be used to examine a “Dose-Response” relationship.
Assist the Program in developing an evaluation information system that will support the steps outlined in the plan.

14. Traveling Observer Technique
There is no substitute for site visits with a structured agenda, including observations of specific program activities and facilities, and requests for data.
Provide each Observer with a protocol, including checklists and schedules.

15. Formative Conclusions
Are there any problems with frontline staff compliance with the implementation plan? Do staff activities match the plan?
Are the participants, or target populations, cooperating with the implementation of the Program?

16. Formative Conclusions
Does the Implementation Fidelity Checklist data demonstrate any problems with poor execution of the plan?
Is the implementation on schedule?

17. Summative Conclusions
What dosage level (amount of the treatment or Program) was actually received by the participants?
Are there any sub-group differences in dosage level?

18. Summative Conclusions
Summarize your conclusions about the implementation of the program.
Based on all of the information you have gained from the Process Evaluation Phase, was the treatment delivered?
Will the summative evaluation evidence be a fair test of Program merit or value?