4 th Biennial SAMEA Conference Meaningful Evaluation: Improving Use and Results Evaluation of Teacher-Directed ICT Initiatives Presenter Mokete Mokone M&E Associate, FeedbackRA 20 September 2013

About the Presentation Purpose of the Evaluation Design & Questions Benefits and Findings Changes to the Evaluation Design Lessons for Evaluations Conclusion

Purpose of the Evaluation To provide baseline, formative and summative information about the project: Improve project implementation strategy Improve project delivery mechanisms Establish the role of ICT solutions in improving teacher content knowledge

Evaluation Design Evaluation TypeComponentEvaluation Questions Design Review To explicate the theory of change underlying the project and investigate implementation arrangements that may have an impact on the implementation of the project =reviewing project plans, concept document and conducting key informant interviews. What is the postulated theory Targeting? Stakeholders How were teachers motivated to participate in the project?

Evaluation Design Evaluation TypeComponentEvaluation Questions Implementation Evaluation Which aimed to comment on the implementation fidelity drawing on sources such as system usage statistics and interviews with teachers and other key informants in the project. To what extent was the pilot project implemented as planned? Service Delivery Drop-off rates Frequency rates Dosage Levels

Evaluation Design (Cont.) Evaluation TypeComponentEvaluation Questions Outcome EvaluationUsing pre-post testing to determine teacher knowledge gains. The testing of teachers was based on a scholastic assessment representative of the content a teacher would focus on in that year. The test was scheduled at project start and completion. Did the project result in the desired / expected process outcomes? What is the level of teachers’ content knowledge or Maths, Science and English at baseline and at the end of the project?

Benefits and Findings Design review (assessing the theory of change) revealed flaws in project design (insufficient training, limited knowledge of ICT, no support structure, curriculum challenges) Implementation review (examining fidelity) showed that participants where not spending the required amount of time on the ICT platform, low dosage levels, weak participation Outcome findings through a pre and post baseline test was planned to determine knowledge pre-intervention and after project implementation. The post test, could not take place due to low levels of participation

Evaluation Scope Change Move away from determining programme effectiveness = drawing lessons on failures Lessons for future ICT teacher directed projects Key informant interviews and participant interviews= 1. What has been learnt from the implementation of Teacher ICT projects, which may be useful to keep in mind when designing similar future projects? 2.What enabling and hindering factors determine the success of Teacher ICT projects? 3. What appropriate systems and infrastructure should be in place before initiating a Teacher ICT project?

Conclusion Evaluations and data collection is often done up front, but it should be responsive to changes in context and implementation High chance of implementation failures requires the evaluation to spend more time to establish if necessary circumstances exist to make implementation possible. Its critical to respond to changes affecting an evaluation plan, at the same time developing an appropriate solution that still allows critical information to be obtained.

Conclusion (cont.) Although the original findings that were sought, were not achieved, usable data was still acquired and helped inform organisational strategy for future Teacher ICT projects. Evaluative findings must also be hugely embedded in the development context and seek to answer a long term view of empowering communities. The success and failure of innovative ICT pilot projects still have lessons that can be applied in developing new and improved interventions.