1. Evaluating Multilingual Education Programs Seminar on Multilingual Education Kabul, 13-18 March 2010 Dennis Malone, Ph.D.

2. Preliminary research Advocacy & mobilization Recruitment Orthography development Graded reading materials Training & supervision Curriculum Supportive political environment Components of successful MLE programs Management & coordination Documentation & evaluation Funding

3. Participatory evaluation in MLE programs 1) Focus What specific part of the program do we need to evaluate? 2) Purpose Why do we need to evaluate this part? What will we do with what we learn about it? 4) Information sources Where will we get the information we need? 3) Indicators What do we want to learn? How will we know if we have achieved our outcomes & outputs? 5) Methods / Tools How will we get the information we need? 7) Time frame When will we begin the evaluation? When will we finish collecting information? When will we finish analyzing the information? When will we finish documenting what we have learned? 6) Responsible persons Who will be responsible for collecting and analyzing the information and documenting what has been learned?

4. Two Evaluation Types Formative Evaluation done during the program Purpose: to identify strengths and weaknesses Leads to actions that maintain what is strong and strengthen what is weak. Summative Evaluation done at the end of the program Purpose: to determine if the program goals and outcomes were achieved Provides relevant information to program funders and other stakeholders

5. Two Evaluation Approaches Qualitative Why did it happen? How did it happen? Who made it happen? Quantitative What happened? How much happened? Compared to what?

6. Teachers’ performance Quality of curriculum and of reading materials Achievement of program goals Costs of the program compared to stakeholders’ satisfaction Evaluations that focus on the quality of the program assess:

7. Focus group discussions Interviews (keep accurate quotes) Observation Portfolios Informal discussions Examination of materials that were produced Evaluation methods that focus on quality:

8. Achievement of outcomes and outputs Learners’ progress in achieving reading and writing skills Costs of the program compared to achievement of program outcomes and outputs. Costs of the program compared to learners’ progress through the system Evaluation methods that focus on quantity:

9. Questionnaires (short answer) Tests/Examinations Analysis of attendance records, student progress reports, other documents Evaluation methods that focus on quantity

10. What information would you need to answer the questions on the next slide? What assessment tools (questionnaires, interviews, etc.) would you use to get the information?

11. Are the Grade 1 parents satisfied with the MLE program? How much progress did the Grade 1 children make in reading and writing in their language? How good are the children’s Stage 1 reading materials? Are people who completed the adult literacy course two (2) years ago reading and writing now? What do they read and why? How well has the program achieved the goals, outcomes and outputs of MLE program plan (for report to funders)?

12. Planning the evaluation process Program Outcomes IndicatorsData neededMethods / instruments Person(s) responsible Time Frame Adapted from Chong project, Thailand

13. Evaluating reading materials— some suggestions Do action research with Grade 2 or 3 students to see if they can read Grade 1 materials easily? Do “cloze” tests with Grade 1 children using Grade 1 reading materials. Analyze reading materials in Grade 1 to see which are “used” most, which least? (Well-used, dirty, wrinkled, torn; OR rarely used, new-looking, clean, unwrinkled.) Interview children about their favorite books.

14. Documentation Results of preliminary research Reports Records Documents Assessment Results

15. Results of Preliminary Research Situation before program begins Language attitudes literacy uses Education records from previous years (in formal education—for children’s programs) to use for comparison Community’s identified goals and needs Context analysis (key factors)

16. Reports Training activities Supervisory visits Public relations activities (e.g., school opening ceremony) Support committee meetings Meetings with donors

17. Records Training activities Public relations activities (e.g., school opening ceremony) Support committee meetings Meetings with donors Budgets / donor reports

18. Documents Copies of all curricula and instructional materials Samples of reading materials Samples of the students’ work Copies of all assessment instruments including exams

19. Assessment Results… of pre-tests of follow-up assessments of learners’ progress in formal or non-formal education of periodic evaluations of the community’s perception of the program of end-of-year evaluations of materials, training, curriculum

20. Questions for group discussions What kind of evaluations are needed for programs in your community? Who can do the needed evaluation? What kind of training is needed for implementing the evaluation? What are challenges to effective evaluation? What are solutions?