Comments on: The Evaluation of an Early Intervention Policy in Poor Schools Germano Mwabu June 9-10, 2008 Quebec City, Canada

Introduction The purpose of the proposed study is to evaluate academic effects of a program, known as Supportive Promotion (SP). SP is like an alternative to grade repetition in a supportive school environment. In Argentina and other Latin American countries ~ 45% of pupils repeat a grade at least once before leaving the school system. The issue of grade repetition is of policy interest in other developing regions as well, especially in Africa, where repetition rates are higher than in Latin America.

Introduction… Why is grade repetition an issue of policy interest? There are at least three reasons: a.Parents and governments spend scarce resources on grade repetition; so there is need to evaluate the returns they get b.Grade repetition is higher among children from poor families; so information in (a) above can show how policies that discourage or encourage repetition (or prevent it) would affect the poor. (In some cases, repeating a grade can enhance prospects of admission into strong schools, and of getting a good job; on the other hand, it can waste students’ time without improving performance). c.Grade repetition can discourage students, through, e.g., peer effects.

Introduction… Relevant literature on grade repetition is reviewed in relation to academic outcomes and gaps in it identified. Two main gaps in the literature are identified. a.No credible way to measure effects of grade repetition due to endogeneity of repetition to education outcomes. b.Clear policies to deal with this problem are lacking in Argentina and elsewhere. The study will fill these gaps

Research Design and its Components An experimental study with a randomized phase-in of students into a treatment group, with students not phased-in, serving as a control group. The innovation of the project is in its design. Treatment group (TG) does not comprise repeating students receiving assistance, with the control group (CG) not getting any. Instead, TG is a regular group of students receiving interventions, with CG being the counterpart group without interventions.

Research Design… The experiment is suitably placed in an going education project (TPA), making its implementation non-controversial. However, since the research is being implemented outside the government system, scaling up of its interventions could be an issue (despite decree 105/2006). The planned research has three interventions.

Research interventions a. Protection of children’s school transition to the next grade: repeating or weak students are identified and assisted. b.Teaching process improvement: sequenced learning, starting with literacy, numeric system, and mathematical operations. c.Changing school institutions and organization: principals are helped to implement the new teaching process. Teachers and principals will be trained in the new methods of classroom instruction and school management. The interventions will be implemented in grades 1-3.

Randomization of interventions The mechanisms for randomizing interventions are not clear enough. If public schools are randomly divided into three groups, with group 1 schools receiving treatment (being phased- in into treatment) in year 1, and 2 & 3 being phased-in later, treatment assignment may not be random. There is need to describe this randomization of interventions in detail as in Miguel and Kremer (2004). First the method for selecting the sample of treatment and control schools should be explained. (The sample of schools in treatment and comparison groups need not be random, although randomness is necessary if results are to be generalized). Second, the interventions must be assigned randomly.

Randomization of interventions… There are several ways of randomizing school treatment phases. A.Miguel and Kremer method (2004): List a random (or non-random sample) of schools by area or alphabetically and count off repetitively 1-2-3…(Any of these groups can be treated in year 1). B.Another Method: A random sample of schools can be split into 3 groups by applying simple random sampling three times, but with replacement. (Any of these can be treated in year 1).

Study Hypotheses The null (the main hypothesis) -- that post- treatment tests scores are the same in treatment and control schools is clear. The distributional hypothesis -- that test scores are more equally distributed in treatment than in control schools is also clear. The study can further test whether grade repetition is the same in treatment and comparison schools, as this is a major concern in the proposal.

Methods Single and double difference methods will be applied and are appropriate, but they are unlikely to yield the same results due to selection or attrition biases. The PRE-test scores are in expectation the same for all students (due to randomization of treatment status of schools); so there is no need to control for them (p. 16). However, control variables at the school level, such as qualification of teachers, and school facilities can be included in the test-score regression (p. 16).

Other Issues It is not clear whether the 3 interventions (improving organization or management of schools, assisting weak students and sequencing of subjects) will be implemented as a package. Each of these on its own could have a major impact on test scores. Duration of exposure to interventions may vary across treatment schools, e.g., some teachers and principals may not attend training early.

The Issue of Grade Repetition This key issue will not be addressed directly. Instead, the study will show how weak students who repeat can be helped if the school environment were supportive. In particular, it will show the type of teaching and school management that can be implemented to prevent repetition, to help weak students progress to the next grade, and to help repeating students improve their test scores and move to the next grade. It will not examine effects of grade repetition on test scores or academic performance generally.

Budget and conclusion If, as indicated, the interventions are expanded to cover other regions, the proposed budget may be inadequate. The proposal is innovative and its objectives and methods are clear. Its findings will be of policy relevance in other regions outside Latin America.

THANK YOU.