THE LOW ACHIEVEMENT TRAP IN MIDDLE INCOME COUNTRIES: COMPARING BOTSWANA AND SOUTH AFRICA A “NATURAL EXPERIMENT” IN BOTSWANA AND SOUTH AFRICA

A LARGE COMPARATIVE STUDY OF STUDENT LEARNING GAINS SOUTHERN AFRICA The study was conducted by the Human Sciences Research Council, the University of Botswana, and Stanford University during the year Sample: 60 primary schools catering to low-income students on each side of the Botswana-South Africa border th grade mathematics classrooms and more than th grade students in those schools. The Study tested students in mathematics near the beginning and end of the year. The Study tested teachers in mathematics, filmed teachers giving a lesson at least once during the year (many twice). The Study analyzed student notebooks in mid- and end-year to determine curriculum coverage (OTL). The Study applied detailed questionnaires to students, teachers, and principals.

THE MAIN POLICY RESULTS OF THE STUDY 1. Classroom differences in our sample are much more important in explaining learner mathematics gains in North West than in Botswana. 2. The most important variables that contribute to learner gains in North West are mathematics teaching quality as measured from analyzing videotaped lessons and the number of lessons taught on test topics as measured from notebook analysis. 3. Teacher mathematics knowledge in North West, as measure by our teacher math test, is significantly positively correlated with better teaching quality and more lessons taught on the test topics.

THE BOTTOM LINE 1. Our study shows that in South Africa the impact on student learning gains of improving mathematics teaching quality (mainly through increasing teacher mathematics knowledge) could be very large. 2. Our study shows that the impact on student learning gains in South Africa of increasing the number of mathematics lessons taught by teachers on required topics could also be very large. 3. It is possible to increase learning gains significantly in Botswana through improving the quality of teaching and teacher mathematics knowledge, but the impact size of such improvement is much smaller than in South Africa. 4. We believe this is the case because Botswana’s educational system is already more “efficient” than South Africa’s, albeit at a rather low performance level.

THE STUDY IS A “NATURAL EXPERIMENT” Students and teachers on either side of the border come from the same cultural grouping, but the school systems have developed in very different historical contexts. Teachers received somewhat different training, curriculum is similar but was formulated more systematically and with greater teacher participation in Botswana. Role of teachers has been different historically, and supervision has been different. We compared how schooling is delivered in the two contexts, and what are the factors that contribute to student achievement gains. We analyzed whether 6 th graders are learning differently in the two contexts.

BORDER AREA INCLUDING FORMER BOPHUTATSWANA HOMELAND

PROPOSAL: STUDENTS IN BOTSWANA DO BETTER BECAUSE OF A SERIES OF SMALL DIFFERENCES IN SCHOOLING DELIVERY From our initial observations and other international tests, low-income students in Botswana seem to do somewhat better than students in South Africa. We proposed that a series of small differences contribute to the observed differences in student learning and learning gains. To measure these differences, we estimated production functions for mathematics achievement gains in each set of classrooms. We used cross section (final test) and value added models in order to compare the more usual kind of results from cross- section data such in SACMEQ with the value-added data we collected.

VALUE-ADDED STUDIES ARE IMPORTANT BECAUSE THEY BETTER IDENTIFY “TRUE” CLASSROOM EFFECTS When we estimate teacher “effects” on student achievement measured at one point in time—say, at the end of the 6 th grade— we cannot control for earlier grades’ teacher contributions tp learning. It is not enough to control for the student’s family background, since there is considerable variation in academic skills of students with similar family backgrounds. Also, teacher skills are probably not randomly distributed among students with different academic skills or SES. “Better” teachers usually end up in schools with “better” students, for various reasons. Thus, we don’t know whether better teachers are “causing” higher student test scores measured in that grade, or whether better teachers have chosen or been chosen to work with higher scoring students. Measuring the change in achievement over a year or from year to year takes care of most of both these problems.

STUDENT ITEM SCORES ON THE INITIAL TEST HAVE THE SAME PATTERN IN NW & BW BUT ARE HIGHER IN BW (RED)

STUDENT ACHIEVEMENT GAINS IN NW ARE NOT LARGE AND ARE SMALLER THAN IN BW (NOT SHOWN)

TEACHER TEST ITEM SCORES IN NW & BW HAVE SIMILAR PATTERN BUT ARE HIGHER IN BW (RED)

TEACHER TEST SCORES IN NW ARE HIGHER IN CLASSROOMS WITH THE HIGHEST AVERAGE SES LEARNERS

THE NUMBER OF LESSONS TAUGHT IN NW CLASSROOMS IS MUCH LOWER THAN IN BW CLASSROOMS StatisticNorth West Recorded LessonsBotswana Recorded Lessons Mean Standard deviation Minimum2133 Maximum97142

THE FREQUENCY OF LESSONS SHOWS THAT IN NW, MANY TEACHERS ARE TEACHING FEWER THAN 50 FROM FEBRUARY-EARLY NOVEMBER

IN BOTH COUNTRIES THE % OF REQUIRED TOPICS COVERED IS LOW On average the North West sample of classrooms covered 52 percent of the test topics relating to Number, operations and relationships 29 percent of the test topics relating to Space and shape (geometry) 28 percent of the test topics relating to Measurement 18 percent of the test topics relating to Patterns, functions and algebra 14 percent of the test topics relating to Data handling On average the Botswana sample of classrooms covered 44 percent of the test topics relating to Number, operations and relationships 32 percent of the test topics relating to Space and shape (geometry) 34 percent of the test topics relating to Measurement 16 percent of the test topics relating to Patterns, functions and algebra 21 percent of the test topics relating to Data handling

LEVELS OF COGNITIVE DEMAND IN TEACHING VIDEOTAPES ALMOST ALL MEMORIZATION OR PROCEDURES W/O CONNECTIONS

FEW TEACHER QUALITY RATINGS FROM VIDEOTAPES ARE “VERY GOOD (3 RATING)

TEACHER TEACHING SKILL IS MUCH MORE RELATED TO TEACHER MATH TEST SCORE IN NW (RED) THAN BW (BLUE)

CROSS-SECTION & VALUE ADDED RESULTS North West ProvinceBotswana Value Added Models Variable Post-Test Only I. Post-Test Controlling for Initial Test Score II. Test Score Gain (Post-test minus Pre-test score) Post-Test Only I. Post-Test Controlling for Initial Test Score II. Test Score Gain (Post-test minus Pre-test score) Initial test score *** *** Teacher experience (years) Teacher experience squared Teacher test score B *0.0005*0.0004* Better teaching quality a *0.0468**0.0508**0.0332**0.0197**0.0154** Best teaching quality a Total lessons on topics * * Observed class size * Average class SES0.0103***0.0048* * Class violence index Learner characteristics included? YES YES YES YES YES YES Constant0.3464***0.2129*** ***0.1875***0.065 Adjusted R Observations

THE IMPACT ON LEARNING GAINS OF IMPROVING TEACHING QUALITY AND LESSONS TAUGHT IS LARGE In NW province improving teaching quality from low to “better” teaching could increase math gains from an average of 3 percentage points to 7-8 percentage points, or almost one-half a standard deviation. If teachers would increase the number of lessons taught on test topics from an average of 60 lessons to 82 (one standard deviation), our estimate show student gains would increase by 3 percentage points. In Botswana, improving teaching also has a large effect on student learning gains, but much smaller than in NW, only 2 percentage points. The effect on learner gains of raising teacher math knowledge in Botswana is positive but very small.

ARE GAINS HIGHER IN BOTSWANA BECAUSE OF MORE RESOURCES OR GREATER EFFICIENCY? Botswana invests more resources per student than North West Province. Our results show greater possibilities of improving student achievement in North West Province than in Botswana using existing resources. There is more “room” for improvement in North West even though resources in neither country are used very effectively: teachers are not very good at teaching mathematics and they teach much less than required. For North West classrooms to improve student achievement to Botswana levels, they need somewhat more resources and to use those resources more effectively.

PRODUCTION POSSIBILITIES AND THE ISSUE OF EFFICIENCY VS. MORE RESOURCES

CONCLUSION 1: SA STUDENTS ARE NOT DOING WELL, BUT THERE ARE CLEAR STEPS TO TAKE TO IMPROVE GAINS. Our study suggests that relatively low-income students on the Botswana side of the border perform better than similar South African students. Reason: The system in Botswana has been more effective in organizing teachers to teach more regularly and has trained them somewhat better in mathematics, and to teach mathematics somewhat more effectively. This creates room for considerable improvement in South Africa through more effective teacher training and more effective accountability measures to increase student opportunity to learn. This seems simple, but it is not, since both those deficiencies are the result of many years of better organization in Botswana schools. On the other hand, the news for Botswana is not so good either. Despite a lot of financial resources and years of better organization, students in Botswana are also learning mathematics at a rather low level. Botswana may also require a large shift in the way it does things in order to get out of relatively low level equilibrium “trap.”

CONCLUSION 2. IMPROVEMENT REQUIRES LONG TERM GOALS & A SERIES OF STEPS TOWARD THAT GOAL 1.Implement teacher in-service summer programs designed to increase teacher mathematics content knowledge and mathematics teaching knowledge. 2.Gradually eliminate academic year, during school in- service training. 3.Implement accountability systems at the provincial level to increase mathematics lessons taught, curriculum coverage, and teaching content demand. 4.Drastically improve teacher pre-service training, with greatly increased requirements of content knowledge instruction.

IS IT WORTH GETTING SA STUDENTS UP TO BW LEVELS, WHICH ARE STILL LOW? Since students in Botswana are only scoring about 6 points higher than students in NW, should be bother to make these big changes just to raise scores to a higher but still low level. The answer is clearly YES. A one half standard deviation increase in student achievement is a big deal anywhere in the world, even from a low level. Perhaps, and this is just speculative, improving teaching in SA and increasing the number of lessons might generate other changes that would have interactive and larger effects. Of course, the idea is also to use such reforms to continue to make more improvements, and to continue to assess the system to find more ways to generate greater gains.