THE WORLD BANK Development, Purposes, Uses of EGRA Session 1.1 1

Outline 1.Purpose: why Early Grade Reading (Assessment)? More focus on quality Using Early Grade Reading as a “marker” of quality 2.Development 3.Uses 2

Outline 1.Purpose: why Early Grade Reading (Assessment)? More focus on quality Using Early Grade Reading as a “marker” of quality 2.Development 3.Uses 3

Purpose: Why? Quality issues ■What are the big international goals? ■How do low income countries compare to high income countries?  LI to HI ratio −Gross primary enrollment: 95% −Net primary enrollment:80% −Gender parity NER:94% −Completion:58% −Learning achievement:Approx 30%? −Learning achievement:Median LI = 3 rd percentile of HI or lower LI= Low income, HI = High income 4

Just as interesting is the internal distribution within the poor countries, and the comparison with the rich countries... 5

PIRLS 2006 Results 6

7

Countries PIRLS 2006 Results 8

9 In path of development of good performance, the biggest and fastest drop is in the number of worst performers… In the “standard” or better path, an educational middle class is developed by helping those with worst performance… And the drop in those with very poor performance is much bigger than the increase in those with good performance But this does not happen by accident…

South Africa Morocco Kuwait Qatar Indonesia Iran Slovenia Slovak Rep. France Denmark Italy PIRLS 2006 Results Percent of learners LowestMediumHighest Reading competency levels 10

11 Need to work at the worst end of the distribution, and with the countries that are really lagging…

Conclusion on quality issues ■So, quality, and above all equality of quality (more equality in learning outcomes) seems to be lagging… ■EFA goals do refer to quality ■But most of the practical emphasis is on access, most of the judgment about “are we on track” is around access ■Some moves afoot to track learning outcomes in a consistent way ■And countries do participate in international and regional assessments, so we are not completely in the dark… So, why a focus on Early Grade Reading? 12

If quality matters, why focus on Early Grade Reading? Early Grade + Reading = ■intervene early, ■intervene on reading, ■have some way to assess orally Let’s see if we can motivate those conclusions 13

Why early? ■Refer to the large and reliable literature on benefits of early (even ECD) intervention ■But let’s focus on a few simple things 14

“For unto every one that hath shall be given, and he shall have abundance: but from him that hath not shall be taken away even that which he hath.” Matthew Effect? Why early? 15

Grade in years and months (thus 1. is 6 months into Grade 1) 6 Why early? Matthew Effect in reading Data from the US Children below a certain level by the end of Grade 1, stay behind forever, and the gap widens And, if they cannot read, they fall behind in everything else Initial SES gets amplified Words per minute 16 Good, Simmons, Smith (1998)

Why early? Reading Trajectories of Low and Middle Readers Words per minute Good, Simmons, Smith (1998) 17

And continues to university & graduate school … Kerchoff and Glennie (1999) This controls for SES and prior academic achievement, so it is the NET effect of a demanding curriculum. 12 th Grade Schooling 12 th Grade Achievement Attainment Four Years Later Attainment 10 Years Later LoSchool LoCurriculum HiCurriculum *** MidSchool LoCurriculum *** 2.054*.266 HiCurriculum 7.587*** *** HiSchool LoCurriculum *** HiCurriculum 7.393*** *** 18

Why early? ■Not only does the problem, if unresolved, amplify over time, but… ■According to various experts, fixing a problem:  in grade 1 takes 30 minutes  in grade 5 takes 2 hours ■Teaching good habits early is key, and it becomes very difficult to remediate later 19

Why reading? ■No, it is not “the only thing that matters” ■But it is a good one to start with  It is a (the?) foundational skill - Hard to imagine anything else going well if children can’t read well and soon  It can be used as a marker - Hard to imagine a good school that can’t teach children to read, if children are not reading, the school (district, country) needs serious help 20

Why oral reading? Reason 1  Oral reading seems to be good predictor  At least in the US (other industrial?) perhaps the best predictor, in early grades, of concurrent and even later success in a broad range of areas 21

Oral reading predictive power Examples: ■ Wilson (2005): 0.74 correlation between oral reading measures and broader cognitive achievement in Arizona ■ Fuchs, Fuchs, Hosp, and Jenkins (2001) survey and explain the rather high correlations between oral reading fluency and a large variety of other tests. ■ Vander Meer, Lentz, and Stevens (2005): 96% of children judged to be at risk using oral reading turned out to be “non proficient” in the Ohio’s more comprehensive test, while of those classified as “low risk” using oral reading fluency, 72% were classified as proficient using a more comprehensive test. ■ Shaw and Shaw (2002): find similar results for the relationship between simple measures of oral fluency and deeper state-wide measures of reading in Colorado ■ Fuchs, Fuchs, and Maxwell (1988): correlation of 0.91 between oral reading fluency and other comprehensive tests ■ Juel (1988) : “The probability of remaining a poor reader at the end of fourth grade, given a child was a poor reader at the end of first grade, was the probability of remaining an average reader in fourth grade, given an average reading ability in first grade, was.87.” ■ Schilling, Carlisle, Scott, and Feng (2007): 80% of at-risk with ORF turned out to be in bottom quartile with Michigan’s own reading test at EOG ■ Wood (2006) finds significant correlations between oral reading fluency and later curriculum- based tests, and finds that oral reading fluency adds explanatory value even when other factors are considered. ■ Some of these recommend adding comprehension and vocabulary (EGRA does comprehension, not vocabulary), but ORF by itself does a pretty good job 22

Why oral reading? Reason 2: If use pencil and paper test too early: children bottom out, and test provides too little information on why children can’t read 23

Why oral reading? Reason 2 Case of country X ■Pencil and paper test ■Mostly attempt to directly measure comprehension  Sufficient: 15%  Basic:24%  Below grade:15%  Less than below grade:46% ■We know they do not comprehend, but there seems to be little sense of why: can they dominate the basics? ■Authorities admit that children bottom out if only try to measure comprehension, so try to measure simpler things: 24

About the easiest level: 25

■If can’t do it, why? ■Can’t decode? ■Don’t have the vocabulary? ■Some other problem? ■Hard to say, it seems… 26 Why oral reading? Reason 2

27 Why oral reading? Reason 3 Elements of oral reading are in accord with curricular framework (at least for some countries) but frequently there are no specific (teacher-level) guidelines on how to assess…

28

29

Mother Tongue, Grade 1 But this almost never happens 30

31 Mother Tongue, Grade 2

Why oral reading? Reason 3 ■Not in all (don’t know really how many – has anyone studied this?) ■Even then maybe not specific enough (compare to any US state)?  (How specific is optimal? How specific is “good enough”?) ■In no country I’ve studied is it operationalized in terms of specific assessment standards or procedures that teachers and officials can actually use So, at least in some countries the curriculum does get down to specifics, but… 32

Outline 1.Purpose: why Early Grade Reading (Assessment)? More focus on quality Using Early Grade Reading as a “marker” of quality 2.Development 3.Uses 33

Development thus far ■“Organic” process: meets “market test” at each step ■First: informal, small samples, see if it was useful at generating awareness, very little funding ■Attention attracted ■Some funding to try it a bit more formally  USAID funding: validate efforts thus far with expert opinion, try some more applications  Experts validate, suggest increased formality, seriousness of trials  World Bank adds some funding, try it in two more international languages, local languages 34

Development thus far ■Tried it in some more countries ■Countries found it useful: “market test” met ■Assess statistical properties, test further country reaction, do a few more countries, “market test” further met ■And here we are: further evaluate results, discuss, chart further direction and uses 35

Outline 1.Purpose: why Early Grade Reading (Assessment)? More focus on quality Using Early Grade Reading as a “marker” of quality 2.Development 3.Uses 36

Possible uses ■Range of uses of the approach not predetermined ■Seek guidance from all of you ■And, different uses would require very different versions of the idea… ■That being said, here are some of the uses that are possible… 37

Possible uses ■But… these uses have been tried or at least discussed (and there is some sequentiality here):  Policy awareness and motivation  Macro  Community-based  Impact tracking and evaluation  Project monitoring  Project impact and evaluation  System monitoring over time  Teacher-based assessment  (Could link to community-based awareness, accountability?) 38

Possible uses  Country comparisons  Within language groups? Not even?  Not a good use, according to many experts’ recommendations:  High-stakes bureaucratic accountability  E.g., bureaucratic reporting for some high-stakes purpose 39