1. 24 October 2018 Lilli Pretorius
ADAPTING EGRA TO ASSESS EARLY LITERACY IN THREE PROVINCES LESSONS FROM THE ZENLIT PROJECT
24 October 2018
Lilli Pretorius

2. OUTLINE Some issues around reading assessment
Theoretical reading framework underlying the Zenlit intervention
EGRA – background, components
Adapting EGRA for reading assessment in the Zenlit context – Zulu, Xhosa and English
Some results
Suggestions for way forward

3. DOES THE LOGIC MAP GET US TO OUR DESTINATION?
High quality PD programme
Increased teacher knowledge
Improved classroom practices
Improved learner performance
Even if PD packages have high-quality features, there is no guarantee that they will work.
Contention around sequence of Steps 2 and 3; if Step 3 comes first, then Step 2 may be “a dividend earned later” (Walpole & McKenna 2015)
In California, significant gains in teacher knowledge, but classroom practice did not solidify or extend the knowledge (Goldschmidt & Phelps 2010). (Step 2 happened)
In a large scale Grade 2 study in the USA, teacher knowledge improved, teachers taught more explicitly, but no difference were found in learner outcomes between interventions and control (Garet et al. 2008). (Step 2 and 3 happened, but not Step 4)

4. SOME ISSUES AROUND ASSESSMENT
who gets tested
what gets tested
when testing occurs
how tests are conducted
why the test is conducted (Sorenson 2015)
“To be useful, tests have to be reasonably accurate. But to be accurate, tests have to be fairly complex, and complex assessment procedures contradict the fundamental notion that assessment should be simple and easy to deliver.”
(Singleton 1997 ‘Screening early literacy’)
Tension between monitoring and evaluating a project and collecting data for research purposes

5. COMPONENTS OF READING LANGUAGE PROFICIENCY EXECUTIVE CONTROL
Working memory
Cognitive self-control
Cognitive flexibility
LANGUAGE PROFICIENCY
Listening comprehension
Vocabulary
Grammar (morphology/
syntax)
COMPREHENSION
READER RESPONSE
DECODING

6. STRUCTURES AND PATHWAYS
SOCIOECONOMIC AND CULTURAL CONTEXT
HOME SCHOOL COMMUNITY
EXECUTIVE CONTROL
Working memory
Cognitive self-control
Cognitive flexibility
LANGUAGE PROFICIENCY
Listening C
Vocabulary
Grammar (morphology/
syntax)
Cognitive-linguistic
COMPREHENSION
READER RESPONSE
DECODING
NEUROLOGICAL
STRUCTURES AND PATHWAYS

7. EGRA use worldwide
It arose from the need for the creation of simple, effective, and low-cost measures of learning outcomes, specifically in reading.
Development of EGRA began in October RTI International was contracted to develop an instrument to assess early grade reading, with input from a panel of reading experts.
The objective was to help countries begin a process of systematically measuring how well children in the early grades of primary school are acquiring reading skills, in order to inform ministries and donors regarding system needs for improving instruction.
It is not intended as a diagnostic tool to screen individual learners. It is also not intended to be a high-stakes accountability measure for determining grade passing.

8. COMPONENTS OF READING ASSESSED BY EGRA (Gove & Cvelich 2011:20)

9. THE EGRA JOURNEY in ZENLIT1.
EGRA from EC 2.
Pilot EGRA 2015 & CEA report 3.
Tweaked & EGRA 2016 (& a little in 2017) 4.
What can we learn from it?

10. 2015 2016 2017 Use to 2016 results to report back to schools
Obtained test from EC D of E
Assessed 5 learners per grade per school - workbooks
Trialled testing time
Budget challenges and constraints
ERA evaluators assessed the learners
Tests evaluated by Prof Sarah Howie at CEA, UP
Adapted the test to reflect orthographic features of Xhosa and Zulu
Increased sample size to 15 learners per grade per school , random selection
Trained the coaches to administer the tests
About learners per morning (3 fieldworkers per school per morning)
Use to 2016 results to report back to schools
The dictation test was dropped – too easy
Tweaked the test a little bit – ironed out problem items
Included 4 control schools in each province

11. ORAL READING COMPREHENSION
EGRA COMPONENTS
Included digraphs and trigraphs
LETTER SOUNDS
Adapted a PA test for the Nguni languages (syllable awareness, phoneme awareness, phoneme deletion – 13 items)
PHONEMIC AWARENESS
Included longer words – 3-, 4- and 5-syllable words
WORD READING
Used grade appropriate text for each grade (longer and more difficult with each grade)
ORAL READING FLUENCY
5-8 questions, depending on grade
at least one inferential question
ORAL READING COMPREHENSION
ASSESSMENT AND EVALUATION

12. DEMOGRAPHICS 2017 Grade 1 Grade 2 Grade 3 Mean Age 6.7 (5 – 9) 7.8
(6 – 10)
8.8
(6 - 11)
Boys
Girls
49%
51%
50%
55%
45%
KZN EC WC
Control
Intervention
180
360
270
301

13. RURAL/URBAN CONTROLS AND INTERVENTIONS

14. CORRELATIONS between EGRA components NGUNI (Zulu/Xhosa) n = 988 correlations** significant at the 0.01 level
Phonemic awareness
Word Reading
ORF
Comprehension
Letter sounds
.65**
.75**
.67**
.58**
Phonemic
Awareness
.62**
.55**
.51**
.89**
.69**
.80**

15. CORRELATIONS between EGRA components English n = 481 correlations
CORRELATIONS between EGRA components English n = 481 correlations** significant at the 0.01 level
Phonemic awareness
Word Reading
ORF
Comprehension
Letter sounds
.63**
.57**
.55**
.51**
Phonemic
Awareness
.68**
.92**
.76**
.80**

16. MULTIPLE REGRESSION: What predicts WORD READING in NGUNI languages(Zulu/Xhosa) n = 988 Adjusted R square = 0.59
Beta
Standard error 𝜷 p
Letter sounds
.37
.017
.60
.000
Phonemic
Awareness
.84
.100
.22

17. MULTIPLE REGRESSION: What predicts WORD READING in English n = 481 Adjusted R square = 0.50
Beta
Standard error 𝜷 p
Letter sounds
.34
.061
.23
.000
Phonemic
Awareness
.35
.275
.54

18. MULTIPLE REGRESSION: What predicts READING COMPREHENSION in Nguni languages(Zulu/Xhosa) n = 988 Adjusted R square = 0.66
Beta
Standard error 𝜷 p
Letter sounds
.006
.002
.08
.004
Phonemic
Awareness
.046
.01
.10
.000
Word reading
-.028
-.22
ORF
.089
.89

19. MULTIPLE REGRESSION: What predicts READING COMPREHENSION in English n = 481 Adjusted R square = 0.74
Beta
Standard error 𝜷 p
Letter sounds
.006
.004
.04
.16
Phonemic
Awareness
.062
.023
.09
Word reading
-.028
-.29
.000
ORF
.075
1.03

20. Falling at the first hurdle… If children do not learn to read in Grades 1-3, they cannot read to learn in Grade 4 onwards
Components of reading
Spaull & Pretorius (2018) Falling at the first hurdle: Early grade reading in South Africa. Springer.

21. SOME TAKEAWAYS
It’s important to adapt the EGRA tool to the orthographic features of the target languages
The adapted EGRA results discriminated between learners’ reading abilities in control and intervention schools
The KZN and EC results provide preliminary data for differences in reading development between rural and urban schools
The results suggest that in transparent agglutinating languages, letter sound knowledge is important for word reading
The results indicate that in transparent agglutinating languages, ORF is a strong predictor of reading comprehension. This has been confirmed by other studies in SA (Spaull, Pretorius & Mohohlwane 2018)
Given the relatively small sample sizes, these results could serve as a starting part for further research into benchmarks for letter-sound knowledge and ORF

22. Building impetus for comprehension
Components of reading

23. DECODING AND COMPREHENSION BENCHMARKS
Independent level
98% decoding accuracy
95% level of comprehension
These are highly skilled readers who can effectively learn from texts appropriate for that specific maturational level.
Instructional level
95% decoding accuracy
75% comprehension
These are readers who do not have major reading problems but who benefit from reading instruction at their maturational level.
Borderline level
90-94% accuracy in decoding
55-74% accuracy in comprehension
These readers need to be given additional reading exposure and practice.
Frustration level
less than 90% decoding accuracy and about 50% or less comprehension
These are readers who have major reading problems and who are reading well below their maturational level. They need intensive reading programmes to increase their reading level.

24. SHARING ZENLIT 2016 ASSESSMENT RESULTS
Teachers and HODS sat at their school tables
How well are your learners reading in Zulu compared to the Zenlit mean results in KZN?
How can we use these results to help us pass the baton and set appropriate high standards (benchmarks)?
HODS to set up a reading and assessment plan for the year and report back (after tea)
Prov X School Y
Grade 1
Grade 2
Grade 3
Prov
School
Phonemic awareness %
31.7
28.2
50.7
40.5
59.5
37.4
Letter sounds score
Min – max
18.9
0- 70
16.3
0-31
36.1
0-96
23.5
0-46
35.7
0-93
21.9
0-55
Word reading score
12.7
10.8
25.9
20.8
55.2
42.2
ORF score
11.3
0-36
7.8
0-13
30.8
0-58
22.9
0-54
0-79
21.4
Oral Reading comprehension %
12.6 8
51.3
13.3

25. GROUP GUIDED READING – identifying cracks, modelling good reading, providing practice opportunities
Components of reading

26. PAIRED READING – provides practice opportunities
Components of reading