1. Larry Condelli Stephanie Cronen American Institutes for Research, USA LESLLA Sixth Annual Symposium Cologne, Germany August 26, 2010 1

2. Overview of Presentation Summary of Study Purpose and Design Impact of the Intervention Content of Instruction Student Attendance Discussion and Implications of Findings 2

3. Evaluation of an ESL Literacy Intervention: Sam and Pat Structured language approach Adapted from Wilson Reading System Never before evaluated for ESL Literacy activities organized around basal reader/workbook Heavily phonics-based 3

4. Sam and Pat: Instructional Approach Direct instruction, transparent Rules explained, modeled, practiced Controlled text, vocabulary and grammar Words match phonics already learned Sequential Easy to hard in defined steps ESL instruction to support literacy 4

5. Sam and Pat: Literacy and Language Skills Covered Phonics for reading and writing Sight words Oral reading for accuracy and fluency Reading comprehension Vocabulary Speaking and listening Grammar 5

6. Research Questions How effective is instruction using the intervention in improving the English reading, speaking and listening skills of low- literate adult ESL learners? Is the intervention more effective for certain groups of students (e.g., language, literacy level)? Do differences in level of implementation of Sam and Pat and other instruction relate to variation in impacts? 6

7. Study Design 10 adult ESOL centers across USA Paired intervention and “normal” ESL literacy classes (34 total) Random assignment of students and teachers 1,344 students participated for one term Minimum 5 hours/week 10-12 weeks instruction with approach Other instruction also provided Each class conducted twice over a year 7

8. Student Flow in the Study 1. Student applies to center 2. Assessed for NLL 3. Recruited into study 4. Gives informed consent 5. Random assignment to class 6. Pretests administered 7. Instruction 8. Posttests administered 8 Intake NL Literacy Post-test Random Assignment Recruited into Study Informed consent Pre-test Instruction

9. Students Assessments Phonics and decoding Word attack Letter/Word ID Reading comprehension Vocabulary (ROWPVT) Listening, oral expression 9

10. Sam And Pat Teachers in the Study All teachers randomly assigned 3-day teacher training on curriculum Follow-up visits by trainers Classroom observations to monitor fidelity Refresher webinar at start of second term 10

11. Students in the Study: “True” Literacy (LESLLA) Little or no literacy in native language Limited oral English Education: 0-3 years Languages: Haitian-Creole Spanish Burmese Others 11

12. Students in the Study: Non-Roman Alphabet Literate Some Literacy in native language with non-Roman script Mean education: 6 and more years Limited oral English Languages: Armenian Arabic Farsi Chinese Others 12

13. Student Test Results 13

14. Main Impact Analyses OutcomeTreatmentControlDifferenceEffect SizeP-value WJID 440.81442.09-1.28 -0.0250.573 WJWA 466.70465.760.94 0.0240.595 SARA Dec 13.2513.37-0.12 -0.0110.809 WJPC 432.83433.52-0.70 -0.0380.295 OWLS 17.8017.790.02 0.0020.974 ROWPVT 28.5329.55-1.02 -0.0610.111 WJPVT 431.65431.240.41 0.0200.663 Sample size: 1137 students 14 No significant impacts on literacy or language outcomes for full sample (below) and subsamples But all students gained on pre-post tests

15. Pre-post Differences, All Students OutcomeMean Pretest Mean Posttest DifferenceP-value WJID 428.315442.12213.8080.000* WJWA 457.400466.5039.1030.000* WJPC 427.061433.7806.7190.000* OWLS 14.23918.0753.8360.000* ROWPVT 22.89829.2856.3870.000* N=1,113 Sam & Pat=567 Control=546 15

16. Lack of Impact of Sam and Pat The study took place in a challenging environment that makes having/finding an impact difficult, but that represents reality Low exposure and instructional time in adult ESL class Short class duration Limited instructional time and irregular attendance Training teachers Lack of specialized training in literacy Short training time available 16

17. Instruction: What is taught? 17

18. Classroom Observations Literacy development instruction: Pre-literacy Phonics Fluency Reading strategies & comprehension Writing ESL Instruction: Oral language development Grammar, etc. Vocabulary Socio-cultural knowledge Functional literacy 18

19. Instructional Content by Interval (%) Content Sam and Pat Mean Control MeanDifferenceP-value Reading Development 65.519.346.30.00 ESL Acquisition 27.367.6-40.20.00 Other Content 5.528.8-23.30.00 Link to Outside 1.37.4-6.10.071 Use of NL 20.743.9-23.20.00 Sample Size Intervals Observations 980 31 1034 33 19

20. Instruction in Literacy Classes In control (“normal”) classes, little literacy taught- Why? Importance of oral language for daily life Student expressed needs Lack of teaching on how to teach LESLLA students More diversity of instruction in control classes More NL use and bringing in outside Less constrained by curriculum and study needs 20

21. Attendance 21

22. Attendance and Outcomes OutcomeRegression Coefficient P=value WJID 0.1040.00 WJWA 0.0710.00 WJPC 0.0430.00 OWLS 0.0270.00 ROWPVT 0.0280.00 WJPVT 0.0560.00 Mean attendance hours (p= n.s.) Sam & Pat : 79.4Control: 71.9 N=1,137, Sam & Pat=587, Control=557 22

23. Effect of Attendance on Test Scores Instruction and outcomes correlated Instruction appears to have an effect on learning Relationship is weak Larger effect on reading outcomes May mean literacy gains more sensitive to gain, may be testing artifact 23

24. Implications of Possible Findings for Practice and Research for LESLLA Students 24

25. Implications for LESLLA Practice Sam & Pat is no more effective than other types of ESL literacy instruction Under what conditions, then, would Sam & Pat be a good choice for instruction? Attendance relates to instruction, weakly Is this what you expect? How could it be increased? Teachers of literacy classes teach little literacy How can we improve this --through curriculum and teacher professional development? What are some potential explanations for these results from your perspective? 25

26. Thank you! Contacts lcondelli@air.org scronen@air.org Enjoy Koeln and LESLLA 26