1. Different Skills? Identifying Differentially Effective Teachers of English Language Learners Ben Master, Susanna Loeb, Camille Whitney, James Wyckoff 5 rd Annual CALDER Conference January 27 th, 2012

2. English Language Learners More than 5 million public school students (~10%) are currently designated as ELLs Typically take 3-5 years to attain oral English proficiency Low overall achievement. e.g. 22% in lowest percentile compared with 3% in highest in 2011 math

3. Teaching of ELLs Many schools, districts and states seek to improve the outcomes for ELLs in particular Teachers often provide few accommodations for ELLS Teacher preparation programs often provide little opportunity to learn about teaching ELLs

4. Teachers Clearly important for student achievement Some evidence that effective teaching can be observed Some evidence that effective teachers, on average have differential knowledge (general and specific) Evidence that teachers improve with experience Some evidence that pre-service and in-service programs can improve teaching

5. Motivation Little corresponding evidence on what teacher characteristics or training are differentially or specifically beneficial for English language learners

6. Today’s study 3 Questions 1. Do teacher characteristics that predict growth for non-ELLs also predict growth ELLs?

7. own test performanceteaching experience

8. Today’s study 3 Questions 1. Do teacher characteristics that predict growth for non-ELLs also predict growth ELLs? 2. Do experiences related to teaching ELLs differentially predict effectiveness with ELLs?

9. teaching experience with ELLs pre-service preparation addressing skills for teaching ELLs in-service preparation addressing skills for teaching ELLs ESL Certification

10. Today’s study 3 Questions 1. Do teacher characteristics that predict growth for non-ELLs also predict growth ELLs? 2. Do experiences related to teaching ELLs differentially predict effectiveness with ELLs? 3. Do theorized teacher characteristics predict differential effectiveness with ELLs?

11. ability to speak Spanish fluently, either native or learned

12. Today’s study 1. Do teacher characteristics that predict growth for non-ELLs also predict growth for ELLs? 2. Do experiences related to teaching ELLs differentially predict effectiveness with ELLs? 3. Do theorized teacher characteristics predict differential effectiveness with ELLs?

13. New York City (NYC) public school system 2000-2001 through 2007-2008 Administrative data: student exams and student characteristics; teacher background characteristics and certification information; school characteristics Survey of first year teachers in 2005: pre-service and in-service experiences, additional background characteristics Data

14. SOME DESCRIPTIVES

15. Percentage of students who are ELLs and standardized math test scores in New York City, by year Percentage of students who are ELLs11.30% Percentage of ELLs in each grade of study Grade 411.90% Grade 510.60% Grade 610.90% Grade 711.20% Grade 811.60% Standardized math test scores-0.63

16. Race/ethnicity and Free or reduced price lunch, by ELL status ELLsNon-ELLs Race/ethnicity White7%15% Black6%36% Hispanic70%35% Asian18%13% Other race/ethnicity0%1% Free or reduced price lunch76%68% Home language is English0%63%

17. Basic distribution of ELLs across classrooms % ELLs in the class Mean across all classrooms11 Percentiles 25th0 50th3 75th9 90th40 95th94

18. Characteristics of teachers serving ELL and non-ELL students, district-wide % Taught by Teachers with 1 or 2 Years of Experience Initial LAST Scores (std. dev.) Mean across NYC25.2243 (30.2) Mean for ELLs23.7237 (33.8) Mean for non-ELLs25.7246 (29.6)

19. Responses and # of respondents for first year teacher survey questions of interest % YesTotal N 1.Fluency in languages other than English: % that self-identified as fluent in Spanish? 15.71185 2. Pre-service opportunity to learn instructional strategies for teaching ELLs: % that “explored in some depth” or “extensively”? 14.11210 3. By mid-year, # of hours of in-service PD focused on ELL instruction: % reporting “>9,” “>17,” or “>33 hours” so far this year? 13.91202

20. Approach Option 1 Compare the learning of ELLs to non-ELLs with one type of teacher vs. another Sorting of teachers and students into school may bias estimates (type 1 teachers sort to better schools) Sorting of teachers to students within schools may bias estimates (easier students sort to type 1 teachers) not particularly believable

21. Option 2 Compare the learning of ELL students with one type of teacher to others within the same school Sorting of teachers and students into school largely accounted for Sorting of teachers to students within schools may bias estimates (easier students sort to type 1 teachers) Better

22. Option 3 Compare the learning of ELL students to learning of non-ELL students within the same classroom Sorting of teachers and students into school accounted for Sorting of teachers to students within schools taken care Better: but can’t see overall effect, just differential

23. 1. Do teacher characteristics that predict growth for non-ELLs also predict growth for ELLs? teachers' own test performance teachers' teaching experience Results

24. Approach 2 (within school): About the same effect of experience Effect of tests only for non-ELLs ELLsNon-ELLs initially failed LAST 0.012-0.014*** 2 nd year teaching in NYC 0.056***0.057*** 3 rd year teaching in NYC 0.091***0.076*** 4 th year teaching in NYC 0.089***0.093*** 6 th year teaching in NYC 0.086***0.097*** 7 th year teaching in NYC 0.079***0.094***

25. Approach 3 (within teacher): No evident difference ELL versus Non-ELL Achievement Gap failed LAST exam x ELLs0.005 2 nd year x ELLs-0.002 3 rd year x ELLs0.007 4 th year x ELLs-0.001 5 th year x ELLs 0.004 6 th year x ELLs-0.005 7 th year x ELLs-0.005

26. 2. Do experiences related to teaching ELLs differentially predict effectiveness with ELLs? past teaching experience with ELLs pre-service addressing skills for ELL teaching in-service addressing skills for ELL teaching ESL certification

27. Approach 2 (within school): Experience with ELLs substantially more important for ELLs ELLs All2 nd year3+ years >6 ELLs last year0.024**0.069**0.015~ Non-ELLs All2 nd year3+ years >6 ELLs last year0.010*0.0120.008~

28. Approach 3 (within teacher): Similar Results ELL versus Non-ELL Achievement Gap 2nd year0.037 2nd year x ELL-0.012~ >6 ELLs in prior year-0.009 >6 ELLs in prior year x ELLs0.031**

29. Approach 2 (within school): Training substantially more important for ELLs ELLs ELL-specific pre-service0.090*0.093*0.027 ELL-specific in-service 0.003 ELL-specific PD (in ‘05) 0.294*** Non-ELLs ELL-specific pre-service-0.0350.0140.016 ELL-specific in-service-0.120*** ELL-specific PD (in ‘05)-0.112* all differences significant **

30. Approach 3 (within teacher): Similar PreIn (same year) ELL pre-service X ELL0.091*0.042-0.062 ELL in-service X ELL 0.043 ELL PD (in ‘05) X ELL 0.226* all differences significant **

31. Approach 3 (within teacher): Similar all differences significant ** All teachersNovices (<=3yrs) Any ESL Cert X ELL0.0170.080** All teachersNovices (<=3yrs) ELL Any ESL Certification0.057**0.069* Non-ELL Any ESL Certification0.033-0.066~ Approach 2 (within school): ESL Certification more Important for ELLs

32. 3. Do theorized teacher characteristics predict differential effectiveness with ELLs? ability to speak Spanish fluently

33. Approach 2 (within school): No Evidence of Effect of Language Hispanic ELLs All Other Students Native fluency in Spanish 0.0050.018 Non-native fluency in Spanish 0.0310.028 Approach 3 (within teacher): Similar, though positive Hispanic ELLs Native fluency in Spanish X Hispanic ELL 0.023 Non-native in Spanish X Hispanic ELL0.034

34. Summary ELLDifferential Test performancenomaybe Experienceno Experience with ELLsyes Pre-sevice ELLyes In-service ELyes ESL certificationyes Spanish fluency?no

35. Conclusions Just a start Indicates Improvement with both experience and education Broadly - indicates differential effectiveness Blunt measures Imperfect identification Highlights importance of opportunities Highlights importance of matching

36. Different Skills? Identifying Differentially Effective Teachers of English Language Learners Ben Master, Susanna Loeb, Camille Whitney, James Wyckoff 5 rd Annual CALDER Conference January 27 th, 2012