1. Early Learning in Mathematics (ELM) The Efficacy of a Kindergarten Curriculum Implemented in Whole Classroom Settings Scott K. Baker, PhD Pacific Institutes for Research / University of Oregon Ben Clarke, PhD Pacific Institutes for Research Hank Fien, PhD University of Oregon Keith Smolkowski, PhD Oregon Research Institute Chris Doabler, PhD Pacific Institutes for Research David Chard, PhD Southern Methodist University IES Conference June 2010

2. Acknowledgements Institute of Education Sciences The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Goal 2 development grant, #R305K040081, and a Goal 3 efficacy grant, #R305A080114, to Pacific Institutes for Research. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education. Additional Oregon Project Staff Kathy Jungjohann / Karen Davis: Curriculum development Mari Strand Cary / Rhonda Griffiths: Coordination and research Chris Doabler: Observation measurement and research

3. Early Learning in Mathematics (ELM) 4-year randomized efficacy control trial Measuring the efficacy of a kindergarten mathematics curriculum in Oregon and Texas. Research Design Randomized Controlled Block Design Classrooms within school matched on full / half day and randomly assigned to treatment (ELM) or control conditions 3

4. Purpose of the 4-year project Study 1: Efficacy trial of the whole group curriculum (ELM) on kindergarten students’ mathematics achievement Study 2: Efficacy trial of small group ELM component – Roots – on the achievement of at-risk students Examine potential mediation variables, dose- response variables, and moderation factors

5. Structure of the Curriculum Daily Calendar Lessons / Activities – 15 minutes daily, whole class “circle” time – Monthly booklets with objectives and application activities 120 Core Lessons divided into 4 quarters – 30 minutes whole class instruction – 15 minutes teacher directed written work – End of quarter assessment of progress

6. ELM Instructional Content National Math Advisory Panel (2008) recommends a focused, coherent progression of mathematics learning with emphasis on proficiency with key topics ELM focuses on key strands rather than a broad array of mathematical content – Numbers and Operations – Geometry – Measurement – Vocabulary (NCTM Process Standard, 2000) NCTM Curriculum Focal Points for K (2006)

7. ELM Conceptual Framework Development of Mathematical Concepts / Models Mathematics-related Vocabulary and Discourse Mathematics-related Vocabulary and Discourse Procedural Fluency

8. Research questions What is the immediate impact of ELM taught in general education kindergarten classrooms on mathematics achievement compared to standard district practice? Is impact moderated by student level of risk for mathematics difficulties? Does rate of teacher models or student practice opportunities mediate a condition effect? Is there evidence of an interaction between condition and student practice on student outcomes?

9. Study Sample Assignment at the classroom level blocked on school – Districts = 3 – Schools = 24 Intervention classrooms = 34 Control classrooms = 30 Students nested in classrooms – Whole class instruction – N = 1,349

10. Student Demographics 56.3% eligible for free or reduced lunch 38.4% English Learners 8.4% special education 49.5% White 36.4% Latino 4.8% Asian / Pacific Islander 2.3% African American

11. ELM Math Instruction Instruction Quality and Math Content Models of Math Representations Procedural Fluency Student Math Achievement Math Vocabulary InterventionMediatorsProximal Outcomes Distal Outcomes Hypothesized Model of ELM Impact

12. Descriptive data on implementation Classroom fidelity observations – treatment and control Classroom observations focusing on instructional interactions – treatment and control Teacher logs addressing content coverage – treatment and control classrooms

13. Implementation Fidelity General ratings (8 items) – Models skills/concepts appropriately and with ease – Engages students in learning throughout the lesson Uses ELM / completes all lesson activities (dichotomous) For each ELM activity (range 1-7 per lesson): Full (2) / Partial (1) / Not Taught (0)

14. Implementation Fidelity Data: ELM Lesson Activities 81 ELM (fidelity) observations during the year Fall:mean = 1.71 (SD =.19) Winter: mean = 1.65 (SD =.33) Spring: mean = 1.62 (SD =.43) Overall:mean = 1.65 (SD =.36) (83% of Full) 2 of 81 lessons had a mean below Partial (1) level of implementation

15. Student measures of impact Test of Early Mathematics Ability (TEMA) Early Numeracy – CBM – Oral Counting – Number Identification – Quantity Discrimination – Missing Number

16. Method and Analysis Framework Competing curricula – All students received instruction – Time balanced across conditions Sample – At risk (some or high risk) < 40 th percentile 66% of student sample – No risk ≥ 40 th percentile 34% of student sample

17. Nested Time × Condition Analysis Outcome: net differences from pre to post Nested students within classrooms – Control for nonindependence (e.g., ICCs) – Controls for teacher effects Maximum likelihood (restricted) – Includes all cases with data at either T1 or T2 – Reduces bias from missing data Moderation added Time × Risk × Condition interaction Effect sizes: Hedges’ g

18. Sample Means, SDs, and Ns T No RiskSome Risk ELMControlELMControl TEMA RawT129.0 (6.8)28.8 (7.3)14.1 (6.0)14.6 (6.2) T239.6 (6.3)39.1 (7.9)28.6 (8.4)26.9 (8.0) TEMA PercentileT166.5 (16.4)65.1 (17.4)14.5 (11.4)15.4 (11.7) T270.4 (18.3)68.1 (20.8)35.5 (23.6)31.0 (22.4) CBM TotalT1120.3 (44.0)116.7 (43.5)45.0 (32.6)45.5 (35.5) T2193.1 (35.8)187.9 (38.1)138.4 (50.3)126.6 (50.5) Sample SizeT1203181397343 T2190174341312 Note. Standard deviations (SDs) presented in parentheses. For students with some risk at T1, we collected TEMAs from 53 fewer students in ELM classrooms and 48 fewer students in control classrooms.

19. TEMA Percentile Scores Gains by Condition Gains – Control: 10.94 – ELM: 14.73 – Difference: 3.79 Test of Condition – t = 2.10 – df = 61 – p =.0396 – ES = +0.14 T1 differences were not statistically significant (t = 0.57)

20. CBM Total Scores Gains by Condition Gains – Control: 77.20 – ELM: 84.87 – Difference: 7.67 Test of Condition – t = 1.99 – df = 61 – p =.0509 – ES = +0.14 T1 differences were not statistically significant (t = 0.60)

21. TEMA Raw Scores Condition by Risk Status Main Effects – Difference in gains: 1.32 – t = 2.41, df = 61, p =.0190 Condition by Risk Status – t = 2.47, df = 61, p =.0162 – No Risk ≥ 40 th %tile Difference in gains: 0.04 t = 0.51, df = 61, p =.9586 – Risk < 40 th %tile Difference in gains: 1.98 t = 3.29, df = 61, p =.0017

22. CBM Total Scores Condition by Risk Status Main Effects – Difference in gains: 7.67 – t = 1.99, df = 61, p =.0509 Condition by Risk Status – t = 2.24, df = 61, p =.0289 – No Risk ≥ 40 th %tile Difference in gains: -0.27 t = -0.05, df = 61, p =.9570 – Risk < 40 th %tile Difference in gains: 10.81 t = 2.54, df = 61, p =.0138

23. Effect Sizes (Hedges’ g) MeasureNot At RiskAt Risk Tema Raw Score+0.006+0.242** EN-CBM Total+0.014+0.215* *p <.05; **p <.01

24. Summary ELM classrooms outperformed controls – TEMA raw and percentile scores – EN-CBM Total Students at risk – Improve on all measures more than no-risk students – Control students at risk catching up on no-risk students TEMA: 14.0 percentile gain on no-risk students EN-CBM: 9.6 point gain on no-risk students – ELM students at risk catching no-risk students faster TEMA: 18.6 percentile gain on no-risk students EN-CBM: 20.63 point gain on no-risk students No condition effects for students with no risk

25. ELM Math Instruction Instruction Quality and Math Content Models of Math Representations Procedural Fluency Student Math Achievement Math Vocabulary InterventionMediatorsProximal Outcomes Distal Outcomes Preliminary Analysis of Association between Observation Data and Student Outcomes

26. Coding of Academic Teacher-Student Interactions (CATS) Observation Instrument CATS uses a frequency count approach to measure teacher-student instructional interactions Observers code behavior occurrences in a continual, serial fashion. Utilizes a strict coding structure CATS based on evidence of effective instruction in early literacy and beginning mathematics, and adapted from the STICO observation instrument (Smolkowski & Gunn, 2010) 26 STUDENT BEHAVIORSTEACHER BEHAVIORS Individual responses Group responses Covert responses Mistakes Teacher models Academic feedback

27. Hypothetical Case of a Instructional Interaction

29. The Role of Teacher Modeling and Student Practice in Student Outcomes

30. Preliminary Mediation Analysis Does rate of teacher models or student practice opportunities mediate condition effect? Rates of (a) teacher models, (b) student group practice opportunity and (c) individual student practice opportunities entered as mediators to determine if they decreased condition effect – Condition effect was still significant – No evidence to support this mediation hypothesis

31. Secondary Analysis If student practice overall is not mediating impact, perhaps the value (quality) of practice differs by classroom and is related to condition

32. Interaction between Rate of Practice and Condition By condition do rates of practice opportunities show the same pattern of impact on student outcomes? Are treatment – control differences on student outcomes similar in classrooms with high rates of practice vs. low rates of practice?

33. Interaction between Rate of Practice and Condition Number of Classrooms by Treatment Condition and Median Rate of Individual and Group Practice Opportunities Practice Opportunity Quartiles in Rate per Minute ControlELMTotal Above Median 62632 Below Median 24832 Total303464 ControlELMTotal Minimum0.41.00.4 25 th %ile0.81.91.2 Median1.32.31.9 75 th %ile1.73.02.4 Maximum4.1

34. TEMA Scores by Rate of Practice Within ELM condition – High-practice classrooms outperform low-practice classrooms – Difference = 2.59, t = 2.58, df = 29, p = 0.0151 Within control condition – No difference between high- and low-practice classrooms – Difference = 0.39, t = 0.31, df = 29, p =.7575 Within classrooms with an above-median practice rate – ELM classrooms (might) outperform control classrooms – Difference = 2.33, t = 1.85, df = 29, p =.0747 Within classrooms with a below-median practice rate – No difference between ELM and control classrooms – Difference = 0.14, t = 0.14, df = 29, p =.8935

35. EN-CBM Scores by Rate of Practice Within ELM condition – No difference between high- and low-practice classrooms – Difference = 9.83, t = 1.46, df = 29, p = 0.1538 Within control condition – No difference between high- and low-practice classrooms – Difference = -7.59, t = -0.92, df = 29, p =.3635 Within classrooms with an above-median practice rate – ELM classrooms outperform control classrooms – Difference = 19.37, t = 2.35, df = 29, p =.0257 Within classrooms with a below-median practice rate – No difference between ELM and control classrooms – Difference = 1.95, t = 0.29, df = 29, p =.7731

36. Next steps Have just completed implementation of Study 1 in Dallas, Texas Have just completed implementation of Study 2 in Oregon Will implement Study 2 in Dallas, Texas in 2010-11 Ongoing analysis to investigate impact of condition and mediation and moderation variables associated with impact