1. Effects of Software Products Findings from the National Evaluation Mark Dynarski Presentation at the 2009 IES Annual Conference Mark Dynarski Presentation at the 2009 IES Annual Conference

2. 2 Study Synopsis Design –Nine reading and six math software products (four grade levels) –132 volunteer schools –Random assignment of volunteer teachers within schools to use products or not: each school is an experiment Implementation –Companies train teachers, provide support –Study purchased upgrades and some hardware components Key Findings –Test scores not statistically different –Most individual products not effective –Few relationships between effects and contextual factors –Experience has mixed effects on effectiveness Design –Nine reading and six math software products (four grade levels) –132 volunteer schools –Random assignment of volunteer teachers within schools to use products or not: each school is an experiment Implementation –Companies train teachers, provide support –Study purchased upgrades and some hardware components Key Findings –Test scores not statistically different –Most individual products not effective –Few relationships between effects and contextual factors –Experience has mixed effects on effectiveness

3. 3 Difference in Technology Use in Treatment and Control Classrooms: First Grade

4. 4 Effect Sizes By School: First Grade

5. 5 Did Net Product Usage (Dose) Matter? First grade: No Fourth grade : YES Sixth Grade: No Algebra: No First grade: No Fourth grade : YES Sixth Grade: No Algebra: No

6. 6 Effects of one year of teacher experience: reading products --First Grade- - --Fourth Grade- - 0.86 First Year Effect -1.28 Second Year Effect 2.65 First Year Effect 4.67 Second Year Effect Effect on Student Test Scores (Normal Curve Equivalent Scores) Neither difference is statistically significant at the 5 percent level.

7. 7 Effects of one year of teacher experience: math products 10 0 -10 --Sixth Grade- - --Algebra I- - -0.44 First Year Effect -3.24 Second Year Effect -0.34 First Year Effect Effect on Student Test Scores 8 6 4 2 -2 -4 -6 -8 2.56 Second Year Effect The difference is statistically significant at the 5 percent level.

8. 8 Dosage: Other Considerations Planned dosage variation studies feasible –Actual dosage related to school and classroom factors that can act as constraints: dosage is only salient within classroom limits Efficacy trials are appropriate setting for dosage questions –Examine dosage within an implementation with some fidelity –Test in large scale effectiveness trial based on model identified in efficacy trial(s) Planned dosage variation studies feasible –Actual dosage related to school and classroom factors that can act as constraints: dosage is only salient within classroom limits Efficacy trials are appropriate setting for dosage questions –Examine dosage within an implementation with some fidelity –Test in large scale effectiveness trial based on model identified in efficacy trial(s)