1. Measuring Large-Scale Program Impact on K-12 Mathematics and Science Learning Norman L. Webb Robert H. Meyer Paula A. White Adding Value to the Mathematics and Science Partnership Evaluations Wisconsin Center for Education Research University of Wisconsin A Seminar at the University of California-Irvine October 3, 2003

3. Historical Context: Curriculum 1957-1972New Math and Science Era—Curriculum Development 1972-1983Back to the Basic and Withdrawal Assessment 1972California Assessment Program statewide committee (3,6,12) 1983Nation At Risk 198638 states with state assessments 1978-1989Accountability ERA—competency testing, state assessments Standards 1989NCTM Standards, Education Summit, National Goals 1991Smith and O’Day, NSF State Systemic Initiatives 199445 states developing challenging standards 199626 SSIs, 16 USIs, 6 RSIs 2001No Child Left Behind Act 200349 states with standards and assessments

4. National Longitudinal Study of Mathematical Abilities (NLSMA) 1962-1967 N=100,000 Students Grade Levels 4 5 6 7 8 9 10 11 12 X PopX X X X X Y Pop X X X X X Z Pop X X X

5. General Components of an Education System Management ProgramPolicy Student Outcomes

6. System Attributes  Enabling  Target  Explanatory

7. Attributes Enabling Capacity Sustainability Target Alignment Saturation Quality Equity Linkages Explanatory Trade-offs Incentives

8. Coordinated Linked Capacities Lead to System Fitness for Change

9. System ComponentElements of Components 1. Pre-SSI PolicyStandards/Frameworks Professional Development Assessment Accountability Other Instructional Guidance Policies 2. Pre-SSI InfrastructureNetworks of Policymakers Professional Developers School Administrators School Restructuring Organizations K-12 Teachers of Mathematics and Science 3. SSI Leadership and Change Strategy Educational Change Strategy Organizational Change Strategy 4. SSI PolicyStandards/Frameworks Professional Development Assessment Accountability Other Instructional Guidance Policies Components and Elements of Systemic Reform

10. System ComponentElements of Components 5. SSI InfrastructureNetworks of Policymakers Professional Developers School Administrators School Restructuring Organizations K-12 Teachers of Mathematics and Science 6. SSI Standards-Based Instructional Reform Individual Capacity Building Organizational Capacity Building Classroom Practice 7. Student OutcomesSignificant Statewide Gain Substantial Statewide Gain Standards-Based Gain Control Group Gain Gap Closing Components and Elements of Systemic Reform (Continued)

11. Operationalizing Elements of Systemic Reform (Continued) Example 1: SSI Policy (and Pre-SSI Policy) Component ElementBreadth (1 to 5)Depth (Low = 1: High = 5) Prior to the SSI, state policies were in place for mathematics/science professional development for 1 = one relevant group of actors (e.g., school administrators; elem., middle, or high school in-service teachers of math/science; pre- service teachers of math/science) 3 = three of above groups 5 = all above groups State policy on mathematics/science professional development is 1 = minimally 3 = substantially 5 = highly aligned to state curriculum standards/frameworks prescriptive (i.e., specific about what teachers need to know and be able to do to successfully engage all students in standards-based learning, and specific about professional development practices that engage teachers in necessary learning) authorized (e.g., by professional organizations for K-16 mathematics/science, SEA) supported by powerful incentives and sanctions (i.e., participating in or ignoring professional development has significant direct, concrete consequences for teachers, administrators, schools, or districts) Professional Development

13. Pre - SSISSI Figure 10. Breadth and depth assessment for selected SSI components: Arkansas.

14. Pre - SSISSI Figure 11. Breadth and depth assessment for selected SSI components: Connecticut.

15. Pre - SSISSI Figure 12. Breadth and depth assessment for selected SSI components: Louisiana.

16. Focus of State SSI and Statewide Achievement Gains from 1992 to 2000 State System and Infrastructure Balanced Close to the Classroom Steady Increase Michigan Texas Louisiana Massachusetts Kentucky New York Some Increase Connecticut Georgia South Carolina Arkansas Little/No ChangeMaine California Nebraska New Mexico

17. Basic Evaluation Model for Judging the Impact of a MSP Δ Achievement = f (MSP) + ε

18. SCALE Evaluation Indicator System Context Project SCALE DistrictSchool Input Teacher Student Disposition To Learn Opportunity To Learn Student Achievement CapacityAction Student Outcomes Student Participation University

19. Number of Hispanic Students Tested by Cohort and Year

20. Number of White Students Tested by Cohort and Year

22. Effectiveness of the Urban Systemic Initiative by Grade and Year for the Rasch Scale and EV1 Model

23. From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between White and Black Students Rasch Scale (EV1 Estimates)

24. From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between White and Hispanic Students Rasch Scale (EV1 Estimates)

25. From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between Advantaged and Disadvantaged Students (EV1 Estimates)

26. Evaluation Design  Indicator System  Experimental Design  Quasi-experimental Design  Qualitative Analysis  Focus Study

27. Experimental Design  Control Group  Random Assignment  Post Measure  Sufficient N for Power Requirements

28. Quasi-Experimental Design  Comparison Group  Pre-Post Measures  Replications

29. Independent Variables  Teacher Knowledge  Teacher Practice  Curriculum  Student Activities  Support  Resources  Professional Development

30. SCALE Evaluation Indicator System Context Project SCALE DistrictSchool Input Teacher Student Disposition To Learn Opportunity To Learn Student Achievement CapacityAction Student Outcomes Student Participation University