Measuring Large-Scale Program Impact on K-12 Mathematics and Science Learning Norman L. Webb Robert H. Meyer Paula A. White Adding Value to the Mathematics and Science Partnership Evaluations Wisconsin Center for Education Research University of Wisconsin A Seminar at the University of California-Irvine October 3, 2003

Historical Context: Curriculum New Math and Science Era—Curriculum Development Back to the Basic and Withdrawal Assessment 1972California Assessment Program statewide committee (3,6,12) 1983Nation At Risk states with state assessments Accountability ERA—competency testing, state assessments Standards 1989NCTM Standards, Education Summit, National Goals 1991Smith and O’Day, NSF State Systemic Initiatives states developing challenging standards SSIs, 16 USIs, 6 RSIs 2001No Child Left Behind Act states with standards and assessments

National Longitudinal Study of Mathematical Abilities (NLSMA) N=100,000 Students Grade Levels X PopX X X X X Y Pop X X X X X Z Pop X X X

General Components of an Education System Management ProgramPolicy Student Outcomes

System Attributes  Enabling  Target  Explanatory

Attributes Enabling Capacity Sustainability Target Alignment Saturation Quality Equity Linkages Explanatory Trade-offs Incentives

Coordinated Linked Capacities Lead to System Fitness for Change

System ComponentElements of Components 1. Pre-SSI PolicyStandards/Frameworks Professional Development Assessment Accountability Other Instructional Guidance Policies 2. Pre-SSI InfrastructureNetworks of Policymakers Professional Developers School Administrators School Restructuring Organizations K-12 Teachers of Mathematics and Science 3. SSI Leadership and Change Strategy Educational Change Strategy Organizational Change Strategy 4. SSI PolicyStandards/Frameworks Professional Development Assessment Accountability Other Instructional Guidance Policies Components and Elements of Systemic Reform

System ComponentElements of Components 5. SSI InfrastructureNetworks of Policymakers Professional Developers School Administrators School Restructuring Organizations K-12 Teachers of Mathematics and Science 6. SSI Standards-Based Instructional Reform Individual Capacity Building Organizational Capacity Building Classroom Practice 7. Student OutcomesSignificant Statewide Gain Substantial Statewide Gain Standards-Based Gain Control Group Gain Gap Closing Components and Elements of Systemic Reform (Continued)

Operationalizing Elements of Systemic Reform (Continued) Example 1: SSI Policy (and Pre-SSI Policy) Component ElementBreadth (1 to 5)Depth (Low = 1: High = 5) Prior to the SSI, state policies were in place for mathematics/science professional development for 1 = one relevant group of actors (e.g., school administrators; elem., middle, or high school in-service teachers of math/science; pre- service teachers of math/science) 3 = three of above groups 5 = all above groups State policy on mathematics/science professional development is 1 = minimally 3 = substantially 5 = highly aligned to state curriculum standards/frameworks prescriptive (i.e., specific about what teachers need to know and be able to do to successfully engage all students in standards-based learning, and specific about professional development practices that engage teachers in necessary learning) authorized (e.g., by professional organizations for K-16 mathematics/science, SEA) supported by powerful incentives and sanctions (i.e., participating in or ignoring professional development has significant direct, concrete consequences for teachers, administrators, schools, or districts) Professional Development

Pre - SSISSI Figure 10. Breadth and depth assessment for selected SSI components: Arkansas.

Pre - SSISSI Figure 11. Breadth and depth assessment for selected SSI components: Connecticut.

Pre - SSISSI Figure 12. Breadth and depth assessment for selected SSI components: Louisiana.

Focus of State SSI and Statewide Achievement Gains from 1992 to 2000 State System and Infrastructure Balanced Close to the Classroom Steady Increase Michigan Texas Louisiana Massachusetts Kentucky New York Some Increase Connecticut Georgia South Carolina Arkansas Little/No ChangeMaine California Nebraska New Mexico

Basic Evaluation Model for Judging the Impact of a MSP Δ Achievement = f (MSP) + ε

SCALE Evaluation Indicator System Context Project SCALE DistrictSchool Input Teacher Student Disposition To Learn Opportunity To Learn Student Achievement CapacityAction Student Outcomes Student Participation University

Number of Hispanic Students Tested by Cohort and Year

Number of White Students Tested by Cohort and Year

Effectiveness of the Urban Systemic Initiative by Grade and Year for the Rasch Scale and EV1 Model

From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between White and Black Students Rasch Scale (EV1 Estimates)

From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between White and Hispanic Students Rasch Scale (EV1 Estimates)

From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between Advantaged and Disadvantaged Students (EV1 Estimates)

Evaluation Design  Indicator System  Experimental Design  Quasi-experimental Design  Qualitative Analysis  Focus Study

Experimental Design  Control Group  Random Assignment  Post Measure  Sufficient N for Power Requirements

Quasi-Experimental Design  Comparison Group  Pre-Post Measures  Replications

Independent Variables  Teacher Knowledge  Teacher Practice  Curriculum  Student Activities  Support  Resources  Professional Development

SCALE Evaluation Indicator System Context Project SCALE DistrictSchool Input Teacher Student Disposition To Learn Opportunity To Learn Student Achievement CapacityAction Student Outcomes Student Participation University