Presentation on theme: "Measuring Large-Scale Program Impact on K-12 Mathematics and Science Learning Norman L. Webb Robert H. Meyer Paula A. White Adding Value to the Mathematics."— Presentation transcript:
Measuring Large-Scale Program Impact on K-12 Mathematics and Science Learning Norman L. Webb Robert H. Meyer Paula A. White Adding Value to the Mathematics and Science Partnership Evaluations Wisconsin Center for Education Research University of Wisconsin A Seminar at the University of California-Irvine October 3, 2003
Historical Context: Curriculum New Math and Science Era—Curriculum Development Back to the Basic and Withdrawal Assessment 1972California Assessment Program statewide committee (3,6,12) 1983Nation At Risk states with state assessments Accountability ERA—competency testing, state assessments Standards 1989NCTM Standards, Education Summit, National Goals 1991Smith and O’Day, NSF State Systemic Initiatives states developing challenging standards SSIs, 16 USIs, 6 RSIs 2001No Child Left Behind Act states with standards and assessments
National Longitudinal Study of Mathematical Abilities (NLSMA) N=100,000 Students Grade Levels X PopX X X X X Y Pop X X X X X Z Pop X X X
General Components of an Education System Management ProgramPolicy Student Outcomes
System Attributes Enabling Target Explanatory
Coordinated Linked Capacities Lead to System Fitness for Change
System ComponentElements of Components 1. Pre-SSI PolicyStandards/Frameworks Professional Development Assessment Accountability Other Instructional Guidance Policies 2. Pre-SSI InfrastructureNetworks of Policymakers Professional Developers School Administrators School Restructuring Organizations K-12 Teachers of Mathematics and Science 3. SSI Leadership and Change Strategy Educational Change Strategy Organizational Change Strategy 4. SSI PolicyStandards/Frameworks Professional Development Assessment Accountability Other Instructional Guidance Policies Components and Elements of Systemic Reform
System ComponentElements of Components 5. SSI InfrastructureNetworks of Policymakers Professional Developers School Administrators School Restructuring Organizations K-12 Teachers of Mathematics and Science 6. SSI Standards-Based Instructional Reform Individual Capacity Building Organizational Capacity Building Classroom Practice 7. Student OutcomesSignificant Statewide Gain Substantial Statewide Gain Standards-Based Gain Control Group Gain Gap Closing Components and Elements of Systemic Reform (Continued)
Operationalizing Elements of Systemic Reform (Continued) Example 1: SSI Policy (and Pre-SSI Policy) Component ElementBreadth (1 to 5)Depth (Low = 1: High = 5) Prior to the SSI, state policies were in place for mathematics/science professional development for 1 = one relevant group of actors (e.g., school administrators; elem., middle, or high school in-service teachers of math/science; pre- service teachers of math/science) 3 = three of above groups 5 = all above groups State policy on mathematics/science professional development is 1 = minimally 3 = substantially 5 = highly aligned to state curriculum standards/frameworks prescriptive (i.e., specific about what teachers need to know and be able to do to successfully engage all students in standards-based learning, and specific about professional development practices that engage teachers in necessary learning) authorized (e.g., by professional organizations for K-16 mathematics/science, SEA) supported by powerful incentives and sanctions (i.e., participating in or ignoring professional development has significant direct, concrete consequences for teachers, administrators, schools, or districts) Professional Development
Pre - SSISSI Figure 10. Breadth and depth assessment for selected SSI components: Arkansas.
Pre - SSISSI Figure 11. Breadth and depth assessment for selected SSI components: Connecticut.
Pre - SSISSI Figure 12. Breadth and depth assessment for selected SSI components: Louisiana.
Focus of State SSI and Statewide Achievement Gains from 1992 to 2000 State System and Infrastructure Balanced Close to the Classroom Steady Increase Michigan Texas Louisiana Massachusetts Kentucky New York Some Increase Connecticut Georgia South Carolina Arkansas Little/No ChangeMaine California Nebraska New Mexico
Basic Evaluation Model for Judging the Impact of a MSP Δ Achievement = f (MSP) + ε
SCALE Evaluation Indicator System Context Project SCALE DistrictSchool Input Teacher Student Disposition To Learn Opportunity To Learn Student Achievement CapacityAction Student Outcomes Student Participation University
Number of Hispanic Students Tested by Cohort and Year
Number of White Students Tested by Cohort and Year
Effectiveness of the Urban Systemic Initiative by Grade and Year for the Rasch Scale and EV1 Model
From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between White and Black Students Rasch Scale (EV1 Estimates)
From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between White and Hispanic Students Rasch Scale (EV1 Estimates)
From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between Advantaged and Disadvantaged Students (EV1 Estimates)
Evaluation Design Indicator System Experimental Design Quasi-experimental Design Qualitative Analysis Focus Study
Experimental Design Control Group Random Assignment Post Measure Sufficient N for Power Requirements