Presentation is loading. Please wait.

Presentation is loading. Please wait.

Measuring Large-Scale Program Impact on K-12 Mathematics and Science Learning Norman L. Webb Robert H. Meyer Paula A. White Adding Value to the Mathematics.

Similar presentations


Presentation on theme: "Measuring Large-Scale Program Impact on K-12 Mathematics and Science Learning Norman L. Webb Robert H. Meyer Paula A. White Adding Value to the Mathematics."— Presentation transcript:

1 Measuring Large-Scale Program Impact on K-12 Mathematics and Science Learning Norman L. Webb Robert H. Meyer Paula A. White Adding Value to the Mathematics and Science Partnership Evaluations Wisconsin Center for Education Research University of Wisconsin A Seminar at the University of California-Irvine October 3, 2003

2

3 Historical Context: Curriculum New Math and Science Era—Curriculum Development Back to the Basic and Withdrawal Assessment 1972California Assessment Program statewide committee (3,6,12) 1983Nation At Risk states with state assessments Accountability ERA—competency testing, state assessments Standards 1989NCTM Standards, Education Summit, National Goals 1991Smith and O’Day, NSF State Systemic Initiatives states developing challenging standards SSIs, 16 USIs, 6 RSIs 2001No Child Left Behind Act states with standards and assessments

4 National Longitudinal Study of Mathematical Abilities (NLSMA) N=100,000 Students Grade Levels X PopX X X X X Y Pop X X X X X Z Pop X X X

5 General Components of an Education System Management ProgramPolicy Student Outcomes

6 System Attributes  Enabling  Target  Explanatory

7 Attributes Enabling Capacity Sustainability Target Alignment Saturation Quality Equity Linkages Explanatory Trade-offs Incentives

8 Coordinated Linked Capacities Lead to System Fitness for Change

9 System ComponentElements of Components 1. Pre-SSI PolicyStandards/Frameworks Professional Development Assessment Accountability Other Instructional Guidance Policies 2. Pre-SSI InfrastructureNetworks of Policymakers Professional Developers School Administrators School Restructuring Organizations K-12 Teachers of Mathematics and Science 3. SSI Leadership and Change Strategy Educational Change Strategy Organizational Change Strategy 4. SSI PolicyStandards/Frameworks Professional Development Assessment Accountability Other Instructional Guidance Policies Components and Elements of Systemic Reform

10 System ComponentElements of Components 5. SSI InfrastructureNetworks of Policymakers Professional Developers School Administrators School Restructuring Organizations K-12 Teachers of Mathematics and Science 6. SSI Standards-Based Instructional Reform Individual Capacity Building Organizational Capacity Building Classroom Practice 7. Student OutcomesSignificant Statewide Gain Substantial Statewide Gain Standards-Based Gain Control Group Gain Gap Closing Components and Elements of Systemic Reform (Continued)

11 Operationalizing Elements of Systemic Reform (Continued) Example 1: SSI Policy (and Pre-SSI Policy) Component ElementBreadth (1 to 5)Depth (Low = 1: High = 5) Prior to the SSI, state policies were in place for mathematics/science professional development for 1 = one relevant group of actors (e.g., school administrators; elem., middle, or high school in-service teachers of math/science; pre- service teachers of math/science) 3 = three of above groups 5 = all above groups State policy on mathematics/science professional development is 1 = minimally 3 = substantially 5 = highly aligned to state curriculum standards/frameworks prescriptive (i.e., specific about what teachers need to know and be able to do to successfully engage all students in standards-based learning, and specific about professional development practices that engage teachers in necessary learning) authorized (e.g., by professional organizations for K-16 mathematics/science, SEA) supported by powerful incentives and sanctions (i.e., participating in or ignoring professional development has significant direct, concrete consequences for teachers, administrators, schools, or districts) Professional Development

12

13 Pre - SSISSI Figure 10. Breadth and depth assessment for selected SSI components: Arkansas.

14 Pre - SSISSI Figure 11. Breadth and depth assessment for selected SSI components: Connecticut.

15 Pre - SSISSI Figure 12. Breadth and depth assessment for selected SSI components: Louisiana.

16 Focus of State SSI and Statewide Achievement Gains from 1992 to 2000 State System and Infrastructure Balanced Close to the Classroom Steady Increase Michigan Texas Louisiana Massachusetts Kentucky New York Some Increase Connecticut Georgia South Carolina Arkansas Little/No ChangeMaine California Nebraska New Mexico

17 Basic Evaluation Model for Judging the Impact of a MSP Δ Achievement = f (MSP) + ε

18 SCALE Evaluation Indicator System Context Project SCALE DistrictSchool Input Teacher Student Disposition To Learn Opportunity To Learn Student Achievement CapacityAction Student Outcomes Student Participation University

19 Number of Hispanic Students Tested by Cohort and Year

20 Number of White Students Tested by Cohort and Year

21

22 Effectiveness of the Urban Systemic Initiative by Grade and Year for the Rasch Scale and EV1 Model

23 From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between White and Black Students Rasch Scale (EV1 Estimates)

24 From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between White and Hispanic Students Rasch Scale (EV1 Estimates)

25 From 1994 to 2000 the Gap in Annual Mathematics Growth as Measured by TAAS Varied by Grade and Year Between Advantaged and Disadvantaged Students (EV1 Estimates)

26 Evaluation Design  Indicator System  Experimental Design  Quasi-experimental Design  Qualitative Analysis  Focus Study

27 Experimental Design  Control Group  Random Assignment  Post Measure  Sufficient N for Power Requirements

28 Quasi-Experimental Design  Comparison Group  Pre-Post Measures  Replications

29 Independent Variables  Teacher Knowledge  Teacher Practice  Curriculum  Student Activities  Support  Resources  Professional Development

30 SCALE Evaluation Indicator System Context Project SCALE DistrictSchool Input Teacher Student Disposition To Learn Opportunity To Learn Student Achievement CapacityAction Student Outcomes Student Participation University


Download ppt "Measuring Large-Scale Program Impact on K-12 Mathematics and Science Learning Norman L. Webb Robert H. Meyer Paula A. White Adding Value to the Mathematics."

Similar presentations


Ads by Google