1. Evaluation Requirements for MSP and Characteristics of Designs to Estimate Impacts with Confidence
Ellen Bobronnikov
March 23, 2011

2. Evaluation is a requirement of all ED MSPs
According to the statute:
Each MSP project is required to develop an evaluation plan that includes rigorous objectives that measure the impact of activities.
However, the type of evaluation design is not specified. The MSP Program only requires projects to report on two aspects of evaluation findings:
Teacher gains in content knowledge based on pre- and post-testing; and
Proficiency levels on state-level assessments of students of teachers who received professional development.
Rigorous evaluations demonstrating that impacts are caused by the program are encouraged, but they may not be appropriate for small projects in their early stages. If your project prepares an evaluation report, this should be attached to the APR.

3. According to the 2007 Report of the Academic Competitive Council
“Successful, large-scale interventions to improve STEM education are unlikely to arise without serious study and trial and error. There is a critical pathway for the development of successful educational interventions and activities, starting generally with small-scale studies to test new ideas and generate hypotheses, leading to increasingly larger and more rigorous studies to test the effect of a given intervention or activity on a variety of students in a variety of settings. Different research methodologies are used along the development pathway, and corresponding evaluation strategies must be used to assess their progress.”

4. Hierarchy of Study Designs for Evaluating Effectiveness

5. Overview
Criteria for Classifying Designs of MSP Evaluations (“the rubric”) created through the Data Quality Initiative (DQI)
Rubric’s key criteria for a rigorous design
Common issues with evaluation reports
Recommendations for better reporting
Preliminary framework for review standards for all evaluation reports

6. Evaluations Reviewed Using the Rubric
All final year evaluations that report using an experimental or quasi-experimental design are considered for review
Evaluations need to include a comparison group to ultimately be reviewed with the rubric
Within each project, we review evaluations of teacher content knowledge, classroom practices, and student achievement

7. 6 criteria used in rubric
Rubric comprises six criterion:
Equivalence of groups at baseline
Adequate sample size
Use of valid & reliable measurement instruments
Use of consistent data collection methods
Sufficient response and retention rates
Reporting of relevant statistics

8. Criterion 1 – Baseline Equivalence
Requirement
Study demonstrates no significant differences in key characteristics between treatment and comparison groups at baseline (for the analytic sample) OR
Adequate steps were taken to address the lack of baseline equivalence in the statistical analysis
Purpose – Helps rule out alternative explanations for differences between groups

9. Criterion 2 – Sample Size
Requirement
Sample size is adequate to detect a difference, based on a power analysis using:
Significance level = 0.05,
Power = 0.8
Minimum detectable effect informed by the literature or otherwise justified
Alternatively, meets or exceeds “rule of thumb” sample sizes:
School/district-level interventions: 12 schools
Teacher-level interventions: 60 teachers (teacher outcomes) or 18 teachers (student outcomes)
Purpose – Increases the likelihood of finding an impact

10. Criterion 3 – Measurement Instruments
Requirement – Data collection instruments used were shown to be valid and reliable to measure key outcomes
Use existing instruments that have already been deemed valid and reliable Refer to TCK instrument database developed by MSP Knowledge Management and Dissemination Project at OR
Create new instruments that have either been:
Sufficiently tested with subjects comparable to the study sample and found to be valid and reliable, OR
Created using scales and items from pre-existing data collection instruments that have been validated and found to be reliable
Resulting instrument needs to include at least 10 items, and at least 70 percent of the items are from the validated and reliable instrument(s)
Purpose – Ensure that instruments used accurately capture the intended outcomes

11. Criterion 4 – Data Collection Methods
Requirement - Methods, procedures, and timeframes used to collect the key outcome data from treatment and comparison groups are comparable
Purpose – Limits possibility that observed differences can be attributed to factors besides the program, such as passage of time and differences in testing conditions

12. Criterion 5 – Attrition Requirement
Need to measure key outcomes for at least 70% of original sample (both treatment and control groups), or evidence that attrition is unrelated to treatment
If the attrition rates between groups equal or exceed 15 percentage points, difference should be accounted for in the statistical analysis
Purpose – Helps ensure that sample attrition does not bias results as participants/control group members who drop out may systematically differ from those who remain

13. Criterion 6 – Relevant Statistics Reported
Requirement
Include treatment and comparison group post-test means and tests of significance for key outcomes OR,
Provide sufficient information for calculation of statistical significance (e.g., mean, sample size, standard deviation/standard error)
Purpose – Provides context for interpreting results, indicating where observed differences between groups are most likely larger than what chance alone might cause

14. Common Issues Found in Evaluation Reports
Information critical for complete assessment of all criteria is often not reported, inconsistently reported, or only reported for the treatment group
Pre & post sample sizes for both groups, means, standard deviations/ errors are frequently missing – these are needed for statistical testing and to calculate attrition rates
Varying sample sizes throughout report without explanations for changes
Validity and reliability testing not reported for locally – developed instruments or cited for pre-existing instruments
Data collection methods are not discussed

15. Key Recommendation – Report the Details
Describe the intervention in detail, including such information as: the format of the training, the background of the trainer(s), the material covered, the frequency of classes, and the numbers of participants.
Report pre & post sample sizes for both groups and explain changes in samples sizes; if reporting sub-groups, indicate their sample sizes as well
Report key characteristics associated with outcomes at baseline (e.g., pretest scores, teaching experience)
Document and describe the data collection procedures
Report means, standard deviations/errors, for both groups on key outcomes; if using a regression model, describe it
Report results from appropriate significance testing of differences observed between groups (e.g., t-statistics or p-values)

16. Preliminary Framework for Review Standards for Evaluation Reports
Preliminary framework based on generally acceptable standards for high quality research studies.
General Framework and Background
Data Sources and Methods
Presentation Of Findings And Placing Them In Context
Report Summary

17. Mathematics and Science Partnership (MSP) Programs
U.S. Department of Education
San Francisco Regional Meeting
March 21-23, 2011 17