Quasi-Experimental Designs For Evaluating MSP Projects: Processes & Some Results Dr. George N. Bratton Project Evaluator in Arkansas

Purposes of Evaluation An evaluator collects, summarizes, and analyzes data to a) inform participants, b) inform project staff, c) inform other stakeholders/partners, and d) satisfy the reporting requirements of funding agencies/sources.

Math-Science Partnerships MSP’s conduct research based interventions which will directly impact teachers’ content knowledge. This impact will be reflected in improved teaching practices which result in improved student achievement as measured by either a norm-referenced or a criterion- referenced examination.

As a project evaluator I have used quasi-experimental designs which employ matched-pairs methodology as a control mechanism. Also, I have relied solely on the results of norm-referenced or criterion-referenced examinations for the quantification of changes. Doing so increases the probability of valid, replicable results.

Quasi – Experimental Designs Not a method of choice, rather a fallback strategy. Useful when the nature of the independent variable precludes randomization or randomization is simply not feasible.

Matched – Pairs Methodology Note that even nonequivalent comparison groups can bolster your design. For teacher-participants – pre-test and post-test For student achievement within a classroom – pre- project to current year comparison of subject area of interest to another subject area For school achievement – cluster analysis to identify a non-participating pair school, then pre- & post- project subject area of interest comparisons

Instruments Teacher Content Knowledge – Diagnostic Teacher Assessments in Mathematics and Science (DTAMS) Arkansas Comprehensive Testing, Assessment, and Accountability Program (ACTAAP) Iowa Tests of Basic Skills (ITBS)

Quantitative Impact on Teachers: Analysis of Gain Scores on DTAMS (MSMP) Participating teachers were given DTAMS examinations Algebraic Ideas, Geometry/Measurement, and Probability/Statistics in the summer of 2005 and again in the spring of During the period of time between these examinations those teachers engaged in the project’s professional development activities. For the n=45 individuals that took all six examinations the paired t-test procedure was employed on the overall score, the Knowledge Type scores, and the Subcategory scores for each pair of exams. For each examination the Knowledge Types are I-memorized/factual knowledge, II-conceptual understanding, III-reasoning/problem solving, and IV-pedagogical content knowledge. The Subcategories are mathematical subject dependent.

Paired t-test For a test – treat – retest experimental design without a control group, one must employ gain scores (Post-test score minus pre- test score) rather than a standard two group test of means because the two sets of scores (pre and post) are not stochastically independent of each other. This set of differences is then tested via the standard t-statistic with degrees of freedom = n-1 as the basis for judging the null hypothesis that the true difference is zero. The alternate hypothesis for these tests was that the true difference is not zero. This t-statistic is the sample mean difference divided by the standard error (standard deviation divided by square root of sample size).

Score TypeMean Gain Standard Dev T- value P-valueSignificance Know I High Know II <.00001High Know III Not Signif Know IV <.00001High Total <.00001High Patterns High Expressions <.00001High Equations Not Signif Algebraic Ideas Results Middle School Math Project

Student Achievement Analysis ACTAAP Math vs ACTAAP Language SEARK Mathematics Project Some test results were reported from n=58 participants, however there were 10 participants who reported all four desired scores (ACTAAP Math & Language 2006 and ACTAAP Math & Language 2007) The teacher is the experimental (aggregation) unit, not the student. Criterion is the percentage of teacher’s students scoring at advanced or proficient level.

Computational Details Null hypotheses is there is no change in the difference between 2006 and 2007 and the alternate hypothesis is that there is a positive change. The t-statistic is t=1.757 with 9 degrees of freedom. The level of significance is

Student Achievement Analysis MSMP ACTAAP Math Cluster Analysis Example Cluster Analysis is a category of procedures and measurements that result in a numerical taxonomy of a set of observations based on some set of quantitative variables. Such procedures are widely used in both the biological and social sciences as well as in marketing research. Using this approach a “match” was determined for each participating district. Note: In MSMP all grade level mathematics teachers in the 10 districts participated, so the district was the experimental unit.

I used the following district variables to base the clustering: enrollment, % of free and reduced lunch, % of gifted and talented, % minority, and millage (tax rate). All variables were standardized so that measurement scale differences wouldn’t play a part in distance computation.

For large sample sizes the difference in sample proportions can be assessed via a standard normal distribution employing the formula, where the t subscript refers to the treatment group, the c subscript refers to the control (non-participating group) and

. For both years, 2005 and 2007, a z-score was obtained for each pair of schools for each of grades 6-8. These z-scores were averaged across all schools for each year so that an average z-score of differences was determined for each year. The only statistically significant difference was at grade 8. This is extremely important since the 8 th grade students in 2007 had been 6 th grade students when the project began.

The instructional practices and assessments discussed or shown in this presentation are not intended as an endorsement by the U.S. Department of Education.