1. Figure 1. From professional development to student achievement We predict that student achievement will improve by providing teachers with improved mathematics content and pedagogy. These treatments are applied to teachers in grades 4-8. Modeling student achievement in K-12 math and science classrooms: An experimental approach M.J. Bryant, R.A. Cardullo, K. Bocian, K.A. Hammond. University of California, Riverside, CA 92521 Figure 2. Comparison of possible experimental designs The treatment (  ) can be applied to either entire schools or to individual classrooms within a school. Schools or classrooms that do not receive the treatment serve as controls (  ). Ignoring issues of human subjects, the treatment would be applied to half of the schools or classrooms. The bottom two panels illustrate and example of increasing the proportion of treatment to control groups which is desirable in an educational experiment. Year 1Year 2Year 3Year 4Year 1Year 2Year 3Year 4 Treatment Applied to Schools Treatment Applied to Classrooms 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Treatment:Control Constant (School) Treatment:Control Increases (School) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 0.00 0.05 0.10 0.15 0.20 0.00.20.40.60.81.00.20.40.60.8 1.0 Alpha (< 0.05 considered significant) Probability of P < Alpha Treatment Applied to: Schools Classrooms Classroom design has the most statistical power School design has the most statistical power The experimental design with the highest statistical power is dependent on where the variation is greatest. Variation greatest among schoolsVariation greatest among classrooms 0.0 Figure 3. Monte Carlo based power analyses inform which design has the highest statistical power. The choice between schools or classrooms as the experimental unit is subject to many logistical considerations. The design impacts statistical power (the probability that a significant difference is found when a difference exists). Computer simulations reveal that statistical power is optimized as a function of where random variation lies. For example if all classrooms in a school are equal but variation exists among schools, randomly assigning treatment to classrooms has greater statistical power. Figure 4. Assignment of treatment and control groups as a stratified random design. In educational settings, demographic factors (i.e. socio-economic status or SES) are known to have a great impact on student performance, but random assignment of treatment is essential. Random treatment assignment may lead to unintended pooling on either end of an SES continuum. By selecting schools at random from matched pairs, we can control for differences in SES. Here schools were matched by principal components analysis. Blue arrows connect matched pairs (  = Treatment,  = Control). Perfect pairing may not be possible for all schools (▲ = schools with SES characteristics outside the range of the other schools). -3 -2 0 1 2 3 -5-4-3-2012345 Schools matched by demographic variables for stratified random assignment Index Based on Aggregated Student Level Characteristics (i.e. % reduced or free lunch) Index Based on Characteristics of the School (i.e. Class Size) Treatment 4th Grade Teachers 5th Grade Teachers 6th Grade Teachers 7th Grade Teachers 4th Grade Students 5th Grade Students 6th Grade Students 7th Grade Students Time (2002-2006) A core team consisting of mathematicians, scientists, and educators develops a treatment The treatment is administered to teachers in the treatment group Teachers teach with learned techniques Efficacy of the treatment is measured through student test scores This several year long study measures the growth in both teachers and students over time INTRODUCTION AND STATEMENT OF PROBLEM Mathematical Achievement through Collaboration with Teachers and Students (Mathematical ACTS) is a multi-faceted NSF-funded research project designed around specifically tailored initiatives for teachers and students in the 4 th – 8 th grades. ACTS seeks to raise achievement and close gaps in mathematics for all students in the Jurupa Unified School District through a series of programs (150 hours annually) for teachers and students. For the purpose of this project, student achievement in mathematics is defined as preparation for and success in Algebra I, the gatekeeper class for higher mathematics courses and entrance colleges and universities. We targeted students in the Jurupa Unified School District 4 th through 9 th grades, beginning in September 2003 through June 2007. Low mathematics performance in K-12 impedes progress in college mathematics and science and its toll is being felt at many institutions of higher education. At the University of California, Riverside the percentage of students who take remedial mathematics (defined as pre-calculus or below) has risen from 65% in Fall, 2000 to 73% in Fall, 2003 (UCR Learning Center Fall, 2003 report). In addition, the majority of under-represented high school students in the 66 th State Assembly District are not prepared for college. Less than 15% of students from this region attend any university (California Postsecondary Education Commission Report, 2003) and the percentage of students that take any mathematics beyond Algebra I is exceedingly low. Left unchecked, any cohort of students that is not well prepared in both mathematics and science will not be prepared to function in a technologically advanced society. The effect of professional development on teaching and the subsequent achievement of students are studied in a research design that includes: Comparison of teacher baseline and post ACTS intervention knowledge and use of mathematics and pedagogy. Assessment of the implementation of ACTS teaching strategies in the classroom. Assessment of the value added by the treatment in the classroom by tracking the rate of student achievement growth in targeted and control (wait-listed) classrooms. Case studies of teacher and school implementation within one school site. Assessment of benchmarks of systemic change for both UCR and JUSD. The initial study group includes teachers and students from 16 elementary and 3 middle schools. In dealing with a small group a number of questions arise concerning the research design, the quality of assessments used for evaluating both content and pedagogy, and the types of analysis used to maximize statistical power in quantifying the effectiveness of treatments. Since scientifically designed and controlled experiments are rare in education, Mathematical ACTS provides an opportunity to quantify the effectiveness of various teacher interventions. Question: How can student achievement in mathematics be improved in public schools so that a larger fraction of students is eligible for Algebra I by eighth grade? Hypothesis: Since students receive mathematics training from teachers, improving student achievement in elementary and middle school math classes can be attained by improving the math proficiency and pedagogy of teachers. Prediction: As a result of ACTS professional development, student achievement, as measured by standardized tests, will significantly increase resulting in a measurable increase in Algebra I enrollment. Figure 5. Mathematics content knowledge and pedagogy are related but distinct. Primary and secondary school teachers tend to have higher pedagogical skills (as measured by a University of Michigan designed pedagogy assessment) when their mathematics content proficiency is higher (CSU/UC Algebra Readiness Test). Secondary school teachers have significantly higher mathematics pedagogical skills, perhaps due to the fact they are single subject teachers. Despite the correlation, there remains considerable variation in pedagogical skill for all levels of content knowledge. The secondary school teachers’ baseline data serves as benchmark goal for the primary schools teachers. 0 0.2 0.4 0.6 0.8 1.0 00.20.40.60.81 Relationship Between Mathematics Proficiency and Pedagogy Score on Mathematics Proficiency Test Score on Mathematics Pedagogy Test Elementary School Teachers Secondary School Teachers Effect of Teacher’s Grade level: P = 0.0002 Percentage of Elementary School Teachers Correctly Answering Question Percentage of Secondary School Teachers Correctly Answering Question 1 to 1 50 60 70 80 90 100 90100 Question 39 Question 1 1)3.01 + 0.301+ 30 = ? (A) 0.631 (B) 3.611 (C) 33.311 (D) 33.32 39) In the right triangle ABC shown to the right, what is the length of AC ? (A) 10 (B) 14 (C) 24 (D) 48 B A C 6 8 Sample Questions Figure 6. Baseline data can serve to inform professional development content. Primary and secondary school teachers do not understand all mathematics subject areas equally. The data points above the 1 to 1 line suggest relatively higher proficiency on a given question by secondary school teachers. By examining the relative performance on each question by the different groups of teachers we can identify concepts where a need for professional development is not indicated (i.e. Question 1). Questions with a great disparity between the two groups can be used to fine tune professional development in areas of content knowledge where it is most needed (Question 39 builds off of State Standards from the grade 4 through grade 7). CONCLUSIONS 1.Any experimental design in education must take into account both traditional scientific design motifs as well as ethical considerations (i.e., ultimately providing treatments for all subjects during the study). 2.To optimize statistical power the choice of schools or classrooms as experimental units depends on the amount of variation within that unit. 3.Pairs of experimental and control groups must be established by taking into account multiple variables (i.e., socio-economic status, school test scores, etc.). 4.In our study, there are significant differences in both math content and pedagogy between elementary (Grades 4-6) and Middle School (Grades 7 & 8) teachers. Further, there are defined areas within mathematics where there are significant differences between these two groups. ACKNOWLEDGEMENTS 1.This work is funded by a Math Science Partnership grant from the National Science Foundation. 2.We would like to acknowledge the support of the ALPHA Center at the UCR for assisting in the preparation of assessment materials and data collection. 3.We thank the Jurupa Unified School District for their cooperation and assistance with this project.