What is program success? Wendy Tackett, Ph.D., Evaluator Valerie L. Mills, Project Director Adele Sobania, STEM Oakland Schools MSP, Michigan

Mathematics Education Resource Center MSP Project Goals 1.Provide high quality professional development to teachers in math or science 2.Positively impact student academic achievement in math or science

MSP Program Logic Oakland Schools

Variables to Consider Teacher attendance & participation in: –Coaching (lesson planning & reflection) –Mathematics institutes/courses –Mathematics instructional materials workshops –Formative assessment workshops (data analysis & planning) –Building based professional development Teacher attitudes about project

Variables to Consider Principal participation in: –Lenses on Learning program –Supporting teachers –Building professional development Building climate –Perceived collegiality –Teacher turnover –Administrative turnover –Support for time out of the classroom

Variables to Consider Student –Level of participation of teachers –Years in classrooms of teachers participating in project Knowledge –Teacher math content knowledge –Teacher pedagogical content knowledge –Teacher instructional knowledge –Student math knowledge –Principal math knowledge

Unintended variables STEM –Learnings from developing professional development for teachers –Learnings from teaching professional development to teachers Coaches –Professional growth of coaches –Impact on the rest of the district if coaches are employed full-time by district

Primary Measures Teacher content & pedagogical knowledge –Learning Mathematics for Teaching (LMT) –sitemaker.umich.edu/lmt Teacher instructional knowledge –Science and Mathematics Program Improvement (SAMPI) lesson observations – Principal knowledge –Lenses on Learning (LOL) – Student knowledge –MEAP (state assessment) –Oakland quarterly assessments (locally developed) Other –Success Case Method teacher interviews –Climate survey –Lesson planning questionnaire

Comparison Group Identified non-participating school district with similar demographics as 5 districts in Oakland Schools Agreement with comparison district: –Test comparison teachers at beginning, middle, & end of project –Collect MEAP information annually –Use Oakland quarterly assessment to test students in 5th grade in fall 2006 & 6th grade in spring 2008 –Pay teachers $200 per testing period & pay district for collecting student achievement data

Treatment Group 5 districts in Oakland Schools –All 3rd-8th grade math teachers participate in a building that is participating Agreement with treatment district: –Test treatment teachers at beginning, middle, & end of project –Observe teachers formally at beginning, middle, & end of project –Collect MEAP information annually –Use Oakland quarterly assessment to test students in 5th grade in fall 2006 & 6th grade in spring 2008 –Pay teachers $200 per testing period

Results: SAMPI Statistically significant improvements in: –Overall rating for classroom instruction –Subcategory ratings of lesson implementation, lesson content, & classroom culture –Substantive student-student interaction –Appropriate pacing for all student levels –Adequate reflection time –Differences between teachers using standards- based math curricula vs. traditional curricula

Using Data Formatively Differences between teachers using standards-based math curricula vs. traditional curricula –MERC staff shared results with partner districts –Those districts using traditional curricula have all now switched to either Everyday Math or the Connected Mathematics Project

Results: SAMPI & Climate 2006 Spring MSP conference in Boston –Iris Weiss discussed impact of building climate on student learning –We developed survey that addressed five elements of building climate Found statistically significant differences (p<.05) in behaviors observed through SAMPI observations in the classroom between teachers in buildings with high climate ratings (3.0 or higher) and the rest of classrooms

Results: SAMPI & Climate

Using Data Formatively Statistically significant differences in behaviors observed through SAMPI observations in the classroom between teachers in buildings with high climate ratings and the rest of classrooms –Coaches started focusing more on creating a sense of collegiality among teachers through building based professional development –MERC project director met with all building principals, curriculum coordinators, and superintendents to more clearly explain the MERC project, potential benefits, and current evaluation findings

Results: LMT Attendance at three or more content-focused or curriculum-based courses had a statistically significant positive impact on the pedagogical content knowledge of teachers LMT Attend 3 or more courses 69% Attend 1-2 courses 54% Comparison group 48%

Using Data Formatively Attendance at three or more content-focused or curriculum-based courses had a statistically significant positive impact on the pedagogical content knowledge of teachers –Offered courses at various times to allow flexibility for teachers who couldn’t previously take them –More concerted efforts to encourage all participants to take at least 3 courses –Developed a fifth content course based on areas of persistently low performance

Results: Coach Observations Teachers are exhibiting deeper understandings of math - leading to more math talk & student discourse Teachers strongly support the professional development sessions Teachers report greater instructional focus and alignment to state standards Teacher concerns moved from logistics to student learning

Using Data Formatively Teachers are exhibiting deeper understandings of math - leading to more math talk & student discourse –Coach observations helped shape future sessions in the Mathematics Institutes and building based professional development –Coaches discuss observations at monthly meetings, sharing ideas, and creating synergies between the coaches –Revised Lesson Planning Protocol based on SAMPI and Success Case interview results

MSP Program Logic Oakland Schools

Our Two Cents Evaluation isn’t just to be done to satisfy grant reporting - more critically it should be used for program improvement Look strategically at the data you’re collecting to show impact along the way…if your logic model is true, it will lead to improved student academic achievement Continue to ask questions and be prepared to adjust your evaluation collection along the way

For More Information Wendy Tackett, Ph.D., iEval – Valerie Mills, Oakland Schools – LearningServices/MathematicsEducationRe sourceCenter/tabid/287/Default.aspx

For a PDF of this presentation: Go to click on “presentations” The instructional practices and assessments discussed or shown in this presentation are not intended as an endorsement by the U.S. Department of Education.