1. Scientific Validation Of A Set Of Instruments Measuring Fidelity Of Implementation (FOI) Of Reform-based Science And Mathematics Instructional Materials Dae Y. Kim, Principal Investigator Amy Cassata, Co-Principal Investigator Jeanne Century, Co-Principal Investigator IES Grantee Panel on Fidelity of Implementation September 7, 2011

2. Instrument Development NSF-Funded Development Project (2007-2010) Developed a framework for measuring the enactment of five reform-based mathematics and science curricula Created a suite of instruments based on this framework Driving Research Questions What are the enacted components of reform-based instructional materials in mathematics and science that are common across specific programs? How can we rigorously and specifically measure multiple components of implementation? How can we collect data on and accumulate knowledge about implementation across multiple programs?

3. Instrument Development Reform-based Curricula Full Option Science System (FOSS), Science and Technology for Children (STC), Science Companion, Science Education for Public Understanding Program, and Everyday Mathematics “Critical Components” The measurable elements of the intended program model that are essential to its implementation May be shared across programs or unique to specific programs Can be organized into general categories Structural: design, procedures, organization, and built-in supports Instructional: expectations for participants’ actions, behaviors and interactions as they enact and engage in the intervention (users and recipients)

4. Framework for Measurement Process of Critical Component Identification Review of written materials to identify explicit and implicit components Interviews with program developers to articulate the “intended” program model and critical components Conversations with program users about critical components Iterative revision, reconciliation, and modification of critical components list and framework for organizing them

5. FOI Framework for Instructional Materials

6. Measuring Implementation Suite of instruments measuring the operationalization of the identified critical components Teacher Instructional Questionnaire Teacher Instructional Log Teacher Instructional Observation Protocol Teacher Interview Protocol School Leader Questionnaire School Leader Interview Protocol School-Wide Observation Protocol Pilot tested in 200 classrooms in CPS (2008) and revised Field-tested in 265 classrooms in CPS (2009)

7. Measuring Implementation

8. Classroom-Level Instruments Critical Components Number of Items Teacher Instructional Questionnaire Teacher Instructional Observation Protocol Teacher Instructional Log Structural Procedural294439 Structural Educative404 Instructional Pedagogical303531 Instructional Student Engagement32

9. Next Steps: Validation IES Funding: Goal 5 (Measurement) Goals Establish validity and reliability within and across the three classroom-level instruments Develop a student questionnaire (grades 3-5) to triangulate measures of student engagement Participants 50 schools across 4 sites (IL, WA, MA) All teachers grades K-5 (N~1000) All consented students grades 3-5 (N~4500) Schools use both Everyday Mathematics and a reform-based science curriculum

10. Year 1: Student Questionnaire 20 to 25-item questionnaire Literature review for items measuring Student Engagement critical components Item development and pilot testing Cognitive interviews Consult with experts Field testing 300 students in grades 3-5 Analyze factorial validity and internal consistency Instrument revision

11. Year 2: Data Collection for Validation All teachers will complete Teacher Instructional Questionnaire – Math Teacher Instructional Questionnaire – Science All students (grades 3-5) will complete Student Questionnaire – Math Student Questionnaire – Science 300 teachers with additional observations and logs 100 teachers: 2 Math observations and 4 Math logs 100 teachers: 2 Science observations and 4 Science logs 100 teachers: 1 Math & 1 Science observation, 4 Math & 4 Science logs Student achievement data (grades 3-5)

12. Year 3: Analysis Inter-rater agreement (observation) Internal consistency of items Factorial structure of items Differences in scores across different groups/content areas Science vs. mathematics Grade level Study site Cross-instrument consistency Teacher Instructional Questionnaire vs. Student Questionnaire Teacher Instructional Observation Protocol vs. Teacher Log Predictive validity Mathematics and science achievement (grades 3-5)

13. Predicting Student Achievement To what extent do composite indices on the four subcategories (SP, SE, IP, ISE) predict student achievement in science and mathematics? Planned analysis for predictive validity 3-level fixed effects HLM model will explore the effect of the FOI composite indices measured at the teacher (classroom) level on individual students’ science and mathematics achievement (Grades 4-5) Prior year’s science or math score will be entered as a covariate Other variables: student demographics, classroom characteristics Data for science and mathematics will be analyzed separately, then pooled Data for questionnaires, logs, and observations will be analyzed separately

14. Summary The FOI framework clearly and specifically describes the nature of reform-based mathematics and science instructional materials so these elements can be measured The framework has resulted in a set of instruments which are currently being validated The framework, and related instruments, have the potential to inform our understanding reform-based STEM curricula and to accumulate knowledge about the elements that make them most effective