Andrew C. Porter Vanderbilt University Measuring the Content of Instruction: Uses in Research and Practice

Tools  Teacher surveys of instruction Daily logs End-of-semester or end-of-year surveys  Content analyses of Standards Tests Curriculum materials  Alignment indices—e.g, alignment between assessment and standards

Content Matrix

Instruction Vertical and Horizontal Alignment Achievement Assessment Standards District State

Example Matrices to Measure Alignment AssessmentStandards Topics Cognitive Demand Alignment Index =1 - X=Assessment Cell Proportions Y=Standards Cell Proportions ∑ |X-Y| 2

Alignment of Assessments with Standards 7th-Grade Math:Goals Study Average Within-State Alignment =.40 Average Between-State Alignment =.39 Average State-Test-to-NCTM Alignment =.39

Alignment of Instruction with Assessment 8th-Grade Math:SCASS Study Average Within-State Alignment =.22 Average Between-State Alignment =.23 Average State-to-NAPE Alignment =.39

Average Alignment =.69 Alignment of Instruction with Instruction 8th-Grade Math: SCASS Study

7th Grade Standards State E State FNCTM

Quality of Data  Response rates  Interrater agreement for content analyses  Validity of teacher self-report Explaining between-teacher variance in alignment to NAEP Predicting student achievement gains [Note: The need for a reform-neutral language]

Uses of Tools Describing Instructional Practices  Research Serve as dependent variable in teacher decision- making research Describe the implemented curriculum Measure implementation of new curricula Assess the validity of transcript studies  Practice Inform teacher reflections on their own instructional practices [Note: Should not be used for teacher accountability.]

Uses of Tools Describing Instructional Materials  Research Research effects of textbooks on instruction Assess the breadth and depth of content in instructional materials  Practice Build tests Write content standards Develop national, state, or district indicator systems

Uses of Indices of Alignment  Research Serve as a control variable Serve as a dependent variable Serve as a descriptive variable  Practice Align state tests to state standards Align instructional materials to standards or course frameworks

Increasing Validity and Value content language  Getting the content language right  Using time samples to describe instruction for an entire school year  Replicating the finding that alignment predicts student achievement gains teacher self-report  Identifying contexts in which teacher self-report on the content of instruction is more or less accurate interrater agreement  Improving the level and consistency of interrater agreement in content analyses distributional properties  Understanding the distributional properties of the alignment statistics powerful professional development  Building powerful professional development programs for data-based decision making  Developing a content language for reading

Conclusions  Much progress has been made in recognizing the importance of instructional content as a variable in education research.  Some progress has been made in building tools for including content in education research.  There have been several innovative uses of these new tools in both research and practice, and more are on the horizon.  But there is much more work to be done.

7th-Grade Standards--Close View State EState FNCTM Number Sense and Numeration

7th-Grade Standards--Close View State EState FNCTM Data Analysis and Probability

Response Rates for Survey Eisenhower Longitudinal Wave 1 75% Wave 2 74% Wave 3 75% Eisenhower Cross-Sectional 72% Reform Up Close 75%

Interrater Agreement AssessmentMeanRange Goals Study to.34 CCSSO Study to.37 Standards Goals Study to.33 [Note: In each study, there was one outlier rater.]

Eisenhower Longitudinal Study 42% of variance explained by level (elementary, middle, high school) and subject 27% of variance explained by teachers in the same school 0% of variance explained by between school or between years Longitudinal data on instruction alignment to NAEP yielded:

Alignment to Predict Achievement Gains

Index Intercorrelations