Integrated Document Analysis Workshop Peter Worth, Assessment Standards & Development Services, WestEd, CA John L. Smithson, Director, Measures of the Enacted Curriculum Wisconsin Center for Education Research, University of Wisconsin-Madison Washington, DC May 5-7, 2008 Describing the Content & Language Demands of Standards, Assessments & Student Work

The intended curriculum: State content standards— What students should learn The enacted curriculum: What teachers teach The learned curriculum: Student outcomes based on school learning The assessed curriculum: State (and other) assessments— tested learning Analyzing Alignment Linquistic Complexity by Language Demands A neutral linguistic taxonomyA neutral content language Topics by Cognitive Demand

Content and Language Coding Form Focus: (A = Academic/S = Social) Language Demand: (A-P, Z) Density: (L = Low/H= High) Mode: Rec. = Receptive (LI=Listening; RD= Reading) Construction: (S=Simple/C=Complex) Pro. = Productive (SP=Speaking, WR=Writing) Overall Complexity: (1= lowest, 4=highest; X= Complexity cannot be determined) Content Codes Language Codes Strand ID TPC1TPC1 CGD1CGD1 Focus (A/S) ModeLanguage Demand (A-P, Z) Density (L/H) Construction (S/C) Overall Complexity (1-4, X) Comments Rec.Pro

Content and Language Coding Grade 8 mathematics item

Content and Language Coding Form Focus: (A = Academic/S = Social) Language Demand: (A-P, Z) Density: (L = Low/H= High) Mode: Rec. = Receptive (LI=Listening; RD= Reading) Construction: (S=Simple/C=Complex) Pro. = Productive (SP=Speaking, WR=Writing) Overall Complexity: (1= lowest, 4=highest; X= Complexity cannot be determined) Content Codes Language Codes Strand ID TPC 1 CGD 1 Focus (A/S) ModeLanguage Demand (A-P, Z) Density (L/H) Construction (S/C) Overall Complexity (1-4, X) Comments Rec.Pro EARDDLC CE 3 K 4 5 6

K-12 Mathematics Content

Basic Algebra

The Goal To render quantitative descriptions of instruction, standards, and assessments using a common language in order to facilitate comparisons and analyses of the three domains of a standards-based approach to education reform and their relationship to one another.

To Facilitate Comparisons Academic Content ELL Lang. Acquisition Academic Content ELL Lang. Acquisition Academic Content ELL Lang. Acquisition

The Two Dimensions Of Content What students should know [Topics] And… Be Able to Do [Expectations for student performance]

Describing Instructional Content SEC utilizes a two-dimensional taxonomy based on: Topic by Cognitive Demand BCDEF

The Content Matrix

… adding levels of relative emphasis yields a 3-D construct

Describing the Cognitive Domain How Many Categories? Bloom’s SEC DOK (Webb) SCASS Science

Dimensions of Knowing & Inquiry AcquireUseExtend (From: Dimensions of Knowing and Inquiring about Science, State Collaborative on Assessments & Student Standards Science Project, Council of Chief State School Officers, 1997)

Depth of Knowledge * Webb, N Alignment of Science and Mathematics Standards in Four State. NISE Research Monograph #18. Madison:Wisconsin Center for Education Research. Skill/Concept Recall Strategic Thinking Extended Thinking Level

Skill/ConceptRecall Strategic Thinking Extended Thinking AcquireUseExtend Exploring Cognitive Demand

Cognitive Demand (or Expectations for Student Performance) Skill/ConceptRecall Strategic Thinking Extended Thinking AcquireUseExtend Memorize Perform Procedures Demonstrate Understanding Conjecture, Generalize Prove Solve non- routine/ make connections Memorize Conduct Investigations Communicate Understanding Analyze Information Apply concepts /make connections Recall Perform Procedures Analyze/ Investigate Evaluate Generate /Demonstrate

Cognitive Demand (or Expectations for Student Performance) Skill/ConceptRecall Strategic Thinking Extended Thinking AcquireUseExtend Memorize Perform Procedures Demonstrate Understanding Analyze Information Evaluate/Apply RecallUnderstandingApplicationAnalyzeEvaluateCreate

Practice Session #1 Exploring Cognitive Demand But first…. Let’s take a 10 minute break!

Exploring Cognitive Demand CgD Immersion Activity Organize into Groups/Tables Each Table w/ CgD Pie Each Person w/ Cgd Descriptors Step 1: Place CgD cards on Pie Slices face-down Step 2: Turn cards over: ID agreements e.g. 2 cards w/ same descriptor in same slice if Group Agrees... discuss key words if not … Discuss … operational definition to distinguish Step 3: Discuss disagreements if consensus reached put in envelope / if not, set aside

Content Analysis Materials Cognitive Demand List Topics Lists Comments & Suggestions Worksheet Coding Forms Documents to be analyzed

Cognitive Demand Lists Five categories of cognitive demand Slightly different for each subject Each category is defined by a list of descriptors The list of descriptors are not exhaustive Each category stands on its own Each category has an associated letter (B-F)

Dimension A: Content Topics Topics List (In your packet of material) Organized at two levels: Content Areas (16 for Mathematics) (27 for Science) (14 for ELAR) Topics (identified by number) (182 Mathematics Topics) (211 Science Topics) (114 ELAR Topics) Plus: non-specific & other

Topics Lists Mathematics Science English Language Arts & Reading Cover grades K-12 Organized into Content Areas Topics & Content Areas have an associated #

Comments & Suggestion Worksheet One for each reviewer - more available Use to: Record coding conventions/decision rules Suggest/identify additional topics not listed Suggest/identify additional CGD descriptors Provide other comments & suggestions Be sure to turn in at end of workshop (and with mail-in materials, as necessary).

The Content Analysis Process ‘Coding’ Teams of 4-5 Content Experts Independent Coding by each Analyst w/ Group Discussion Goal for Process: Generalizability not Inter-rater Reliability Should not be necessary to discuss every item – select by team Pick-up and return documents / coding sheets to Alissa Sign & return to Alissa non-disclosure forms

Practice Exercise #1 (Math)

Practice Exercise #2 (ELAR)

A neutral content language Topics by Cognitive Demand The intended curriculum: State content standards— What students should learn The enacted curriculum: What teachers teach The learned curriculum: Student outcomes based on school learning The assessed curriculum: State (and other) assessments— tested learning Content Analysis Workshop

Coding Forms Assessment Coding Forms Standards/Benchmark Coding Forms Each is used to record content descriptions Each content description consists of A topic number A cognitive demand category letter The Integrated Analysis Coding Form Combines Content & Linguistic Information in a single data collection form

Assessment Coding Forms

Standards Coding Forms

Gr. 5-8 National Science Standards