1. Issues Related to Judging the Alignment of Curriculum Standards and Assessments Norman L. Webb Wisconsin Center for Education Research University of Wisconsin-Madison Annual Meeting of the American Educational Research Association Montreal, April 11, 2005 This work was supported by a subgrant from the U. S. Department of Education (S368A030011) to the State of Oklahoma and a grant from the National Science Foundation, (EHR 0233445) to the University of Wisconsin—Madison. Any opinions, findings, or conclusions are those of the author and do not necessarily reflect the view of the supporting agencies.

2. Webb Alignment Process Identify Standards and Assessments Select 6-8 Reviewers (Content Experts) Train Reviewers on DOK Levels Part I: Code DOK Levels of the Standards/Objectives Part II: Code DOK Levels and Corresponding Objectives of Assessment Items

3. Reports Standards DOK analysis Degree of alignment by standard Source of Challenge Reviewers’ Notes General Comments Made by Reviewers Reliability Among Reviewers Overall Finding of the Degree of Alignment

4. Data Tables Acceptable levels on four alignment criteria Source of challenge DOK by item and intraclass correlation Notes DOK level and objective code by item Objectives by item Items for each objective Number of reviewers coding an item by objective

5. Alignment Criteria  Categorical Concurrence Number of items per standards  Depth-of-Knowledge Consistency Percent below, at, and above  Range-of-Knowledge Correspondence Percent of objectives with at least one item  Balance of Representation Index 0 to 100

6. What are appropriate acceptable levels for each criterion? Number of items for one standard Distribution of items by complexity Number of standard-objectives with at least one item Distribution of items among objectives under a standard

7. Alignment Levels Using the Four Criteria Alignment Level Categorical Concurrence Depth of Knowledge Range of Knowledge Balance of Representation Acceptable6 items per standard 50% 70% Weak---40%-49% 60%-69% UnacceptableFewer than 6 items per standard Less than 40% Less than 60%

8. State A Categorical Concurrence for Grade 3 Science (N=55 items) StandardsHits Cat. Concurr. TitleMeanS.D. 3.1 - History/Nature10NO 3.2 - Inquiry17.382.12YES 3.3 - Unifying Themes7.54YES 3.4 - Subj Matter/Conc33.51.94YES 3.5 - Design/Applic2.121.27NO 3.6 - Personal/Social4.751.09NO Total66.255.78

9. State B Depth-of-Knowledge Consistency High School Mathematics (N=51 items) Standards # Hits% Under% At% Above DOK Consistency TitleMMMM I - Patterns, Relationships and Functions 10.4483170NO II - Geometry and Measurement13205129YES III – Data Analysis and Statistics13.4458402WEAK IV - Number Sense and Numeration 2.78256114YES V - Numerical and Algebraic Operations and Analytical... 10.67305712YES VI - Probability and Discrete Mathematics 6.8942562YES Total57.22434711

11. State B Range of Knowledge Correspondence High School Mathematics (N=51 items) Standards # Hits# Objs Hit% Objs HitRng. of Know. Title Goals # Objs # Mean I - Patterns, Relationships and Functions 21110.444.2238NO II - Geometry and Measurement 318135.7832NO III - Data Analysis and Statisti 31413.44535NO IV - Number Sense and Numeration 3142.782.4417NO V - Numerical and Algebraic Operations and Analytical... 2910.675.2255YES VI - Probability and Discrete Mathematics 2116.893.6733NO Total1578.1157.224.3935

13. State B Balance of Representation High School Language Arts (3 of 12 standards) (N=116 items) Standards Balance Index Bal. of Represent. % Hits in Std/Ttl Hits Index Title Goals # Objs # MeanS.D.MeanS.D. I. - Meaning and Communication— Reading 15.112880.570.12NO II. - Meaning and Communication— Writing 144870.680.14WEAK VIII. - Genre and Craft of Language 151760.630.16WEAK Total1255.338180.360.21

15. What considerations should be given to different item types? How to consider a multiple-point assessment item? What is the trade off between multiple-choice items and open- ended items?

16. What are issues related to vertical alignment? Appropriate progression of complexity across grades Appropriate progression of content across grades

17. State A Mathematics DOK Levels for Objectives by Grade

18. State A Reading Language Arts DOK Levels for Objectives by Grade

19. Vertical Alignment Questions What level of concurrence is there between objectives for the two grades? To what extent do comparable objectives increase in depth from one grade to the next? To what extent does the range of content increase from one grade to the next? How does the balance of representation change from one grade to the next?

20. Type of Vertical Relationships Broader Deeper Prerequisite New Identical