1. An Analysis of Three States Alignment Between Language Arts and Math Standards and Alternate Assessments Claudia Flowers Diane Browder* Lynn Ahlgrim-Delzell UNC Charlotte

2. Alignment Deep alignment Creation of maximum pedagogical and environmental congruence between teaching and the testing situations Fenwick W. English

3. Alignment Curriculum AssessmentInstruction

4. Alternate Assessment State Standards (General Curriculum) Alternate AssessmentInstruction

5. Methods of Alignment Low complexity E.g., asking content experts to examine assessment items and match them to content standards using a Likert scale High complexity Multiple criteria considered

6. Research Questions How do exemplary states’ alternate assessments align to their states’ academic content standards in language arts and math? “Exemplary”: researchers in alternate assessment nominated states as having best general curriculum foundation in their alternate assessments; strong consensus for three states How applicable is a complex alignment method developed for general assessments to alternate assessments?

7. Webb’s (1997) Criteria for Alignment Utilized Categorical concurrence Depth of knowledge Range-of-knowledge correspondence Balance of representation

8. States Reviewed State A- performance based State B- portfolio State C- portfolio

9. Research Steps Identified criteria to be applied Collected sample alternate assessment & state standards from each state in language arts and math; interviewed states to clarify assessments Developed coding matrices Convened panel: 6 curriculum experts; 2 state directors of AA; 4 grant researchers Panel trained; then did coding individually Entered data into database; analyzed Invited feedback from states on findings

10. Results Number of items in language arts and math assessments ranged from 5 to 54 Percent of items that could be aligned to the state’s content standards ranged from 77.5% to 94.1% This low complexity alignment step verified that all three states did have strong match between alternate assessment items and state standards

11. Categorical Concurrence Consistency of categories of content in both the content standards and assessments Webb (1997) recommends at least six items per standard Computed as % of content standards with at least six items aligned

12. Findings for Categorical Concurrence State A LA 66.7%Math 66.7% State B LA 40%Math 40% State C -LA 33.3%Math 0% Question: How feasible/ desirable to have six items for every state standard in an alternate assessment?

13. Depth of Knowledge Consistency Level 1: Recall, observe, one step procedures, “right there” thinking Level 2: Comprehension, comparison, organization; classification Level 3: Reasoning, planning, conjecture, connect ideas, inference, prediction Level 4: Complex reasoning, planning and developing, multiple possible solutions, requires explanation and justification We added- Level 0: Requires little/no thinking (someone else is making the response)

14. Computation of DOK Panel first rated each of the state’s standards for DOK Became familiar with standards and DOK levels prior to viewing alternate assessments Then rated DOK for each alternate assessment entry Reliability (coeffficient alpha) of reviewers’ ratings was.69-.97 (adequate)

15. Findings DOK for all states AA negatively skewed as compared to DOK of the state standards Most items at levels 1 and 2 for AA But items at all four levels in all three states’ alternate assessments Question: Is an overall negatively skewed DOK an alternate achievement standard? Does including items at all four levels help to ensure setting high expectations for AA?

16. Range-of-Knowledge Correspondence Examines the alignment of assessment items to the multiple objectives within the content standards Calculated as the % of objectives (for standards) that have at least one hit (one assessment item aligned to that standard) Recommended criteria is 50% of objectives have at least one aligned item

17. Findings Not calculated for State B because the have a single level of standards State A: range-of-knowledge congruence was 36.5% State C: range-of-knowledge was 0 Question: Range of knowledge criteria, developed for typical large scale tests, may be most difficult to apply to AA. Once again, is the sampling of fewer objectives in the assessment another type of alternate achievement standard? What should be the criteria for range-of-knowledge for AA?

18. Balance of Representation Emphasis the assessment gives to different objectives within a content standard…how evenly distributed assessment items are Calculated using balance of representation index equation

19. Findings Balance index score of 0 indicates unbalanced representation; 1.0 is balanced Only State A approached balanced representation; 2 of 6 standards had balance index>.70 Other two states had balance index below.6 for all standards Question: If alternate assessments focus on smaller number of objectives within the content standards, how is this subset determined? Vary across years? By teachers developing the portfolio? By stakeholders designing the AA?

20. Feedback from States Alignment method seemed appropriate and outcomes useful Noted that much of lack of congruence between assessment and standards was intentional…narrowing focus for alternate achievement standards In two states with portfolios, teachers make decision about which subset of objectives students will demonstrate…individualization valued as way to capture performance of students with complex disabilities

21. Is it fair to apply alignment criteria developed for general assessments? All state representatives wanted the application of higher expectations for their alternate assessments “We want the higher expectations, even though we may not get there.” Also wanted higher expectations for students Given it may be skewed to lower level items, important to have AA sample full range of depth-of-knowledge

22. Conclusions Many general education assessments also do not meet criteria for alignment (Webb, Horton, & O’Neal, 2002) Overall, these states had high match between assessment items and state standards (low complexity alignment) Variance from Webb’s (1997) criteria may reflect these states’ progress in applying alternate achievement standards

23. Recommendations States need to conduct alignment studies for their alternate assessments Align to state standards Articulate how alternate achievement standards are expected to affect alignment Be sure to have items at all levels of depth of knowledge Need for discussion about what the alignment criteria should be for alternate assessment Lower criteria? Different criteria?

24. Need for further curriculum development Research on teaching Ss with significant cognitive disabilities skills at multiple levels of depth of knowledge (e.g., problem solving) Research on teaching academic skills beyond sight words and money

25. Support for this research was funded in part by Grant No. H324C010040 of U.S. Department of Education, OSEP, awarded to UNC Charlotte. The opinions expressed do not necessarily reflect the position or policy of the Department of Education and no official endorsement should be inferred. We express our appreciation to the three states who participated in this research. Correspondence concerning this presentation and research should be sent to : Dbrowder@email.uncc.edu