1. Redesign of Alternate Assessments 1 Redesign of Alternate Assessments for Students with Significant Disabilities: National and State Perspectives CCSSO June 16, 2008

2. Redesign of Alternate Assessments 2 Presentation Team Moderator: Patty McDivitt, Data Recognition Corporation Aran Felix, Alaska Department of Education and Early Development Melissa Fincher, Georgia Department of Education Claudia Davis, Louisiana Department of Education Discussant: Rachael Quenemoen, National Center for Educational Outcomes

3. Redesign of Alternate Assessments 3 Presentation Focus Purpose and Rationale Goals Development Challenges Lessons Learned

4. Redesign of Alternate Assessments 4 Alaska Aran Felix Alaska Department of Education and Early Development

5. Redesign of Alternate Assessments 5 Purpose and Rationale Alaskas previous Alternate Assessment Portfolio assessment Housed in the Special Education Unit Assessed English/Language Arts, Math, and Skills for Healthy Living (only E/LA and Math used for AYP) Intended use for grades 3, 6, 8, 11 only Stressed academic content Alternate Performance Standards (content standards) Used as Goals on student IEPs Stressed inclusion (Generalization dimension of scoring) Department of Education organized scoring sessions Single set of achievement standards Scoring dimensions: Skill (student achievement), Generalization, Appropriateness

6. Redesign of Alternate Assessments 6 Purpose and Rationale Steps to explore a solution: Alternate Assessment moved to Assessment Unit Teacher and Parent Survey conducted Reliability-Validity Study conducted Considered removing Generalization & Appropriateness dimensions from scoring for AYP; report only to districts. Needed a new standard setting and better overall technical quality OSEP Condition drove development of a Data-folio assessment for grades 4, 5, 7, 9, 10, Portfolio for all grades overwhelming teachers. Accomplished the original intent of inclusion and teaching/assessing content standards Moved scoring to test vendor Passed Peer Review requirements

7. Redesign of Alternate Assessments 7 Goals Assess a broader range of content standards Manage test size and testing window Increase standardization content assessed (apples to apples) training of teachers to administer the assessment scoring of the assessment training of the protégés Establish a higher technical quality Create an online assessment (training, scoring, reporting)

8. Redesign of Alternate Assessments 8 Development 1) Funding Issues Small population of alternate-eligible students Request for Information 2) What type of assessment to develop? Research other states approaches Research using another states item bank Decision needed: Retrofit portfolio with performance tasks or move to a performance task system?

9. Redesign of Alternate Assessments 9 Development Look of new assessment Used another states item bank while examining alignment the first year Performance Tasks in content areas 4-8 items per task Paper/pencil materials allowing for some adaptation Assessors administer one-on-one to student Online test materials (scoring protocols, student materials, training manuals) Online training for test administrators plus proficiency modules to maintain reliability Online scoring, reporting, unofficial report Original 3-year plan compressed to 2 years (peer review)

10. Redesign of Alternate Assessments 10 Development Year 1: 2005-2006 ( Portfolio still in place) Developed Request for Proposals Contracted with new vendor Convened committees of educators Developed Proficiency level descriptors and Developed Extended Grade Level Expectations Piloted online system with technology coordinators Created security access levels Developed online security rules Trained Qualified Assessors for Pilot Pilot tested the new assessment Created a crosswalk of test items to Alaska content standards (phase 1 blueprint)

11. Redesign of Alternate Assessments 11 Development Year 2: 2006-2007 Developed cousin items Conducted bias review Conducted statewide trainings and orientations Developed a qualified assessor and qualified mentor- trainer path and materials Included mentor pre-test and debrief audios Administered assessment (one test) Piloted science assessment Conducted standard setting for RWM Received Technical Report Submitted new assessment to Peer Review (and received approval) AYP and student reports to parents/districts/website

12. Redesign of Alternate Assessments 12 Development Year 3: 2007-2008 Developed cousin items and conducted bias review Repackage all content tests into grade clusters vs. grade level within content areas New look for website Trained Mentors on science assessment Administered all assessments including science Teacher Survey of Consequential Validity, Mentor Audios Conducted standard setting for science Conducted standard validation for Received Technical Report Submitted new assessment to Peer Review (and received approval) Piloted science assessment AYP and student reports to parents/districts/website

13. Redesign of Alternate Assessments 13 Development Future Plans for years 2008-2011 Develop larger item bank, conduct reviews Field test new items Construct new operational test forms, A & B Conduct a new Standard Setting on Forms A and B of each content-area assessment Consider a program review by external evaluator

14. Redesign of Alternate Assessments 14 Challenges Compressing the schedule from 3-4 years to 2 years Possibility of administering two Alternates simultaneously Using another states items Alaskanizing the assessment Moving from a one-size model to a grade-cluster model

15. Redesign of Alternate Assessments 15 Lessons Learned The importance of flexibility and responsiveness Understanding capacity when huge demands (such as compressed time schedule) are imposed by second peer review deadlines The role of an outside consultant as EED advisor for the first years of the project Involvement of TAC prior to issuing RFP

16. Redesign of Alternate Assessments 16 Recommendations RFP: get help from other states or hire a consultant; include any customizing details; require experienced project manager; include right to own test items Link pattern of test development to general education cycle and include content experts in all aspects of test development (standards to test items) Communicate with the field regularly through the process of developing extended content standards. Provide these standards to the field early. If using any online pieces conduct pilot of online system to explore platform issues and get sign-off from someone in charge at district level Exercise caution when updating online system during test window Include an errata page on online system for notifying field of updates Have a backup server if you have online components Remain cheerful !

17. Redesign of Alternate Assessments 17 Georgia Melissa Fincher Georgia Department of Education

18. Redesign of Alternate Assessments 18 Purpose and Rationale Why did Georgia redesign our alternate? In a word: COMPLIANCE In a few words: OPPORTUNITY TO LEARN

19. Redesign of Alternate Assessments 19 Goals The GAA is designed to ensure that students with significant cognitive disabilities are: Provided access to the state-mandated curriculum. Given the opportunity to demonstrate progress toward achievement of curriculum knowledge, concepts, and skills.

20. Redesign of Alternate Assessments 20 Development The GAA is a portfolio of student work provided as evidence that a student is making progress toward grade-level academic standards. Evidence provided must show instructional activities and student work that is aligned to specific grade-level standards.

21. Redesign of Alternate Assessments 21 Development Georgia elected to go with a portfolio format to allow flexibility for a very diverse group of students. The portfolio format allows the teacher, who knows the student best, to design and document instructional tasks that are meaningful and purposeful for the individual student.

22. Redesign of Alternate Assessments 22 Development Collaboration between Assessment, Special Education, and Curriculum Significant investment in training of educators surrounding curriculum access Ongoing documentation of decisions surrounding development and implementation Involvement of Georgias Technical Advisory Committee Augmented with an AA-AAS expert

23. Redesign of Alternate Assessments 23 Challenges Documenting technical quality Traditional indices do not lend themselves easily to alternate assessments Federal Peer Review Complicated assessment program Teacher buy-in

24. Redesign of Alternate Assessments 24 Lessons Learned Investing in teacher training pays off, but it needs to be continual as the assessment evolves. Technical documentation of alternate assessments looks different, but it is worth pursuing. Validity is an ongoing journey. Students are doing things we never thought possible.

25. Redesign of Alternate Assessments 25 Louisiana Claudia Davis Louisiana Department of Education

26. Redesign of Alternate Assessments 26 Purpose and Rationale Purpose: Louisianas alternate assessment did not receive approval in the USDOE peer review process.

27. Redesign of Alternate Assessments 27 Goals Louisianas alternate assessment (LEAP Alternate Assessment or LAA) lacked: Academic focus (although linked to state standards) Alignment with grades or grade spans USDOE advised a redesign of the LAA to be implemented in spring of 2008 LDE began the redesign in July of 2007

28. Redesign of Alternate Assessments 28 Development First, Louisiana developed Extended Standards (ES), i.e., extensions of state content standards. What should students with significant cognitive disabilities know and be able to do? How do we identify expectations regarding the breadth and depth of the standards, benchmarks, and grade-level expectations (GLEs)? How should students demonstrate knowledge and skills based on the GLEs? What is the appropriate range of tasks to be used to measure this knowledge and these skills?

29. Redesign of Alternate Assessments 29 Development LDE contracted with the testing vendor to develop Extended Standards for ELA, Mathematics, and Science. Recommendations from an initial committee of state special educators regarding selected standards and a template for the ESs, including Complexity Levels, were used to guide the development. Committees of Louisiana educators (general and special educators) reviewed the proposed ESs with Complexity Levels. Content-area groups across four grade spans (grades 3-4, 5-6, 7-8, and 9-11) Review was time intensive -- one intensive week

30. Redesign of Alternate Assessments 30 Development Complexity Levels of the Extended Standards: Each ES has 3 levels of complexity (1 being least difficult) that provide access to general education concepts and skills. They serve as guidelines for the development of assessment tasks at 3 levels of complexity. Math Example: 3. Add and/or subtract to solve simple problems. 2. Identify simple addition and subtraction concepts within daily living problems. 1. Count to solve simple problems.

31. SAMPLE PAGE AND KEY FOR ENGLISH LANGUAGE ARTS Standard One: Students read, comprehend, and respond to a range of materials, using a variety of strategies for different purposes. BenchmarksGrade-Level ExpectationsExtended StandardsComplexity Levels ELA-1-E4: recognizing story elements (e.g., setting, plot, character, theme) and literary devices (e.g., simile, dialogue, personification) within a selection 8. Identify story elements, including: theme conflict character traits, feelings, and motivation (ELA-1-E4) 5. Identify a variety of story elements, including: the impact of setting on character multiple conflicts first- and third-person points of view development of theme (ELA-1-E4) ES-8-5: Identify story elements, including: character 3. Identify the main character in a story 2. Identify two characters in a story 1. Identify one character in a story GLE from grade 3 (top) and grade 4 (bottom) Extended Standard: ES-8 /5 refers to GLE 8 from grade 3 ES-8/ 5 refers to GLE from grade 4 O O 3 is most complex

32. Redesign of Alternate Assessments 32 Development Extended standards and complexity levels were finalized in late September. Extended Standards Handbook (draft) disseminated to school districts in October. Assessment development began.

33. Redesign of Alternate Assessments 33 ELA, Math, Science ELA, Math ELA, Math, Science ELA, Math Science Alternate Assessments Grade SpanContent Area 3 and 4 5 and 6 7 and 8 High school 9 10 11

34. Redesign of Alternate Assessments 34 Alternate Assessments 25 performance tasks per content and grade span Each task administered and scored by the teacher Tasks scored on a 0-1 point or 0-2 point scale using a rubric

35. Redesign of Alternate Assessments 35 Development Alignment Study Alignment of performance tasks to the Extended Standards Panel of eight expert independent reviewers (4 state reviewers/4 national reviewers) Based on Webbs Depth-of-Knowledge alignment process for use in aligning ES to performance tasks (2007) Level 1: Recall of Information Level 2: Basic Reasoning Level 3: Complex Reasoning

36. Redesign of Alternate Assessments 36 Challenges Time was the biggest challenge. Short time frame: 8 months between first development activity (July, 07) and administration window for the assessment (February–March, 08) Lack of adequate time for development activities Lack of preparation time for teachers to provide instruction and to prepare themselves and their students for the assessment Lack of time for adequate delivery of professional development to district personnel Lack of time to field test the items (2008 scores will not be included in state accountability results for schools and districts)

37. Redesign of Alternate Assessments 37 Challenges Other challenges Scheduling collaborative meetings with colleagues in Special Populations on a short timeframe Student IEPs completed based on former alternate assessment Necessary changes in Participation Criteria for AA Decisions regarding Test format Accommodations for students Manipulatives for tasks Assistive technology

38. Redesign of Alternate Assessments 38 Lessons Learned Adequate preparation time for students and teachers is a MUST! Adequate professional development is a MUST! The assessment is not appropriate for all students with significant cognitive disabilities (SCD). Teachers of students with SCD are not so familiar with standardized testing and need more training on administration procedures and security. Opinions from the field of the new assessment ranged from loved it to hated it based primarily on their students abilities to access the test. Assistive technology personnel need to be included at the beginning of the development phase.

39. Redesign of Alternate Assessments 39 To End on a Positive Some Teacher Comments: Im so pleased to have a curriculum (Extended Standards) to guide my instruction. The new test is easier to administer. A good test for content. IEP goals will have to be rewritten to address the new assessment. Great improvement. Great design! Pleasantly positively surprised. The test reflected what I teach.

40. Redesign of Alternate Assessments 40 National and State Perspectives …. And Next Steps Rachel F. Quenemoen, Senior Research Fellow, NCEO National Center on Educational Outcomes

41. Redesign of Alternate Assessments 41 NCEO STATE SURVEY REPORTS 2005 State Special Education Outcomes: Steps Forward in a Decade of Change 2005 State Special Education Outcomes: Steps Forward in a Decade of Change 2003 State Special Education Outcomes: Marching On 2003 State Special Education Outcomes: Marching On 2001 State Special Education Outcomes: A Report on State Activities at the Beginning of a New Decade 2001 State Special Education Outcomes: A Report on State Activities at the Beginning of a New Decade 1999 State Special Education Outcomes: A Report on State Activities at the End of the Century 1999 State Special Education Outcomes: A Report on State Activities at the End of the Century Thompson & Thurlow (1999, 2001, 2003) Thompson, Johnstone, Thurlow, & Altman (2005)

42. Content Addressed by Alternate Assessments: Change Over Time YearFnct. skill No link Stnds Fnct. skill Link Stnds Stnds Plus Fnct. skills Exp/ ext St stnd* Grade level stnd** IEP team deter cntnt OtherRevising 199916---119--- 24--- 2000 9 3728--- 3 2001 415919--- 3 2003 2---436---3 32 2005--- 1211017 *Category possibly included grade level standards prior to 2005 ** Category introduced in 2005

43. 2005 - Outcomes Measured by Rubrics on Alternate Assessments 25 (40) 25 (32 independence) 23 (23) 20 (21) 18 15 (18) 13 (20) 10 (20) 10 9 7 (12) 7 05101520253035404550 Skill/Competence_ Level of Assistance Degree of Progress Number/Variety of Settings Alignment with Academic Content Standards Ability to Generalize Appropriateness Staff Support Social Relationships Self Determination Participation in General Education Settings Support Number of Regular States (Numbers in parentheses from 2001)

44. Alternate Assessment Approaches 2000-2005 (from 2005 Survey) YearPortfolio or Body of Evidence Rating Scale or Checklist IEP Analysis OtherIn Develop- ment/ Revision Regular States 199928 (56%) 4 (8%)5 (10%) 6 (12%) 7 (14%) 200124 (48%) 9 (18%)3 (6%)12 (24%) 2 (4%) 200323 (46%)15 (30%)4 (8%) 5 (10%) 3 (6%) 2005*25 (50%)** 7(14%)***2 (4%) 7 (14%) 8 (16%) Unique States 2003 4 (44%) 0 (0%)1 (11%) 3 (33%) 2005 1 (11%) 0 (0%) 1 (11%) **Of these 25 states, 13 use a standardized set of performance/events/tasks/skills. ***Of these 7 states, three require the submission of student work.

45. Redesign of Alternate Assessments 45 Flexibility and Standardization Nominal categories are NOT often useful for characterizing the technical aspects of the assessment (see Gong & Marion, 2006). The evaluation of technical adequacy interacts with the types of alternate assessments (i.e., choices/ degree of flexibility-standardization) being employed. This does NOT mean that standardization is good and flexibility is badit all depends on purposes!

46. Redesign of Alternate Assessments 46 Where Are We Now? Content coverage – National Alternate Assessment Center work – University of Kentucky: Is it reading? Is it math? Is it science?; University of North Carolina: Links for Academic Learning; other methodologies for alignment. Peer Review suggests great variability, near and far linkages, but a steady trend is toward academic content. Key questions: is it measureable bits or big ideas and concepts or both? What is grade level for these students? Scoring criteria and procedures – What does student performance look like? Student vs. system? How do we measure independence? Who scores? Who checks? Trust but verify? Flexibility vs. standardization issue. Peer Review suggests great variability on this. Concerns related to support, prompts, communications issues, academics interaction.

47. Redesign of Alternate Assessments 47 Where Are We Now? Part 2 Performance/achievement descriptors and standard setting Is there achievement on the content? Is the content clearly referenced? How have accessibility issues been factored in? What does independence mean? How good is good enough? What should these students know and be able to do? How well? This needs careful monitoring over time, consequential validity studies. Approach Degree and logic of flexibility and standardization choices Nominal categories are not particularly useful descriptors. Unfortunately, …the naked eye is drawn to test format not educational soundness (Baker, 2007)

48. Redesign of Alternate Assessments 48 More or Less Than Meets the Eye? BECAUSE of the number of uncertainties still in play, we need: Transparency Integrity Consequential validity studies Planned improvement over time

49. Redesign of Alternate Assessments 49 Questions