National Center on Educational Outcomes N C E O What the heck does proficiency mean for students with significant cognitive disabilities? Nancy Arnold, Washington Ron Cammaert, Riverside Publishing Dan Wiener, Massachusetts Ed Roeber, Measured Progress Rachel Quenemoen, NCEO

National Center on Educational Outcomes N C E O “…would allow States to use a documented and validated standards- setting process to define academic achievement standards for students with the most significant cognitive disabilities, as defined in proposed Sec (d)(2), who take an alternate assessment.” March 20, 2003 NPRM for Title I

National Center on Educational Outcomes N C E O Many states have completed a careful standards-setting process for their alternate assessment, just as they have done for their general assessments – these efforts are needed to produce alternate achievement standards Examples shown in NCEO reports (see Synthesis Reports 42, 47, 48, plus one more on its way)

National Center on Educational Outcomes N C E O “These [alternate achievement] standards must be aligned with the State's academic content standards and reflect professional judgment of the highest learning standards possible for those students.” March 20, 2003 NPRM for Title I

National Center on Educational Outcomes N C E O WHO are the students who participate in alternate assessment aligned to alternate achievement standards?

National Center on Educational Outcomes N C E O What do we measure? What does good learning look like for this small group of children? What is “achievement?”

National Center on Educational Outcomes N C E O A brief history of educational goals for these students Developmental approaches Functional approaches Academic approaches – access to the general curriculum, standards-based content, grade level contexts * Diane Browder, 2001

National Center on Educational Outcomes N C E O Development of Alternate Assessments Quenemoen, Rigney, & Thurlow, Careful stakeholder and policymaker development of desired student outcomes for the population, reflecting the best understanding of research and practice, thoughtfully aligned to same content expected for all students. 2. Careful development, testing, and refinement of assessment methods. 3. Scoring of evidence of content aligned student work, according to professionally accepted standards, against criteria that reflect best understanding from research and practice. 4. Standard-setting process to allow use of results in reporting and accountability systems. 5. Continuous improvement of the assessment process.

National Center on Educational Outcomes N C E O Alternate Assessments are works in progress Alignment to content standards varies: Reading and math skills in context of grade level curriculum contexts Reading and math skills in functional contexts Reading and math skills in isolation Weak linkages to reading and math All of these exist within current state approaches

National Center on Educational Outcomes N C E O Alternate Assessment Strategies Thompson & Thurlow, 2001 Evidence/Portfolio Checklist Combination Performance event IEP Analysis [2 states undecided]

National Center on Educational Outcomes N C E O Case Studies: Common Criteria Quenemoen, Thompson, & Thurlow, Content Standards Linkage. 2. Independence. 3. Generalization. 4. Appropriateness. 5. IEP Linkage. 6. Performance. level of skill or mastery and multiple settings; progress and appropriateness; accuracy, mastery, progress, independence, multiple settings, multiple occasions, or multiple purpose.

National Center on Educational Outcomes N C E O Validity of Alternate Assessments * Face Validity – Are scoring procedures consistent with important best practice indicators in the lives of students with significant disabilities? Concurrent Validity – Do scores correlate with other measures of students achievement and indices of quality programming at the school level? *Concepts developed by Harold Kleinert, U of KY Predictive Validity – How well do scores predict post-school success?

National Center on Educational Outcomes N C E O Development of Alternate Assessments Quenemoen, Rigney, & Thurlow, Careful stakeholder and policymaker development of desired student outcomes for the population, reflecting the best understanding of research and practice, thoughtfully aligned to same content expected for all students. 2. Careful development, testing, and refinement of assessment methods. 3. Scoring of evidence of content aligned student work, according to professionally accepted standards, against criteria that reflect best understanding from research and practice. 4. Standard-setting process to allow use of results in reporting and accountability systems. 5. Continuous improvement of the assessment process.

National Center on Educational Outcomes N C E O Copies of the papers cited and presented are at: or Search for NCEO Alternate Assessment Topic, Resources

National Center on Educational Outcomes N C E O What (the Heck) Does “Proficient” Mean? Standard Setting on the WAAS Nancy Arnold Washington Department of Public Instruction CCSSO - San Antonio, TX June 2003

National Center on Educational Outcomes N C E O Steps Taken to Determine Standard Setting Process Review of NCEO Synthesis Reports Detailed Study of Standard Setting for Alternate Assessment in Other States Review of Relevant Literature –Standards for Educational and Psychological Testing (AERA, APA, NCME) – Setting Performance Standards: Concepts, Methods, and Perspectives (Cizek) Validation of Process with National Technical Advisory Committee and Advisory Panel

National Center on Educational Outcomes N C E O Standard Setting Methodology Selection of panelists –Teachers, parents and administrators –Stratified sample –Portfolio experts and novices Selection of standard setting materials –Performance descriptors –Scoring patterns –Exemplar portfolios Orientation and training of panelists

National Center on Educational Outcomes N C E O Standard Setting Methodology (continued) Determining alternate achievement performance descriptors Set cut scores in three rounds –using scoring patterns –revise cut scores using exemplar portfolios –finalize cut scores using impact data Evaluate standard setting process