Assessing Learning for Students with Disabilities Tom Haladyna Arizona State University

Useful Sources Standards for Educational & Psychological Testing, AERA, APA, NCME (1999) Standards for Educational & Psychological Testing, AERA, APA, NCME (1999) Tindal & Haladyna (2002) Tindal & Haladyna (2002) Large-scale assessment programs for all students: Validity, technical adequacy, and implementation. Downing & Haladyna (2006) Handbook of test development Downing & Haladyna (2006) Handbook of test development Haladyna & Downing (2004). Haladyna & Downing (2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice. Kane (2006) Content-related validity evidence. Handbook of test development. Kane (2006) Content-related validity evidence. Handbook of test development. Kane (In press). Validation. Educational Measurement (4 th ed.) Kane (In press). Validation. Educational Measurement (4 th ed.)

Assessment vs. Testing Assessment is the act of judging the indicators of student achievement for the benefit of planning future instruction. Assessment is the act of judging the indicators of student achievement for the benefit of planning future instruction. Testing is a way of providing one valid source of information for assessment Testing is a way of providing one valid source of information for assessment A test is NEVER a valid source of information for assessment unless corroborated by other evidence.—Multiple indicators A test is NEVER a valid source of information for assessment unless corroborated by other evidence.—Multiple indicators

Validity of a Test Score Interpretation or Use Way of reasoning about test scores. Way of reasoning about test scores. Concerned about the accuracy of any interpretation or use. Concerned about the accuracy of any interpretation or use. Involves an argument about how an assessment or a test score can be validly interpreted or used. Involves an argument about how an assessment or a test score can be validly interpreted or used. Involves a claim by the developer/user Involves a claim by the developer/user Involves evidence that might support this claim Involves evidence that might support this claim

Validation’s Steps Developmental Phase Developmental Phase State a purpose for the test. State a purpose for the test. Define the trait (construct). Define the trait (construct). Content Content Cognitive demand Cognitive demand Develop the test. Develop the test. Validate—conduct the study. Validate—conduct the study. Investigative phase Investigative phase

Two Types of Evidence That supports our claim That supports our claim That weakens or threatens validity That weakens or threatens validity Construct under representation Construct under representation Construct-irrelevant variance Construct-irrelevant variance

Two Types of Evidence Includes procedures known to strengthen our argument and support our claim. Includes procedures known to strengthen our argument and support our claim. Includes statistical/empirical information that also strengthens our argument and supports our claim Includes statistical/empirical information that also strengthens our argument and supports our claim

More Types of Evidence Content-related Content-related Reliability Reliability Item quality Item quality Test design Test design Test administration Test administration Test scoring Test scoring Test reporting Test reporting Consequences Consequences

Content Structure—sub scores??? Structure—sub scores??? Concurrent—how it correlates with other information Concurrent—how it correlates with other information Does it represent the construct (content)? Does it represent the construct (content)?

Reliability Very important type of validity evidence. Very important type of validity evidence. Can be applied to individual or group scores. Can be applied to individual or group scores. Group scores tend to be very reliable. Group scores tend to be very reliable. Can focus at reliability at a decision point. Can focus at reliability at a decision point. Subjective judgment a factor in reliability. Subjective judgment a factor in reliability.

Random Error Basis for reliability Basis for reliability Can be large or small Can be large or small Can be positive or negative Can be positive or negative We never know. We never know. We just guess. We just guess. Guessing allows us to speculate about where a student’s true score lies and what action we take. Guessing allows us to speculate about where a student’s true score lies and what action we take.

Item Quality Universal item design Universal item design Format issues Format issues Item reviews Item reviews Field tests Field tests

Test Design Breadth Breadth Scope Scope Depth Depth Length Length Formats Formats

Test Administration Standardized Standardized Accommodations Accommodations Standards Standards

Test Scoring Avoid errors. Avoid errors. Quality control is important. Quality control is important. Invalidate scores when evidence suggests that. Invalidate scores when evidence suggests that.

Score Reporting Helpful to teachers for assessment Helpful to teachers for assessment Meets requirements for accountability Meets requirements for accountability Meet Standards (Ryan, 2006) Meet Standards (Ryan, 2006)

Advice Document what you do. Technical Report Document what you do. Technical Report Build the case for validity. Build the case for validity. Do validity studies when possible. Do validity studies when possible. Stay focused on the real reason for assessment and testing: helping students learn not satisfying someone in DC. Stay focused on the real reason for assessment and testing: helping students learn not satisfying someone in DC.