1. Jamal Abedi University of California, Davis/CRESST Presented at The Race to the Top Assessment Program January 20, 2010 Washington, DC RACE TO THE TOP ASSESSMENT PROGRAM PUBLIC & EXPERT INPUT MEETING

2. Q1. How “a through-course summative assessment system” can be developed and implemented with the following characteristics? A. A system that includes components of assessments delivered periodically throughout the school year B.If we do this, how should we ask applicants to describe their approaches? C.What evidence should we request if such summative results are part of an accountability system?

3. A. A system that includes components of assessments delivered periodically throughout the school year Potentials: Provides “value added components” for a more thorough assessments of student performance Multiple measures from different sources with different formats (e.g., performance assessment) could improve the reliability and validity of the summative assessments Accountability system will be based on a more comprehensive picture of what students know and are able to do Provides better assessment opportunities for the special needs students specially for ELLs

4. A. A system that includes components of assessments delivered periodically throughout the school year Challenges: Extra efforts and expenses in developing and implementing the new components Extra time taken from instructions Comparability issues in content, construct, linguistics and psychometrics of the different components within and across the consortia of states Burden on teachers and schools or whoever is responsible for providing data Redundancy of the information

5. B. How should we ask applicants to describe their approaches? (Who is responsible for this? The Department, the applicants? Or in a collaborative effort?) i. Clearly identify the “through-course” components and their relevance to the assessment and accountability system (new assessments? Interim assessment outcomes? Performance assessments?) ii. Clearly document the value added by the “through-course” components to the assessment and accountability system iii. Design and field test clear methodology for creating and interpreting the composite system iv. Identify burden on teachers, schools, districts, consortia ofstates or whoever responsible for collecting the components and incorporating them into the assessment system v. Present evidence on problem of redundancy of added criteria

6. C. What evidence should we request if such summative results are part of an accountability system? a. Validity – including construct, content, consequential, and predictive validity b. External validity for postsecondary preparedness c. Reliability – including inter-rater reliability if human scored d. Fairness e. Precision across the full performance continuum (e.g. from low to high performers) f. Comparability across years

7. a. What evidence should we request: Validity, including construct, content, consequential, and predictive validity i. Construct: Different components measure the same construct; construct- irrelevant sources are controlled; the components would provide added values ii. Content: Evidence of alignment of the components (“end-of-course” and the “through-course”) with the common set of K-12 internationally benchmarked, college and career ready standards iii. Consequential: Evidence in achieving the goal of measuring a common set of standards, examine both intended (determination of student college and career readiness, high school exit decision, college course placement) and unintended consequences (focusing on a limited outcomes, the possibility of teaching to the test) iv. Predictive: High power in predicting college and career readiness

8. b. What evidence should we request: External validity for postsecondary preparedness Measures or indices of: i. College and career readiness ii. High school completion iii. College performance iv. Maintaining high performance at college v. Student attributes for college/career success

9. c. What evidence should we request: Reliability – including inter-rater reliability if human scored i. Identify sources of systematic error of measurement (bias) and suggest ways to reduce the impact of such sources ii. Test reliability: internal consistency (examining dimensionality), in addition, estimate reliability through either a parallel form or a test-retest approach iii. Evidence of reliability for subgroups, at the minimum, separate reporting for ELLs and students with disabilities iv. Standard error of measurement by subgroups

10. c1. What evidence should we request: Inter-rater reliability i. Clearly identify factors affecting validity and consistency of scoring open-ended test items such as rater’s background and experience ii. Involve teachers in scoring of open-ended items at the same time estimate and discuss burden on teachers in getting involved in scoring iii. Provide detailed information on the scoring rubric including a discussion on how sensitive is the rubric to students’ backgrounds iv. Not relying solely on percent of agreement approach, use other approaches that are more robust such as Kappa, intra-class correlation, and William’s Index v. Examine and report sources of measurement error to inter-rater reliability

11. d. What evidence should we request: Fairness i. Evidence on examining and controlling for sources of cultural and linguistic biases ii. Evidence on the equal opportunity to learn the content being assessed specially for ELL students iii. Evidence on the appropriateness of accommodations to level the playing field iv. Evidence on the validity and fairness of accommodations

12. e. What evidence should we request: Precision across the full performance continuum i. Enough test items at different level of difficulty to address different levels of content knowledege ii. Evidence on the discrimination power across the full performance continuum iii. Scoring rubric to be sensitive to student different cultural and linguistic backgrounds iv. Evidence of plans and efforts to make assessments more accessible across the full performance continuum

13. Q2. Evidence on the proposed “system” that will ensure consistent and high level of rigor Evidence of convergent validity (MTMM) Evidence of discriminant validity (MTMM) Details of comparability evidence and a clear description of approaches for obtaining such evidence Evidence on the relevance of assessments to all students with different backgrounds

14. Q3. Computer-based test administration Potentials Capable of incorporating accessibility features for students at risk for example, ELLs and students with disabilities Computer assessment system can include a variety of accommodations that would be difficult to implement most or all in a paper-and-pencil assessment format Examples for ELLs include: English and native language glossaries (pop-up), adaptive based on student’s level of English proficiency, Native language or bilingual version of the assessments, extra time, and dictionaries Examples for SWDs include: Read aloud, flexible schedule, one-on-one testing, audio visual assistance As an example of computer capability for subgroups, adaptive testing for ELLs can be mentioned

15. Q3. Computer-based test administration Challenges: Comparability between the computer-based and paper-and-pencil versions Design and logistical issues The challenge that a full comparability may undermine the potential capabilities of computer-based testing For the added accessibility features by computer, applicants must provide clear research-based evidence that computer testing does not alter the construct being measured after incorporating those features All possible sources of threats to application of such system should be clearly indentified and field tested

16. 4. Process or approach leading to innovation and improvement over time Evidence of flexibility of the assessment system and its capability in utilizing innovations over time Evidence that incorporating innovations does not alter the construct being measured Plans for studies for carefully examining the impact of innovations on the assessment outcomes before incorporating them into the assessment system The department should have ongoing quality control system for making sure appropriate innovations have been considered

17. 5. Issues that seem to require additional, focus research Research and research-based evidence on the value-added methodology is a major component of a good RFP Evidence on the comparability and generalizability of performance tasks should be provided by the applicants who are creating such assessments and have access to information on the test content and psychometric properties Studies on how to make assessments more accessible for ELLs and students with disabilities may help consortia of states to be less dependent on accommodations that may not produce valid results Sources of measurement error/construct irrelevant differentially affecting performance of subgroups and ways to control the impact of these sources Effectiveness and validity of accommodations used for ELLs and students with disabilities

18. Other suggestions It is important to incorporate an ongoing quality control procedure (by the Department or its representative) into the process so appropriate corrections can be made Any departure (even minor) from the approved plans should be discussed with the Department and justifications for such changes must be provided The consortia should provide assurance that the decisions on the development and implementation of the assessments are made collectively by all stake holders It is important that people in charge of developing the assessments are quite familiar with research findings in the area of assessments particularly with the recent work on assessments for ELLs and students with disabilities

19. For more information please contact Jamal Abedi at UC Davis/CRESST (530) 754-9150 or jabedi@ucdavis.edu