1. © Regents of University of California 1 Functional Validity: Extending the Utility of State Assessments Eva L. Baker, Li Cai, Kilchan Choi, Ayesha Madni UCLA/CRESST Comparing Expectations for Validity Models and for New Assessments: Goals, Approaches, Feasibility, and Impact Council of Chief State School Officers (CCSSO) 2015 National Conference on Student Assessment San Diego, California – June 24, 2015

2. © Regents of University of California 2 So What’s New? Opting Out Salient target Evidence of benefit Displaced anger Transparency

3. © Regents of University of California 3 Transparency: Expectations Clear and Sensible? Better test transparency and utility for public Specificity in the right places Support student learning and persistence

4. © Regents of University of California 4 Today: Feature Analysis Argue that tests for “summative” purposes can contribute to transparency of findings to improve learning By conducting qualitative and quantitative analyses of tests (and interventions) the veil of obscurity—and what to teach—can be lifted FA key element of data-mining

5. © Regents of University of California 5 Features for Analysis and Design of Assessments

6. © Regents of University of California 6 How It Works Rate components of items/tasks Low inference features Features recombined in tasks, items (game levels, episodes) Meta-tagged in data Performance summaries across individual or clusters of features Criteria: Significant difficulty, growth, or complexity

7. © Regents of University of California 7 Sample CRESST Features: Content, Cognition, Task, Linguistics Knowledge—mapped to standards and prerequisites – Content—topics, memory, concepts, procedures, systems – Representations Cognitive requirements and skills – Problem solving components – Communication, inferencing – Pattern detection, situation awareness

8. © Regents of University of California 8 Task Features Surface requirements – Format – Stimulus content, prompts, resources, representations – Game mechanic or interaction engine – Affordances, accessibility, accommodations – Team work requirements – Narrative or scenario content and structure Response Requirements – Answer formats – Criteria or scoring rules – Actions or number of types and steps in a response – Essay elements or particular demands

9. © Regents of University of California 9 Linguistic Features Discourse – Complexity or number of ideas in passage or directions – Length – Literal or inferential comprehension – Academic structure, domain-dependent or independent Syntax – Sentence patterns, type and variation – Sentence length – Context cues Word choice – Academic vocabulary-specific domain – Academic language, type, density

10. © Regents of University of California 10 Problem Solving Constraints Single: Increase Vector’s speed to reach stars by reducing amount of friction. Multiple: Increase Vector’s speed to reach stars but not too fast to avoid hitting dynamite.

11. © Regents of University of California 11 State Assessment Study - 1 Purpose: To predict performance from three years of standards use and attribute results – Rated features of content, cognition, linguistics, and tasks with high consistency – Tagged every test item in English Language Arts (ELA) and math for grades 3 & 4 and 7 & 8 for years 2011, 2012, and 2013 by feature – Features accounted for on average 50% of variance on item difficulty

12. © Regents of University of California 12 Assessment Study - 2 Math Grades 4, 8, 11 Previous features augmented by results of student think-alouds A total of 70 features identified and tagged on a sample of math items

13. © Regents of University of California 13 Assessment Study - 2 Findings 4th grade: 16 features were significantly related to difficulty, 10 harder, 6 easier 8th grade: 10 features significantly related to difficulty, 6 harder, 4 easier 11th grade: 12 features related to difficulty, 7 harder, 5 easier

14. © Regents of University of California 14 Feature Relationships Features across grades – Cognitive load – Representation type – Constructed or multiple responses – Guidance – Linguistics Features across ELA and math – Linguistics amount Able to predict item difficulty by features

15. © Regents of University of California 15 Current R&D Continuing FA of state level assessments, refining definitions, protocols, and training Sub-group and feature interactions FA of interventions—PBS learning games and videos, classroom instructional assignments Linking features of interventions and assessments to predict performance Developing two ways of automated feature extraction Designing assessments and games using features Engaging in FA validity studies across projects Looking for partners

16. © Regents of University of California 16 Summary Feature analysis may make “summative” results useful for improvement Multiple purposes for tests Development implications for tests and for designing and predicting effects of interventions

17. Copyright © 2014 The Regents of the University of California. Do Not Distribute Eva L. Baker eva@ucla.edu

18. © Regents of University of California 18 Back up slides Back Up Slides

19. © Regents of University of California 19 State Assessment Functional Validity Data to determine year-to-year cohort performance changes – instructional sensitivity? Summarized across specified features significantly related to high and low difficulty Resulting feature sets accounted on average for 50% of variation of performance If confirmed by instructional studies, findings may guide teachers and professional development to improve test performance using invariant Guide procurement for re-designed specifications Note: Cai, Baker, Choi, Buschang, 2014; Baker, Cai, Choi, 2014; Choi, Madni, 2015

20. © Regents of University of California 20 How It Is Done – Feature Parsing Elements are defined and rated which comprise test items and tasks or learning requirements, e.g., linguistics, content elements, detailed cognitive processes Each item is re-rated by pairs of trained staff for each feature. More granular and operational level of analysis than many currently used approaches Features tagged to items in data

21. © Regents of University of California 21 Purposes of Assessments Beyond accountability Policy linking accountability and improvement Accountability analyses interfere with guidance supporting teaching and learning Can improvement of learning become a useful function of large-scale tests? Feature analysis of item and test properties can yield useful instructional information

22. © Regents of University of California 22 Mapping Features: Ontologies: Networks of Relationships SEL Problem Solving Content