1. Some Implications of Expertise Research for Educational Assessment Robert J. Mislevy University of Maryland National Center for Research on Evaluation, Standards, and Student Testing (CRESST) 2005 CRESST Conference UCLA, Los Angeles, CA September 8-9, 2005

2. The Central Idea lExpertise research seeks to understand in detail the knowledge, representations, development, and contexts of expertise. lResults can ground assessment arguments & task design. lMust recast in terms of purpose, perspective, principles, and structures of assessment. lEspecially valuable with complex proficiencies, situations, or performances.

3. Outline lAn example: Architectural design lImplications of expertise research lAssessment arguments lGenerative schemas for task design lConclusion

4. The Architectural Registration Examination lArchitectural design; CAD-like environment. (ETS: Katz, Bejar, Braun, Hone, Brittingham, Bennett, et al.) lTo replace 10-hour hand-drawn design problem lReflects changing of profession to CAD lPremium on thinking, not drawing

5. Cognitive task analysis Planning the firestation site: A “block diagram” design problem

6. An Example of a Task Prompt for the ARE

7. An Illustrative Base Diagram for an ARE Task

8. A Sample Solution to an ARE Task

9. Some Results Differences between novices and experts Success rate: 98% vs. 88% Planning time & execution time Patterns of revision involving rework

10. Assessment Arguments What complex of knowledge, skills, or other attributes should be assessed? What behaviors or performances should reveal those constructs? What tasks or situations should elicit those behaviors? (Messick, 1994)

11. Implications for task design (1) Experts perceive/understand/act via fundamental principles rather than surface features Chi, Feltovich, & Glaser on physics In architecture, Constraints Number Variation in importance/difficulty Degree of conflict Implicit constraints

12. Implications for task design (2) Importance of interaction with situation l Intelligence built into situations, tools, processes l “Attunement to affordances” l Cycles of …  Approach to solution, as in ARE  Hypothesize, test, revise, as in inquiry & troubleshooting

13. Implications for task design (3) External knowledge representations Maps, insurance forms, symbol systems, diagrams, Punnett squares, wiring charts, blueprints Roles in practice l Gather/create/share/hold/transform knowledge l Embody generative principles of domain l Central to practice in the domain Roles in assessment l Environment / stimuli / work of tasks l Design KRs to create / implement assessment

14. CTA for Assessment CTA can help ground assessment, re validity argument and principled/explicated task construction. l Must aim to specify elements of Student-, Evidence-, and Task-models l in light of the intended assessment’s purpose, l and constraints of the assessment-building and assessment-delivery contexts.

15. Design patterns Look across domains to find recurring difficulties, blockages, overloads (Salthouse, 1991) Classes of expertise / observables / situations that arise in many domains. Can create “Design patterns” in, e.g., Design under constraints (engineering) Problem-solving in finite domains (troubleshooting) Model-based reasoning (science)

16. Advances in assessment design A cognitive design system approach to generating valid tests (Embretson,1998) Model-basel assessment (Baker, 1997) Constructing measures (Wilson, 2004) Understanding by design (Wiggins, 1998) On the structure of educational assessments (Mislevy, Steinberg, & Almond, 2003)

17. Conclusion Insights from expertise research can improve the practice of assessment. Suitable conceptual frameworks, tools, and exemplars are now appearing. Design & delivery frameworks are key to making technology-based / complex assessment practical.