IAEA 2008 Slide 1 September 8, 2008 Some Implications of Expertise Research for Educational Assessment Robert J. Mislevy, PhD Professor of Measurement & Statistics University of Maryland Keynote address at the 34th International Association for Educational Assessment (IAEA) Conference, Cambridge University, September 8, 2008.

IAEA 2008 Slide 2 September 8, 2008 Introduction l These are exciting times in assessment. l Developments in psychology and technology. l Insights from expertise research with implications for assessment design. l Let’s start with a quiz.

IAEA 2008 Slide 3 September 8, 2008 What is this a picture of? (

IAEA 2008 Slide 4 September 8, 2008 Which cards need to be turned over? E4T7 Each card has a letter on one side and a number on the other. Consider the rule “If there is a vowel on one side, there is an even number on the other.” Which cards do you need to turn over to make sure the rule is not violated? (Wason,1966)

IAEA 2008 Slide 5 September 8, 2008 A Little Story I will read a little story and ask you some questions about it.

IAEA 2008 Slide 6 September 8, 2008 Was this sentence in the story? 1. Mary gave Ed a stuffed hippo. 2. Harold received ninety gifts. 3. Ralph gave Tina a toy car.

IAEA 2008 Slide 7 September 8, 2008 There were five sentences in the story. In order, what were the second-to-last words in each of the sentences?

IAEA 2008 Slide 8 September 8, 2008 What is this a picture of? (

IAEA 2008 Slide 9 September 8, 2008 Which cards need to be turned over? E4T7 Each card has a letter on one side and a number on the other. Consider the rule “If there is a vowel on one side, there is an even number on the other.” Which cards do you need to turn over to make sure the rule is not violated? (Wason,1966) Whose IDs do we need to check? Each person has an age and a beverage. Consider the rule “If you are under 21, your beverage must be non-alcoholic.” Which people do you need to check to make sure the rule is not violated? 15 Year old Water drinker 30 Year old Wine drinker

IAEA 2008 Slide 10 September 8, 2008 Was this sentence in the story? 1. Mary gave Ed a stuffed hippo. 2. Harold received ninety gifts. 3. Ralph gave Tina a toy car. YES NO Ralph gave a toy car to Tina.

IAEA 2008 Slide 11 September 8, 2008 There were five sentences in the story. In order, what were the second-to-last words in each of the sentences? One, stuffed, to, four, holiday.

IAEA 2008 Slide 12 September 8, 2008 Limitations and Difficulties Processing limitations l Limited attention l Limited working memory Knowledge limitations l Not knowing what information is relevant l Don’t know how to integrate information l Not knowing what to expect l Not knowing what to do and when to do it l Lack of production proficiency

IAEA 2008 Slide 13 September 8, 2008 Capabilities l Reasoning in terms of patterns l Many patterns simultaneously (language) l Can make perception, procedures, strategies automatic with practice l Can think about our thinking (metacognition) l Benefit from procedures, methods, tools, external knowledge representations Expertise as the circumvention of human processing limitations (Salthouse, 1991)

IAEA 2008 Slide 14 September 8, 2008 A Closer Look at Cognition Knowledge as patterns, at many levels… Assembled to understand, to interact with, and to create particular situations in the world Developed, strengthened, modified by use Associations of all kinds, including applicability, affordances, procedures, strategies, affect “The user’s knowledge of the language rules is interlocked with his knowledge of when, where, and with whom to use them” (R. Ellis, 1985)

IAEA 2008 Slide 15 September 8, 2008 Walter Kintsch’s CI Theory of Reading Comprehension More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition. TextText baseSituation ModelContext Context 1 LTM E.g., reading tasks in Occupational English Test (OET; McNamara, 1996) call upon patterns re language, but also genre, medical knowledge, use of information in clinical settings A relevant pattern from LTM may be activated in contexts but not others (e.g., physics models, use of conditionals). If a pattern hasn’t been learned, it won’t be activated (although it may get constructed in the interaction). A relevant pattern from LTM may be activated in contexts but not others (e.g., physics models, use of conditionals). If a pattern hasn’t been learned, it won’t be activated (although it may get constructed in the interaction).

IAEA 2008 Slide 16 September 8, 2008 Walter Kintsch’s CI Theory of Reading Comprehension More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition. TextText baseLTMSituation ModelActionContext Context 1 Context 2

IAEA 2008 Slide 17 September 8, 2008 Walter Kintsch’s CI Theory of Reading Comprehension More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition. TextText baseLTMSituation ModelActionContext Context 1 Context 2

IAEA 2008 Slide 18 September 8, 2008 Walter Kintsch’s CI Theory of Reading Comprehension More focused research areas within cognitive psychology today differ as to their foci, methods, and levels of explanation. They include perception and attention, language and communication, development of expertise, situated and sociocultural psychology, and neurological bases of cognition. TextText baseLTMSituation ModelActionContext Context 2 Context 3

IAEA 2008 Slide 19 September 8, 2008 Expertise Research Cognitive task analysis (e.g., Simon & Chase) l Compare experts & novices in replicable conditions l What knowledge is needed? How is it represented? How is it used? What makes tasks hard? Ethnographic research (e.g., Lave) l Expertise in situ l “Critical incident” studies (NBME) Replication possible in simulations lFlight simulators, football kick in video situations

IAEA 2008 Slide 20 September 8, 2008 Expertise Research Experts organize their knowledge effectively l Perceive / understand / act in terms of fundamental principles rather than surface features (Chi, Feltovich, & Glaser) Importance of interaction with situation l Cycles of “perceive / understand / act” External knowledge representations (KRs) l Nexus of info-processing & sociocultural POV l Supported cognition / distributed cognition

IAEA 2008 Slide 21 September 8, 2008 So… How do you use this improved understanding of the nature and acquisition of expertise to design and conduct assessments?

IAEA 2008 Slide 22 September 8, 2008 Assessment Arguments What complex of knowledge, skills, or other attributes should be assessed? What behaviors or performances should reveal those constructs? What tasks or situations should elicit those behaviors? (Messick, 1994)

IAEA 2008 Slide 23 September 8, 2008 Examples l The Architectural Registration Examination (ARE) Architectural design; CAD-like environment l DISC simulator Simulations for problem-solving in dental hygiene l NetPASS (Cisco) Computer network design & troubleshooting

IAEA 2008 Slide 24 September 8, 2008 ARE Example (Irv Katz, ETS) l To replace 10-hour hand-drawn design problem l Reflects changing of profession to CAD l Premium on thinking, not drawing l Planning the firestation site

IAEA 2008 Slide 25 September 8, 2008 An Example of a Task Prompt for the ARE

IAEA 2008 Slide 26 September 8, 2008 An Illustrative Base Diagram for an ARE Task AB D

IAEA 2008 Slide 27 September 8, 2008 A Sample Solution to an ARE Task

IAEA 2008 Slide 28 September 8, 2008 NCARB Example (Irv Katz, ETS) Differences between novices and experts Success rate: 98% vs. 88% Planning time & execution time Patterns of revision involving rework Implications for task design: Constraints Number Variation in importance/difficulty Degree of conflict Implicit constraints? (c.f. writing expertise)

IAEA 2008 Slide 29 September 8, 2008 Design Patterns “Design under constraints” is common to many domains: e.g., engineering, assessment design, wedding planning, apparel design Could define a Design Pattern with these as foci What are characteristics of performances (i.e., observables) that evidence these knowledge / capabilities / attunements? What are characteristics of situations that elicit these observables?

IAEA 2008 Slide 30 September 8, 2008 Other design pattern possibilities Troubleshooting finite systems Medical diagnosis Inquiry cycle Model-based reasoning

IAEA 2008 Slide 31 September 8, 2008 DISC Example The Dental Interactive Simulations Corporation (DISC) The DISC Simulator & Scoring Engine Cognitive Task Analysis to support design rationale

IAEA 2008 Slide 32 September 8, 2008 Goals of the DISC CTA What are the kinds of knowledge and skills that hygienists call upon to solve problems and make decisions in dental hygiene? What do they say and do that gives you evidence about their use of this knowledge? What kinds of situations call upon this knowledge? How do you make re-usable schemas to evoke evidence and construct ‘stories’ around these recurring patterns?

IAEA 2008 Slide 33 September 8, 2008

IAEA 2008 Slide 34 September 8, 2008

IAEA 2008 Slide 35 September 8, 2008

IAEA 2008 Slide 36 September 8, 2008 An example with the Cisco Learning Institute: The NetPass Prototype Create on-line performance assessment of networking skills Focus on learner feedback rather than high- stakes testing Expert/Novice studies ground design

IAEA 2008 Slide 37 September 8, 2008 The task starts with a scenario and description of goals

IAEA 2008 Slide 38 September 8, 2008 To capture their mental model of the network, we ask them to draw the network with a diagramming tool

IAEA 2008 Slide 39 September 8, 2008 The diagram is created by dragging and dropping icons

IAEA 2008 Slide 40 September 8, 2008 Configuring the devices

IAEA 2008 Slide 41 September 8, 2008 When students are done, they press Submit”…

IAEA 2008 Slide 42 September 8, 2008 And the graphical representation is converted into a text representation in XML format

IAEA 2008 Slide 43 September 8, 2008 The text file is scored following detailed rules, which result in characterizations of the work

IAEA 2008 Slide 44 September 8, 2008 And feedback is created for the student… both diagnostic and a summary profile.

IAEA 2008 Slide 45 September 8, 2008 Conclusion Insights from expertise research can improve the practice of assessment, and support deeper learning. Doing so requires a deeper understanding of assessment design. »More explicit arguments and representations »Generativity, re-usability, and inter-operability “Too many notes” ? (Emperor Joseph II) Suitable conceptual frameworks, tools, and exemplars are now beginning to appear.