University of Maryland Slide 1 May 2, 2001 ECD as KR * Robert J. Mislevy, University of Maryland Roy Levy, University of Maryland Eric G. Hansen, Educational Testing Service (builds on work with Linda Steinberg and Russell Almond) March 6, 2003 * Evidence-centered design as knowledge representation

University of Maryland Slide 2 May 2, 2001 Knowledge Representations l A knowledge representation (KR) is a structure for expressing, communicating, and thinking about important entities and relationships in some domain. Maps, wiring diagrams, physics equations, nested lists. Object models, for business systems and computer systems. Evidence-centered design models & structures l KRs are surrogates for something else--a real world situation, or a class of situations, or a representation in other KRs. They capture some entities and relationships, but ignore others. The included entities, relationships, and processes are the ontology of the KR -- what kinds of things you think about, and how.

University of Maryland Slide 3 May 2, 2001 KRs are useful when they highlight important relationships and make them easier to work with. l KRs facilitate analogies across problems and domains. In what ways are AP Studio Art, the SAT, Hydrive, and a language proficiency oral interview alike? l KRs make it easier to acquire and structure information. E.g., ECD design process in ETS Teaching & Learning programs l KRs can facilitate working together. ECD object model for sharing, re-using, repurposing the elements and processes in assessments. l KRs are significant in planning. What will a solution have to look like? What elements in assessments can vary substantially, but what relationships must hold? l Overlapping KRs coordinate work in complex systems. Multiple ECD KRs, with bridges among them, for different, interrelated parts of assessment (substance to argument to specs & models to operation to reporting).

University of Maryland Slide 4 May 2, 2001 *** Warning -- cognitive overload ***

University of Maryland Slide 5 May 2, 2001 Where you usually start: What are all the kinds of things that are important to know and do, when and how? What does good work look like? Not generally organized according to assessment arguments. (In ECD, “domain analysis”) KRs: Idiosyncratic from domains, as evolved to suit domain purposes.

University of Maryland Slide 6 May 2, 2001 Where you usually want to go: Operational assessment system: Pieces and processes that gather, evaluate, and report, to achieve assessment purpose. (In ECD, “assessment delivery system.”) KRs: Object model for delivery system.

University of Maryland Slide 7 May 2, 2001 How do you get from here to there? That is, from knowledge about the domain, to objects and processes that meet the purposes you had in mind?

University of Maryland Slide 8 May 2, 2001 What’s in between (1): Assessment argument: What knowledge, skill, accomplishments, etc., of students do you want do draw inferences about? What do you need to see them say, do, or make? What circumstances can evoke this evidence? (Messick, 1984) KRs: Toulmin & Wigmore diagrams % EH: How about: What’s in between (part 1).. Just a thought..

University of Maryland Slide 9 May 2, 2001 What’s in between (2): Organizing argument in KRs that presage the structure of assessment elements and processes. Still substantively meaningful. (In ECD, “Domain Modeling”) KRs: ETS “paradigms”; T&L forms; PADI Design Patterns; Bayes nets for arguments; BEAR construct map structure.

University of Maryland Slide 10 May 2, 2001 What’s in between (2): Organizing argument in KRs that presage the structure of assessment elements and processes. Still substantively meaningful. (In ECD, “Domain Modeling”) KRs: ETS “paradigms”; T&L forms; PADI Design Patterns, Bayes nets for arguments. What’s in between (2, continued): Also has implications for what information you need before administration of the assessment, and how you can interpret the results. % EH: Very good point for disability access..

University of Maryland Slide 11 May 2, 2001 What’s in between (2): Organizing argument in KRs that presage the structure of assessment elements and processes. Still substantively meaningful. (In ECD, “Domain Modeling”) KRs: ETS “paradigms”; T&L forms; PADI Design Patterns, Bayes nets for arguments. Need to establish correspondence between the common assessment KRs and the domain-specific KRs that address key entities and relationships from that domain, as they need to be organized into the assessment argument, thence assessment structures. E.g., BioKIDS’ structure/demand matrices and FOSS’s filled-in construct maps

University of Maryland Slide 12 May 2, 2001 What’s in between (3): Models and specifications for operational elements and processes. (ECD “Conceptual Assessment Framework”) KRs: Student, Evidence, & Task models; Bayes nets; Measurement-model equations; Task Templates; Generalized rubrics, scoring algorithms

University of Maryland Slide 13 May 2, 2001 The upshot: Work through KRs, get machinery that embodies the substantive assessment argument, to meet the purposes you had in mind.