1. On Layers and Objects in Assessment Design Robert Mislevy, University of Maryland Michelle Riconscente, University of Maryland Robert Mislevy, University of Maryland Michelle Riconscente, University of Maryland

2. 2 Evidence-Centered Design  Conceptual design framework  Broad applicability  Based on  Evidentiary reasoning principles  Demands of assessment production & delivery  Assures  Validity of assessment  Coordination among collaborators (Mislevy, Almond, & Lukas, 2003)

3. 3 Defining Assessment  Reasoning particular things  Reasoning from the particular things students do, say, or make…  to inferences  to inferences about their knowledge, skills & abilities.

4. 4 Some Fundamentals  Evidentiary Reasoning  Assessment as Argument

5. 5 Evidentiary Reasoning C R B unless since on account of W A so supports Toulmin (1958) D

6. 6 Example I C R B unless on account of W A so supports D since Toulmin (1958)

7. 7 Example II John caused the car accident and Marg was seriously injured. John’s BAC was.13. Marg was unconscious and bleeding. John’s speech was slurred. Marg had the right of way. W1: Since an intoxicated driver is generally presumed to be at fault in an accident…. W2: Since the impact of a 2000lb. auto moving at 50mph on a human will generally cause serious injury… B1: State Code: drunk driving at.10 BAC and common law doctrine of negligence per se. B2: Emergency medical records at Wishard Hospital indicate this type of collision will result in serious condition. Marg was also intoxicated… Toulmin (1958) C R B unless on account of W A so supports D since

8. 8 Assessment as Argument C D since W Inferences Observations Situations  Inferences  Observations  Observations needed to ground them  Situations  Situations that will evoke them reasoning  Chain of reasoning connecting them since

9. 9 Some Fundamentals  Evidentiary Reasoning  Assessment as Argument  Knowledge Representations (KRs)  Common Language

10. 10 Knowledge Representations  Directions from Campus to the White House  http://www.mapquest.com/directions/main.adp?go=1&do=nw&un=m& 2tabval=address&cl=EN&ct=NA&1tabval=address&1y=US&1a=&1c=& 1s=&1z=&1ah=NXXzjLrv%252bgGeW7JeQ4fDrUNya%252b4DtKsjxh eMTbVThQJ3lE9WrNx7DAJWKfgZBsxDNU8KMBJ3MT8wsLFk85Pi6j LRqQHxBFYMlklq5k82hIODOViBiLYDxDz3sk%252bXxGlUcWB2Xe c5x21sAggO4xXFyg%253d%253d&2y=US&2a=1600+Pennsylvania+Av enue&2c=Washington&2s=DC&2z=&2ah=&idx=0&id=417d7541- 00047-01632-400c2551&aid=417d7541-00048-01632-400c2551  OR ?

11. 11 Some Fundamentals  Evidentiary Reasoning  Assessment as Argument  Knowledge Representations (KRs)  Common Language  Layers  Leveraging Varied Expertise  Common Structures

12. 12 A Layered Approach  Leveraging Varied Expertise  e.g., Housing Developments  Common Structures  e.g., Architecture, Software Design  Iterate through Assessment Argument  narrative  technical specifications InferencesObservationsSituations

13. 13 Assessment Delivery ECD Layers Conceptual Assessment Framework Domain Modeling Domain Analysis What is important about this domain? What tasks are central to proficiency in this domain? What KRs are central to this domain? What is important about this domain? What tasks are central to proficiency in this domain? What KRs are central to this domain? Represent key aspects of the domain in terms of assessment argument. What tools and materials do we need to implement this kind of assessment? How do we move to implementation with actual examinees?

14. 14  Valued work  Task features  Representational forms  Performance outcomes  Valued knowledge  Knowledge structure and relationships  Knowledge-task relationships Domain Analysis

15. 15 Assessment Delivery ECD Layers Conceptual Assessment Framework Domain Modeling Domain Analysis

16. 16 Domain Modeling  Express the content of the domain analysis in terms of the assessment argument.  Takes narrative form  PADI KR: Design Patterns InferencesObservationsSituations Domain Modeling

17. 17 Design Patterns  Identify in narrative form:  Knowledge, skills, and abilities (KSAs)  Observations to support inference  Features of task situations that elicit target KSAs  Related content or inquiry standards  Do not provide a concrete design or implementation of an assessment task

18. 18

19. 19 Assessment Delivery ECD Layers Conceptual Assessment Framework Domain Modeling Domain Analysis

20. 20 Conceptual Assessment Framework  Move from narrative to more technical specifications.  Express assessment argument in terms closer to implementation  ECD Models Conceptual Assessment Framework

21. 21 STUDENT MODEL EVIDENCE MODEL TASK MODEL InferencesObservationsSituations From Argument to Models (Mislevy, Almond, & Lukas, 2003)

22. 22 EVIDENCE MODEL How do we measure it? Quality Relevance STUDENT MODEL TASK MODEL From Argument to Models (Mislevy, Almond, & Lukas, 2003) What are we measuring? Where do we measure it?

23. 23 The Passenger There are perfectly satisfactory answers to all your questions, but I don’t think you understand how little you would learn from them… Your questions are much more revealing about yourself than my answers would be about me. (Peploe, Wollen, & Antonioni, 1975)

24. 24 HIDDEN SLIDE – READ JOKE While crossing the US-Mexican border on his bicycle, the man was stopped by a guard who pointed to two sacks the man had on his shoulders. "What's in the bags?", asked the guard. "Sand," said the cyclist. "Get them off - we'll take a look," said the guard. The Cyclist did as he was told, emptied the bags, and proving they contained nothing but sand, reloaded the bags, put them on his shoulders and continued across the border. Two weeks later, the same thing happened. Again the guard demanded to see the two bags, which again contained nothing but sand. This went on every week for six months, until one day the cyclist with the sand bags failed to appear. A few days later, the guard happened to meet the cyclist downtown. "Say friend, you sure had us crazy", said the guard. "We knew you were smuggling something across the border. I won't say a word - but what is it you were smuggling?" "Bicycles!"

25. 25  Move from narrative to more technical specifications.  Express assessment argument in terms closer to implementation  ECD Models pre-blueprints  Create pre-blueprints from which we will eventually generate blueprints and operational assessments.  PADI KR: Task Templates Conceptual Assessment Framework

26. 26 Task Templates  Support the specification of technical details  Link CAF components  Serve as pre-blueprints: abstractions of multiple assessment tasks  Become task specifications when all template components are specified

27. 27

28. 28

29. 29 Summing Up  ECD: assessment design as development of an assessment argument  Coordinate work across expertise  Use of layers  Knowledge Representations  Illustrated with PADI work  In addition:  Explication of reasoning behind assessment design decisions  Identification of reusable elements and pieces of infrastructure

30. 30 An ECD “Flight Check”  Conceptual design framework  Broad applicability  Based on  Evidentiary reasoning principles  Demands of assessment production & delivery  Assures  Validity of assessment  Coordination among collaborators (Mislevy, Almond, & Lukas, 2003)