Presentation is loading. Please wait.

Presentation is loading. Please wait.

Shute, V. J., Levy, R., Baker, R., Zapata, D., & Beck, J. (2009). Assessment and learning in intelligent educational systems: A peek into the future.

Similar presentations


Presentation on theme: "Shute, V. J., Levy, R., Baker, R., Zapata, D., & Beck, J. (2009). Assessment and learning in intelligent educational systems: A peek into the future."— Presentation transcript:

1

2 Shute, V. J., Levy, R., Baker, R., Zapata, D., & Beck, J. (2009). Assessment and learning in intelligent educational systems: A peek into the future. In S. D. Craig, & D. Dicheva (Eds.), Proceedings of the Artificial Intelligence and Education (AIED '09) Workshop on Intelligent Educational Games (pp. 99 – 109). Brighton, UK.

3 Education is a social process. Education is growth. Education is not a preparation for life; education is life itself. –Dewey Education, Writ Large

4 The world has changed a lot in the past 100 years. Education has not. Classroom photo, 1910.Classroom photo, Why Education Needs to Change

5 Why Assessment Needs to Change

6 Learning & Assessment Definitions ASSESSMENT ASSESSMENT: Collecting, analyzing, and interpreting info about student understanding and/or performance re: educational goals. Various procedures used for evaluating student work & learning. LEARNING: LEARNING: Lifelong process of accessing, interpreting, and evaluating information & experiences into knowledge, skills, behaviors, values, identities, and dispositions.

7 CurrentAssessmentPractices StudentLearning Teach. Stop. Test. Repeat.

8 Assessment

9 Collaborative Learning

10 Black & Wiliam, 1998a; 1998b Stiggins, 2002 Assessments for Learning Bass & Glaser, 2004 Informative Assessments Razzaq, Feng, Nuzzo-Jones, Heffernan, Koedinger, Junker, et al., 2005 Assistments Wiggins, 1998 Educative Assessments Shute, Hansen, & Almond, 2008 You can’t fatten a hog by weighing it, or can you? ACED Fuse Learning & Assessment

11 ThemesThemes

12 Comprehensive Learner Models Seamless & Ubiquitous Assessment Assessment Info for Decision Making Aligned along 3 dimensions: vertically, horizontally, and temporally. Theme 1Theme 2Theme 3

13 On June 26, 1974, the first product with a bar code was scanned at a check-out counter. It was a 10-pack of Wrigley's Juicy Fruit chewing gum. Today, the pack of gum is on display at the Smithsonian Institution's National Museum of American History Setting Up A Metaphor for Theme 1

14  Before 1974, businesses had to close down once a year to take inventory.  Barcodes enabled access to a continuous stream of information to monitor inventory.  Now businesses continue without interruption, supporting real-time inventory management.  Before 1974, businesses had to close down once a year to take inventory.  Barcodes enabled access to a continuous stream of information to monitor inventory.  Now businesses continue without interruption, supporting real-time inventory management. Theme 1 Metaphor (via Pellegrino, Chudowsky, & Glaser, 2001) Business

15  Schools shouldn’t have to interrupt normal instruction to give external tests.  Instead, assessment should be continual & invisible, supporting real-time, JIT learning.  This is the first of 3 themes envisioned, 20 years hence (seamless & ubiquitous assessment).  Schools shouldn’t have to interrupt normal instruction to give external tests.  Instead, assessment should be continual & invisible, supporting real-time, JIT learning.  This is the first of 3 themes envisioned, 20 years hence (seamless & ubiquitous assessment). Theme 1 Metaphor (via Pellegrino, Chudowsky, & Glaser, 2001) School

16  Advances in measurement let us administer evidence-based assessments to:  Extract ongoing information from a learner  Make accurate inferences of competencies  React in immediate and helpful ways.  Accomplished via automated scoring and machine-based reasoning techniques.  Seamlessly woven into the fabric of the learning environment. Stealth Assessment

17  Stealth assessment can be used to improve learning:  Gathering evidence for accurate estimates of competencies  Info on competencies used by teachers, the system/game, and/or students  Current estimates of competency levels integrated into the game and displayed as progress indicators.  This elevates valued competencies to the same level as health and weapons! How Does It Work?

18 Theme 1: Summary “Seamless and Ubiquitous Assessment” Idea: A continuous process fusing assessment and learning. Seamless: Removal of false boundaries between learning and assessment. Ubiquitous: Constant nature and need to feedback back results of assessment into learning.

19 Current State Theme 2: Comprehensive Models Goal State

20 Theme 2: Comprehensive Models  Research needed to identify, model, and assess 21 st century competencies as well as other attributes, like affective variables, values, dispositions, etc.21 st century competencies  These will comprise rich evolving profiles representing a synopsis of what’s known, what can be done, what’s believed, preferred/valued, and felt.

21

22 ECD focuses on the evidence of competencies. It provides a way to reason about assessment design, and a way to reason about learner performance. As an assessment framework, it can support both Theme 1 (seamless/ubiquitous assessment) and Theme 2 (comprehensive models). ECD focuses on the evidence of competencies. It provides a way to reason about assessment design, and a way to reason about learner performance. As an assessment framework, it can support both Theme 1 (seamless/ubiquitous assessment) and Theme 2 (comprehensive models).

23 Competency Model What do you want to say about the person? Evidence Model What observations would provide best evidence for what you want to say? Task/Action Model Model What kinds of tasks let you make the necessary observations? Assessment Design

24 For more information about ECD, please see ECD for Dummies.ECD for Dummies Design and Diagnosis

25 Example Conceptual Model

26 Example Computational Model

27 Summary of ECD ECD should be used as the framework for new assessments because it can:  Yield valid assessments for different purposes  Provide accurate estimates of complex competencies and dynamic performances  Aggregate information from various sources  Afford transparency to stakeholders via evidentiary reasoning to support claims.  Yield valid assessments for different purposes  Provide accurate estimates of complex competencies and dynamic performances  Aggregate information from various sources  Afford transparency to stakeholders via evidentiary reasoning to support claims.

28  New competencies need to be identified, modeled, assessed and supported.  Assessments need to be valid and reliable to meet our educational needs (e.g., ECD and stealth assessment).  Immersive games are fun, and enable learning within complex and relevant environments. Wrapping It Up

29 Themes over time Theme20 years… Seamless assessment Learning-related programs/devices that students use (games, learning environments, applications) employ seamless assessment. Ubiquitous assessment Different platforms are interoperable re: delivery, analysis, and support of key comps, relevant data, and inferences linking them altogether. Comprehensive Model: Identifying key comps Wide range of important skills and other attributes identified and justified (e.g., causal reasoning, collaboration, critical thinking, reflexivity, self-regulation, adaptability, focused attention, systems thinking, interdisciplinary thinking, problem solving, compassion, etc.) Comprehensive Model: Modeling Robust (conceptual & computational) models of all comps, as well as their interactions. Vertical scales for key competencies (e.g., trajectories). Comprehensive Model: Assessing Most assts intimately combined within learning experiences thus rarely need summative asst (e.g., only for auditing purposes). Assessment data accumulated into lifelong portfolios/profiles. Theme 1 Theme 2

30  Fusing assessment and learning  Understanding full complement of characteristics involved with learning  Rendering assessments useful to all parties Major Funding Targets

31 Thank you! Questions? Website :

32 Curiosity Critical Thinking Collaboration Grit Systems Thinking Empathy Creativity Go Back

33 BenefitsEfficiency Retention/ Transfer National Trend Validity/Accuracy

34 Where’s Context? General-izabilityBuy-inInterop-erabilityPrivacyBarriers (or Opportunities!)


Download ppt "Shute, V. J., Levy, R., Baker, R., Zapata, D., & Beck, J. (2009). Assessment and learning in intelligent educational systems: A peek into the future."

Similar presentations


Ads by Google