Presentation is loading. Please wait.

Presentation is loading. Please wait.

Data mining with DataShop Ken Koedinger CMU Director of PSLC Professor of Human-Computer Interaction & Psychology Carnegie Mellon University.

Published byAshley Shaw Modified over 9 years ago

Similar presentations

Presentation on theme: "Data mining with DataShop Ken Koedinger CMU Director of PSLC Professor of Human-Computer Interaction & Psychology Carnegie Mellon University."— Presentation transcript:

1 Data mining with DataShop Ken Koedinger CMU Director of PSLC Professor of Human-Computer Interaction & Psychology Carnegie Mellon University

2 “Knowledge components are the germ of transfer” Goal of the week: What does Ken mean by this?

3 Overview Motivation for data mining  Better understanding of students => better instructional design Exploratory Data Analysis  Data Shop demo, Excel Learning curves & Learning Factors Analysis Example project from last summer

4 Data Mining Questions & Methods What is going on with student learning & performance?  Exploratory data analysis Summary & visualization tools in DataShop Tools in Excel: Auto filter, Pivot Tables, Solver How to reliably model student achievement?  Item Response Theory (IRT) Basis for standardized tests, SAT, GRE, TIMSS… Version of “logistic regression”

5 Data Mining Questions & Methods 2 What’s the nature of knowledge students are learning? How can we discover cognitive models of student learning that fit their learning curves?  Learning Factors Analysis (LFA) Extends IRT to account for learning Search algorithm: Discover cognitive model(s) that capture how student learning transfers over tasks over time What features of a tutor lead to the most learning?  Learning Decomposition Extends LFA to explore different rates of learning due to different forms of instruction How to extract reliable inferences about causal mechanisms from correlations in data?  Causal modeling using Tetrad

6 Overview Motivation for data mining  Better understanding of students => better instructional design Exploratory Data Analysis  Demo: DataShop, Excel Learning curves & Learning Factors Analysis Example project from last summer Next

7 Data Shop Demo …

8 Before going to DataShop, let’s look at a tutor (1997 version!) that generated the example data set we’ll look at

9 TWO_CIRCLES_IN_SQUARE problem: Initial screen

10 TWO_CIRCLES_IN_SQUARE problem: An error a few steps later

11 TWO_CIRCLES_IN_SQUARE problem: Student follows hint & completes prob

12 How to get to the DataShop: Go to http://learnlab.org & click … http://learnlab.org 1 2 3

13 PSLC’s DataShop Researchers get data access, visualizations, statistical tools Learning curves track student learning over time Discover what concepts & skills students need help with

14 PSLC’s DataShop Learning curves reveal over- and under-practiced knowledge components Rectangle-area has an initial low error rate, but is practiced often

15 Other DataShop Features Error Reports  Identify misconceptions by looking for common student errors  When do students ask for hints?  Are there alternative correct strategies? Performance Profiler Export Data  Get all or part of the data in tab-delimited file  Use your favorite analysis tools …

16 Exported File Loaded into Excel

17 Overview Motivation for data mining  Better understanding of students => better instructional design Exploratory Data Analysis  Data Shop demo, Excel Learning curves & Learning Factors Analysis Example project from last summer Next

18 3(2x - 5) = 9 6x - 15 = 92x - 5 = 36x - 5 = 9 Cognitive Model drives behavior of intelligent tutor systems … Cognitive Model: expert component of intelligent tutors that models how students solve problems If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d If goal is solve a(bx+c) = d Then rewrite as bx+c = d/a Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

19 3(2x - 5) = 9 6x - 15 = 92x - 5 = 36x - 5 = 9 Cognitive Model drives behavior of intelligent tutor systems … Cognitive Model: expert component of intelligent tutors that models how students solve problems If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction Hint message: “Distribute a across the parentheses.” Bug message: “You need to multiply c by a also.” Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing Known? = 85% chanceKnown? = 45%

20 Cognitive Modeling Challenge Problem: Intelligent Tutoring Systems depend on Cognitive Model, which is hard to get right  Hard to program, but more importantly …  A high quality cognitive model requires a deep understanding of student thinking  Cognitive models created by intuition are often wrong (e.g., Koedinger & Nathan, 2004)

21 Significance of improving a cognitive model A better cognitive model means:  better feedback & hints (model tracing)  better problem selection & pacing (knowledge tracing) Making cognitive models better advances basic cognitive science

22 How can we use student data to build better cognitive models? Cognitive Task Analysis methods  Think alouds, Difficulty Factors Assessment General lecture Tuesday  Peer collaboration dialog analysis TagHelper track  Newer: Data mining of student interactions with on-line tutors

23 Back to DataShop to illustrate

24 Use log data to test alternative knowledge representations Which “knowledge component” analysis is correct is an empirical question! Log data from tutors provides data to compare different KC analyses  Find which “germ” accounts for student learning behaviors

25 Not a smooth learning curve -> this knowledge component model is wrong. Does not capture genuine student difficulties.

26 This more specific knowledge component (KC) model (2 KCs) is also wrong -- still no smooth drop in error rate.

27 Ah! Now we are getting a smooth learning curve. This even more specific decomposition (12 KCs) better tracks the nature of student difficulties & transfer for one problem situation to another.

28 Overview Motivation for data mining  Better understanding of students => better instructional design Exploratory Data Analysis  Demo: DataShop, Excel Learning curves & Learning Factors Analysis Example project from last summer Next

29 Example project from 2006 Rafferty (Stanford) & Yudelson (U Pitt) Analyzed a data set from Geometry Applied Learning Factors Analysis (LFA) Driving questions:  Are students learning at the same rate as assumed in prior LFA models?  Do we need different cognitive models (KC models) to account for low-achieving vs. high- achieving students?

30 Rafferty & Yudelson Results 1 Different student learning rates? Yes

31

32 Rafferty & Yudelson Results 2 Is it “faster” learning or “different” learning?  Fit with a more compact model is better for low pre for high learn  Students with an apparent faster learning rate are learning a more “compact”, general and transferable domain model (Became basis of Anna Rafferty’s masters thesis)

33 Data Mining-Data Shop Offerings Tomorrow Lectures in 3501 Newell-Simon Hall, activities here (Wean 5202) 1. Educational data mining overview & introduction to using the DataShop  Follow-up activities: Exercise in using DataShop for exploratory data analysis Use tutor/course that generated target data set. Begin data export, data scrubbing, exploratory data analysis 2. Learning from learning curves: Item Response Theory, Learning Factors Analysis 3. Other data mining techniques: Learning decomposition, causal models with Tetrad Define metrics to address driving question, begin analysis

34 Questions?

35 What’s next? Tomorrow:  Do you know which offerings you will go to tomorrow?  Any conflicts -- two you want to go to that are at the same time?

36 END

Download ppt "Data mining with DataShop Ken Koedinger CMU Director of PSLC Professor of Human-Computer Interaction & Psychology Carnegie Mellon University."

Similar presentations

Effective Instructional Conversations + TuTalk Instruction Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute.

Effective Instructional Conversations + TuTalk Instruction Carolyn Penstein Rosé Language Technologies Institute/ Human-Computer Interaction Institute.
Scientifically Informed Web- Based Instruction Financial and Intellectual Support: The William and Flora Hewlett Foundation Carnegie Mellon University.

Scientifically Informed Web- Based Instruction Financial and Intellectual Support: The William and Flora Hewlett Foundation Carnegie Mellon University.
Causal Data Mining Richard Scheines Dept. of Philosophy, Machine Learning, & Human-Computer Interaction Carnegie Mellon.

Causal Data Mining Richard Scheines Dept. of Philosophy, Machine Learning, & Human-Computer Interaction Carnegie Mellon.
Educational Data Mining Overview Ryan S.J.d. Baker PSLC Summer School 2012.

Educational Data Mining Overview Ryan S.J.d. Baker PSLC Summer School 2012.
Educational Data Mining Overview Ryan S.J.d. Baker PSLC Summer School 2010.

Educational Data Mining Overview Ryan S.J.d. Baker PSLC Summer School 2010.
Bridgette Parsons Megan Tarter Eva Millan, Tomasz Loboda, Jose Luis Perez-de-la-Cruz Bayesian Networks for Student Model Engineering.

Bridgette Parsons Megan Tarter Eva Millan, Tomasz Loboda, Jose Luis Perez-de-la-Cruz Bayesian Networks for Student Model Engineering.
What can CTAT do for you? Overview of the CTAT track Vincent Aleven, Bruce McLaren and the CTAT team 3rd Annual PSLC LearnLab Summer School Pittsburgh,

What can CTAT do for you? Overview of the CTAT track Vincent Aleven, Bruce McLaren and the CTAT team 3rd Annual PSLC LearnLab Summer School Pittsburgh,
Educational data mining overview & Introduction to Exploratory Data Analysis Ken Koedinger CMU Director of PSLC Professor of Human-Computer Interaction.

Educational data mining overview & Introduction to Exploratory Data Analysis Ken Koedinger CMU Director of PSLC Professor of Human-Computer Interaction.
Knowledge Inference: Advanced BKT Week 4 Video 5.

Knowledge Inference: Advanced BKT Week 4 Video 5.
1 LearnLab: Bridging the Gap Between Learning Science and Educational Practice Ken Koedinger Human-Computer Interaction & Psychology, CMU PI & CMU Director.

1 LearnLab: Bridging the Gap Between Learning Science and Educational Practice Ken Koedinger Human-Computer Interaction & Psychology, CMU PI & CMU Director.
Improving learning by improving the cognitive model: A data- driven approach Cen, H., Koedinger, K., Junker, B. Learning Factors Analysis - A General Method.

Improving learning by improving the cognitive model: A data- driven approach Cen, H., Koedinger, K., Junker, B. Learning Factors Analysis - A General Method.
Tradeoffs Between Immediate and Future Learning: Feedback in a Fraction Addition Tutor Eliane Stampfer EARLI SIG 6&7 September 13,

Tradeoffs Between Immediate and Future Learning: Feedback in a Fraction Addition Tutor Eliane Stampfer EARLI SIG 6&7 September 13,
An Individualized Web-Based Algebra Tutor D.Sklavakis & I. Refanidis 1 An Individualized Web-Based Algebra Tutor Based on Dynamic Deep Model Tracing Dimitrios.

An Individualized Web-Based Algebra Tutor D.Sklavakis & I. Refanidis 1 An Individualized Web-Based Algebra Tutor Based on Dynamic Deep Model Tracing Dimitrios.
Supporting (aspects of) self- directed learning with Cognitive Tutors Ken Koedinger CMU Director of Pittsburgh Science of Learning Center Human-Computer.

Supporting (aspects of) self- directed learning with Cognitive Tutors Ken Koedinger CMU Director of Pittsburgh Science of Learning Center Human-Computer.
Projects March 29, 2006. Project Requirements Think Aloud –At least two people OR Difficulty Factors Assessment –Ideally >25 (at least one class), but.

Projects March 29, Project Requirements Think Aloud –At least two people OR Difficulty Factors Assessment –Ideally >25 (at least one class), but.
Data mining with DataShop Ken Koedinger CMU Director of PSLC Professor of Human-Computer Interaction & Psychology Carnegie Mellon University Ryan S.J.d.

Data mining with DataShop Ken Koedinger CMU Director of PSLC Professor of Human-Computer Interaction & Psychology Carnegie Mellon University Ryan S.J.d.
Searching for Patterns: Sean Early PSLC Summer School 2007 Question: Which is a better predictor of performance in a cognitive tutor, error rate or assistance.

Searching for Patterns: Sean Early PSLC Summer School 2007 Question: Which is a better predictor of performance in a cognitive tutor, error rate or assistance.
+ Doing More with Less : Student Modeling and Performance Prediction with Reduced Content Models Yun Huang, University of Pittsburgh Yanbo Xu, Carnegie.

+ Doing More with Less : Student Modeling and Performance Prediction with Reduced Content Models Yun Huang, University of Pittsburgh Yanbo Xu, Carnegie.
Educational Data Mining Overview John Stamper PSLC Summer School 2011 7/25/2011 1PSLC Summer School 2011.

Educational Data Mining Overview John Stamper PSLC Summer School /25/2011 1PSLC Summer School 2011.
1 Learning from Learning Curves: Item Response Theory & Learning Factors Analysis Ken Koedinger Human-Computer Interaction Institute Carnegie Mellon University.

1 Learning from Learning Curves: Item Response Theory & Learning Factors Analysis Ken Koedinger Human-Computer Interaction Institute Carnegie Mellon University.

Similar presentations

Ads by Google