1. 1 Studying and achieving robust learning with PSLC resources Ken Koedinger HCI & Psychology CMU Director of PSLC

2. 2 This is the 4th Annual PSLC Summer School 8th overall –ITS was focus in 2001 to 2004 Goals: –Learning science & technology concepts –Hands-on project you present on Fri

3. Pittsburgh Science of Learning Center (PSLC) 170+ researchers from CA to Germany Ken Koedinger - Carnegie Mellon Co-Director Kurt VanLehn - University of Pittsburgh Co-Director Charles Perfetti - Chief Scientist & Future Co-Director Executive Committee: Vincent Aleven, Maxine Eskenazi (Diversity Director), Julie Fiez, David Klahr (Education Director), Marsha Lovett, Tim Nokes, Lauren Resnick + representatives of Jr. Faculty, Post-docs & Grad students Michael Bett – Managing Director

4. 4 Advancing a Science of Academic Learning Challenge: Chasm between learning science & educational practice Empirical –Lots of rigorous principle-testing lab studies –Lots of realistic classroom design research –Too few experiments combine both Theoretical –Almost as many theories as there are results! –Need for “rigorous, sustained scientific research in education” (National Research Council, 2002)

5. 5 Overview Background –Intelligent Tutoring Systems –Cognitive Task Analysis PSLC Methods, Resources, & Theory –In vivo experimentation –LearnLab courses –Robust learning theoretical framework –Enabling technologies Summary Next

6. 6 PSLC is about much more than Intelligent Tutors But tutors & course evaluations were a key inspiration Quick review …

7. 7 Past Success: Intelligent Tutors Bring Learning Science to Schools! Intelligent tutoring systems –Automated 1:1 tutor –Artificial Intelligence –Cognitive Psychology Andes: College Physics Tutor –Replaces homework Algebra Cognitive Tutor –Part of complete course Students: model problems with diagrams, graphs, equations Tutor: feedback, help, reflective dialog

8. Cognitive Tutor Approach

9. 9 Cognitive Model: A system that can solve problems in the various ways students can Strategy 1: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + ac = d Strategy 2: IF the goal is to solve a(bx+c) = d THEN rewrite this as bx + c = d/a Misconception: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + c = d Cognitive Tutor Technology

10. 10 3(2x - 5) = 9 6x - 15 = 92x - 5 = 36x - 5 = 9 Cognitive Tutor Technology Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d If goal is solve a(bx+c) = d Then rewrite as bx+c = d/a Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

11. 11 3(2x - 5) = 9 6x - 15 = 92x - 5 = 36x - 5 = 9 Cognitive Tutor Technology Cognitive Model: A system that can solve problems in the various ways students can If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction Hint message: “Distribute a across the parentheses.” Bug message: “You need to multiply c by a also.” Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing Known? = 85% chanceKnown? = 45%

12. 12 Cognitive Tutor Course Development Process 1. Client & problem identification 2. Identify the target task & “interface” 3. Perform Cognitive Task Analysis (CTA) 4. Create Cognitive Model & Tutor a. Enhance interface based on CTA b. Create Cognitive Model based on CTA c. Build a curriculum based on CTA 5. Pilot & Parametric Studies 6. Classroom Evaluation & Dissemination

13. Cognitive Tutor Approach

14. 14 Difficulty Factors Assessment: Discovering What is Hard for Students to Learn Which problem type is most difficult for Algebra students? Story Problem As a waiter, Ted gets $6 per hour. One night he made $66 in tips and earned a total of $81.90. How many hours did Ted work? Word Problem Starting with some number, if I multiply it by 6 and then add 66, I get 81.90. What number did I start with? Equation x * 6 + 66 = 81.90

15. 15 Algebra Student Results: Story Problems are Easier! Koedinger, & Nathan, (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences. Koedinger, Alibali, & Nathan (2008). Trade-offs between grounded and abstract representations: Evidence from algebra problem solving. Cognitive Science.

16. 16 Expert Blind Spot: Expertise can impair judgment of student difficulties 0 10 20 30 40 50 60 70 80 90 100 Elementary Teachers Middle School Teachers High School Teachers % making correct ranking (equations hardest)

17. 17 “The Student Is Not Like Me” To avoid your expert blindspot, remember the mantra: “The Student Is Not Like Me” Perform Cognitive Task Analysis to find out what students are like

18. 18 Cognitive Tutor Course Development Process 1. Client & problem identification 2. Identify the target task & “interface” 3. Perform Cognitive Task Analysis (CTA) 4. Create Cognitive Model & Tutor a. Enhance interface based on CTA b. Create Cognitive Model based on CTA c. Build a curriculum based on CTA 5. Pilot & Parametric Studies 6. Classroom Evaluation & Dissemination

19. 19 Tutors make a significant difference in improving student learning! Andes: College Physics Tutor –Field studies: Significant improvements in student learning Algebra Cognitive Tutor –10+ full year field studies: improvements on problem solving, concepts, basic skills –Regularly used in 1000s of schools by 100,000s of students!!

20. 20 Prior achievement: Intelligent Tutoring Systems bring learning science to schools A key PSLC inspiration: Educational technology as research platform to generate new learning science

21. 21 Logic of Pittsburgh Science of Learning Center (PSLC) Support experimental studies that –Test fundamental principles, not whole courses –Are internally & externally valid Create a theory of “robust learning” Leverage technology & computational modeling

22. 22 Overview Background –Intelligent Tutoring Systems –Cognitive Task Analysis PSLC Methods, Resources, & Theory –In vivo experimentation –LearnLab courses –Robust learning theoretical framework –Enabling technologies Summary Next

23. 23 PSLC statement of purpose To yield theoretically sound and useful principles of robust learning, we have created LearnLab to facilitate in vivo learning experimentation. Outline of Overview 1.Robust learning 2.Theory 3.Method 4.LearnLab Next

24. 24 What is Robust Learning? Robust Learning is learning that –transfers to novel tasks –retained over the long term, and/or –accelerates future learning Robust learning requires that students develop both –conceptual understanding & sense-making skills –procedural fluency with basic foundational skills

25. 25 PSLC statement of purpose To yield theoretically sound and useful principles of robust learning, we have created LearnLab to facilitate in vivo learning experimentation. Outline of Overview 1.Robust learning 2.Theory 3.Method 4.LearnLab Next

26. 26 In Vivo Experiments Principle-testing laboratory quality in real classrooms

27. 27 In Vivo Experimentation Methodology Methodology features: What is tested? –Instructional solution vs. causal principle Where & who? –Lab vs. classroom How? –Treatment only vs. Treatment + control Lab experiments Design research & field trials Instructional solution Classroom Lab Causal principle What is tested? Where? Generalizing conclusions: –Ecological validity: What instructional activities work in real classrooms? –Internal validity: What causal mechanisms explain & predict?

28. 28 Methodology features: What is tested? –Instructional solution vs. causal principle Where & who? –Lab vs. classroom How? –Treatment only vs. Treatment + control Lab experiments Design research & field trials In Vivo Experimentation Methodology In Vivo learning experiments Instructional solution Causal principle What is tested? Where? Generalizing conclusions: –Ecological validity: What instructional activities work in real classrooms? –Internal validity: What causal mechanisms explain & predict? Classroom Lab

29. 29 PSLC statement of purpose To yield theoretically sound and useful principles of robust learning, we have created LearnLab to facilitate in vivo learning experimentation. Outline of Overview 1.Robust learning 2.Theory 3.Method 4.LearnLab Next

30. 30 LearnLab A Facility for Principle-Testing Experiments in Classrooms

31. 31 LearnLab courses at K12 & College Sites 6 + cyber-enabled courses: Chemistry, Physics, Algebra, Geometry, Chinese, English Data collection –Students do home/lab work on tutors, vlab, OLI, … –Log data, questionnaires, tests  DataShop Researchers Schools Learn Lab Chemistry virtual lab Physics intelligent tutor REAP vocabolary tutor

32. 32 PSLC resources enable theoretical development …

33. 33 PSLC statement of purpose To yield theoretically sound and useful principles of robust learning, we have created LearnLab to facilitate in vivo learning experimentation. Outline of Overview 1.Robust learning 2.Theory 3.Method 4.LearnLab Next

34. 34 Theoretical Framework Levels Macro level –What instructional principles explain how changes in the instructional environment lead to changes in robust learning? Micro level

35. 35 Explanation at the macro-level Novice knowledge state Learning Processes Instructional Method I T vs. I C Pre-test Accelerated future learning TransferLong-term retention Robust Learning measures: Expert knowledge state Normal Post-test Why is instruction I T better than I C ?

36. 36 Hausmann & VanLehn 2007 study: Macro level description Research question: Does providing explanations or eliciting “self- explanations” from students better enhance robust learning? General instruction: Students alternate between –Watching videos of worked examples of physics problems –Solving new problems in the Andes intelligent tutor Treatment variables: –Videos include justifications for steps or do not –Students are prompted to “self-explain” or paraphrase

37. 37 Self-explanations => greater robust learning Justifications: no effect! Immediate test on electricity problems: Transfer to new electricity homework problems Instruction on electricity unit => accelerated future learning of magnetism!

38. 38 Key features of H&V study In vivo experiment –Ran live in 4 physics sections at US Naval Academy –Principle-focused: 2x2 single treatment variations –Tight control manipulated through technology Use of Andes tutor => repeated embedded assessment without disrupting course Data in DataShop (more later)

39. 39 Theory Integration Strategy PSLC has supported some 130+ studies How do they fit together?

40. 40 Robust Learning Sense- Making Foundational Skills Macro-level of robust learning framework: Taxonomy of outcomes, processes, & treatments Outcomes: Instructional Treatments: Collaboration, tell vs. ask, question insertion Visual-verbal coordination, example-rule coordination Explicit instruction, learner control … Schedules, … Construction, elaboration, discrimination Co-Training Refinement of Features Streng- thening Learning Processes:

41. 41 Robust Learning Sense- Making Foundational Skills 3 Original Research clusters Outcomes: Instructional Treatments: Collaboration, tell vs. ask, question insertion Visual-verbal coordination, example-rule coordination Explicit instruction, learner control … Schedules, … Construction, elaboration, discrimination Co-Training Refinement of Features Streng- thening Learning Processes: Interactive Communication Coordinative Learning Fluency and Refinement

42. 42 Robust Learning Sense- Making Foundational Skills Hausmann & VanLehn in Interactive Communication Cluster Outcomes: Instructional Treatments: Collaboration, tell vs. ask, question insertion Explicit instruction, learner control … Schedules, … Visual-verbal coordination, example-rule coordination Construction, elaboration, discrimination Co-Training Refinement of Features Streng- thening Learning Processes: Interactive Communication Coordinative Learning Fluency and Refinement

43. 43 Interactive Communication Research Question What properties of interactive communication promote robust learning? –20+ projects/studies have addressed versions of this question

44. 44 Use of wiki toward macro- level theory development Experimenters & instructors: Create study summary pages on wiki Work out shared meanings of terms –Like, what does “self-explanation” mean? –Create & edit glossary pages in the wiki Seek out commonalities with others –Tie their hypotheses to general principles –Create shared instructional principle pages

45. 45 PSLC wiki supports theory integration Links to H&V study page: Interactive Communication cluster page:

46. 46 PSLC wiki supports theory integration 182 concepts in glossary Self-explanation glossary entry

47. 47 PSLC wiki supports theory integration Points back to Hausmann’s study page Instructional Principle pages unify across studies

48. 48 Theoretical Framework Levels Macro level –What instructional principles explain how changes in the instructional environment lead to changes in robust learning? Micro level –Can learning be explained in terms of what knowledge components are acquired at individual learning events?

49. 49 Key Micro-level Concepts Knowledge component (unobservable) –A mental structure or process that a learner uses to accomplish steps in a task or a problem Learning event (unobservable) –Points in time where knowledge components are learned or used Instructional event (observable) –Element of learning environment designed to evoke a particular learning event –P(Learning_event | Instructional_event) < 1.0 Learner factors like metacognition & motivation affect this conditional probability

50. 50 Kinds of Knowledge Components (KCs) Mental representations of: Domain knowledge –Facts, concepts, principles, rules, procedures, strategies Prerequisite knowledge –Feature encoding knowledge Integrative knowledge –Schemas or procedures that connect other KCs Metacognitive knowledge –About knowledge or controlling use of knowledge Beliefs & interests –What one likes, believes Cross-cutting distinctions –Correct vs. incorrect –Verbal (explicit) vs. non-verbal (implicit) –Probabilistic vs. discrete NOT KCs: Any external representation of knowledge –Textbook descriptions Generic cognitive structures –Working memory Continuous parameters on knowledge representations –Strength, level of engagement, implicit value of a goal, affect

51. 51 Macro-level analysis treats knowledge states & learning processes as black boxes Novice knowledge state Learning Processes Instructional Method I T vs. I C Pre-test Accelerated future learning TransferLong-term retention Robust Learning measures: Expert knowledge state Normal Post-test

52. 52 Macro-level analysis treats knowledge states & learning processes as black boxes Novice knowledge state Learning Processes Instructional Method I T vs. I C Pre-test Accelerated future learning TransferLong-term retention Robust Learning measures: Expert knowledge state Normal Post-test

53. 53 Unpacking learning: 3 kinds of events Novice knowledge state Pre-test Robust Learning measures Expert (desired) knowledge state Normal Post-test Learning Event Instruct i onal Event Assessment event Learning Event Instruct i onal Event Assessment event Learning Event Instruct i onal Event

54. 54 PSLC’s DataShop Vast open data repository, analysis tools, visualizations Generates: –learning curves –statistical model fit (blue line) Based on micro level analysis: –learning event opportunities –Averaged across knowledge components

55. 55 Greatest impact of technology on 21st century education? Benefits to student learning from use of educational technology? Perhaps But my bet is: –Scientific advances coming from data mining of the vast explosion of learning data that will be coming from educational technologies

56. 56 Example1 Example2Example3 Back to H&V study: Micro-analysis Learning curve for main KC Self-explanation effect tapers but not to zero

57. 57 Overview Background –Intelligent Tutoring Systems –Cognitive Task Analysis PSLC Methods, Resources, & Theory –In vivo experimentation –LearnLab courses –Robust learning theoretical framework –Enabling technologies Summary Next

58. 58 PSLC Enabling Technologies Tools for developing instruction & experiments –CTAT (cognitive tutoring systems) SimStudent (generalizing an example-tracing tutor) –OLI (learning management) –TuTalk (natural language dialogue) –REAP (authentic texts) Tools for data analysis –DataShop –TagHelper

59. 59 Summary Learning can be greatly improved! –Address robust learning –Increase certainty –Leverage technology PSLC provides tools & processes to help learning researchers achieve these goals Robust Learning Principles Existing Solo Theories In Vivo Studies Enabling Technology LearnLab Courses

60. 60 END