2. 1 In-vivo research on learning Charles Perfetti PSLC Summer School 2009

3. PSLC summer school 2009 2 In-vivo experiments In Vitro In Vivo

4. PSLC summer school 2009 3 Features of in-vivo experiments in learning “On-Line” course? An Intelligent Tutoring System? A real class; real students; an intervention that counts.

5. PSLC summer school 2009 4 The value of in-vivo experiments in learning Noisy, uncontrolled environment Content of intervention is validated by course goals So: Built in generalization to classroom learning

6. PSLC summer school 2009 5 Problems faced by an in-vivo researcher Noisy, uncontrolled environment As for your experiment: Students have other things to do Instructors have other things to do

7. PSLC summer school 2009 6 Examples of in-vivo studies Algebra, Physics, Chemistry, Geometry, French, Chinese,English Some with computer tutors in major role ITS Practice tutors Some without tutors or tutors in minor role

8. PSLC summer school 2009 7 Pre-requisites for an in-vivo experiment Knowledge components analysis Mapping of KCA to a learning or instructional hypothesis Theory based Empirical precedent Mapping instructional hypothesis to specific intervention

9. Knowledge Components vs. curriculum topic Single Topic (Area) as unit 12 separate KCs as units Enabled by Data Shop

10. Mapping a KCA onto an instructional hypothesis The case of Chinese characters PSLC summer school 2009 9 zao3 Whole Character = early morning Radical = sun

11. Mapping an instructional hypothesis to an instructional intervention Learning event space PSLC summer school 2009 10

12. 11 Instructional Event Space Learning Events Instructional Events Assessment Events Performance Explicit or implicit Focus on Valid Features Make Knowledge Accessible Promote Active Processing Schedule events effectively Coordinate multiple events

13. 12  Knowledge Components Analysis 2 (+2) Knowledge Components: 1. the character as a whole; (plus its meaning) 2. the radical that is part of the character (plus its meaning)  Two approaches based on this analysis  (1) Dunlap, Liu, & Perfetti; (2) Pavlik  Two different Instructional Events manipulations  Illustrate 1 here: Feature focus 1. Learning meanings of Chinese characters zao3 Whole Character = early morning Radical = sun

14. 13 Instructional Event Space Associate character form with meaning Assessment Events Performance Whole Character means x Default (typical) Instructional event Early morning

15. 14 Associate radical with x’ and whole character with x Part of character means x’ Assessment Events Performance Dunlap et al: Instructional event manipulation: semantic radical instruction Early morning Highlighted radical = sun/day Instructional Event Space

16. 15 Learning English Spelling (Background knowledge and feature focusing themes) Dunlap, Juffs, Friedline, Perfetti

17. KC analysis of English spelling phonology—orthography /breit/--brate /hiyl/--heel /hiyl/--heal So: phonology-semantics-orthography 16

18. 17 Feature focusing interventions 130 students in levels 3 4, & 5 Interventions: “Pure” feature focus: form only (pronunciation-spelling pairs) Meaning mediated focus: form + meaning (pronunciation-meaning-spelling triads) 7 sessions, 30 minutes per session over 7 weeks

19. Dunlap, Juffs, Friedline, Perfetti

20. PSLC summer school 2009 19 Learning Measures Across-session error rates (transfer to new items) Post-test tone judgments presented by tutor Two successive syllables heard. Are they same or different in tone? (transfer to different task) Nature of syllable pairs Tone same, segments different /duan/3 /liang/3 Same onset and rime, shi2 -- shi3; Share rime only, e.g. dao2 – kao3; Share neither onset nor rime, e.g., duo2 -- gong3.

21. PSLC summer school 2009 20 Studies with major role for a computer tutor Formative evaluation. How can the tutor be improved? Summative evaluation. Is the tutor effective? Both of these apply to all instructional interventions, whether tutor based or not

22. PSLC summer school 2009 21 Formative Evaluation Examples User interface testing Early, before the rest of the tutor is built Engage students and instructors Get detailed response from students viewing tutor with talk-aloud procedures Wizard of Oz Human (the Wizard) in the next room watches a copy of screen Responds when student presses Hint button or makes an error User interface evaluation Does the wizard have enough information? Can the wizard intervene early enough? Tutor tactics evaluation. What did the Wiz do when?

23. PSLC summer school 2009 22 Formative Example 3: Snapshot critiques Procedure: ITS log file Select student help events from log file Experts examine context leading up to the help message noting the help they would provide Examine match between help from experts and that from ITS. Compare with match between two experts. Modify ITS help messages according to reliable expert input.

24. PSLC summer school 2009 23 Summative evaluations Question: Is the tutor (or other instructional intervention) more effective than a control? Typical design Experimental group gets the instructional intervention (the tutor). Control group learns via the “traditional” or “current practice” method Pre & post tests Data analysis Did the tutor group “do better” than the control?

25. PSLC summer school 2009 24 Control conditions for in-vivo experiments Typical control conditions Existing classroom instruction Textbook & exercise problems For cog tutors: Another tutoring system Human tutoring A control intervention; 2 plausible interventions—which is more effective

26. PSLC summer school 2009 25 Learning Assessments 1. Immediate Learning 2. Long-term retention 3. Transfer Over content, form, testing situations 4. Accelerated Future Learning New content; learning measure

27. June 2009 NSF Site Visit Instructional Event Space Learning Events Instructional Events Assessment Events Performance Explicit or implicit Focus on Valid Features Make Knowledge Accessible Promote Active Processing Schedule events effectively Coordinate multiple events Learning Long term retention Transfer Accelerated future learning

28. PSLC summer school 2009 27 Transfer illustrated: Liu, Wang, Perfetti Chinese tone perception study In-vivo study Traditional classroom (not online) Materials from students’ textbook New materials each week for 8 weeks of term 1 Term 2 continued this, and added novel syllables unfamiliar to the student 3 instructional conditions tone number + pin yin, contour + pin-yin; contour only Hint system (CTAT) Tutors presented materials in 3 different instructional interfaces, according to the 3 conditions Data shop logged individual student data

29. PSLC summer school 2009 28 Illustration of 2 conditions from Liu et al shi

30. PSLC summer school 2009 29 Data from Liu et al tone study Learning curves week-by-week

31. PSLC summer school 2009 30 Multiple kinds of transfer Liu et al shows 2 kinds of materials transfer Within term 1, learning sessions, each syllable to be learned was different but familiar. So transfer of learning to familiar items At second term, there were unfamiliar syllables. So transfer of learning to unfamiliar items. (Not so good.)

32. PSLC summer school 2009 31 Example of acceleration of future learning (Min Chi & VanLehn) First probability, then physics. During probability only, Half students taught an explicit strategy Half not taught a strategy (normal instruction) PrePost Probability Training Score PrePost Physics Training Score Accelerated future learning Ordinary transfer

33. PSLC summer school 2009 32 Creating assessments General strategy: Guided by cognitive task analysis (pre-test as well) including learning goals and specific knowledge components Include some items from the pre-test Check for basic learning Some items similar to training items Measures near-transfer Some problems dissimilar to training problems Measures far-transfer

34. PSLC summer school 2009 33 Mistakes to avoid in test design Tests that are Too difficult Too easy Too long Tests that Fail to represent instructed content Missing content; over sampling from some content Depend too much on background knolwedge Notice problems in test means Notice variances

35. PSLC summer school 2009 34 Interpreting test results as learning Post-test in relation to pre-test. 2 strategies: ANOVA on gain scores First check pre-test equivalence Not recommended if pre-tests not equivalent Pre-test, post test as within-subjects variable (t-tests for non-independent samples) ANCOVA. Post-tests scores are dependent variable; pre-test scores are co-variate

36. PSLC summer school 2009 35 Plot learning results Bar graphs for instructional conditions Differences due to conditions Learning Curves Growth over time/instruction

37. PSLC summer school 2009 36 Bar graphs (with error bars!)

38. PSLC summer school 2009 37 Learning Curves Weekly sessions over 2 terms Error rate

39. PSLC summer school 2009 38 Learning Curves Weekly sessions over 2 terms Error rate

40. A final word on experiments In-vivo limitations The role of (in-vitro) laboratory studies

42. 41 The end