1. On the Difficulty, and Increasing Importance, of Becoming an Effective Learner—Online and Offline Robert A. Bjork University of California, Los Angeles C ENTER FOR E XCELLENCE IN T EACHING U NIVERSITY OF S OUTHERN C ALIFORNIA Los Angeles, California May 3, 2015

2. The need to manage one’s own learning is increasing More and more learning is happening outside the classroom – And is our responsibility And our ever more complex and changing world requires acquiring new skills and updating old skills – Often, sometimes exclusively, on our own – And across our lifetimes, not simply during the years of formal schooling

3. The opportunities to manage one’s own learning are increasing as well Computer-based technologies provide opportunities to learn on our own – Whether as an adjunct to formal instruction – Or motivated by career changes, hobbies, or new interests The bottom line – Knowing how to learn effectively has never been more important – And the opportunities to do so have never been more available

4. But … becoming metacognitively sophisticated as a learner is no easy task Optimizing self-regulated learning requires: – Managing one’s learning activities effectively Spacing, variation, generation, retrieval practice, … Organizing one’s knowledge, using technology, engaging in cooperative learning, … – Monitoring the state of one’s learning effectively Interpreting the meaning and predictive value of objective indices of performance Interpreting the meaning and predictive value of subjective indices of performance – Controlling one’s learning activities and decisions effectively, given such monitoring

5. Adapted from Nelson and Naren’s (1990) framework for metamemory. From Bjork, Dunlosky, Kornell (2013).

6. Examples of learners being fooled Predicted Actual Predicted Actual Judged Actual Judged Actual Rating of Training Satisfaction Actual Predicted Actual

7. Examples of learners being fooled Predicted Actual Predicted Actual Judged Actual Judged Actual Rating of Training Satisfaction Actual Predicted Actual Generation Interleaving SkillsSpacing Pre-testing Interleaving Concepts Stability Bias

8. Why are learners fooled? (1) Objective indices of performance can be misinterpreted – Learning ≠ performance (For an “integrative review” see Soderstrom & Bjork, 2015, Perspectives on Psychological Science) – What we can observe is performance, what we must infer is learning (2) Subjective indices of performance can be misinterpreted – Perceptual fluency or familiarity – Retrieval fluency—how readily something “comes to mind” – Egocentrism in social communication

9. Why are learners fooled? (3) We can be prone to counter-productive beliefs and attitudes –Errors, rather than being viewed as an essential component of effective training, are assumed to reflect inadequacies of the instructor, the student, or both –Differences in performance across individuals are over-attributed to innate differences in ability; the power of training, practice, and experience is under- appreciated Fixed versus growth mind sets –Efficient learning is easy learning The styles-of-learning idea

10. Interpreting one’s current performance The basic problem: – Conditions of instruction that make performance improve rapidly often fail to support long-term retention and transfer, …whereas – Conditions of instruction that appear to create difficulties for the learner, slowing the rate of apparent learning, often optimize long-term retention and transfer

11. Manipulations that introduce “desirable difficulties” (Bjork, 1994) for the learner Varying the conditions of learning Distributing or spacing study or practice sessions Using tests (rather than presentations) as learning events Providing “contextual interference” during learning (e.g., interleaving rather than blocking practice)

12. 1. Varying the conditions of learning Two examples

13. Varying the conditions of learning; Example: Kerr & Booth (1978) Two age groups: –8-year-olds and –12-year-olds Task: beanbag toss to target on floor (occluded) –Conditions of Practice: Fixed group: All practice at a fixed distance (the criterion distance) Varied group: Practice at criterion distance +/- one foot (i.e., never at the criterion distance) –Final test after practice: Criterion distance

14. Absolute error (inches) on final test (3-foot distance for 8-year-olds) Age of Participant Practice Condition 8 years12 years Fixed (criterion) 8.315.55 Varied (criterion +/- 1 ft) Kerr and Booth (1978): Results

15. Absolute error (inches) on final test (3-feet distance for 8-year-olds) Age of Participant Practice Condition 8 years12 years Fixed (criterion) 8.315.55 Varied (criterion +/- 1 ft) 5.424.63

16. Varying the environmental context of learning (Smith, Glenberg, & Bjork, 1978)   Study in Room A Study in Room A Study in Room B Tested Room C Tested Room C Study in Room A Tested in Room A Tested in Room B

17. 2. Distributing, rather than massing, repeated study sessions: The “spacing effect”

18. Spacing versus massing repeated study or practice sessions: The spacing effect Restudying information or learning procedural or motor skills: Study Test Delayed test

19. Spacing versus massing repeated study or practice sessions: Short retention interval Restudying information or learning procedural/motor skills Study Test Immediate test

20. Example: Baddeley & Longman (1979) Training Schedule 1 x 1 hr2 x 1 hr1 x 2 hr2 x 2 hr Distributing/Spacing of Practice Hours to Learn Keyboard 34.942.643.249.7 Mean Satisfaction Rating 1 (Very Satisfactory) to 5 (Very Unsatisfactory) 2.401.862.001.73

21. 3. Using tests, rather than presentations, as learning events: Testing as pedagogy rather than as assessment

22. Three virtues of testing from a learning standpoint 1.Retrieving information or procedures is a “memory modifier” (Bjork, 1975) – Recalled information/procedures become more recallable in the future than they would have been otherwise; – Recalling is a substantially more powerful as a learning event than is being presented information or shown procedures (inflatable life vest example) – Other Information/procedures associated with the same cues becomes less recallable (retrieval-induced forgetting; Anderson, Bjork, & Bjork, 1994) – Basically, using our memories shapes our memories, and in adaptive ways

23. Three virtues of testing from a learning standpoint 2. Testing provides far better feedback as to what has or has not been learned than does restudying or being shown information or procedures, and … 3. Testing potentiates the effectiveness of subsequent studying, even when few or no correct answers can be produced.

24. Example of the power of tests—versus presentations—as learning events (Roediger & Karpicke, 2006) Participants studied a to-be-learned passage on the sun or on sea otters (about 30 idea units per passage) Two learning conditions of interest –SSSS: four consecutive 5-min study periods –STTT: one study period plus four consecutive tests of recall for the passage (no feedback)

25. Study Test Final Test 5 mins or 1 week 5 mins or 1 week * No feedback given after tests Read entire passage 14.2 times Read entire passage 3.4 times Example: Restudying versus testing as learning events (Roediger & Karpicke, 2006) 5 mins * No feedback given during tests

26. Results

27. Rate: How much will you remember in one week’s time?

28. Overall, tests potentiate subsequent learning, but what about failed tests? (Kornell, Hays, & Bjork, 2009) What happens when learners produce errors on those pre-tests? –Maybe producing errors impairs subsequent learning… –Maybe we become prone to remembering those errors, rather than the correct information or procedures? Example: trivia questions

29. Whale ???? ______ Whale Mammal 8 seconds5 seconds Whale Mammal 13 seconds Do we learn our errors? Is an unsuccessful test better than a presentation?

30. Frog ???? ______ Frog Pond 8 seconds5 seconds Frog Pond 13 seconds To give you a feeling for the procedure

31. Results (Kornell, Hays, & Bjork, 2009)

32. What do people think about errors? (“which helped you remember better?”) (Huelser & Metcalfe, 2012) Even after the final test, (in which they do better on the tested pairs), learners still thought that restudying is better than testing.

33. Semantic activation hypothesis WHALE-? WHALE: MAMMAL DOLPHINOCEAN

34. The benefits of anticipatory errors appear to be very general Many replications in this and other countries of the basic findings using variations of the “Whale- Mammal” procedure Benefits persist over long retention intervals (Kornell, Hays & Bjork, 2009) Holds for facts from an expository passage (Richland, Kornell & Kao, 2009) And has been demonstrated in classroom contexts –(Bjork, Soderstrom & Little, 2015) –“Productive failure” (e.g., Kapur & Bielaczyc, 2012)

35. 3. Introducing “contextual interference”: Interleaving, rather than blocking, the separate components of to-be- learned knowledge and skills

36. Blocked versus random practice; Example: Shea & Morgan (1979)

40. Are learners themselves fooled by their performance during training? (Simon & Bjork, 2001) Day 1: Blocked or random/interleaved practice on three different keystroke patterns Day 2: Criterion test The new feature: During practice and just prior to the day-2 test, participants were asked to predict how well they would do on the day-2 criterion task

41. A very basic point: Learning ≠ performance What we can observe is performance, what we must infer is learning, and the former is an unreliable index of the latter Students and teachers alike become susceptible to illusions of comprehension and learning when performance is interpreted as learning Predicted Actual

42. Advantages of interleaved practice have been replicated with many other skills

43. Blocked versus interleaved practice Ste-Marie, Clark, Findlay, & Latimer (2004)

44. A new direction: The role of interleaving in the learning of perceptual, verbal, and conceptual skills Examples: Learning mathematical procedures and learning categories and concepts

45. Wedge V = Spheroid V = Spherical Cone V = Half Cone V= Does interleaving improve learning formulas? (Rohrer & Taylor, 2007)

46. Mixed Blocked Practice Final Test

47. Email from William Emeny, a teacher in England “After learning about desirable difficulties, in particular spacing and interleaving, I started trying to put them into practice. I did this in a number of ways. “Firstly, I realised we did interleaving as standard practice in our department, but only during the last 6 months of the course. … I wanted to check the impact of this [interleaving] and so I looked into our assessment data. –“We … assess students approximately 3 times per year and so I looked at the mean grade of students throughout the 3 year course. –“This led to the graph … referred to in [my] email, which shows the impact of the interleaving:

49. Optimizing inductive learning The ability to generalize concepts and categories through exposure to multiple exemplars.

50. Gentoo

51. Where’s the Gentoo?

52. Hypothesis: Inductive learning is one instance in which spacing is not ideal Blocking/massing allows the learner to notice characteristics that unify a category Gentoo

53. Lachesis ReinhardGentoo Whereas interleaving makes doing so difficult

54. Design (Kornell & Bjork, 2008) Paintings shown one at a time Study phase: 72 paintings –12 artists, 6 paintings by each Half the artists: 6 paintings shown in succession (“Blocked”) Other six artists: 6 paintings interleaved with paintings by other artists (“Interleaved”} Test phase: 48 new paintings –Participants asked to say which artist painted each new painting

55. Example of a massed block:

56. Lewis

62. Example of an interleaved block

63. Pessani

64. Wexler

65. Schlorff

66. Stratulat

67. Hawkins

68. Mylrea

69. Lewis PessaniStratulatSchlorffWexlerJurasMylreaPessaniStratulatSchlorffWexlerJurasMylreaHawkins M S S M M S S M M S S M

70. Feedback Test

71. Kornell & Bjork (2008) Results Replications with other categories: Butterflies (Birnbaum, Kornell, Bjork, & Bjork, 2013) Birds (Wahlheim, Dunlosky, & Jacoby, 2011) Categories of novel objects (Vlach, Ankowski, & Sandhofer, 2012) Women’s voices (Yan, Vetter, & Bjork, 2012) Non-parametric statistics (Sana, Yan, Kim, Bjork, & Bjork, 2014) Replications with other populations of learners Older learners (Castel, Kornell, Eich, & Bjork, 2010) Three-year-olds (Vlach, Sandhofer, & Kornel, 2008)

72. Results ActualResponses

73. Why is the judgment that blocking enhances inductive learning so difficult to overcome? Blocking provides a sense of fluency; interleaving a sense of confusion and difficulty The participants come to the experiment with a belief that blocking is better – Blocking is everywhere in the real world (the participants’ teachers blocked instruction, e.g.) – Recent evidence (Tauber, Dunlosky, Rawson, Wahlheim, & Jacoby, 2012): 8 families of birds (warblers, finches, etc.); 6 exemplars of each (48 photos in total) “Do you want to study a bird from the same family or a different family?”

75. What does it take to overcome learners’ belief that blocking, not interleaving, enhances learning? (Yan, Bjork, & Bjork, in press) Prior research findings (Koriat & Bjork, 2006) suggest that overcoming such metacognitive illusions requires both – Experience-based debiasing – Theory-based debiasing What mixture of experience and/or theory can overcome learners’ belief that blocking is optimal for learning?

76. Metacognitive Judgment: What schedule led to better learning? Blocked, interleaved or equal? What does it take to overcome participants’ preference for blocking? Study 6 exemplars by each of 12 artists (half blocked; half interleaved) Test (4 new exemplars per artist): Identify artist, receive feedback [None] 90% do better interleaved 90% + even though massing feels easier 90% + even though massing feels easier + because intermixing enhances discrimination

77. To what do participants attribute the interleaving benefit? Participants presented 6 paintings by each of 12 artists – Paintings by 6 of the 12 artists presented blocked; – Paintings by the other 6 artists presented interleaved. Final test: 24 new paintings (“who, among the artists you studied, painted this painting?”) Post-test survey – Participants given their true scores: “You got X/24 of the blocked artists correct, and you got Y/24 of the interleaved artists correct. You did better on the [blocked/interleaved] artists.” – And then asked the following questions

78. Post-test questions What was the biggest reason for your better performance in the [blocked/interleaved] condition? – I made lucky guesses – That particular set of artists was easier to learn – That schedule is more effective Imagine you were an art teacher: What schedule would you choose to use in teaching your students?

79. Classification performance N = 75 p <.001 d =.53 Number of participants for whom... I > B: 45 (average difference =.24) B > I: 20 (average difference =.13) B = I: 10

80. To what did participants attribute their performance? χ 2 (1) = 5.41, p =.02

81. What schedule would you use to teach your students? Metacognitive Judgment Actual PerformanceBlockedInterleaved Blocked > Interleaved (n = 20) 191 Equal (n = 10) 91 Interleaved > Blocked (n = 45) 2223

82. What do students already know, and not know, about how to study? Survey of 472 introductory psychology students (Kornell & Bjork, 2007)

83. How do you decide what to study next? 59%Whatever's due soonest/overdue 4%Whatever I haven't studied for the longest time 4%Whatever I find interesting 22%Whatever I feel I'm doing the worst in 11%I plan my study schedule ahead of time, and I study whatever I've scheduled

84. If you quiz yourself while you study (either using a quiz at the end of a chapter, or a practice quiz, or flashcards, or something else), why do you do so? 18%I learn more that way than I would through rereading 68%To figure out how well I have learned the information I’m studying 4%I find quizzing more enjoyable than rereading 9%I usually do not quiz myself

85. Do you usually return to course material to review it after a course has ended? 14%Yes 86%No

86. When you study, do you typically read a textbook/article/other source material more than once? 16%Yes, I re-read whole chapters/articles 60%Yes, I re-read sections that I underlined or highlighted or marked 23%Not usually 16%Yes, I re-read whole chapters/articles

87. 20%Yes 80%No Would you say that you study the way you do because a teacher (or teachers) taught you to study that way?

88. Finally: The importance of avoiding “egocentrism” in social communication We are all vulnerable to such egocentrism, also called “the curse of knowledge” –Which leads us to over-estimate how much of what we are saying our students, friends, and colleagues is actually understandable Newton’s (1990) study as a parable of teaching.

91. Piaget (1962) “Every beginning instructor discovers sooner or later that his first lectures were incomprehensible because he was talking to himself, so to say, mindful only of his point of view. He realizes only gradually and with difficulty that it is not easy to place one’s self in the shoes of students who do not yet know about the subject matter of the course.”

92. The end

93. Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417-444. Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. In M. A. Gernsbacher, R. W. Pew, L. M. Hough, & J. R. Pomerantz (Eds.), Psychology and the real world: Essays illustrating fundamental contributions to society (pp. 56-64). New York: Worth Publishers. Soderstrom, N., & Bjork, R. A. (2015). Learning versus performance: An integrative review. Perspectives on Psychological Science, 10, 176-199. Relevant readings