Training complex judgment The effects of critical thinking instruction and contextual interference Anne Helsdingen 1, Tamara van Gog 1, Jeroen van Merriёnboer 1, 2, 1 CELSTEC, 2 Netherlands Laboratory for Lifelong Learning (NeLLL), Open University of the Netherlands

An example

Your task: Judge the priority for the police Which variables to take into account? –E.g. weapon, condition of the victim, intention of perpetrator, any camera pictures,… How do they relate to the priority? –Priority higher when victim is seriously wounded? –Priority lower when there is no description or camara image of perpetrator? First priority estimate is a guess, but then you learn –On the basis of outcome feedback –The relationship between relevant variables and the criterion

Complex judgment tasks Many judgment and decision problems are similar: –Military command and control –Fire commander –Medical diagnosis Studying experienced decision makers revealed: –Recognition based decisions –Meta-cognition skills to monitor decision strategies Critical thinking instruction

Critical thinking instruction (Cohen, Freeman, & Thompson, 1998) 1) Story telling: try to incorporate all observations, all cues in one comprehensive explanation, one story. (templates) 2) Story testing: what is missing, what is conflicting with other information? 3) Story evaluation: the Devil’s Advocate tells you that your critical assumption is not true. Develop an alternative story that als explains all observations. 4) Develop plans and contingencies

Optimizing complex judgment training Does critical thinking instruction lead to better post training performance in complex judgment? How to prepare decision makers for the variety of problems? How to enhance transfer from learning task to other tasks? Manipulating sequence of learning tasks: Random sequence to enhance contextual interference

Contextual Interference Random sequence of learning tasks benefits learning (compared to blocked) In random: Learners compare between different tasks - elaborate In random: learners have to recollect different strategy vs having a strategy ready - reconstruct In blocked: an illusion of competence develops

Research questions Does critical thinking instruction enhance post training performance in complex judgment tasks? Does the contextual interference effect manifest itself in learning complex judgment tasks? What are the combined effects of critical thinking instruction and practice schedule?

Research questions Does critical thinking instruction enhance post training performance in complex judgment tasks? Yes Does the contextual interference effect manifest itself in learning complex judgment tasks? Sure What are the combined effects of and critical thinking instruction and practice schedule?

Critical thinking instruction 2 field experiments (N= 16, N=16) military officers in a command and control training program Scenario based exercises, 6 x 1 hour, critical thinking instruction vs regular instruction manipulated between subjects. Post training test (2x1 hour scenario’s), judgment performance evaluated by blind evaluators Critical thinking improves post training judgment performance Performance difference between experimental and control group was greatest for far transfer test. Conclusion: critical thinking instruction leads to better judgment making strategy and deeper understanding of domain

Effect of practice schedule on learning complex judgment 2 lab experiments (N=54, N=64), University students. Task: estimate the priority (0-100) of a fictituous crime for fictituous police force on the basis of four dichotomous variables Blocked vs random sequence of learning tasks between subjects Results show better retention and transfer for random sequence group No difference in performance during learning

Critical thinking instruction in random and blocked practice Lab experiment (N=120), University students. Task: estimate the priority (0-100) of a case for the police force –Priority based on four dichotomous variables that were present in each case. –Identify the variables and determine relationship with priority –Outcome feedback only 2x3 between subjects design: –Blocked vs random sequence of learning tasks between subjects –Critical thinking prompts: before task, after task and no prompts Performance measurement is difference between real and estimated priority During learning, on near transfer and far transfer

Critical thinking instruction

Critical thinking prompts - examples Are (Were) there any similarities between the following (the last) two cases? Which are (were) they? And what is (was) different between these cases? The following (last) two cases (will) differ (-ed) from the former two (preceding) cases on one specific aspect. Can you tell me what that is (was) ? And does (did) it have an effect on the priority of a case? Do you consider some aspects to be more important than others when estimating a case’s priority?

Hypotheses Transfer paradox: –Blocked practice yields better performance during learning –Random practice leads to better performance on retention and transfer tests Critical thinking prompts enhance transfer test performance: –Blocked practice with proactive prompts yield higher transfer test scores than without or with retrospective prompts –Random practice with retrospecitive prompts yield higher transfer test scores than without or with proactive prompts During a blocked practice, learners develop an illusion of competence –Higher certainty during blocked practice learning than during random practice –This is counteracted by critical thinking: blocked practice with proactive promts lower certainty than without prompts

Performance Performance during learning –Blocked practice showed better performance during learning phase –In random practice, retrospective prompts yielded better performance Performance on near transfer –Random practice resulted in better performance than blocked –After random practice with refrospective prompts better performance Performance on far transfer –After random practice better far transfer performance –After random with retrospective prompts best scores –After blocked practice with proactive prompts performance at similar level as after random practice

Certainty Certainty during learning –No effect of practice schedule –In blocked practice, proactive prompts resulted in higher certainty –In random practice, no prompts condition yielded higher certainty Certainty on near transfer –After random practice higher certainty (+performance ) –After blocked practice with proactive prompts higher certainty (+performance) Certainty on far transfer –Higher certainty after random practice schedule (+performance) –Higher certainty after blocked with pro-active prompts (+performance)

Discussion & conclusions √ Transfer paradox: –Blocked practice yields better performance during learning √ –Random practice leads to better performance on transfer tests √ Critical thinking prompts enhance transfer test performance: –Blocked practice with proactive prompts yield higher transfer test scores than blocked practice without or with retrospective prompts –Random practice with retrospecitive prompts yield higher transfer test scores than random practice without or with proactive prompts ? During a blocked practice, learners develop an illusion of competence – Higher certainty during blocked practice learning than during random practice – This is counteracted by critical thinking: blocked practice with proactive promts lower certainty than without prompts

How about…. ….critical thinking? Better judgment strategy & deeper understanding of underlying mechanisms What aspects support learning: reflection, self-explanation, ….. …contextual interference? Benefits learning complex judgment: deeper understanding Extra processing or illusion of competence ? ….combining the two? Timing is everything Then even blocked practice may be adequate for learning It doesn’t hurt to try …future research Are people capable to judge their own learning in these complex tasks? How can we measure judgments of learning continuously and unobtrusively?