1. Northwestern University
Bilateral semantic processing: Inferences in language, insight in problem solving
Mark Jung-Beeman
Northwestern University
Department of Psychology
Neuroscience Institute
Cognitive Brain Mapping Group \

2. Northwestern University Drexel University
Bilateral semantic processing: Inferences in language, insight in problem solving
Northwestern University Drexel University
Zoe Clancy John Kounios
Jason Haberman (UCDavis) Debbie Green
Sandra Virtue (Depaul U) Jennifer Frymiare (U Wisc)
Stella Arambel (deceased) Jessica Fleck
Dianne Patterson Richard Greenblatt
Todd Parrish
Paul Reber Bar-Ilan University
Terri Swan Miriam Faust
Karuna Subramaniam Nira Mashal
Ed Bowden
Research sponsored by NIDCD/NIH

3. Bilateral semantic processing: Inferences in language, insight in problem solving
OUTLINE:
• Drawing inferences from stories -- bilateral comprehension
• Three bilateral component semantic processes (to start)
• Insight -- bilateral, parallel processing during problem solving

4. Semantic activation - “Wernicke’s area”
Bilateral Activation, Integration, and Selection model of semantic processing
Semantic activation - “Wernicke’s area”
Bottom-up lexical-semantic activation: index of semantic representations (pMTG)
Semantic integration - anterior Sup. Temp. Gyrus
Compute semantic overlap - detect or generate (aSTG)
Semantic selection - Inf. Frontal Gyrus
Select among competing activated concepts (IFG)

6. Bilateral semantic processing: Inferences in language, insight in problem solving
OUTLINE:
• Drawing inferences from stories -- bilateral comprehension
• Three bilateral component semantic processes (to start)
• Insight -- bilateral, parallel processing during problem solving

7. Problems with view that language is purely a LH function
General anatomical symmetry
RH damaged patients - some language problems
Recovery from aphasia, hemispherectomy, callosotomy
Neuroimaging - always some RH signal, some tasks RH>LH
Some tasks lvf-RH better than rvf-LH

8. Brain bases of comprehension
of natural language
Natural language, stories, discourse
• Higher level semantic processing (plus all lower levels)
As language input more complex (and natural):
• More anterior temporal lobes
• More bilateral processing

9. Brain bases of cognitive processes when
people draw inferences from stories
Causal bridging (coherence) inferences
“Before going to the wedding, John was sitting around in his jeans, so he went to his bedroom to find some clothes.”

10. Brain bases of cognitive processes when
people draw inferences from stories
Causal bridging (coherence) inferences
“Before going to the wedding, John was sitting around in his jeans, so he went to his bedroom to find some clothes.
He came out wearing his tuxedo, which had belonged to John's father, but looked like new.”

11. Brain bases of cognitive processes when
people draw inferences from stories
Causal bridging (coherence) inferences
“Before going to the wedding, John was sitting around in his jeans, so he went to his bedroom to find some clothes.
He came out wearing his tuxedo, which had belonged to John's father, but looked like new.”
CHANGE

12. Brain bases of cognitive processes when
people draw inferences from stories
We know people make such causal inferences
We know a lot about other types of inferences that people make - types of text, motivation, knowledge, capacity
We still don’t know much about component processes that support this seemingly complex behavior

13. RH semantic processing and inferences
RHD patients have difficulty drawing inferences
• Answer questions about inferable events less accurately than control subjects; intact on explicitly stated facts
(Brownell et al., 1986; Beeman, 1993)
• Do not show inference-related priming; control subjects do
(Beeman, 1993)

14. Proposed component processes of inference generation
1) Activation / integration (detect overlap)
2) Selection
3) Incorporation / integration (map overlap)
Hemispheric cooperation
RH activates information that may support inferences. Weak activation not reach consciousness.
I interpret these results as providing support for multiple component processes involved in drawing inferences.
Optimally drawing inferences requires hemi coop’n… in which the RH

15. Time course of inference related semantic activation in both hemispheres during story comprehension.
“Before going to the wedding, John was sitting around in his jeans,1 so he went to his bedroom to find some clothes.2 After a few minutes,3 he came out wearing his tuxedo,4 which had belonged to John's father5, but was still fashionable and looked like new.”
- CHANGE
(1) and (2): Predictive inference.
(3): Transition.
(4): Coherence or bridging inference.
(5): Resolved and incorporated.

16. Left visual field
Right visual field
Left Hemisphere
Right Hemisphere

17. Priming: Inference faster than Unrel
“Before going to the wedding, John was sitting around in his jeans,1 so he went to his bedroom to find some clothes.2 After a few minutes,3 he came out wearing his tuxedo,4 which had belonged to John's father5, but was still fashionable and looked like new.” Brain and Language, 2000
Priming:
Inference
faster than
Unrel
This graph shows priming….

18. Asymmetric dynamic semantic fields: relatively coarser coding in RH; better selection in LH
Left Hemisphere
Right Hemisphere
RULER
foot
foot
TOES
CUT
Large but weakly activated;
Diffuse, including secondary
and less relevant concepts
- hard to select, output
Small but strongly activated;
Focused on dominant or
contextually relevant concepts
- easy to select, interpret, output

19. RH coarse semantic coding: Increased likelihood of semantic overlap
for distant semantic relations
foot
foot
pain
glass
CUT
CUT
pain
glass

20. Semantic activation - “Wernicke’s area”
Bilateral Activation, Integration, and Selection model of semantic processing
Semantic activation - “Wernicke’s area”
Bottom-up lexical-semantic activation: index of semantic representations (pMTG)
Semantic integration - anterior Sup. Temp. Gyrus
Compute semantic overlap - detect or generate (aSTG)
Semantic selection - Inf. Frontal Gyrus
Select among competing activated concepts (IFG)

21. RH Middle & superior temporal gyrus involved in computing semantic integration
Deriving theme from paragraphs (St. George et al.)
Generating best ending (Kirchner et al.)
Generating inferences? - moderately related sentence pairs (Mason & Just)
Metaphoric over literal sentences (Bottini et al.)
Detecting temporal/emotional inconsistency (Ferstl)
Generating insight solutions (Jung-Beeman et al; Kounios et al)

22. Brain activity when people draw inferences on-line, as indexed by fMRI
Three ways to contrast inference versus no-inference conditions:
- Text: infernce versus no-inference; strong vs. weak constraint
- Individual differences: high versus low Working Memory
- Behavioral measures: recall of inferences
General Results:
Bilateral activity in pMTG; aSTG; IFG
- modulated by constraint, WM, time

23. Brain activity when people draw inferences on-line, as indexed by fMRI
… John was going to a wedding, but he had been sitting around the house in his jeans, so he went to his bedroom to find some clothes. Soon he came out wearing his tuxedo, * …
Explicit:
…went to his bedroom to change his clothes. Soon he came out wearing his tuxedo ,* …
- High baseline, ongoing stories; small input difference

24. Semantic integration at moment of implied events:
Predominantly RH aSTG

25. Semantic integration at event point:
Bilateral anterior Superior Temporal Gyrus
L R
Post Ant L R
L R
Lower (ns) threshold, selected for LH STG

26. Semantic activation and integration at coherence break (“tuxedo”):
Predominantly LH STG

27. Semantic selection: High versus low working memory
High WM (reading span) subs show stronger, earlier evidence of semantic selection of inferences
(St. George et al; many behavioral)
• Completion requires selection, incorporation

28. Semantic selection: Inferior frontal gyrus
Selecting some concepts over competitors
• Usually IFG in LH (Thompson-Schill et al; Barch; Friston)
Some instances, RH IFG
(Seger 2000; Friederici et al., 2000; Jung-Beeman et al.)

29. Semantic selection: Inferior frontal gyrus
Selecting some concepts over competitors
• Usually IFG in LH (Thompson-Schill et al; Barch; Friston)
Some instances, RH IFG
Unusual verb generation (cake -> “decorate”) (Seger 2000)
Repair grammatical errors (Friederici et al., 2000)
Utilize unintended meaning of ambiguous words in sentence
(Jung-Beeman et al.)

30. Semantic selection: fMRI signal in IFG (LH > RH) at coherence break in High WM subs only
(Fig: High WM > Low WM)

31. Replication and extension: Working memory and predictability
Unpredictable inferences: LH activation, IFG, pSTG
• searching for connections
Predictable inferences: Bilateral activation, IFG, pSTG
• building on connections
Higher WM (n=13) > lower WM (n=13):
• facile comprehension

32. Successful integration versus continued activation:
STG in High vs. Low WM subs at coherence break,
Predictable inferences RH
IFG
High WM subs
show bilateral
(stronger in RH)
Low WM show
LH only RH
pSTG
p<.001

33. Replication and extension: Working memory and predictability
Unpredictable inferences: LH activation, IFG, pSTG
• searching for connections
Predictable inferences: Bilateral activation, IFG, pSTG
• building on connections
Higher WM (n=13) > lower WM (n=13):
RH activation, pSTG, IFG, and a little aSTG
• facile comprehension

34. Successful integration versus continued activation:
STG in High vs. Low WM subs at coherence break,
Predictable inferences RH
aSTG
High WM subs
show bilateral
(stronger in RH)
Low WM show
LH only, no aSTG
p<.005

35. Replication and extension: Working memory and predictability
Unpredictable inferences: LH activation, IFG, pSTG
• searching for connections
Predictable inferences: Bilateral activation, IFG, pSTG
• building on connections
Higher WM (n=13) > lower WM (n=13): RH activation, pSTG, IFG, and a little aSTG
• facile comprehension

36. Conclusions about inferences
Semantic integration builds up as story hints that some event might occur: anterior STG; RH (?)
At coherence break: integration and activation (STG), especially in LH
completing the inference requires selection (IFG)
RH contributes to facile inferencing/comprehension, not just kick in when demands are high

37. Current projects, Future directions
Shift semantic distance for integration --> shift hemi asymmetry
Closely tie to behavioral markers of inference activation, selection, incorporation
Recall of inferences √
Priming of inferences
Successful integration versus effort of difficult integration
Incorporation (recall study)

38. Recalled inferences
If inferences recalled, must have been incorporated
Working Memory correlates with
total recall
Recall of inferences
NOT with recall of episodes w/o inferences
Contrast fMRI signal of recalled infs versus recall episode, no infs

39. L R Post Ant L R Inferences recalled versus
Episode recalled, inf not recalled
L R
Post Ant L R
L R R R
Bilateral pMTG, stronger in RH RH aSTS, bilat IFG
p<.005 , positive only

40. So what? Knowing where processing occurs informs and
constrains what and how it occurs

41. Bilateral semantic processing: Inferences in language, insight in problem solving
OUTLINE:
• Drawing inferences from stories -- bilateral comprehension
• Three bilateral component semantic processes (to start)
• Insight -- bilateral, parallel processing during problem solving

42. Semantic activation - “Wernicke’s area”
Bilateral Activation, Integration, and Selection model of semantic processing
Semantic activation - “Wernicke’s area”
Bottom-up lexical-semantic activation: index of semantic representations (pMTG)
Semantic integration - anterior Sup. Temp. Gyrus
Compute semantic overlap - detect or generate (aSTG)
Semantic selection - Inf. Frontal Gyrus
Select among competing activated concepts (IFG)

43. Why does the RH code more coarsely?
Asymmetries in neural microcircuitry

45. Given topographic mapping of brain, broader input/output fields => coarser semantic coding
Left Hemisphere
Right Hemisphere
RULER
foot
foot
TOES
CUT
Large but weakly activated;
Diffuse, including secondary
and less relevant concepts
Small but strongly activated;
Focused on dominant or
contextually relevant concepts

47. RH coarse semantic coding: Increased likelihood of semantic overlap
for distant semantic relations
foot
foot
pain
glass
CUT
CUT
pain
glass

49. Why a separate area for semantic integration?
Could form associations in “activation” area
BUT
Higher level relations, correlated co-occurrence, indirect
Ability to extract, attend to, & manipulate relations
Analogous to individual areas within vision (e.g., motion)

50. Why anterior STS/STG for semantic integration?
Again, neural architecture

51. Patchy organization and multisensory integration
(Beauchamp 2004)
L R
Post Ant L R

53. Why anterior STS/STG for semantic integration?
Again, neural architecture
More anterior = longer intrinsic conxns, better to integrate across patches
RH = longer than LH

55. Important clarifications
Not an “inference area”
Semantic integration - participates in many functions
Not specific to categories of inferences - varies with demand
Tight comparison not reveal whole network
Just areas that differ when storied imply versus explicitly state events
RH and LH cooperate

58. Bilateral semantic processing: Inferences in language, insight in problem solving
OUTLINE:
• Drawing inferences from stories -- bilateral comprehension
• Three bilateral component semantic processes (to start)
• Insight -- bilateral, parallel processing in problem solving

59. Brain bases of insight during problem solving:
Aha! and antecedents
Most problems solved with mix of analytic and insight processing
• Distinct computations, distributed across hemispheres, allows two approaches to proceed simultaneously (partially interactive)
• Hemispheric components, task shielding/switching

60. Archimedes and the crown
King’s crown - gold, or silver
Archimedes knew gold and silver differed in density
Archimedes knew weight, but couldn’t geometrically measure to obtain volume (and compute density)

62. Archimedes and the crown
Why has story persisted so long?

63. Archimedes and the crown
Why has story persisted so long?
• Resonates with our own experiences of
solving insight problems solving problems with insight

64. Archimedes and the crown
Solvers reach impasse (dead-end) - couldn’t measure
Must reinterpret some aspect of problem
Volume by water displacement
Unconscious processing important
If not thinking of crown, how recognize importance of water?
Solution accompanied by “Eureka!”

65. Insight component processes?
Insight solutions associated with
Switching to new strategy or associations (“restructuring”)
Semantic integration -- solvers see connections that previously eluded them
Right hemisphere?

66. Solving problems with insight
Characteristics of both “insight problems” and solving processes similar to characteristics of discourse and comprehension processes for which the Right Hemisphere (RH) seems to make contributions
Drawing inferences, understanding the gist
Getting jokes, metaphors, connotations
2ndary word meanings

67. Solving problems with insight
Solvers reach impasse (dead-end)
Must reinterpret some aspect of problem
Unconscious processing important
Solution accompanied by “Aha!”

68. Short insight problems:
Remote Associates Test: The RAT (Mednick, 1962)
RAT Compound Remote Associate Problems
Bowden & Jung Beeman, 1998
child
tennis
scan
lame
strike
same

69. brain match RAT Compound Remote Associate Problems
Bowden & Jung Beeman, 1998
child
tennis
scan
lame
brain
match
strike
same

71. Aha! experience Solution appears sudden and obvious
As soon as you think of solution, you “just know” it works for all three words
Comes as a whole, not part by part
(vs strategic, step-by-step testing, etc)

72. Event-related fMRI design
Insight solutions versus noninsight solutions
Very “tight” comparison
Not reveal whole network of problem solving
Highlights just components that are uniquely engaged (or at least emphasized) for insight solutions

73. Insight effect in RH anterior Superior Temporal Gyrus:
FMRI signal for insight > noninsight solutions.
L R
Post Ant L R
L coronal R axial sagittal
p < .005, cluster > 500 mm3

74. RH aSTG: Singal change across the active region
Signal change for insight Insight effect
and noninsight solutions (Ins - non)
Percent
Signal
change
Percent signal change
Time (sec)

75. “Best” cluster
within each
hemisphere!!

76. Parallel study with 128 channel EEG
Temporal specificity
Processing specificity - frequencies

77. Gamma band insight effects

81. Insight solving conclusions
Insight solutions associated with increased activity in RH aSTG
Binding and conscious accessibility (gamma) over RH aSTG
Preceded by visual gating (alpha) - RH temp/ occipital areas

82. Insight solving conclusions
Insight solutions associated with increased activity in RH aSTG
Binding and conscious accessibility (gamma) over RH aSTG
- Lexical or semantic integration
Preceded by visual gating (alpha) - RH temp/ occipital areas
- Sensory gating indicates cognitive control?

83. Replication plus… more areas New data set: improved N, scanner, protocol
RH aSTG (distant semantic integration)
Anterior Cingulate (monitoring response competition, switching)
Posterior Cingulate - same?
Hippocampus/parahippocampal gyri - memory, reorgnzn?

84. Insight effect in RH Superior Temporal Gyrus:
FMRI signal for insight > noninsight solutions.
L R
Post Ant L R
L coronal R axial sagittal
p < .001, cluster > 1000 mm3 ant and post STG

85. NONinsight effect in LH Inf. Frontal Gyrus:
FMRI signal for NONinsight > insight solutions.
LH IFG - dominant semantic retrieval
or selection
- turns on at problem onset
- off at solution, esp’y Insight
RH IFG - unusual retrieval / selection
- off at problem onset
- on at solution (I>NI, ns)
sagittal
p < .005, cluster > 1000 mm3

86. General vs specific mechanisms - Visual Aha!

87. Visual Aha! effect in RH anterior Mid Temporal Gyrus:
FMRI signal for insight > noninsight recognition
L R
Post Ant L R
L coronal R axial sagittal
p < .01, cluster > 500 mm3

88. Visual Aha! effect in RH anterior Mid Temporal Gyrus:
FMRI signal for insight > noninsight recognition
L R
Post Ant L R
L coronal R axial sagittal
p < .01, cluster > 500 mm3

89. Visual Aha! effect in RH Angular Gyrus:
FMRI signal for insight > noninsight recognition
L R
Post Ant L R
L coronal R axial sagittal
p < .01, cluster > 500 mm3 Also: RH Sup Frontal Gyrus

90. Visual Aha! effect in Bilateral M. Occipital Gyri:
FMRI signal for NONinsight > insight recognition
L R
Post Ant L R
L coronal R axial sagittal
p < .005, cluster > 500 mm3

91. Visual Aha! conclusions
NOT just for verbal problems
Similarities - shared mechanisms (not “insight”, but…)
Insight: top-down, cognitive control, integration
RH -- unconscious, weak but mutually constraining, integration
Recognition comes as a whole, not part by part
Noninsight: bottom-up
Some differences - Angular Gyrus somewhat surprising

92. General vs specific mechanisms - Visual Aha!

93. Insight solving conclusions
Insight solutions associated with increased activity in RH aSTG
Binding and conscious accessibility (gamma) over RH aSTG
Preceded by visual gating (alpha) - RH temp/ occipital areas

94. Insight solving conclusions
Insight solutions associated with increased activity in RH aSTG
Binding and conscious accessibility (gamma) over RH aSTG
- Lexical or semantic integration
Preceded by visual gating (alpha) - RH temp/ occipital areas
- Sensory gating indicates cognitive control?

95. Insight solving conclusions
Insight solutions associated with
Semantic integration -- solvers see connections that previously eluded them
When “the light goes on…”

96. Semantic activation - “Wernicke’s area”
Bilateral Activation, Integration, and Selection model of semantic processing
Semantic activation - “Wernicke’s area”
Bottom-up lexical-semantic activation: index of semantic representations (pMTG)
Semantic integration - anterior Sup. Temp. Gyrus
Compute semantic overlap - detect or generate (aSTG)
Semantic selection - Inf. Frontal Gyrus
Select among competing activated concepts (IFG)

97. Insight preparation
Do different mental states influence how you solve problems?
Brain activity during a “rest period” (fMRI) or at a “Ready?” prompt (EEG), prior to getting a problem
Problems solved with insight versus without insight

98. Preparation for Insight
Is there a general form of preparation for insight that begins before a problem is presented?
We examined neural activity during the 2 sec immediately before each problem was presented.
Compared neural activity preceding problems solved with insight to activity preceding problems solved without insight.

99. t
Noninsight: More neural activity over right occipital cortex. More attention to the visual stimulus?
Insight: More neural activity over left temporal cortex and medial frontal cortex. Medial frontal cortex is involved in control and mobilization of cognitive resources. Left temporal cortex represents concept. Preparing to retrieve solution from area likely to have the strongest activity. Fits hemispheric theory. Not area where insight occurs.

102. Conclusions Two forms of preparation.
Noninsight: Increased visual attention to displayed problem.
Insight: Mobilization and control of cognitive resources; activation of temporal lobe semantic regions; suppression of irrelevant thoughts.

103. Summary Insight is different from ordinary problem solving.
Insight involves a sudden, discrete, awareness of the solution to a problem.
Insight involves different neural structures and mechanisms.
Insight is the result of a special form of preparation involving cognitive regulation by medial frontal region.

104. Is insight really sudden? Part II: Antecedents of insight
Positive mood facilitates insight and creative problem solving (Isen et al.)

105. Insight and mood Positive mood associated with increased creativity
Better access to more distant associations
Increased cognitive flexibility
Anxiety associated with decreased creativity
narrower focus of attention

106. Positive mood and insight
Positive mood enhances (anxiety impedes):
Total solution rate
% solved with insight
Insight-like preparatory activity in ACC

107. Positive mood modulates prep activity in ACC
Insight >Non
Prep activity
Pos Aff>Neg
in prep activ
Convergence

109. General vs specific mechanisms - Visual Aha!

110. Thank you!