1. Neuropsychological Evidence for Category Structure Then: The Functional Role of Mental Imagery
Psychology 355: Cognitive Psychology Instructor: John Miyamoto 05/19/2016: Lecture 08-4
Note: This Powerpoint presentation may contain macros that I wrote to help me create the slides. The macros aren’t needed to view the slides. You can disable or delete the macros without any change to the presentation.

2. Lecture probably ends here
Outline
Neuropsychological evidence for category representations.
Mental Imagery
Imagery Debate – Do mental images play a functional role in human cognition?
Evidence in favor of the functional role of imagery.
Are perception and imagery similar processes?
Behavioral evidence
Neuropsychological evidence
Lecture probably ends here
What Is the Neural Representation of Category Knolwedge?
Psych 355, Miyamoto, Spr ‘16

3. What Is the Neural Representation of Category Knowledge?
Are there object-specific or person-specific neural representations?
Are there specific neurons that are sensitive to specific objects or persons? Are there grandmother cells?
How is knowledge of categories of objects represented at the neural level?
Category Specific Neurons
Psych 355, Miyamoto, Spr '16

4. Evidence for Category-Specific Neurons
Kreiman, G., Koch, C., & Fried, I. (2000).
Single-cell recordings from 11 epilepsy patients awaiting surgery.
Neurons found in the temporal lobe that respond best to specific classes of objects.
Category-specific neurons for:
faces;
famous faces;
animals;
cars;
buildings;
spatial layouts;
abstract patterns
Multimodal Category Representations
Psych 355, Miyamoto, Spr '16

5. Multimodal Category Representations
Category knowledge includes knowledge of ....
an object's visual appearance, e.g., what a dog looks like;
typical sounds, e.g., what kinds of sounds you would expect from a dog;
function, e.g., what you would use a hammer for;
smell, e.g., what a dog smells like; what a flower smells like; what a fire smells like;
taste, especially true of foods
typical body actions that relate to it, e.g., we have a representation of the body actions with respect to a hammer, a chair, a chest of drawers
Multimodal representations involve multiple sensory modes, vision, audition, touch, kinesthesis, and so forth.
Example: The Grasping Circuit
Psych 355, Miyamoto, Spr '16

6. Example of a Multimodal Category Representation: The Grasping Circuit for Manipulable Objects
The Grasping Circuit – a neural circuit associated with use of manipulable objects, like hammers, screwdrivers, tennis rackets, etc.
Involves pathways in parietal cortex
Question: Are the neural circuits involved in handling a hammer part of the category representation of "hammer"?
fMRI Images While Viewing Hammers, Buildings, Animals & People,
Psych 355, Miyamoto, Spr '16

7. Neural Evidence for Multimodal Representations of Category Knowledge
Hammer use activates the left ventral premotor cortex & left posterior parietal cortex (grasping circuit).
Subjects are immobile in the scanner so they could not grasp anything.
Nevertheless, viewing a hammer activated the grasping circuit.
Supports the hypothesis that we activate object-appropriate motor association areas when we access category knowledge.
Supports the view that category knowledge is multimodal.
Psych 355, Miyamoto, Spr '16
Summary: Neural Representations of Category Knowledge – END

8. Summary – Neural Representations of Category Knowledge
Some neurons may be specific to particular people or objects. (Controversial Issue)
Some neural mechanisms are specific to particular kinds of objects.
Not necessarily single neurons – the mechanism may involve distributed processing.
Neural category representations are multimodal.
As yet, we only have a preliminary understanding of neural representations of category knowledge. This is a good field for someone with a scientific pioneering spirit.
Visual Images - What Are They?
Psych 355, Miyamoto, Spr '16

9. Visual Images – What Are They?
Mental Imagery: Experiencing a sensory impression in the absence of sensory input
Visual imagery: “seeing” in the absence of a visual stimulus
Table Showing the Pro and Con View of the Functional Importance of Imagery
Psych 355, Miyamoto, Spr '16

10. The Debate Over Mental Imagery
Pro-Mental Imagery
Anti-Mental Imagery
Mental images play a functional role in human cognition – they play an important role in memory, problem solving and reasoning.
Advocated by Stephen Kosslyn
Mental images are epiphenomenal;
Mental images accompany real mechanism but plays no functional role.
Cognitive theory does not need a theory of mental images.
Advocated by Zenon Pylyshyn
Human cognition makes use of mental images and propositional representations.
All cognitive representations are propositional representation.
Same Slide with Emphasis Rectangles
Psych 355, Miyamoto, Spr '16

11. The Debate Over Mental Imagery
Pro-Mental Imagery
Anti-Mental Imagery
Mental images play a functional role in human cognition – they play an important role in memory, problem solving and reasoning.
Advocated by Stephen Kosslyn
Mental images are epiphenomenal;
Mental images accompany real mechanism but plays no functional role.
Cognitive theory does not need a theory of mental images.
Advocated by Zenon Pylyshyn
Human cognition makes use of mental images and propositional representations.
All cognitive representations are propositional representation.
Review: Basic Pattern of Argument in the Imagery Debate
Psych 355, Miyamoto, Spr '16

12. Basic Approach to Arguing for the Functional Role of Mental Imagery
Find behavioral or neuropsychological evidence that is ...
easy to explain if we assume that humans possess and use perception- based representations, but ...
hard to explain if we assume that we have only propositional representations.
In general, these arguments are not air tight, but they can be strong and convincing.
Example of Ptolmaic circles in astronomy
Related Hypothesis: Humans use analog representations. I.e., Humans use representations that are analogous to working with real physical objects.
Review: Shepard's Mental Rotation Study
Psych 355, Miyamoto, Spr '16

13. Shepard's Mental Rotation Experiments
TASK: As quickly as possible, decide whether the two figures have the same shape or different shapes.
Mental rotation experiments: Influential argument for the importance of mental imagery in cognitive processes.
Review: Mental Rotation Experiment - Results
Psych 355, Miyamoto, Spr '16

14. Mental Rotation - Results
2-D Rotation
3-D Rotation
Angle of Rotation (Degrees)
Mean Reaction Time (sec.)
Response time for "identical" figures is a linear (straight-line) function of the angle of rotation between the figures.
Result is easy to explain if subjects are rotating a mental image.
Result is hard to explain if mental representation is exclusively propositional.
Image Scanning – Kosslyn's Experiment with Island Map
Psych 355, Miyamoto, Spr '16

15. Mental Scanning (Image Scanning)
Distance (cm)
Response Time (sec)
Figure 10.4: Image scanning is discussed in Goldstein, pp
Subjects study the map of the island. Then take it away.
Using only the mental image of the island, imagine a dot moving from one point to another, e.g., from the mountain to the tree. Push a button when you are done.
Zooming In or Zooming Out on a Mental Image
Psych 355, Miyamoto, Spr '16

16. Zooming In and Zooming Out on a Mental Image
Top: Imagine a rabbit next to an elephant.
Yes or No: Does the rabbit have whiskers?
Bottom: Imagine a rabbit next to a fly.
Same Slide with Results Added
Psych 355, Miyamoto, Spr '16

17. Zooming In and Zooming Out on a Mental Image
Top: Imagine a rabbit next to an elephant.
Yes or No: Does the rabbit have whiskers?
Bottom: Imagine a rabbit next to a fly.
RT result is hard to explain if you assume that information is retrieved from a semantic network without imagery.
RT = 2,020 ms slower
RT = 1,870 ms faster
Can Imagery Prime Perception
Psych 355, Miyamoto, Spr '16

18. Can Imagery Prime Perception?
Priming – Stimulus A can prime Stimulus B if A is similar to B.
Why? Theoretical reason is that A and B activate similar brain areas or processes, so the recent activation by A facilitates the processing of Stimulus B.
Question: Can imagining a visual representation prime the perception of a real visual stimulus? Farah, M. J. (1985). Psychophysical evidence for a shared representational medium for mental images and percepts. Journal of Experimental Psychology: General, 114,
Farah's Demonstration that Imagery Can Prime Perception
Psych 355, Miyamoto, Spr '16

19. Images Can Act as Primes for Real Visual DisplaysH
Step (a): Create mental image of H or T while staring at a blank screen.
Step (b): After forming a good image, subject presses a button that causes 2 screens to be displayed one after the other. One screen has an H or a T, the other screen is blank.
Task: Say whether the letter was on the 1st or the 2nd screen.
"1"
"2"
"1"
"2"
Will Priming Occur?
Psych 355, Miyamoto, Spr '16

20. Images Can Act as Primes for Real Visual DisplaysH
Will priming occur?
Will these responses be slower?
Will these responses be faster?
Results of Farah's Experiment
Psych 355, Miyamoto, Spr '16

21. Results: Images Act as Primes for Real Visual Displays
% Correct Detection
Results: When mental image is same as target, percent correct is higher.
Interpretation: Forming the mental image requires similar brain activity to actual perception. Therefore forming the mental image primes the perception of the target.
Conclusion: Imagery Plays a Functional Role in Cognition
Psych 355, Miyamoto, Spr '16

22. Yes – Mental Imagery Plays a Functional Role in Cognition
Lots of evidence –
Image scanning (RT pattern consistent with mental image)
Zooming in or out with image (RT pattern consistent with mental image)
Image can prime detection task
Evidence is easy to explain if we postulate that people use mental imagery;
Evidence is hard to explain if we claim that people only use propositional representations.
Next Question: How similar are perception and imagery?
Neuropsych Evidence for Similarity Between Perception & Mental Imagery
Psych 355, Miyamoto, Spr '16

23. Neuropsychological Evidence for Similarity Between Perception & Mental Imagery
Neural response is similar when perceiving an object or imagining the object.
Single cell studies - neurons that respond to perceiving or imagining an object.
fMRI studies - similar brain activity when perceiving an object or imagining the object.
Transcranial magnetic stimulation (TMS) has similar effect (slower response) on tasks based on perception and on imagery.
Single-Cell Evidence for Similarity Between Perception & Imagery
Psych 355, Miyamoto, Spr '16

24. Single-Cell Studies of Perception & Imagery
Imagine the Baseball
Perceive the Baseball
Kreiman, Koch & Fried (2000) Single-cell recording in medial temporal lobe as precursor to surgery for epilepsy.
Cell is sensitive to perception and mental image of baseballs, but not to faces.
Same Slide with Emphasis on Face Stimuli
Psych 355, Miyamoto, Spr '16

25. Single-Cell Studies of Perception & Imagery
Imagine the Face
Perceive the Face
Kreiman, Koch & Fried (2000) Single-cell recording in medial temporal lobe as precursor to surgery for epilepsy.
Cell is sensitive to perception and mental image of baseballs, but not to faces.
Same Slide with Final Comment re Selectivity of Other Individual Cells
Psych 355, Miyamoto, Spr '16

26. Single-Cell Studies of Perception & Imagery
Kreiman, Koch & Fried (2000) Single-cell recording in medial temporal lobe as precursor to surgery for epilepsy.
Cell is sensitive to perception and mental image of baseballs, but not to faces.
Also found other cells that were selective for perception & imagery of animals, famous people, and food.
Similarity in fMRI Activation During Perception & Imagery
Psych 355, Miyamoto, Spr '16

27. fMRI Studies of Perception and Imagery
Perception of objects and imagining an object produces similar activation in visual cortex.
Goldstein Figure 10.13
Results are from: Le Bihan, D., Turner, R., Zeffiro, T. A., Cuenod, A., Jezzard, P., & Bonnerdot, V. (1993). Activation of human primary visual cortex during visual recall: A magnetic resonance imaging study. Proceedings of the National Academy of Sciences, USA, 90, 11802–11805.
O'Craven & Kanwisher Study - Better Evidence of the Same Type
Psych 355, Miyamoto, Spr '16

28. Location of the fusiform face area and parahippocampal gyrus
Parahippocampal place area (PPA)
and
Fusiform face area (FFA)
Diagram of Brain From the Side Facing Left
Graphic from the article: Haynes, J-D., & Rees, G. (2006). Decoding mental states from brain activity in humans. Nature Reviews Neuroscience, 7,
Next: fMRI Results for Face & Place Recognition
P 355, Miyamoto, Winter '09

29. fMRI Study of Face and Place Perception/Imagery
% Signal Change
FFA: Fusiform face area. Specialized for faces.
PPA: Parahippocampal place area. Specialized for representing location info.
% Signal Change
Perception
Imagine
Subjects either view or imagine a face or place
Next: Look at Just the Upper Left Quadrant of this Graph
P 355, Miyamoto, Winter '09

30. fMRI Study of Face and Place Perception/Imagery
% Signal Change
FFA: Fusiform face area. Specialized for faces.
PPA: Parahippocampal place area. Specialized for representing location info.
% Signal Change
Perception
Imagine
Subjects view a face or place Face stimulus activates FFA; place stimulus does not.
Next: Look at Just the Lower Left Quadrant of this Graph
P 355, Miyamoto, Winter '09

31. fMRI Study of Face and Place Perception/Imagery
% Signal Change
FFA: Fusiform face area. Specialized for faces.
PPA: Parahippocampal place area. Specialized for representing location info.
% Signal Change
Perception
Imagine
Subjects view a face or place Place stimulus activates PPA; face stimulus does not.
Next: Discuss Results for the Left Half of this Graph
P 355, Miyamoto, Winter '09

32. fMRI Study of Face and Place Perception/Imagery
% Signal Change
FFA: Fusiform face area. Specialized for faces.
PPA: Parahippocampal place area. Specialized for representing location info.
% Signal Change
Perception
Imagine
Subjects view a face or place Face stimulus activates FFA; place stimulus does not.
Next: Look at Just the Upper Right Quadrant of this Graph
P 355, Miyamoto, Winter '09

33. fMRI Study of Face and Place Perception/Imagery
% Signal Change
FFA: Fusiform face area. Specialized for faces.
PPA: Parahippocampal place area. Specialized for representing location info.
% Signal Change
Perception
Imagine
Subjects view a face or place Place stimulus activates PPA; face stimulus does not.
Next: Discuss Results for the Left Half of this Graph
P 355, Miyamoto, Winter '09

34. fMRI Study of Face and Place Perception/Imagery
% Signal Change
FFA: Fusiform face area. Specialized for faces.
PPA: Parahippocampal place area. Specialized for representing location info.
% Signal Change
Perception
Imagine
Face and place stimuli have opposite effects on FFA and PPA.
Double dissociation when perceiving faces or places.
Next: Discuss the Right Half of the Graph
P 355, Miyamoto, Winter '09

35. fMRI Study of Face and Place Perception/Imagery
% Signal Change
FFA: Fusiform face area. Specialized for faces.
PPA: Parahippocampal place area. Specialized for representing location info.
% Signal Change
Perception
Imagine
Face and place stimuli have opposite effects on FFA and PPA.
Double dissociation when imagining faces or places.
Do Neurological Impairments Have Equivalent Effects on Perception & Imagination?
P 355, Miyamoto, Winter '09

36. Do Neurological Impairments Have Similar Effects on Perception and Imagery?
Kosslyn's TMS study supports this hypothesis. (See Goldstein, p. 287 and Figure on p. 288)
What about permanent neurological impairments due to lesions or strokes?
Reminder re Hemispatial Neglect – Transition to HemiSpatial Neglect in Imagery
Psych 355, Miyamoto, Spr '16

37. Hemispatial Neglect (Goldstein calls this "unilateral neglect")
Hemispatial Neglect (Unilateral Neglect): A deficit of attention in which one entire half of a visual scene is simply ignored.
The cause of unilateral neglect is often a stroke that has interrupted the flow of blood to the right parietal lobe.
Figure to the right: Patient’s copy of an image (model) shows systematic deficits.
This slide is based on instructional material that was downloaded from the Pearson Publishers website ( for Smith & Kosslyn (2006; ISBN ).
The patient’s copy in the right column neglects the left side of the visual field (opposite to the side of brain damage).
Unilateral Neglect in Perception & Images
Psych 355, Miyamoto, Spr '16

38. Left Unilateral Neglect in Perception & Images
Bisiach, E., & Luzzatti, C. (1978). Unilateral neglect of representational space. Cortex, 14,
Patient with left unilateral neglect was asked to imagine himself standing at one end of the Piazza del Duomo in Milan.
Patient neglected left side of the visual image (in his description) just as he neglected the left side in actual perception.
“Duomo” is pronounced do-mo.
Transition to Question: Are Perception & Imagery Always Similar?
Psych 355, Miyamoto, Spr '16

39. Are Perception & Imagery Always Similar?
Due to brain injury, Patient CK has visual agnosia (inability to recognize objects)
Figure (a) – incorrect identifications
Dart labeled “feather duster”
Tennis racquet labeled “fencer’s mask”
Asparagus labeled “rose twig with thorns”
Figure (b) – drawings from memory
Outline of England
Guitar
Figure (b) – If you show CK his drawings at a later time, he cannot recognize (label) what they are.
Figure (p. 284). (a) Pictures incorrectly labeled by CK who had visual agnosia. (b) Drawings from memory by CK. From study by Behrmann, Moscovitch, & Winocur (1994).
(a)
(b)
Dissociations Between Perception & Imagery
Psych 355, Miyamoto, Spr '16

40. Dissociations Between Imagery & Perception
Case
Perception
Imagery
Guariglia (1993) OK
Unilateral neglect
Farah et al. (1993): Patient RM
OK: Recognizes objects & can draw pictures of objects
POOR: Can't draw objects from memory or answer questions that require mental imagery
Behrmann et al. (1994): Patient CK
POOR: Visual agnosia (can't recognize objects)
OK: Can draw objects from memory
Same Slide without Emphasis Rectangles
Psych 355, Miyamoto, Spr '16

41. Dissociations Between Imagery & Perception
Case
Perception
Imagery
Guariglia (1993) OK
Unilateral neglect
Farah et al. (1993): Patient RM
OK: Recognizes objects & can draw pictures of objects
POOR: Can't draw objects from memory or answer questions that require mental imagery
Behrmann et al. (1994): Patient CK
POOR: Visual agnosia (can't recognize objects)
OK: Can draw objects from memory
Diagram Showing Bottom Up & Top Down Processing of Images
Psych 355, Miyamoto, Spr '16

42. When Are Perception & Imagery Similar? When Are They Different?
Behrmann et al. (1994) point out that perception is more bottom up; Imagery is more top down.
Same Slide with Explanation of Behrmann’s Hypothesis
Psych 355, Miyamoto, Spr '16

43. When Are Perception & Imagery Similar? When Are They Different?
Hypothesis:
CK's injury blocks the bottom up input for object perception.
RM injury blocks the top down construction of a mental image.
Conclusion - END
Psych 355, Miyamoto, Spr '16

44. Conclusion
Mental manipulation of images is similar to perception of scenes as they undergo the analogous physical alterations.
Perception and imagery engage similar cognitive processes, but they are not perfectly equivalent.
Perception has more bottom-up influence than imagery.
Imagery has more top-down influence than perception.
END
Psych 355, Miyamoto, Spr '16