Chapter 9: Visual Knowledge Valerie A. Kline, PhD Spring 2019

Chapter 9: Visual Knowledge Lecture Outline Visual Imagery Long-Term Visual Memory The Diversity of Knowledge

Nonverbal knowledge Abstract thoughts Pictures Sounds Tastes Others Lack concrete form Pictures Sounds Tastes Others

What is the nature of these mental images? Visual Imagery A variety of day-to-day problems seem to require the use of visual imagery How many windows are in your apartment? Was David in class yesterday? Will this sweater look good with your blue pants? How do you get to Starbucks from this classroom? What is the nature of these mental images?

Francis Galton (1883) Introspection to study mental imagery Visual Imagery Francis Galton (1883) Introspection to study mental imagery Self-reports suggested they could inspect mental images as pictures Individual differences Need more objective approach His participants’ self reports suggested that they could inspect their images in much the same way as a picture. The participants also differed widely in the amount of detail their mental images seemed to contain. (Or were these differences in self-reporting style?)

Chronometric studies Ask participants to manipulate the mental images Visual Imagery Chronometric studies Ask participants to manipulate the mental images Observe how long these manipulations take How picture-like are mental images? Draw a cat Write a few sentences describing a cat Studies of visual imagery in the last 50 years have avoided introspection and instead asked participants to do something with their images—to read information off them or manipulate them in some way. Chronometric studies measure the amount of time required by a cognitive process of interest.

How do we represent images in our mind? Depictive: Descriptive: The boy is inside the box. The blocks are inside the box. The blocks are on top of the boy.

A small carnivorous mammal… …with a strong flexible body, quick reflexes, sharp teeth and retractable claws adapted to killing small prey.

Visual Imagery Kosslyn (1976) asked participants to answer yes/no questions about their mental images. Imagined cat, confirm that cats have heads faster than confirm cats have claws The reverse was true if the participants were asked to think about cats, not to imagine them Mode of representation changes pattern of information available Kosslyn (1976) asked participants to answer yes/no questions about their mental images. If participants imagined a cat, they were faster to confirm that cats have heads, compared to confirming that cats have claws. The reverse was true if the participants were asked to think about cats, not to imagine them. This suggests that as the mode of representation changes, so does the pattern of information availability.

Image-scanning procedure, Kosslyn et al. (1978) Visual Imagery Image-scanning procedure, Kosslyn et al. (1978) Memorize this map Scan from one landmark to another on the imagined map Other studies have used the image-scanning procedure to study mental imagery. Kosslyn et al. (1978) first asked participants to memorize this map. They were then asked to mentally scan from one landmark to another on the imagined map.

Imagined distance corresponds to real distance Visual Imagery The time it took to scan the image corresponded to the distance on the map. Thus, mental images seem to preserve the spatial layout and geometry of the represented scene. Imagined distance corresponds to real distance Participants scan image at constant rate

Does a mouse have whiskers? Visual Imagery Does a mouse have whiskers? Easier to image a mouse next to a paper clip than next to an elephant.

The nature of mental images Visual Imagery The nature of mental images Preserve distance relations Spatial layout Geometry Depict rather than describe

Visual Imagery Mental-rotation task Which of these pairs is the same? The mental-rotation task also suggests that mental images preserve spatial information in three dimensions. In each pair, are the objects identical despite being viewed from different perspectives? The further the distance, the longer it takes

Shepard & Metzler (1971) Stimuli: pairs of 3D block figures rotated between 0 – 180° Task: Shown one object, then removed. Shown second object and asked if was the same or different as the first.

same or different?

same or different?

Shepard & Metzler (1971) Results: Correct “Yes” Trials

Visual Imagery The greater the angle, the longer the time The data from this task suggest that the greater the angle of rotation between the two pictures, the longer the response time. It is as if the participants were rotating their mental images of the two objects into alignment. The greater the angle, the longer the time As if they were rotating the images in real life

Mental images more like mental sculptures What does this mean? Mental images more like mental sculptures Depictive vs. Descriptive? Mental images are depictive How do we use images? We mentally manipulate the mental image to make a decision If it was a verbal description it would be the same amount of time no matter how far it was rotated.

Demand character Did experimenters somehow cue people? Visual Imagery Demand character Did experimenters somehow cue people? Do people just simulate the time they think it would take in real life? Even without instruction, participants still form images Finke & Pinker, 1982 Did the image-scanning and mental-rotation experiments have demand character, or cues with which the experimenter signaled to the participants how they were “supposed to behave”? These two results are observed even without instruction to use imagery, suggesting that participants spontaneously form mental images and scan them.

shown dots, then removed 2 sec later, saw arrow Finke & Pinker (1982) shown dots, then removed 2 sec later, saw arrow Task: Is arrow pointing to location of any of the dots (X)? Results: RT increases (takes longer) with increasing distance from arrow to dots Depictive vs. descriptive

Imagining Vs. Perceiving What is the relationship? Brooks, 1968 Segal & Fusella, 1970, 1971 Farah, 1985 O’Craven & Kanwisher, 2000

Two ‘yes/no’ tasks: Two response types: Brooks (1968) Two ‘yes/no’ tasks: Letter task (visual): “Yes” to top/bottom corners Sentence task (verbal): “Yes” to nouns Two response types: Verbal Spatial/visual Is responding spatially/visually worse when you are holding an image in mind?

Visual/Verbal Instructions I’ll show you a block letter. Start at the star and scan around the perimeter of the letter. If the corner is on the top or bottom of the letter say “yes” L * The only trick is you will have to imagine the block letter

T *

Visual/Visual Instructions On the next trial, you will point to the letter Y for “yes” and N for “no” instead of responding verbally. Move down one row of letters for each response.

Verbal/Visual Instructions Memorize this sentence: “The dirty sock hung from the clothesline” I’ll remove the sentence and I want you answer whether each consecutive word is a noun? Point to a Y for “yes” and an N for “no”

Verbal/Verbal Instructions Memorize this sentence “After several hours the dog ran home from the lake” I’ll remove the sentence and I want you answer verbally whether each consecutive word is a noun? Now say “yes” or “no”

Image block letters or sentences Respond vocally or by pointing Brooks (1968) Image block letters or sentences Respond vocally or by pointing

Segal & Fusella (1970, 1971) Detect faint stimuli (visual, auditory) Imagine stimuli (visual, auditory) How similar are imagery and perception? What do we know about attention to stimuli that share similar modalities?

Visual Imagery More likely to wrongly choose that the stimulus matches the image when signal and image are both visual or auditory Less accurate when signal and image are the same Participants asked to form a visual or auditory image while having to detect a visual or auditory signal. Mental imagery seems to use perceptual mechanisms. Visual imagery interferes with detecting dim visual stimuli, and auditory imagery interferes with detecting quiet tones. Segal & Fusella (1970)

Imagery Can interfere with perception (mismatching) Visual Imagery Imagery Can interfere with perception (mismatching) Segal & Fusella (1970) Can facilitate perception (matching) Farah (1985) However, if we are imagining a stimulus related to the one we are about to perceive, facilitation occurs. Interference between imagery and perception occurs when we try to visualize one thing while perceiving another.

Are visual imagery and visual perception related? Farah (1985) Are visual imagery and visual perception related? Task 1: Imagine a letter. Task 2: Which came first? Better at detecting letter position when you were imagining the same letter

O’Craven & Kanwisher (2000) Does imagining and seeing an object engage the same brain area? Saw pictures (perception) vs. imagined pictures (imagery)

O’Craven & Kanwisher (2000)

Occipital areas used for early visual processing Visual Imagery Occipital areas used for early visual processing Active during visual imagery Transcranial magnetic stimulation (TMS) disrupts mental imagery The same occipital areas used for early visual processing are activated during visual imagery. Patients with unilateral neglect may also neglect the left side of space in their mental images. Disrupting visual-processing areas with transcranial magnetic stimulation (TMS) also disrupts mental imagery.

Studies of brain damage Lose ability to perceive color Lose ability to perceive fine detail Neglect syndrome attention

Visual Imagery Patient can only see the right side of the plaza Patients with unilateral neglect may also neglect the left side of space in their mental images. If this patient was shown a picture, he seemed to see only the right side of it; if asked to read a word, he read only the right half. The same pattern of neglect was evident in the patient’s imagery: In one test, the patient was urged to visualize a familiar plaza in his city and to list the buildings “in view” in the image. If the patient imagined himself standing at the northern edge of the plaza, he listed all the buildings on the plaza’s western side (i.e., on his right), but none on the eastern. If the patient imagined himself standing on the southern edge of the plaza, he listed all the sights on the plaza’s eastern side, but none on the western.

Visual Imagery Patient can only see the right side of the plaza Patients with unilateral neglect may also neglect the left side of space in their mental images. If this patient was shown a picture, he seemed to see only the right side of it; if asked to read a word, he read only the right half. The same pattern of neglect was evident in the patient’s imagery: In one test, the patient was urged to visualize a familiar plaza in his city and to list the buildings “in view” in the image. If the patient imagined himself standing at the northern edge of the plaza, he listed all the buildings on the plaza’s western side (i.e., on his right), but none on the eastern. If the patient imagined himself standing on the southern edge of the plaza, he listed all the sights on the plaza’s eastern side, but none on the western.

Do we use the same mental processes for imagined and actual images? Segal & Fusella (1970,1971) and Brooks (1968) – yes, forming visual images interferes with visual perception Farah (1985) – yes, actual images enhanced by imagery if they are related O’Craven & Kanwisher (2000) – yes, same brain areas are active (though less activity for imagined)

Functional equivalence between imagery and perception. Visual Imagery Functional equivalence between imagery and perception. Visual acuity higher—can see two dots Research also demonstrates a functional equivalence between aspects of visual imagery and visual perception. For instance, studies of visual acuity measure how close two dots can be to each other and still appear distinct. For both perception and imagery, acuity is greatly reduced if the dots are not in the center of vision. Visual acuity lower—need more space to see two dots

Shown 2 line segments, asked which is longer Moyer (1973) Shown 2 line segments, asked which is longer It took takes longer to make the judgment if the two are close in size

Moyer (1973) Asked people to imagine two animals and judge which of two animals was larger. The closer in size the two animals were, the longer the judgement took bee vs. cockroach moose vs. bee

Why is Moyer (1973) important? So what? Why is Moyer (1973) important? Because when shown actual images (line segments) or when imagining images (animals), the same result holds – it takes longer if the two are similar in size Must be using the same mental process to do both

Close relationship between imagery and perception Mental imagery in congenitally blind

Visual Imagery People who have been blind since birth also demonstrate the same effects in mental-rotation or image-scanning tasks, with response time being proportional to the distance traveled Thus, we need to distinguish between visual imagery and spatial imagery Spatial imagery may be based in movement or body imagery, or it may be abstract and not tied to any one sense People who have been blind since birth also demonstrate the same effects in mental-rotation or image-scanning tasks, with response time being proportional to the distance traveled. Thus, we need to distinguish between visual imagery and spatial imagery. Spatial imagery may be based in movement or body imagery, or it may be abstract and not tied to any one sense.

Individual differences Vivid imagers versus non-imagers Visual Imagery Individual differences Vivid imagers versus non-imagers Report seeing images better The same for spatial imagery Vivid imagers better for visual imagery ~10% non-imagers Lab findings Real life findings Individual differences in the ability to form mental imagery also suggests a distinction between visual and spatial imagery. Self-reported “vivid imagers” perform no differently than “non-imagers” on tasks that depend on spatial imagery. However, vivid imagers are better on tasks that specifically require visual imagery.

Eidetic or photographic memory Visual Imagery Eidetic or photographic memory Extremely rare Found in some autistic individuals Some researchers believe that 5% of all children have what is known as eidetic imagery, an extremely detailed form of visual memory.

Mental images different from pictures Visual Imagery Mental images different from pictures Can be more like sculptures than pictures Differences between visual imagery and spatial imagery Perception is not neutral and goes beyond the information given Interpretations are present in images In some critical ways, our mental images are different from pictures. Recall that although a picture may be ambiguous, our perception of the picture is not neutral. Perception “goes beyond the information given” to organize and interpret the image, making it unambiguous. Similarly, mental images correspond to these interpretations, not to the ambiguous images.

Imagery only preserves one interpretation Visual Imagery Imagery only preserves one interpretation For instance, if participants view an ambiguous image like the Necker cube or the duck/rabbit, they will only be able to imagine whichever interpretation they initially perceived.

Thus, mental images (like percepts) are organized depictions Visual Imagery Thus, mental images (like percepts) are organized depictions One way to think about mental images is as a package that includes the depiction itself as well as a perceptual reference frame For instance, the duck/rabbit image, understood as a duck, is associated with the reference frame “facing to the left” Thus, images (like percepts) are organized depictions. One way to think about mental images is as a package that includes the depiction itself as well as a perceptual reference frame. For instance, the duck/rabbit image, understood as a duck, is associated with the reference frame “facing to the left.”

Close your eyes and rotate it 90º clockwise Visual Imagery Don’t know what this is? Close your eyes and rotate it 90º clockwise Participants who fail to recognize this shape can sometimes recognize it using mental imagery. However, the perceptual reference frame must change. Participants must be told to rotate the image 90º clockwise and that the form’s left edge is the top.

Visual Imagery Sometimes putting an idea down on paper can help make a discovery that requires a change in the reference frame

Mental images Alternative to verbal description Visual Imagery Mental images Alternative to verbal description Spatial layout and geometry are preserved Reflect perceptual interpretation and are associated with reference frames Depiction- form Description- idea Mental images provide a distinctive means of representing the world compared to a verbal description. Mental images are like pictures in that spatial layout and geometry are preserved. However, they are different from pictures in that they reflect perceptual interpretation and are associated with reference frames.

HOMEWORK 2 Due Tuesday 3/26 @ 9am

Long-Term Visual Memory What is the nature of visual imagery taken from long-term memory?

Long-Term Visual Memory Images in long-term memory Stored in a piecemeal fashion Must activate representation of image frame Elaborate on this frame Images that have more parts or detail take longer to create Images seem to be stored in long-term memory in a piecemeal fashion. To form a mental image, you must activate a representation of an image frame and then elaborate on this frame to make the image more detailed. In agreement with this hypothesis, images that have more parts or detail take more time to create.

Long-Term Visual Memory Generating three rows faster For example, a mental image of four columns of dots takes more time to generate and is more difficult to maintain than a mental image of three rows of dots. Generating four columns slower

Long-Term Visual Memory Image files Recipes or instructions for how to construct an active mental image of the object or shape May represent visual information in terms of propositions, or verbal labels Information stored in long-term visual memory is sometimes thought of in terms of image files, which specify what particular objects or shapes look like. Image files are used as recipes or instructions for how to construct an active mental image of the object or shape.

Long-Term Visual Memory Will have more accurate memory for something that is either blue or green This may explain, for instance, why speakers of languages with a large number of color categories have better memories for color—the memories may be the verbal labels and not perceptual representations of color. Will have less accurate memory

Long-Term Visual Memory Interpretation changes the reconstruction of the image Carmichael, Hogan, & Walters, 1932 Other evidence that long-term visual memory may be propositional comes from studies showing that verbal labels influence later re-creations of a studied figure.

Long-Term Visual Memory Which is farther south, New Orleans or Tijuana? Which is farther north, Seattle or Montreal? Errors of reasoning about geography also suggest that this spatial information is remembered in terms of propositions. Which is farther north, Seattle or Montréal? (Answer: Seattle) Stevens & Coupe, 1978

In general imagery improves memory Paivio & Colleagues (1968, 1969) 1st – rate nouns on scale from 1 to 7 for how readily it evokes an image House elephant ego fear Calm chair peanut butter curiosity 2nd – do these ratings predict memory performance for new Ps?

Long-Term Visual Memory It is also clear that, in most cases, imagery helps memory. For instance, words that evoke imagery are easier to remember than those that do not. Further, when learning pairs of words, a useful mnemonic is to imagine the two concepts interacting. Imagery helps memory, especially with an interaction Bower & Winzenz, 1970 rehearsal sentences imagery

Long-Term Visual Memory Dual coding (Pavio, 1971) High-imagery words, for instance, can be coded as both word and image Low-imagery words only have a verbal code Information can be stored as a verbal description/code or as a visual description/code The two are independent, so 2 is better than 1 Tree (concrete word) – could be either Verbal description is TREE Visual description is What memory aid does imagery provide? One possibility is dual coding, or a double representation in memory. High-imagery words, for instance, can be coded as both word and image. At the time of retrieval, either code will provide the sought-after information.

Long-Term Visual Memory Studies of memory for pictures illustrate ways in which long-term visual memory reflects general principles of memory, such as Primacy and recency Encoding specificity Schemata or generic knowledge Spreading activation and priming Familiarity and source memory

Long-Term Visual Memory Schematic retrieval (Friedman, 1979) found that participants failed to notice differences between previously seen and new pictures if both were consistent with a schema (e.g., a kitchen or barnyard) Pictures that contained violations of a schema (e.g., kitchen with a fireplace) were readily noticed One study illustrating the effects of schemata (Friedman, 1979) found that participants failed to notice differences between previously seen and new pictures if both were consistent with a schema (e.g., a kitchen or barnyard). However, pictures that contained violations of a schema (e.g., a kitchen with a fireplace) were readily noticed.

Long-Term Visual Memory Boundary extension Information is filled in that was not present in the picture Another effect of schemata is boundary extension—pictures are often recalled as having depicted more information than they actually did.

The Diversity of Knowledge Visual working memory is based in imagery and uses perceptual, spatial representations Image scanning, rotation, zooming Visual long-term memory is based on propositional knowledge and shares many representational principles with other forms of long-term memory Spreading activation, priming, schematic knowledge