Language and Perception Ling 411 – 17

Perception: Starting view  Perception is a bottom-up process From primary perceptual area upwards  E.g. primary auditory, for auditory perception  Takes place in a single perceptual area E.g. auditory cortex for auditory perception  Works by processing input to the sense organ E.g. auditory input for auditory perception  Each of these points is wrong!

The McGurk Effect  Acoustic syllable [ba] presented to subjects  with visual presentation of articulatory gestures for [ga]  Subjects typically heard [da] or [ga]  “Evidence has accumulated that visual speech modifies activity in the auditory cortex, even in the primary auditory cortex.” Mikko Sams (2006) How does it work? (1)Visual input (2)Top-down processing

Perception – Refining a simple-minded view 1. It is not confined to a single perceptual modality The McGurk effect  Auditory perception affected by visual input Conceptual structure affects auditory perception  The influence of context on speech perception

Perception – Refining a simple-minded view 1. Not confined to a single perceptual modality The McGurk effect  Visual input affects auditory perception Conceptual structure affects auditory perception 2. Not just bottom-up Top-down processing fills in unsensed details 3. Not even confined to posterior cortex Can also use motor neurons  Experiment: left hand or right hand?  Mirror neurons

Top-down processing in perception T CUP MADE OF GLASS CERAMICSHORT HAS HANDLE Properties Conceptual and perceptual information Node for CUP in conceptual area for drinking vessels Visual properties are in occipital and lower temporal areas

Bidirectional processing and inference T CUP MADE OF GLASS CERAMIC SHORT HANDLE These connections are bidirectional

Pertinent neuroanatomical findings: Bidirectional Processing  An established fact of neuroanatomy: A connection from point A to point B in the cortex is generally accompanied by a connection from point B to point A  Separate fibers (axons): (1) A to B, (2) B to A  In short, cortico-cortical connections are generally bidirectional

Bidirectional processing: reciprocal links excitatory inhibitory

Bidirectional processing and inference T CUP SHORT HANDLE Thought process: 1. The cardinal concept node is activated by a subset of its property nodes 2. Feed-backward processing activates other property nodes Consequence: We “apprehend” properties that are not actually present in the sensory input

Bidirectional processing and inference T CUP MADE OF GLASS CERAMIC SHORT HANDLE These connections are bidirectional Separate fibers for the two directions; shown as one line in the notation

Cortical Structure and Inference: Perceiving things that are not in the input T Category Properties A B F E Consequence: If A and B, then E and F C D

Examples  Looks like a duck Probably quacks  Ceramic, cup-shaped, handle Probably holds coffee (without breaking)  Dark clouds, thunder It’s going to rain  ATM Probably has money

Perception depends mainly on cortical structures already present before sensory input “Perception is hallucination constrained by sensory data” Shepherd

A terminological problem  We need to distinguish Perception narrowly conceived  The basic process of recognition  Single perceptual modality  Bottom-up processing  No motor involvement Perception broadly conceived  Two different terms needed Recognition (a.k.a. ‘microperception’)  Bottom-up process in a single perceptual modality Perception (the broad conception) (a.k.a. ‘macroperception’)

“Micro-perception” and “macro-perception”  Microperception and macroperception  Microperception A.k.a. recognition The local process of integrating features Performed in one perceptual modality Bottom-up  Macroperception The overall process of perception Uses multiple modalities Uses top-down processing

Perception – Refining a simple-minded view 1. Not just bottom-up Top-down processing fills in unsensed details 2. Not confined to a single perceptual modality The McGurk effect  Visual input affects auditory perception Conceptual structure affects auditory perception 3. Not even confined to posterior cortex Can also use motor neurons  Experiment: left hand or right hand?  Mirror neurons

Left hand or right hand?

Left or right hand?  Imaging experiment  Subjects were shown pictures of one hand  Asked to identify: left or right  Functional imaging showed increased CBF in hand area of motor cortex Peter Fox, ca. 2000

Motor structures in perception  The left-hand vs. right-hand experiment  ‘Mirror neurons’ in motor cortex  Articulation as aid to phonological perception  Articulation in reading  Motor activity in listening to music  Watching an athletic event

Mirror Neurons  NY Times: “One mystery remains: What makes them so smart?” (Jan. 10, 2006)  Answer: They are not smart in themselves Their apparent smartness is a result of their position: at top of a hierarchy Compare:  The general of an army  The head of a business  Similarly, high-level conceptual nodes The “grandmother node”

Mirror Neurons  What makes mirror neurons appear to be special? Ans.: They receive input from visual perception The superior longitudinal fasciculus  Connects visual perception to motor areas  How can a motor neuron receive perceptual input? Motor neurons are supposed to operate top-down Answer: bidirectional processing  They also receive perceptual information Bottom-up processing

Superior Longitudinal Fasciculus From O. D. Creutzfeldt, Cortex Cerebri (1995)

Are some neurons “smarter” than others?  Claim: A grandmother node would have to be very smart Identifies very complex object Even in many varieties  Alternative: the head of a hierarchy It is the hierarchy as a whole that has those ‘smarts’ Similarly, mirror neurons  They get visual input since they are connected to visual areas Superior longitudinal fasciculus

Implications of hierarchical organization  Nodes at a high level in a hierarchy may give the appearance of being very “smart”  This appearance is a consequence of their position — at top of hierarchy  As the top node in a hierarchy, a node has the processing power of the whole hierarchy Grandmother nodes Mirror neurons Compare:  The general of an army  The head of a business organization

Multi-Modal Perception 1. Perception is not just bottom-up Top-down processing fills in unsensed details 2. It is not confined to a single perceptual modality The McGurk effect  Visual input affects auditory perception Conceptual structure affects auditory perception 3. It is not even confined to posterior cortex Can also use motor neurons  Motor activation in speech perception  Mirror neurons

Perceptual structures in motor production  Perceptual structure is used in two ways 1. Planning (e.g. visualizing while painting) 2. Monitoring  Examples Phonological recognition in speech production  Cf. Wernicke’s aphasia Painting Musical production Baseball, soccer, tennis, etc.

The Influence of language on non-linguistic perception  As we have seen, non-speech input affects phonological perception  It is also the case that language affects non-linguistic perception E.g., visual perception

Language and (Visual) Perception PhonologicalPhonological Object ProductionPerception Categories Vision Language N.B.: These connections are bidirectional

Recent experiments of Kay et al.  Experiments at UC Berkeley Color perception: do differences in color naming across languages influence color perception?  Main finding: Lateralized influence of language on perception Response time faster for between-category discrimination – especially for RVF presentation A left hemisphere (RVF) phenomenon green blue

Perception: A simple-minded view, revisited  Perception is a bottom-up process From primary perceptual area upwards  E.g. primary auditory, for auditory perception  Takes place in a single perceptual area E.g. auditory cortex for auditory perception  Works by processing input to the sense organ E.g. auditory input for auditory perception  Each of these points in wrong

Another hypothesis of Whorf  Grammatical categories of a language influence the thinking of people who speak the language  Can we explain this too in terms of brain structure?

Mechanisms of operation 1. Entrenchment Strengthening of connections through repeated activation  An automatic brain process  Important in learning 2. Reverberation of activation 3. Priming 4. Language as a major means of learning conceptual and perceptual distinctions

Entrenchment and thinking: a mechanism  Connections become stronger with use (entrenchment)  Grammatical categories make speakers constantly heed selected phenomena  Connections for phenomena which speakers must constantly heed.. Will be repeatedly traversed Therefore will get progressively stronger

Example: Grammatical gender  Does talking about inanimate objects as if they were masculine or feminine actually lead people to think of inanimate objects as having a gender?  Could the grammatical genders assigned to objects by a language influence people’s mental representation of objects? Boroditsky (2003)

Plausibility of the possibility  Children learning to speak a language with grammatical gender may suppose that gender indicates a meaningful distinction between types of objects  Other grammatical distinctions do reflect actual perceptual differences: singular:plural

Children learning a language with gender  “For all they know, the grammatical genders assigned by their language are the true universal genders of objects.” Boroditsky et al, 2003

Experiment: Gender and Associations (Boroditsky et al. 2002)  Subjects: speakers of Spanish or German All were fluent also in English English used as language of experiment  Task: Write down the 1 st 3 adjectives that come to mind to describe each object All the (24) objects have opposite gender in German and Spanish  Raters of adjectives: Native English speakers

Examples:  Key (masc in German, fem in Spanish) Adjectives used by German speakers:  Hard, heavy, jagged, metal, serrated, useful Adjectives used by Spanish speakers:  Golden, intricate, little, lovely, shiny, tiny  Bridge (fem in German, masc in spanish) Adjectives used by German speakers:  Beautiful, elegant, fragile, peaceful, pretty Adjectives used by Spanish speakers:  Big, dangerous, long, strong, sturdy, towering

Results of the Experiment (Boroditsky et al. 2002)  Raters of adjectives were native English speakers  Result: Adjectives were rated as masculine or feminine in agreement with the gender in subject’s native language

end