Martin J. Pickering, Simon Garrod, Behavioral and Brain Sciences, An Integrated Theory of Language Production and Comprehension Computer Science & Engineering Sang-Woo Lee

Background - Aphasia

 Expressive aphasia  Agrammatic aphasia  Understand what other people say, but cannot speak the sentence well.  Caused by damage to, or developmental issues in the anterior regions of the brain  Including (but not limited to) the Broca’s area 3 Broca’s Aphasia

 Also known as Receptive Aphasia  Fluent apahsia, or sensory aphasia  Speak the sentence fluently, but not well-organized sense in their speech  Traditionally associated with neurological damage to Wernicke’s area in the brain  (Actually it is not just simply associated to Wernicke’s area in current experimental result, but anyway…) 4 Wernicke’s Aphasia

5 There are modules which specify some function The “classical Lichtheim-Broca-Wernicke” Model perception action

 Background - Aphasia  Traditional independence of production and comprehension  Interweaving in action and action perception  Perception process in action  Predict next action of other  Joint Action  Interweaving in Production and Comprehension  Comprehension process in Production  Predict next speech of other  Interactive Language  Professor’s Question Contents 6

Traditional independence of production and comprehension

8 Traditional model of communication  Discrete stages  A produces, B comprehends  B produces, A comprehends

 Assumes “horizontal split” between production a nd comprehension  Arrows-within-arrows indicate feedback (in interactive accounts)  But this feedback is internal to production or comprehension  It may involve “general knowledge”  But production does not involve comprehension processes  And comprehension does not involve production processes 9 Horizontal Split

 Interlocutors are not static, as the traditional model assumes, but are “moving targets” performing a joint activitiy (Garrod & Pickering, 2009) 10 Example of predict in Behavioral Instance

11 Example of predict in Neuroscience Big(neuter) Painting (neuter) Big (common) Bookcase (common) (Pickering & Garrod, 2007) große Gemälde großen Bücherschrank

 Also, many experiments demonstrate effects of one on the other  Picture-word interference (Schriefers et al., 1990)  Word identification affected by externally controlled cheek movement (Ito et al., 2009)  And strongly overlapping neural circuits for production and comprehension (e.g., Pulvermüller & Fadiga, 2010; Scott et al., 2009) 12 Other Counterexample

 Prediction process could be naturally understood with production module  When they comprehend the utterance, they also use production model internally. 13 Result - There is forward model To predict perception caused by their own utterance - Fast alert when you say something wrong

Interweaving in action and action perception  Perception process in action  Predict next action of other  Joint Action

Interweaving in action and action perception  Close links between action and action perception, e. g.  participants’ arm movements affected by observing another person’s arm movements (Kilner et al., 2003)  And making hand movements can facilitate concurrent visu al discrimination of deviant hand postures (Miall et al., 2006 )  Such links could have various purposes  Supporting overt imitation  facilitating memory or understanding (“postdictively”)  But authors propose that they aid prediction of own a nd others’ actions, by use of a forward model  Based on computational neuroscience (Wolpert, 1997; see Grush, 2004)

Forward modelling in action  In our terms, the action command causes the action implementer to move the hand and the perceptual implementer to construct the perc ept  And the efference copy causes the forward action model to generate the predicted hand movement and the forward perceptual model to c onstruct the predicted percept

Forward Modeling in Action -Just act -Feel involved percepts of your own act e.g. Own coordination info Feeling of wind blowing to your arm Gravity info …

18 Efference Copy

Forward Modeling in Action - Predict perception caused by their own action e.g. Own coordination info Feeling of wind blowing to your arm Gravity info …

Prediction-by-Simulation - Predict perception of other’s next action by seeing other’s current action

Joint Action  People are highly adept at joint activities (Seban z et al., 2006).  ballroom dancing, playing a duet, carrying a large obj ect together  Precise timing is crucial  To succeed, A predicts B’s action and B predicts B’s action

22 Joint Action

Interweaving in Production and Comprehension  Comprehension process in Production  Predict next speech of other  Interactive Language

Forward modeling in language production  Action implementer  production implementer  Perceptual implementer  comprehension implementer  Action command  production command  Drives the production implementer  Efference copy drives the forward models  Comparator  monitor  compares the utterance percept and the predicted utterance percept

Unifying production and comprehension  Production and comprehension are interwoven  Tight coupling in dialogue (Clark, 1996; Pickering & Garrod, 2004)  Behavioural experiments show effects of comprehension processes on production and vice versa (e.g., Schriefers et al., 1990)  Overlap of brain circuits for production and comprehension (e.g., Pulvermuller & Fadiga, 2010)  Such interweaving facilitates prediction of self and other’s utterances

26 Classical modeling in language production -Just say utterance, -Listen what you say.

27 Forward modeling in language production - Predict perception caused by their own utterance - Fast alert when you say something wrong

Self-monitoring  Speaker wishes to say kite  In the past, she has always constructed the kite- concept and then uttered /k/  She therefore constructs forward model p^[phon](t) = /k/  If she then incorrectly constructs p[phon] = /g/, the monitor notices the mismatch  If she believes the forward model, she will detect an error (and perhaps reformulate)  Otherwise, she will alter her forward model

29 Prediction-by-simulation - Predict perception of other’s next utterance by listening other’s current utterance Big(neuter) Painting (neuter)

30 Interactive Language  Joint action involves combining accounts of action an d action perception  Similarly, interactive language involves combining ac counts of production and comprehension  Facilitates coordination (e.g., short intervals between speak ers; Wilson & Wilson, 2005)  Facilitates alignment (developing same representations; Pic kering & Garrod, 2004)  Alignment in turn facilitates comprehension (better predictio n of others)

31 Interactive Language

Conclusion  We propose that language production and comprehensio n are interwoven  It assumes a central role to prediction in production, com prehension, and dialogue  Speakers construct forward models to predict aspects of their upcoming utterances  Listeners covertly imitate speakers and use forward mod els to predict the speakers  Our account helps explain the efficiency of production an d comprehension and the remarkable fluidity of dialogue

Thank you

Professor’s Question  Comprehension process in Production  Predict next speech of other  Interactive Language

 Q1: Give the evidence for how language production and comprehension are tightly interwoven. How does this relate to the perception-action cycle theory of cognitive systems? 35 Question 1

 Q2: Explain and give the evidence for how action, action perception, and joint action are interwoven. Explain how the authors use this to develop accounts of production, comprehension, and interactive language. 36 Question 2  Action implementer  production implementer  Perceptual implementer  comprehension implementer  Action command  production command  Comparator  monitor  participants’ arm movements affected by observing anothe r person’s arm movements (Kilner et al., 2003)  And making hand movements can facilitate concurrent visual discrimination of deviant hand postures (Miall et al., 2006)

 Q3: Give examples of what behavioral and neuroscientific data on language processing can be explained by the integrated theory of language production and comprehension explains, while modular theory does not. 37 Question 3 - Behavioral data - Neuroscientific data (Pickering & Garrod, 2007)