1. Situation Models and Embodied Language Processes Franz Schmalhofer University of Osnabrück / Germany 1)Memory and Situation Models 2)Computational Modeling of Inferences 3)What Memory and Language are for 4)Neural Correlates 5)Integration of Behavioral Experiments and Neural Correlates (ERP; fMRI) by Formal Models

2. Acknowledgements Charles A. Perfetti, Michal Balass, Jessica Nelson, Chin-Lung Yang & Edward Wlotko University of Pittsburgh funded by Alexander von Humboldt Foundation Uwe Friese, Markus Raabe, Karin Pietruska Ho-Ming Chow Niki Vavatzanidis Anke Karabanov University of Osnabrück Mark Greenlee Roland Rutschmann University of Oldenburg

3. Outline Behavioral Data and Computational Modeling on Inferencing in Text Comprehension Event-related potentials (ERP) on the reading of inference-related words An fMRI-experiment on inference processes and the verification of inference related statements

4. Text comprehension and Inferencing Mary heard the ice-cream van coming. She remembered the pocket money. She rushed into the house.

5. A blueprint of the reader (Perfetti, 1999) Visual Input Mapping to phonology Linguistic System Meaning and Form Selection Comprehension Processes Situation Model Parser Text Representation Inferences Word Identification Phonological Units Orthographic Units Word Representation Lexicon Meaning Morphology Syntax -argument structure Writing System Orthography General Knowledge Phonology Syntax Morphology

6. The KiWi-model (Schmalhofer, 1998) related domain knowledge situation model Sensory encoding text repres. common sense direct experiencetext

9. A mapping of mental and brain processes Beeman, Bowden & Gernsbacher (2000) –Information supporting predictive inferences initially activated in RH Long & Baines (2002) –Hemispheric differences in representations Tapiero & Fillon: Emotional inferences and hemispheric differences Mason & Just (2004) –Inferencing by right hemisphere language network and reasoning network (dorso- lateral prefrontal cortex) Ferstl (2003, HC); Ferstl & von Cramon (2001) –Shift from local to global aspects  precuneus –Inferencing  fronto-median cortex (FMC) –Situation model (dep. on content)  vFMC

10. Predictions from the KIWi-model (explicit and paraphrase)

11. Predictions from the KIWi-model (inference and control conditions)

12. How many nodes and links had to be newly constructed NodesLinks Explicit11 Paraphrase11 Inference22 Control33

13. Text Materials in ERP experiment

14. ERP study of inferencing Use ERPs to examine inference processes. Vary the accessibility of referents required by the integration (new construction versus preexisting traces) referent previously introduced, possibly inferred or control condition Measurements taken on a single word that occurs early in the second sentence

15. The N400 component in ERP-studies N400: A negativity shift around 400 ms is associated with an incongruent meaning. “He smeared the bread with socks.” A marker for semantic processing; sensitive to semantic congruence effects in sentence contexts (Van Petten & Kutas, 1990) Sensitive to global comprehensibility effects in text (St. George, Mannes, & Hoffman, 1994)

16. General Procedure of ERP-study Participants: 32 adult readers Procedure: Slow-SVP presentation 600 ms SOA word-to-word 300ms exposure duration 300ms interval Last word of sentence followed by additional 300ms interval Materials: 120 two-sentence passages for each participant; 30 in each condition; Comprehension Probes (T-F) at random after 25% of trials

17. Summary (ERP) A common set of assumptions about construction and integration processes in text comprehension is useful for accounting for behavioral data as well as neural correlates (e.g. ERP) Inference possibilities yield a cognitive preparation for target concept, but different from explicit and control conditions Distinction between text and situation representation accounts for behavioral data (priming) and ERP-data quite well

18. Text Materials of fMRI Experiment

19. fMRI-Lab

20. fMRI Lab 1.5 T Siemens Sonata fMRI-Scanner (Siemens Medical Solutions) 8 Channel Head Coil Lumitouch optical response device Visual Stimulation and recording of responses were controlled by E-Prime and PC

21. Behavioral Results Results in Milliseconds Differences are significant except the following pairings: –Explicit – Paraphrase –Filler – Predictive

22. fMRI: General Procedure 13 adult skilled readers 108 (72 + 36) trials; 18 trials per condition Experiment is divided into 3 Sessions à 36 trials (6 per condition) Pseudorandom order of trials Conditions and domains (themes) were counterbalanced by a Latin Square Design Different response delays were equalized at the end of each trial to receive a constant trial length

23. Time sequence of one trial

24. fMRI - Procedure BOLD sensitive T2 weighted functional sequence (TR = 3s, TE = 50 ms, FOV = 192mm) Rotated app. 10° relating to AC-PC line, covering prefrontal, parietal and temporal regions in full and the majority of the occipital cortex 3 Sessions à 324 continuously acquired scans (app. 48 Minutes) T1 weighted structural sequence (TR= 1900 ms, TE=3.93ms, FOV = 256 mm) Acquisition at the end of the experiment (app. 10 Minutes)

25. fMRI – Data Analysis Data were preprocessed and analyzed using The SPM2 Software Package (Functional Imaging Laboratory, Welcome Department of Imaging Neuroscience, London) Statistical Analysis was conducted by calculating a Random Effects Analysis Referring to the problem of multiple comparisons only those clusters with a corrected p < 0.05 were considered as significantly activated

26. fMRI – Data Analysis The last third of reading –modeled as a block Statement Verification Task –modeled as a block Reading Sentence versus Pseudoword Reading Sentence Reading in Inference Condition versus Sentence Reading in Explicit Condition Statement Verification Inference vs Explicit Inference vs Paraphrase Control vs Explicit

27. Word-Reading vs Pseudoword-Reading

28. Summary (fMRI-Experiment) fronto-median wall: –reasoning, inferencing, situation model integrations (Ferstl, Mason & Just; Robertson & Gernsbacher) Precuneus, Cuneus, occipital lobe: –Imagery, perception tasks, memory related imagery operations Posterior Cingulate: –Memory retrieval Prefrontal: Inferencing BA9/10: theory of mind

29. Integration of Fields and Methods How the brain enables the mind? Cognitive Neuroscience Cognitive Science How the mind works? Brain Research Components of the brain, How does it work? Philosophy Linguistics Computer Science Psychology Language sytems Electronic machines Human Behavior Medicine - Physiology

30. What the course was about: Sth. Old and Std. New Memory and Situation Models Computational Modeling of Inferences What Memory and Language are for Neural Correlates (ERP, fMRI) Integration of Behavioral Experiments and Neural Correlates (ERP; fMRI) by Formal Models