How do we process text with spatial information? Marijn E. Struiksma*, Matthijs L. Noordzij**, Bas F.W. Neggers*** & Albert Postma* *Universiteit Utrecht **Radboud Universiteit Nijmegen *** Universitair Medisch Centrum Utrecht

Spatial language: spatial configuration of the world Simple sentences Search directions, identifying people Complex route finding instructions Generate mental map Also blind people Spatial Language

Aim: gain insight in underlying process and nature of mental representations Research with sighted and blind Different input modalities Spatial Language

Propositional model Verbal strategy Strategic model Verbal strategy or Visuo-spatial strategy Spatial Language Processing

Noordzij et al. (2005) Sentence-sentence Sentence-picture 80% expectancy Different Strategies

+ triangle left of circle + triangle left of circle 80%20% Spatial condition

Different Strategies + triangle and circle + triangle and circle 20%80% Non-spatial condition

Different Strategies Spatial: slower on unexpected pictures Propositional model: unexpected stimuli – general switch cost Identical for spatial and non-spatial Strategic model: unexpected stimuli – general switch cost and incompatibility for both sentence and picture Fig. 1 from Noordzij et al. (2005)

Automatic propositional representation Additional visual-spatial representation Strategically dependent on context Neuroimaging parietal areas: Understanding spatial terms Visual-spatial representation Dual-representational model

Noordzij et al Similar sentence-sentence and sentence-picture paradigm Similar behavioral pattern Event-related potentials (ERPs) EEG: processing spatial sentences

ERPs EEG: processing spatial sentences Fig. 4 from Noordzij et al. (2006) Parieto-occipital activation for spatial sentences, expecting a picture. visual-spatial strategy

Sentence-sentence and sentence- picture paradigm 100% expectancy for S2 RT faster for picture than sentence, especially for spatial fMRI: processing spatial sentences

fMRI fMRI: processing spatial sentences Fig. 2 from Noordzij et al. (2008) Spatial > Non-spatial for both stimulus modalities: activity in left Supramarginal gyrus (SMG)

How is spatial language processed in the absence of vision? Is language processing different for different input modalities? Is the nature of mental representations modality specific? Nature of mental representations

Sentence-sentence paradigm Comparing spatial and non-spatial Auditory version Scanner-details: 3T Philips Achieva scanner PRESTO-SENSE sequence TR = 500ms Voxel-size: 4*4*4 mm fMRI blind & sighted

13 congenitally blind Age 36.5 ± male, 6 female 5 right-handed, 5 left-handed, 3 ambidexter 13 sighted controls Age 37.2 ± male, 5 female 6 right-handed, 4 left-handed, 3 ambidexter Subjects

Compare two sentences Do they describe the same situation? Spatial Sentence Comprehension

Block-design: 4 sessions, 18 blocks per session, 2 trials of 7.5s per block 4 conditions SpatialNonspatial Compound preposition left of right of (LR) together with (TW) Compound adverb smaller than taller than (ST) older than younger than (YO) Spatial Sentence Comprehension

Position = Left of/Right of Size = Taller than/Smaller than Conjunction = Together with Age = Older than/Younger than Position (15s) Interval (6-9s) Size (15s) Age (15s) Conjunction (15s) Interval (6-9s) Presented pseudo-randomly Spatial Sentence Comprehension

Performance Performance is good No difference CB or SC

Behavioral Results: RT Main effects of Space and Category No group difference

Contrast: Position > Combination ROI around SMG from Noordzij et al. (2008) Conjunction CB and SC Results: Supramarginal Gyrus ROI, T=2.5, p =.045 (corrected)

Contrast: Relational > Combination ROI around SMG from Noordzij et al. (2008) Conjunction CB and SC Results: Supramarginal Gyrus ROI, T=2.5, p =.045 (corrected)

Contrast Estimates SMG

fMRI results Evidence for reorganization Difference CB and SC for task vs. rest Occipital areas No dimension modulation Whole brain, T=4.0,k=10 voxels, p < 0.03

Behavioral results: sighted can generate propositional and visual- spatial representations Target area: left SMG Left SMG not influenced by context (picture or sentence) input modality (visual or auditory) Conclusions 1

Blind also activate left SMG and visual areas Visual areas are not modulated by Space or Category Left SMG important for processing spatial prepositions Other function: ordering on 1 dimension Activation in left SMG is modality- independent + hardwired Conclusions 2

Questions ?

Block-design 3 types of blocks Rest Non-words Words Verb Generation

Results Conjunction CB+SC Language areas bilateral Broca, Wernicke Covert verb generation: precentral gyrus supplementary motor area Cerebellum Word - Nonword Whole brain, T=4.0, k=20 voxels, p < 0.007

Results: Reorganization Difference CB-SC CB significantly activate occipital cortex BA 19, 31, 37 CB vs. SC Whole brain, T=4.5, k=10 voxels, p < 0.019