Multisensory Integration in the Brain Paul J. Laurienti, M.D., Ph.D. Department of Radiology

McGurk Illusion

Why Multisensory? 1.Survey of Language Knowledge LOUISA C. MOATS, Ed.D 2.VALERIE G. TUCKER, M.Ed., CALT President, The Alliance for Accreditation and Certification of Multisensory Structured Language Education. 3.Panel of Experts Moderator: VALERIE G. TUCKER, M.Ed. A panel of experts will address the National Reading Panel Competencies and give teachers practical ideas for implementing multisensory structured activities for each identified area. 4.Capturing Elusive Images: Practical Strategies for Building Orthographic Recall MICHELE BERG, Ph.D 5.Teaching the Teachers: Effective Models for Colleges and Universities 6.Dyslexia and the English Language Learner ELSA CÁRDENAS-HAGAN, Ed.D, 7.Beyond Basic Phonics: The MultiSyllabic “Missing Link” to Literacy POLLY BAYRD, M.A and KRISTIN JOHNSON, M.A. 8.Strengthening Vocabulary with Direct Instruction and Morphology BRYAN K. CLAYBACK, B.A. et al. 9.Expressive Phonological Impairment and Spelling: Impact of a Multisensory Intervention DEBORAH USREY, Ed.D 10.Teaching Non-Literate and Brain-Injured Adults Successfully with Sounds in Syllables JENNIE LYNCH, B.A. and SANDRA DILLON, M.A. 11.Effects of a Multisensory-Based Comprehensive Reading District Reform Model PAUL WORTHINGTON 12.The ABCs of Grammar Instruction: Keeping it Structured, Sequential and Multisensory PHYLLIS C. HUTSON 13.Hands-On, MultiSensory Reading Instruction GERI MARSHALL MOHLER, Ed.D. and BETH ANN BADER-PAETSCHOW, Ed.D. 14.The Association Method: Successful Strategies for Oral and Written Language Intervention MAUREEN K. MARTIN, Ph.D. 15.Make it Fun! Make it Challenging! Make it Multisensory! SUE GNAGY FEGAN, M.S., F/AOGPE and S C OTT MURRILL, M.A. 16.Considerations of Private Practice ELLEN J. LINSKY, Ed.D., M/AOGP 17.Implementing Undergraduate and Graduate Multisensory Structured Language Teacher Training Programs: Two Examples KEICHER and SCHNEIDER 18.The Student Notebook: An Important Component of Multisensory Structured Language Instruction. KAREN K. LEOPOLD, M.Ed. 19.The Roots of Algebra Revealed: Concepts Before Procedures MARILYN ZECHER, M.A., C.A.T. 20.Preparing Preschool Children to Read: A Randomized Trial Study MARCIA DAVIDSON, Ph.D., JENNAE BULAT, M.A., M.B.A., and LAUREL COCO, M.A. 21.Part B: The Painted Essay: A Multisensory Approach for Essay Writing and Assessment CARLENE BADEAU, B.A. 22.Part A: Reading Remediation in the Middle School Through Explicit Phonics Instruction GERI MARSHALL MOHLER, Ed.D 23.The Slingerland Multisensory Approach to Reading BONITA COLONY and SUE WINTERS-BROW N, M.Ed. 24.Teaching the Teachers: Effective Models for Colleges and Universities MAUREEN K. MARTIN, Ph.D. et al 25.Fun and Games in an OG Lesson COREY ZYLSTRA 26.Does Multisensory Instruction Meet the Criteria for Scientifically-Based Reading Research (SBRR)? DEBORAH F. KNIGHT, Ph.D. and MARGIE GILLIS, Ph.D 27.Building Bridges Through Collaboration: Reading Disabilities Prevention and Intervention DEBORAH MIDDEL–KAT Z E N M E Y E R, M.A. et al 28.Prevention and Early Intervention: Multisensory Instruction in K-3 General Education Classrooms ROBIN CARLO, M.A 29.Slingerland Multisensory Approach for Upper Elementary through High School Students S U SAN HEINZ, Ph.D.

Why Multisensory?

We live in a “multisensory” world Information can be processed on a modality- specific basis Information converges and is integrated in the brain Performance enhancement is larger for multisensory than unisensory stimuli

Color Discrimination Task (Multisensory) (Unisensory)

Multisensory Enhancement AuditoryMultisensoryVisual * Cross-modal Visual 1Visual 2Dual-Visual ns Dual-visual

Factors Modulating Multisensory Integration Temporal/spatial coincidence Semantic congruence Sensory attention –Selective –Divided

Temporal Coincidence Hairston et al. Exp Brain Res (2005) 166: 474–480 +

Temporal Coincidence Hairston et al. Exp Brain Res (2005) 166: 474–480 +

Temporal Coincidence Hairston et al. Exp Brain Res (2005) 166: 474–480

Factors Modulating Multisensory Integration Temporal/spatial coincidence Semantic congruence Sensory attention –Selective –Divided

Semantic Congruence VisualMultisensory- Congruent Multisensory- Incongruent

Factors Modulating Multisensory Integration Temporal/spatial coincidence Semantic congruence Sensory attention –Selective –Divided

Selective Attention – Multisensory Processing

Selective attention suppresses multisensory integration Unpublished data removed

Behavioral studies Redundant multisensory benefits >> redundant unisensory benefits Multisensory Integration is modulated stimulus features –Timing –Location –Meaning Congruent – enhancements Incongruent- decrements Sensory attention modulates multisensory integration

Multisensory Brain Regions Superior temporal sulcus –STS (BA 22) Inferior temporo-occipital –BA 19/37 Angular gyrus –BA 39/40 Cingulate gyrus –BA 32/9/10 Brodmann Areas

Superior Temporal Sulcus Calvert et al, Current Biology 2000, 10:649–657 Beauchamp et al. – Nature Neuroscience 2004, Atteveldt et al. Neuron , 271–282.

Inferior Temporo-occipital L. Cohen et al. NeuroImage 23 (2004) 1256–1270 Hashimoto et al, Neuron, Vol. 42, (2004) 311–322 Beauchamp et al, Neuron, Vol. 41, 809–823, March 4, 2004,

Angular and Cingulate Gyri Saito et al Cereb Cortex. 2005, 15: Nonlinguistic Linguistic Laurienti et al. (2003) de Zubicaray et al. (2002)

Unisensory Brain Regions Brodmann Areas Do the unisensory cortices interact? How does attention modulate unisensory activity? Are unisensory interactions altered in dyslexia?

Unisensory Brain Regions Cross-modal deactivations are modulated by selective attention Unpublished data removed AuditionVision Laurienti et al. (2002)

Phoneme Recognition Task Dyslexic readers do not deactivate visual cortex during an auditory processing task Unpublished data removed

Imaging Multisensory Processes Multiple multisensory cortical regions Factors that modulate behavior (coincidence, congruence, attention) also modulate fMRI Unisensory and multisensory regions are closely situated Cross-modal deactivations Altered multisensory processing in dyslexia