Reading, Language, Brain: The role of computational models Mark S. Seidenberg University of Wisconsin-Madison

I am very happy to be here. And to visit the National Key Laboratory of Cognitive Neuroscience and Learning The modern study of reading in Chinese began with the pioneering work of Shu Hua. In the West, this is the most famous laboratory for scientific studies of reading and language in China. And China is a very big country.

My Own Work Children, adults Normal, dyslexic English, Serbian, Chinese, others Brain circuits Behavior Computational models Connectionist models that simulate detailed aspects of acquisition, skilled performance. Dyslexia = anomalies in how system develops Mainly at Medical College of Wisconsin. Jeff Binder, Lab director.

Computational models Seidenberg & McClelland, 1989 (about the time neuroimaging came into psychology)

Computational models Harm and Seidenberg, 1999: phonology, dyslexia Zevin and Seidenberg, 2006: consistency effects, nonwords, individual differences

Computational models Joanisse & Seidenberg, 2000: verb morphology. The “past tense debate”

Computational Models Harm & Seidenberg, 2004: computing MEANING

Many other models Many people, labs

Inferior frontal/insular cortex Inferior SMG buffer Supramarginal gyrus Primary auditory area/ and surroundings (Input) Inferolateral anterior temporal region (Semantics) Temporal lobe Insular-Opercularis (Output) Opercularis-Triangularis Anterior STG/STS Middle STG/STS Sound + auditory word input Activation of meaning (from other modalities) and input to spontaneous speech production 9 Articulatory speech

Hidden layer Elman Input Sound Semantics (Input & Output) Output sound Hidden layer 10 Ueno, Lambon Ralph, Rogers, 2012

The reading models instantiated several theoretical principles. General, not linked to details of any single model Are they correct? Are they still relevant? Needed? What good are these models???

But, how are these components, processes realized? What is in the ovals? What does an arrow represent? How is the system learned? How is it represented in the brain? Every theory/model must have these basic elements: What reading IS. What is different about “dual-route” models (e.g., DRC, CDP+) meaning spellingsound context 1. only about pronunciation aloud: no semantics 2. a second mechanism for pronouncing some words: adds word frequency 3. “functional architecture”: independent of brain 4. no learning

Let’s look at some of the basic principles. And what has happened to them.

1. Distributed representations Current status: fMRI evidence for word-specific representations univariate methods biased against finding highly distributed representations? Much research using other methods (MVPA and related tools) Chris Cox (Wisconsin grad student), Tim Rogers, M. Seidenberg How are categories represented? Artifacts, animals, etc.

2. Variable mappings between codes Arbitrary, correlated, to what degree? Quasiregularity Depends on writing system More about this later! Current status: many investigations across different writing systems Dual-route models: DRC: GPC rules. Very different. CDP+: rules are gone. Replaced by connectionist network.

3. Statistical learning Learning based on frequencies, distributions of events gradual, structures emerge over time Current status: 1. Using models to look at reading development under atypical conditions perceptual, learning, experiential deficits dialect differences in US teaching methods

3. Statistical learning Learning based on frequencies, distributions of events gradual, structures emerge over time Current status: 1. Using models to look at reading development under atypical conditions perceptual, learning, experiential deficits dialect differences in US teaching methods 2. Statistical learning in language acquisition

Related mechanisms in reading Chinese? 刘川生书记在致辞中代表学校向莅临大会的领导和专家表示热烈的欢迎，对 北京市委、市政府长期以来关心和支持北师大建设表示衷心的感谢。刘川生 书记指出，文化是民族的精神家园，推进文化创新发展是时代赋予我们的神 圣使命。大学是文化传承创新的重要阵地，在中华文化创新和传播的伟 Where are the words?

4. Processing by satisfying multiple constraints. A general way of solving complex problems. Part of what makes humans intelligent! Key idea: nonlinear combination of clues in isolation, not very informative together, very informative What is it? A living thing

4. Processing by satisfying multiple constraints. A general way of solving complex problems. Part of what makes humans intelligent! Key idea: nonlinear combination of clues in isolation, not very informative together, very informative What is it? A living thinguniversity professor

4. Processing by satisfying multiple constraints. A general way of solving complex problems. Part of what makes humans intelligent! Key idea: nonlinear combination of clues in isolation, not very informative together, very informative What is it? A living thinguniversity professorVisiting NKLCNL

4. Processing by satisfying multiple constraints. A general way of solving complex problems. Part of what makes humans intelligent! Key idea: nonlinear combination of clues in isolation, not very informative together, very informative What is it? A living thinguniversity professorVisiting NKLCNL American ME!

In reading aloud orthographyphonologysemantics Activation from both parts

Recent research: is semantics used in reading words aloud (in English)? Dual-route models: No Some studies show such effects (Strain, Patterson, Seidenberg, 1996) but not all.

Will Graves, Medical College of Wisconsin  Rutgers University Imageability effects on reading words aloud Skilled adult readers Highly educated Individual differences among readers?

DTI volume of sem-phon pathways AG – pSTG ITG – pMTG Correlate with imageability effect For skilled readers of English

So: skilled readers (in English) differ. Related to anatomical differences in relevant parts of reading circuit. Need to look at other individual differences.

Combining constraints in reading for meaning: orthographyphonologysemantics

Reading Chinese “ radical ”

But contributions vary with type of character Transparency of radical? Consistency of phonetic? Characters with other structures N-N compounds like TELEVISION 电视 more semantics, less phonology?

“Division of labor” depends on writing system, word, task, reader

About morphology Much debate about what is “morphological” in Chinese. Much debate about what is “morphological” in English. Simple view: minimal units of meaning. Discrete. Combined like beads on a string BOAT HOUSEHOUSE BOAT

But, many partial regularities BAKER:person who bakesBAKE + ER TALKER:person who talksTALK + ER GROCER:person who sells food*GROCE CORNERa vegetablenot CORN + ER DISLIKEDIS + LIKEnot LIKE DISAGREEDIS + AGREEnot AGREE with DISCOVERDIS + COVER*not COVER SWEETBREADSnot sweet, not bread BOOTLEGmake illegal alcohol SLAPSTICKa kind of humor

Similarity ratings: TEACHERTEACH BACKERBACK CORNERCORN

Seidenberg and Gonnerman, TICS, 2000 Morphemes are graded, not discrete Reflect correlations between form and meaning Which vary in degree Many similarities to Chinese

5. Interconnectivity Representations determined by functions in circuits

6. Models perform TASKS Activation is task dependent. Circuits both represent and process information Many studies showing task-dependence of activation in areas like pOTS Yang et al. Mano et al. (in press): in naming vs. visual discrimination tasks others

Conclusions The models are a useful tool. They are not literally correct. The principles that govern them are relevant to understanding brain, behavior. More so than the specific architectures that were proposed; too simple! 1. distributed representations 2. variable mappings 3. statistical learning 4. constraint satisfaction 5. Interactivity, feedback 6. task orientation (others)

Thank you! Thanks to collaborators Modeling Tim Rogers (Wisconsin) Chris Cox (Wisconsin) Jason Zevin (Weill Cornell Sackler) Michael Harm (Google) Marc Joanisse (Western Ontario) David Plaut (CMU) Jay McClelland (Stanford) Imaging (Medical College of Wisconsin Jeff Binder, lab director Will Graves Rutvik Desai Quintino Mano Chinese language Tianlin Wang (University of Wisconsin)

Binder, Graves, Desai, Seidenberg articles