Emergence of Semantics from Experience Jay McClelland Department of Psychology and Center for Mind, Brain, and Computation Stanford University 1

The Parallel Distributed Processing Approach to Semantic Cognition Representation is a pattern of activation distributed over neurons within and across brain areas. Bidirectional propagation of activation underlies the ability to bring these representations to mind from given inputs. The knowledge underlying propagation of activation is in the connections. Experience affects our knowledge representations through a gradual connection adjustment process language 2

Distributed Representations: and Overlapping Patterns for Related Concepts dog goat hammer dog goat hammer 3

Emergence of Meaning in Learned Distributed Representations Learned distributed representations that capture important aspects of meaning emerge through a gradual learning process in simple connectionist networks The progression of learning captures several aspects of cognitive development: Differentiation of Concepts Illusory Correlations Overgeneralization And many other things 4

5

The Rumelhart Model 6

The Training Data: All propositions true of items at the bottom level of the tree, e.g.: Robin can {grow, move, fly} 7

Target output for ‘robin can’ input 8

Forward Propagation of Activation aj ai wij neti=Sajwij wki 9

Back Propagation of Error (d) aj wij ai di ~ Sdkwki wki dk ~ (tk-ak) Error-correcting learning: At the output layer: Dwki = edkai At the prior layer: Dwij = edjaj … 10

11

12

Early Later Later Still E x p e r i e n c e 13

Why Does the Model Show Progressive Differentiation? Learning is sensitive to patterns of coherent covariation Coherent Covariation: The tendency for properties of objects to co-vary in clusters Figure shows attribute loadings on the principal dimensions of covariation. These capture: 1. Plants vs. animals 2. Birds vs. fish 3. Trees vs. flowers Same color = features that covary Diff color = anti-covarying features 14

Trajectories of Concept Representations During Differentiation 15

Illusory Correlations 16

A typical property that a particular object lacks e.g., pine has leaves An infrequent, atypical property 17

Overgeneralization of Frequent Names to Similar Objects “tree” “goat” “dog” 18

19

Other Applications of the Model Expertise effects Conceptual reorganization Effects of language and culture Effects of brain damage: Loss of differentiation Overgeneralization in object naming Illusory correlations camel swan 20

Conclusion We represent objects using patterns of activity over neuron-like processing units These patterns depend on connection weights learned through experience Differences in experience lead to differences in conceptual representations. 21