1. Does the Brain Use Symbols or Distributed Representations?
James L. McClelland
Department of Psychology and Center for Mind, Brain, and Computation Stanford University

2. Parallel Distributed Processing Approach to Semantic Cognition
Representation is a pattern of activation distributed over neurons within and across brain areas.
Bidirectional propagation of activation underlies the ability to bring these representations to mind from given inputs.
The knowledge underlying propagation of activation is in the connections.
language

3. Development and Degeneration
Learned distributed representations in an appropriately structured distributed connectionist system underlies the development of conceptual knowledge.
Gradual degradation of the representations constructed through this developmental process underlies the pattern of semantic disintegration seen in semantic dementia.

4. Differentiation, ‘Illusory Correlations’, and Overextension of Frequent Names in Development

6. The Rumelhart Model

7. The Training Data:
All propositions true of items at the bottom level of the tree, e.g.:
Robin can {grow, move, fly}

8. Target output for ‘robin can’ input

9. Forward Propagation of Activationaj ai
wij
neti=Sajwij
wki

10. Back Propagation of Error (d)aj
wij ai
di ~ Sdkwki
wki
dk ~ (tk-ak)
Error-correcting learning:
At the output layer: Dwki = edkai
At the prior layer: Dwij = edjaj …

13. Early
Later
Later Still
E x p e r i e
n c e

15. Why Does the Model Show Progressive Differentiation?
Learning in the model is sensitive to patterns of coherent covariation of properties
Coherent Covariation:
The tendency for properties of objects to co-vary in clusters

16. Patterns of Coherent Covariation in the Training Set
Patterns of coherent covariation are reflected in the principal components of the property covariance matrix of the training patterns.
Figure shows attribute loadings on the first three principal components:
1. Plants vs. animals
2. Birds vs. fish
3. Trees vs. flowers
Same color = features covary in component
Diff color = anti-covarying features

17. Sensitivity to Coherence Requires ConvergenceA A A

18. An Approximate Probabilistic Characterization of the Model: The Naïve Bayes Classifier Model
One-layer Naïve Bayes Classifier Model:
P(a|i,{C}) = Skp(a|Ck)(p(Ck|i))
Mixture of Nested Bayes Classifiers:
P(a|i,{D,S,B,I}) = aDP(a|i,D) + aSP(a|i,{S}) + aBP(a|i,{B}) + aIP(a|i,{I})
Alternative formulation under development:
P(a|i,{D,S,B,I}) = aIP(a|i,{I}) + (1-aI)aBP(a|i,{B}) + (1-aB)aSP(a|i,{S}) + (1-aS)aDP(a|i,D)

19. A One-Class and a Two-Class Naïve Bayes Classifier Model
Property
One-Class Model
1st class in two-class model
2nd class in two-class model
Can Grow
1.0
Is Living
Has Roots
0.5
Has Leaves
0.4375
0.875
Has Branches
0.25
Has Bark
Has Petals
Has Gills
Has Scales
Can Swim
Can Fly
Has Feathers
Has Legs
Has Skin
Can See

20. Accounting for the network’s representations with classes at different levels of granularity
Living Thing
Plant
Tree
Pine
Bias
Regression Beta Weight
Epochs of Training

21. Illusory Correlations
Rochel Gelman found that children think that all animals have feet.
Even animals that look like small furry balls and don’t seem to have any feet at all.
A tendency to over-generalize properties typical of a superordinate category at an intermediate point in development is characteristic of the PDP network.

22. A typical property that a particular object lacks
e.g., pine has leaves
An infrequent,
atypical property

23. Overgeneralization of Frequent Names to Similar Objects
“tree”
“goat”
“dog”

24. Why Does Overgeneralization of Frequent Names Increase and then decrease?
In the simulation shown, dogs are experienced 10 times as much as any other animal, and there are 4 other mammals, 8 other animals, and ten plants.
In a one-class model, goat is a living thing:
P(name is ‘Dog’|living thing) = 10/32 = ~.3
In a two-class model, goat is an animal:
P(name is ‘Dog’|animal) = 10/22 ~.5
In a five class model, goat is a mammal:
P(name is ‘Dog’|mammal) = 10/15 = .67
In a 23 class model, goat is in a category by itself:
P(name is ‘Dog’|goat) = 0

26. Development and Degeneration
Sensitivity to coherent covariation in an appropriately structured Parallel Distributed Processing system underlies the development of conceptual knowledge.
Gradual degradation of the representations constructed through this developmental process underlies the pattern of semantic disintegration seen in semantic dementia.

27. Disintegration of Conceptual Knowledge in Semantic Dementia
Progressive loss of specific knowledge of concepts, including their names, with preservation of general information
Overgeneralization of frequent names
Illusory correlations

28. Picture naming and drawing in
Sem. Demantia

30. Inference and Generalization in the PDP Model
A semantic representation for a new item can be derived by error propagation from given information, using knowledge already stored in the weights.
Crucially:
The similarity structure, and hence the pattern of generalization depends on the knowledge already stored in the weights.

31. Start with a neutral representation on the representation units
Start with a neutral representation on the representation units. Use backprop to adjust the representation to minimize the error.

32. The result is a representation similar to that of the average bird…

33. Use the representation to infer what this new thing can do.

34. Proposed Architecture for the Organization of Semantic Memory
action
name
Medial Temporal Lobe
motion
Temporal
pole
color
valance
form

35. Conclusion
Distributed representations gradually differentiate in ways that allow them to capture many phenomena in conceptual development.
Their behavior is approximated by a blend of Naïve Bayes classifiers across several levels of granularity, with the blending weights shifting toward finer grain categories as learning progresses.
Effects of damage are approximated by a reversal of this tendency: degraded representations retain the coarse-grained level knowledge but loose the finer-grained information.
We are currently extending the models to address the sharing of knowledge across structurally related domains, I’ll be glad to discuss this idea in response to questions.

36. Rogers et al (2005) model of semantic dementia
name
assoc
function
temporal pole
vision

37. Errors in Naming for As a Function of Severity
Simulation Results
Patient Data
omissions
within categ.
superord.
Severity of Dementia
Fraction of Neurons Destroyed

38. Simulation of Delayed Copying
Visual input is presented, then removed.
After several time steps, pattern is compared to the pattern that was presented initially.
name
assoc
function
temporal pole
vision

39. Simulation results IF’s ‘camel’ DC’s ‘swan’ Omissions by feature type
Intrusions by feature type

40. Differential Importance (Marcario, 1991)
3-4 yr old children see a puppet and are told he likes to eat, or play with, a certain object (e.g., top object at right)
Children then must choose another one that will “be the same kind of thing to eat” or that will be “the same kind of thing to play with”.
In the first case they tend to choose the object with the same color.
In the second case they will tend to choose the object with the same shape.

41. Adjustments to Training Environment
Among the plants:
All trees are large
All flowers are small
Either can be bright or dull
Among the animals:
All birds are bright
All fish are dull
Either can be small or large
In other words:
Size covaries with properties that differentiate different types of plants
Brightness covaries with properties that differentiate different types of animals

44. Similarities of Obtained Representations
Size is relevant
for Plants
Brightness is relevant for Animals