1. Distributed versus Sparse Representations and Unsupervised Learning
Chapter 5, Anastasio

2. Distributed, Sparse, Unsupervised
Learning objectives:
explain the difference between “sparse” and “distributed” neuronal encoding
explain the difference in purpose between Hopfield networks and two-layer neural networks
simulate the development of “broadly-tuned specialist” neurons using two-layer neural networks

3. “Face cells” in the inferior temporal cortex of the monkey have been shown to respond
to faces of other monkeys by elevating their firing rates
So face cells are “specific” in the sense that they respond only to faces…
…but they are “broadly tuned” because they respond to varying degrees to a spectrum
of face-like images

4. Fusiform face area
Part of both temporal lobe (involved in memory) and occipital lobe (involved in vision)

5. Using fMRI to demonstrate the role of the fusiform gyrus in facial recognition
N. Kanwisher, J. McDermott, & MM Chun. “The fusiform face area: a module in human extrastriate cortex specialized for face perception.” J. Neurosci., 1997.

6. Face blindness (prosopagnosia)
People with face blindness can’t readily recognize faces because their fusiform
face area does not function properly; they have to rely on the brain’s non-facial object
recognition system, which is not nearly as sensitive
It would be like trying to identify people’s upside-down faces
You can still do it, but it’s a lot harder
Our brains are hard-wired to recognize faces

7. “Jennifer Aniston cells”
A 2005 Nature paper showed that a single cell in the medial temporal lobe (hippocampus and surrounding cortex) responded to pictures of Jennifer Aniston
Another neuron in the same area responded to pictures of Halle Barry, as well as the word “Halle Barry”
RQ Quiroga, I Fried, C Koch. “Brain Cells for Grandmother.” Scientific American, Feb

8. R.Q. Quiroga, L. Reddy, et. al. “Invariant visual representation by single neurons in the human brain.” Nature, 2005.

9. R.Q. Quiroga, L. Reddy, et. al. “Invariant visual representation by single neurons in the human brain.” Nature, 2005.

10. Concept cells
A better name for these cells is “concept cells,” because it turns out they respond to broader concepts than just one person
For example, the so-called “Jennifer Aniston cell” also responded to pictures of Lisa Kudrow, another actress on Friends
A cell that responded to pictures of Luke Skywalker also responded to pictures of Yoda
Concept cells are “broadly tuned specialists”

11. Distributed versus sparse representation
Millions or billions
of cells
Thousands of cells
Related concepts are encoded by
overlapping sets of cells, which
would constitute the
biophysical basis
for cognitive associations
between the two concepts
--there is almost certainly no such thing as a “grandmother cell,” in the sense of one single cell that encodes for a particular concept (for one, it would be an incredibly delicate memory—if that one neuron died, you would lost that memory! We know that memory is more stable than that)
--instead, the debate is whether neurons store memories and encode concepts using a sparse (on the order of thousands of cells) or a distributed representation (millions to billions of cells)
Unrelated concepts are encoded
by non-overlapping sets of cells
RQ Quiroga, I Fried, C Koch. “Brain Cells for Grandmother.”
Scientific American, Feb

12. How might the brain form representations of sensory objects?
Think of each sensory object as having a pattern of
different “features”
Train the units of a neural network to be specialized
(but broadly tuned) for a different pattern

13. Representing different sensory objects with “feature vectors”

14. Neural network for identifying animals
Fur
Barks
Meows
Fins
--one pattern is presented to the input layer of the network; the weights are then modified according to some rule
--this process is repeated, and eventually one neuron in the output layer will “specialize” for this input pattern
--other output neurons will specialize for other input patterns
--since there are give output neurons, each one could specialize for a different animal
--but some neurons will partially activate in response to an animal that is not their “preferred” animal, if it shares some of the same features as their preferred animal (the “bat neuron” will partially respond to a “cat” input pattern, for example)
--this is called unsupervised learning because we don’t label the input patterns; in supervised learning, we do label the input patterns
Gills
Feathers
Wings

15. Neural network for identifying animals
Cat feature vector
Weights are modified 1
Fur
Barks
Responds to cat
feature vector 1
Meows
Fins
--one pattern is presented to the input layer of the network; the weights are then modified according to some rule
--this process is repeated, and eventually one neuron in the output layer will “specialize” for this input pattern
--other output neurons will specialize for other input patterns
--since there are give output neurons, each one could specialize for a different animal
--but some neurons will partially activate in response to an animal that is not their “preferred” animal, if it shares some of the same features as their preferred animal (the “bat neuron” will partially respond to a “cat” input pattern, for example)
--this is called unsupervised learning because we don’t label the input patterns; in supervised learning, we do label the input patterns
Gills
Feathers
Wings

16. Neural network for identifying animals
Bat feature vector
Weights are modified 1
Fur
Barks
Meows
Fins
Responds to bat
feature vector
--one pattern is presented to the input layer of the network; the weights are then modified according to some rule
--this process is repeated, and eventually one neuron in the output layer will “specialize” for this input pattern
--other output neurons will specialize for other input patterns
--since there are give output neurons, each one could specialize for a different animal
--but some neurons will partially activate in response to an animal that is not their “preferred” animal, if it shares some of the same features as their preferred animal (the “bat neuron” will partially respond to a “cat” input pattern, for example)
--this is called unsupervised learning because we don’t label the input patterns; in supervised learning, we do label the input patterns
Gills
Feathers 1
Wings

17. Two-layer neural networks (TLNN) vs. Hopfield networks
Both learn patterns by modifying the strength of connections between neural units
Hopfield networks are intended to complete partial or corrupted patterns, while TLNN’s are intended to sparsely represent input patterns
Hopfield networks only require one exposure to each pattern, while TLNN’s require many

18. TLNN Learning Rule “feature vector” (input) Output vector
Anastasio, p. 139

19. Biological plausibility
Hebbian plasticity
Homeostatic plasticity
(connections strengths
do not get too large)

20. Example Problem
Initially set V = [0 1/sqrt(2) 0 1/sqrt(2); ]; and the pattern matrix P to P = [ ; ]; Set the training rate a=1. Then conduct 5 iterations of training on the first input pattern (x=P(1,:)’), followed by 5 iterations of training on the second input pattern (Use a=1, and the ‘max’ function for Eqn. 5.2 and ‘norm’ for Eqn 5.3). What is your final matrix, V? Interpret what it means—does the first output neuron encode the first or second input pattern? How do you know? Save your code as simple_example.m.
--ask, “How many neurons are there in the input layer? How about the output layer?”

21. Extension Problem (in lab)
Modify your code from Problem 1 so that V starts out as a 2x4 matrix of random numbers between 0 and 1. Run this several times, and look at the final matrix V each time. Does the first output neuron encode the first or second input pattern? What determines this? Submit this code as random_example.m.

22. Another Extension Problem (lab also)
Modify your code from the first exercise so that you have overlapping patterns, P = [1/sqrt(2) 1/sqrt(2) 0 0; 0 1/sqrt(2) 1/sqrt(2) 0 ]; If you set V = [0 1/sqrt(2) 0 1/sqrt(2); ]; does the network successfully encode each pattern? Why? Now modify the code so that the matrix V is initially random. Does the network now successfully encode each pattern? Why? Submit this code as overlap_example.m.

23. Unsupervised learning

24. Training a neural network to represent 3D basis vectors
Feature Vector y 1 x z

25. Training a neural network to represent 3D basis vectors
Feature Vector y x z 1

26. Training a neural network to represent 3D basis vectors
Feature Vector y x z 1

27. The point of unsupervised learning: the outputs become specialized for specific input patterns.

28. Start with the matrix of input patterns, InPat=[1 0 0; 0 1 0; 0 0 1] Then start with random connectivity between neurons, V=rand(3,3); Without any training, what is the output when you present the first pattern?

29. You can compute the output of each neuron, for each pattern, by just typing Out = V*InPat’

30. To represent each pattern as a row, type Out = (V
To represent each pattern as a row, type Out = (V*InPat’)’ Set nPat=3, nOut=3, and nIn=3, then visualize input and output by running SOMoutImage.m. What should the output image look like if training of the neural network is successful?

31. Run KohonenSOM.m. This trains the weights of the neural network and generates a new Out matrix. Visualize input and output with SOMoutImage.m. Was the training successful? Which output neuron represents the x vector?

32. Look at weight matrix V. Why does it take the form it does?

33. Training few output neurons to represent many input patterns

34. Write code which normalizes the input patterns, then plots them as vectors in 3D space using the quiver3 function. Name the m-file plot3d_vectors.m.

35. Set V to be a random matrix with values uniformly distributed between 0 and 1. If we require that the network have three output neurons, what size should V be? Calculate the output patterns for all input patterns, and display them. Is there any inherent specificity in the output neurons for the input patterns?

36. Now run KohonenSOM. m, and visualize the results
Now run KohonenSOM.m, and visualize the results. Each output neuron has become specialized for its own input pattern cluster.

37. What if there is an input pattern that is halfway between two clusters?
Set
newpattern=[1/sqrt(2) 1/sqrt(2) 0]’;
What are the activations of the three output neurons in response to this new pattern?

38. Wilder Penfield
Neurosurgeon who in the 1930’s mapped the somatosensory cortex
Stimulating different parts of the cortex resulted in patients feeling
sensations at different locations of their body

39. Cerebral cortex Midbrain Pons Medulla Medulla Spinal cord
Neural pathways from different parts of the body
converge on the spinal cord, where they ascend
and eventually reach the somatosensory cortex
Pons
Medulla
Medulla
Spinal cord
Principles of Neural Science, Kandel, p. 361

40. The human sensory homunculus
The somatosensory cortex forms a
somatotopic map of the body surface
The area of the cortex devoted to each
body part is proportional not to the mass
of that part, but to the density of innervation
It is a fractured map; for example, the part
part of cortex that is associated with the head
is far from the part associated with the face
Principles of Neural Science, Kandel, p. 364

41. Different species are more sensitive to sensation in different parts of their body
Principles of Neural Science, Kandel, p. 377

42. Brain maps
A brain map is a region of the brain in which stimulus patterns that are similar are encoded in nearby brain regions. How does the brain form these maps? And why?

43. Add cooperation to
competition, letting
the winner and its
nearest neighbors
have their weights
be modified

45. Problem 5.2 (lab)