1. Presented by Hanan Shteingart Physiology A – 2009, ICNC May 2009
Macaca Mulatta
Presented by Hanan Shteingart
Physiology A – 2009, ICNC
May 2009

2. Something to Wake You Up
Go to icnc.wordpress.com
to make your own dragon

3. How does the brain (IT) codes 3D objects?
Main Question
How does the brain (IT) codes 3D objects? ?
macaque monkey inferotemporal cortex (IT)

4. Abstract Complex shape codes - focused on 2D. 3D requires :
higher-D coding (more info)
computation (harder to decipher)
The Good News: evidence for an explicit neural code.
The Bad News: very complicated paper! (fancy object creation, evolutionary stimulus strategy, linear/nonlinear models with and crazy statistical tests)

5. Introduction object boundary fragments (features)
integration into explicit signal
component level shape (or holistic via learning) V2 IT V4
~isomorphic (106 pixels)
unwieldy and unstable  not useful for object perception
2D studies

6. The Hypothesis
H0: IT neurons encode three-dimensional spatial configurations of surface fragments.
H1: Complex shape perception is based primarily on two-dimensional image processing.
Previous studies
differential responses across a small number of 3D shapes or tuning along a single depth-related dimension
Question: how does H0 and H1 different? The paper says that “However, these results and the multiple-views
hypothesis itself are also compatible with three-dimensional representation29
(see Discussion).”

7. The 3D Decoding Problem
Which 3D shape factors are associated with neural responses?
3D  complex, multi tuning properties  Large stimulus set
wide range of 3D elements
combined in many different ways
A conventional random or systematic (grid-based) approach can never produce sufficiently dense combinatorial sampling.
This has not been attempted before.

8. Decoding Solution Evolutionary stimulus strategy Two advantages:
Focused
Variant
This evolutionary stimulus strategy made it possible for the first time to test the three-dimensional configural coding hypothesis at the neural level.
The second advantage is not clear (page 4 left)
Focused around the response range of the neuron (far less time)
shape characteristics encoded by the neuron are repeatedly varied and recombined with other shape features

9. Evolutionary Algorithms
Population Initialization
Fitness Scoring
Selection
Variation
Termination
Initial generation of 50 random 3D shapes
Averaged response = probability to reproduce
Descended morphed, either locally or globally

10. Evolutionary Stimulus Strategy

11. Response Model
Feature Space: Stimuli were characterized by 7 surface fragments:
X,Y,Z
XY, YZ angles,
max/min curvature
Two Gaussian function amplitude at the stimulus point closest to the Gaussian peak + nonlinear interactions.

12. Results Overview Strong cross-prediction of responses Model order = 21
Strong cross-prediction of responses
Model order = 2
Most fragments are outside plane (3D)
3D shape tuning independent of lighting, position, size and depth.
wide range of tuned configurations.
3D representation in IT is not holistic
Convexity bias 2
geometric similarities between runs are imposed by the neuron itself and thus serve to confirm the generality of the tuning model
The neuron remained responsive when either disparity or shading cues for depth were present, but failed to respond when both were eliminated

13. Tuning Functions Consistency: Non consistence: Lighting
Depth, Position, Size
Non consistence:
orientation
Light angle
depth
position
rotation
size
xy plane

14. 3D Verification
The modulation index is the response difference between the high- and low response stimuli, normalized by maximum response across all conditions.
three-dimensional shape tuning was largely independent of lighting direction, stimulus position, stimulus size and stimulus depth.
Separability is represented here by the fraction of response variance (r2) explainable by a matrix product between separate tuning functions for shape and shading, depth, position or size ??????????????

15. Surface-Fragment Configurations
Tuning models spanned a wide range of surface-fragment configurations
3D shape representation in IT is not generally holistic (model covers partially the whole shape)

16. Convexity Bias
stimuli
model
Tuning was markedly biased in the curvature domain toward high values, especially on the convex end of the scale.
spatial
curvature

17. Discussion
Convexity bias may reflect functional importance (emphasize pointy parts)
Substantial fraction of IT neurons followed H0: they were simultaneously tuned for 3D shape
Neurons tuned for multiple regions in this domain
consistent with classic theories of configural shape representation (‘geons’) with 2 differs:
No rotating of reference frame
Multi-part configuration for single neuron
H0 = neurons encode the three-dimensional shape, three-dimensional orientation and relative three-dimensional position of object parts

18. The Need for 3D Encoding?
Why would the brain explicitly represent complex 3D object shape, considering the computational expense of inferring 3D structure from the two-dimensional retinal image and the higher neural tuning dimensionality required?
Speculation: Representation of 3D object structure supports other aspects of object vision beyond identification (e.g. usage).

19. Configural coding of three-dimensional object structure
Henry Moore’s ‘‘Sheep Piece’’ (1971–1972)

20. Question to Eli
“The average neural response to each stimulus determined the probability with which it produced morphed descendants in subsequent stimulus generations” (pg 4) but in methods it says: a typical second generation would contain ten stimuli generated de novo, four descendants of stimuli in the highest response bin, four descendants from the second highest bin, etc. [equal probabilities]
Fig 3b. Response consistency was measured by separability of tuning for shape? Why is this consistency? It’s more independence of variables or something.