2. The Time Dimension for Scene Analysis DeLiang Wang Perception & Neurodynamics Lab The Ohio State University, USA

3. Presentation outline l Introduction l Scene analysis and temporal correlation theory l Oscillatory Correlation l LEGION network l Oscillatory Correlation Approach to Scene Analysis l Image segmentation l Object selection l Cocktail party problem l Concluding remarks

4. Scene analysis problem

5. Binding problem Feature binding (integration) is a fundamental problem in neuroscience and perception (and perceptrons) Binding problem in Rosenblatt’s perceptrons

6. Temporal correlation theory l Temporal correlation theory proposes a solution to the nervous integration problem (von der Malsburg’81; also Milnor’74) l Application to cocktail party processing (von der Malsburg & Schneider’86)

7. Physiological evidence (Gray et al.’89)

8. Oscillatory correlation theory Oscillators represent feature detectors Binding is encoded by synchrony within an oscillator assembly and desynchrony between different assemblies

9. Computational requirements Need to synchronize locally coupled oscillator population Need to desynchronize different populations, when facing multiple objects l Synchrony and desynchrony must be achieved rapidly

10. LEGION architecture l LEGION - Locally Excitatory Globally Inhibitory Oscillator Network (Terman & Wang’95)

11. Relaxation oscillator as building block Typical x trace (membrane potential) With stimulus Without stimulus

12. Analytical results l Theorem 1. (Synchronization). The oscillators in a connected block synchronize at an exponential rate l Theorem 2. (Multiple patterns) If at the beginning all the oscillators of the same block synchronize and different blocks desynchronize, then synchrony within each block and the ordering of activations among different blocks are maintained l Theorem 3. (Desynchronization) If at the beginning all the oscillators of the system lie not too far away from each other, then the condition of Theorem 2 will be satisfied after some time. Moreover, the time it takes to satisfy the condition is no greater than N cycles, where N is the number of blocks

13. Connectedness problem l Minsky-Papert connectedness problem is a long- standing problem in perceptron learning l The problem exposes fundamental limitations of supervised learning, and illustrates the importance of proper representations

14. Connectedness problem: LEGION solution l Basic idea: Synchronization within a connected pattern and desynchronization between different ones

15. Presentation outline l Introduction l Scene analysis and temporal correlation theory l Oscillatory Correlation l LEGION network l Oscillatory Correlation Approach to Scene Analysis l Image segmentation l Object selection l Cocktail party problem l Concluding remarks

16. Oscillatory correlation approach to scene segmentation l Feature extraction first takes place l An visual feature can be pixel intensity, depth, local image patch, texture element, optic flow, etc. l An auditory feature can be a pure tone, amplitude and frequency modulation, onset, harmonicity, etc. l Connection weights between neighboring oscillators are set to be proportional to feature similarity l Global inhibitor controls granularity of segmentation l Larger inhibition results in more and smaller regions l Segments pop out from LEGION in time

17. Image segmentation example: Demo Input image

18. Image segmentation example Input imageSegmentation result

19. Object selection l The slow inhibitor keeps trace of each pattern, which can be overcome by only more salient (larger) patterns l Unlike traditional winner- take-all dynamics, selection (competition) takes place at the object level l Consistent with object- based attention theory l Binding precedes attention, rather than attention precedes binding (Treisman & Gelade’80)

20. Results of object selection Input image LEGION output Selection output Input LEGION segmentation Selection

21. Cocktail party problem In a natural environment, target speech is usually corrupted by acoustic interference, creating a speech segregation problem l Popularly known as cocktail-party problem (Cherry’53); also ball- room problem (Helmholtz, 1863) Human listeners organize sound in a perceptual process called auditory scene analysis (Bregman’90) l Auditory scene analysis (ASA) takes place in two conceptual stages: l Segmentation. Decompose the acoustic signal into ‘sensory elements’ (segments) l Grouping. Combine segments into groups, so that segments in the same group likely originate from the same sound source

22. Oscillatory correlation for ASA (Wang & Brown’99) Frequency

23. Auditory periphery: Cochleagram l Cochleagram representation of the utterance: “Why were you all weary?” mixed with phone ringing

24. Grouping layer: Example l Two streams emerge from the group layer l Foreground: left (original mixture ) l Background: right l More recent results (Hu & Wang’04):

25. Back to physiology l Chattering cells recorded by Gray & McCormick’96 l Burst oscillations are best modeled by relaxation oscillators

26. Versatility and time dimension l The principle of universality: “Give me a concrete problem and I will devise a network that solves it.” (von der Malsburg’99) l It characterizes artificial intelligence l The principle of versatility: “Given the network, learn to cope with situations and problems as they arise.” (von der Malsburg’99) l It characterizes natural intelligence l Time dimension is necessary for versatility l Flexible and infinitely extensible l Irreplaceable by spatial organization

27. Conclusion l Advances in dynamical analysis overcome computational obstacles of oscillatory correlation theory l Major progress is made towards solving the scene analysis problem l From Hebb’s cell assemblies to von der Malsburg’s correlation theory, time is an indispensable dimension for scene analysis