An Oscillatory Correlation Approach to Scene Segmentation DeLiang Wang The Ohio State University

IPAM'022 Outline of Presentation l Introduction l Scene segmentation problem l Oscillatory Correlation Theory l LEGION network l Oscillatory Correlation Approach to Image Segmentation l Circuit Implementation of LEGION l Summary

IPAM'023 Scene Analysis Problem

IPAM'024 Simplified Scenario

IPAM'025 Temporal Correlation Theory Feature binding is a fundamental problemn l In neuroscience l In perception Temporal correlation as a representation l Extra degree of freedom l A plausible mechanism See von der Malsburg’81, Milner’74, Abeles’82

IPAM'026 Neurophysiological Evidence l Gray & Singer (1989)

IPAM'027 Neurophysiological Evidence - continued l Gray et al. (1989)

IPAM'028 Oscillatory Correlation Theory

IPAM'029 Computational Requirements for Oscillatory Correlation Need to synchronize locally coupled oscillator population l Extensive literature in theoretical physics and applied mathematics Need to desynchronize different populations, when facing multiple objects l Critically, the above functions must be achieved very rapidly

IPAM'0210 Three Possible Ways to Reach Synchrony l Comparator model l Globally connected l Locally connected

IPAM'0211 LEGION Architecture l LEGION - Locally Excitatory Globally Inhibitory Oscillator Network (Terman & Wang’95)

IPAM'0212 Single Relaxation Oscillator With stimulus Without stimulus Typical x trace (membrane potential)

IPAM'0213 Model of a Relaxation Oscillator l Model definition l Coupling between oscillators l Where N(i) is the set of neighboring oscillators that connect to oscillator i

IPAM'0214 l Global inhibitor Model of a Relaxation Oscillator - continued

IPAM'0215 Somers & Kopell’93 Fast Threshold Modulation for Two Oscillators LB RB E LB E RB

IPAM'0216 Selective Gating for Two Oscillators RB LB LB I

IPAM'0217 Summary of Analytical Results Definitions: A pattern is a connected region, and a block a subset of oscillators stimulated by a given pattern. The following results are established for  > 0 sufficiently small l Theorem 1. (Synchronization). The parameters of the system can be chosen so that all of the oscillators in a block synchronize. Moreover, the rate of synchronization is exponential

IPAM'0218 Summary of Analytical Results - continued Theorem 2. (Multiple patterns) If at the beginning all the oscillators of the same block synchronize with each other and the distance between any two oscillators belonging to two different blocks is greater than some constant, then: l Synchronization within each block is maintained l The ordering of the activation among different blocks is fixed l Given a certain period of time, at least one block is in its active phase l At most one block is in its active phase at any time

IPAM'0219 Summary of Analytical Results - continued Theorem 3. (Desynchronization) If at the beginning all the oscillators of the system lie not too far away from each other, then the condition of Theorem 2 will be satisfied after some time. Moreover, the time it takes to satisfy the condition is no greater than N cycles, where N is the number of patterns. l The entire mechanism is called Selective Gating (Terman & Wang’95)

IPAM'0220 LEGION Example Input image Successive snapshots

IPAM'0221 LEGION Example - Temporal Traces

IPAM'0222 LEGION Example: Segmentation Capacity Input image

IPAM'0223 Oscillatory Correlation Approach to Image Segmentation l Feature extraction first takes place l An image feature can be pixel intensity, depth, local image patch, texture element, optic flow, etc. l Connection weights between two neighboring oscillators are set to be proportional to feature similarity l Global inhibitor controls granularity of segmentation l Larger inhibition results in more and smaller regions l Segments pop out from LEGION in time

IPAM'0224 Image Segmentation Example Input imageSegmentation result

IPAM'0225 Image Segmentation Example Input imageSegmentation result

IPAM'0226 3D MRI Image Segmentation Left: LEGION results Right: Manual results

IPAM'0227 3D MRI Image Segmentation: a 2D View Left: Input Middle: Segmentation results Right:Further segmentation into white and gray matter

IPAM'0228 Aerial Image Analysis Extraction of hydrographic objects

IPAM'0229 Texture Segmentation Upper: input; Lower: segmentation result

IPAM'0230 Motion Segmentation

IPAM'0231 Circuit Implementation of LEGION l Circuit implementation of LEGION using PWM/PPM l PWM: Pulse-width modulation PPM: Pulse-phase modulation l By Ando et al. (2000) Pulse modulation

IPAM'0232 Circuit Implementation of a Single Oscillator Circuit of a single relaxation oscillator SPICE simulation result

IPAM'0233 Circuit Implementation of a LEGION Network Connection circuit Global inhibitor circuit

IPAM'0234 Circuit Implementation of LEGION - continued Input image l Application to gray-level image segmentation Output at two different times

IPAM'0235 LEGION on a Chip (Cosp 2000) The chip area is 6.7 mm 2 (Core 3 mm 2 ) and implements a 16x16 LEGION network

IPAM'0236 Back to Biology Gray & McCormick’96

IPAM'0237 Summary l Mathematical analysis of the selective gating mechanism and LEGION dynamics l Both synchronous oscillations and the structure of the model are neurally plausible l Oscillatory correlation approach to scene segmentation l VLSI implementation A neural theory for perceptual organization

IPAM'0238 Collaborators l David Terman l Erdogan Cesmeli l Ke Chen l Xiuwen Liu l Naeem Shareef