1. Learning complex visual concepts
early motion abilities
detection
segmentation
tracking
learn to perceive coherent objects
Dorfman, Harari, Ullman
hands
gaze direction
learning complex visual concepts (e.g. when infant, unsupervised)
- leverage basic motion abilities may be present at birth, learn hands
- leverage hands and face (also early) to learn direction of gaze
- object segregation – complex perceptual process, integrate multiple cues
- computationally challenging – humans outperform current models
- infants limited early – 3m treat as single object, 4.5m segregate/motion
- use of static cues develops slowly over first year (color, texture, continuity)
- adult – count distinct cars, delineate boundaries, occlusion relations
- use motion abilities to learn object segregation cues, leverage to learn to
perceive coherent objects (static), learn object concepts (movable things)
- how learn cues/object segment, what initial capacities make possible?
- model starts 2 capacities: group image regions by common motion, detect
motion discontinuities, learns aspects of object segmentation/static images
- (as before) unsupervised, observing videos of objects in motion
object segregation

2. Learning to segregate objects – Dorfman et al. (2013)
- video clips from Danny’s project page (details), two cues:
- infants group adjacent regions together, using common motion,
can later identify same regions in static images
- motion discontinuities – infants sensitive boundaries, used in model
to learn useful static boundary cues (teaching signal to extract image
features along object boundaries, figure/ground (encoded in color),
use to locate object boundaries, ID figure direction, new static images
- outset: no ability to segregate objects in static images, only motion

3. Computing the 2D motion field
used an “off-the-shelf”
motion algorithm
(Sun et al., 2010)
divide image into stationary background
+ moving regions
select a moving region for further processing…

4. From moving regions to static object segregation
cover image region with SIFT image descriptors
(Lowe, 2004)
capture distribution of image gradients at multiple scales
computed at “keypoints”
store location of each SIFT descriptor relative to center of moving region
search for similar distribution of SIFT descriptors in new static image
selected moving region

5. Testing object segregation in static images
learn object models from each movie (5 sec, 40 sec segments)
test on static images with variety of backgrounds, pose, lighting
overall good performance…
... but some errors
boundaries not dilineated well
Goal object-based segmentation: learn appearance of specific object (doll,
fruit, etc. in movies, detect/extent/separate from background in new images -

6. Learning boundary features
use motion discontinuities to learn static cues for occluding boundaries
in each frame of the training movies:
detect motion discontinuities
extract image patches along boundaries (5 sizes)
represent each patch with a SIFT descriptor
label figure (moving) & ground (stationary) sides of boundary
in new static images:
look for image locations with similar SIFT descriptors for same 5 sizes of image patch centered on this location
T-junctions convexity extremal edges

7. Combining information sources

8. Summary
object segregation is a complex task, learned gradually from infancy
static object segregation can be learned from motion
two mechanisms work in synergy:
from motion discontinuities, learn cues to occlusion boundaries
from regions of common motion, learn object forms
enough to get started…
... but object segregation by adults is much more complex!