1. What should be done at the Low Level?
16-721: Learning-Based Methods in Vision
A. Efros, CMU, Spring 2009

2. Class Introductions Name: Research area / project / advisor
What you want to learn in this class?
When I am not working, I ______________
Favorite fruit:

3. Analysis Projects / Presentations
Wed: Varun
note-taker: Dan
Next Wed: Dan
note-taker: Edward
Dan and Edward need to meet with me ASAP
Varun needs to meet second time

4. Four Stages of Visual Perception
Ceramic
cup on
a table
David Marr, 1982
© Stephen E. Palmer, 2002

5. Four Stages of Visual Perception
The Retinal Image
An Image
(blowup)
Receptor Output
© Stephen E. Palmer, 2002

6. Four Stages of Visual Perception
Retinal
Image
Image-based
Representation
An Image
Primal Sketch
(Marr)
Image-
based
processes
Edges
Lines
Blobs
etc.
(Line Drawing)
© Stephen E. Palmer, 2002

7. Four Stages of Visual Perception
Image-based
Representation
Surface-based
Representation
Primal Sketch
2.5-D Sketch
Surface-
based
processes
Stereo
Shading
Motion
etc.
© Stephen E. Palmer, 2002

8. Koenderink’s trick

9. Four Stages of Visual Perception
Surface-based
Representation
Object-based
Representation
2.5-D Sketch
Volumetric Sketch
Object-
based
processes
Grouping
Parsing
Completion
etc.
© Stephen E. Palmer, 2002

10. Geons
(Biederman '87)

11. Four Stages of Visual Perception
Object-based
Representation
Category-based
Representation
Volumetric Sketch
Category-
based
processes
Pattern-
Recognition
Spatial-
description
Basic-level Category
Category: cup
Color: light-gray
Size: 6”
Location: table
© Stephen E. Palmer, 2002

12. We likely throw away a lot

13. line drawings are universal

14. However, things are not so simple…
Problems with feed-forward model of processing…

15. two-tone images

18. “attached shadow” contour
hair (not shadow!)
“cast shadow” contour
inferred external contours

20. Cavanagh's argument
Finding 3D structure in two-tone images requires distinguishing cast shadows, attached shadows, and areas of low reflectivity
The images do not contain this information a priori (at low level)

21. Feedforward vs. feedback models
Marr's model (circa 1980) Cavanagh’s Model (circa 1990s)
object recognition by matching 3D models
Object
basic recognition with 2D primitives
memory
3D model
3D shape
2½D sketch
2D shape
feedback
primal sketch
reconstruction of shape from image features
stimulus
stimulus

22. A Classical View of Vision
Object and Scene
Recognition
High-level
Figure/Ground
Organization
Mid-level
Grouping /
Segmentation
In fact, here is a classical view of visual perception, in which figure/ground organization actually plays a very important role. In this linear architecture, first we have the image on the retina, then pixels in the image are grouped together into regions and contours. After grouping, figure/ground origination takes place, which assigns the ownership of contours and forms the perception of shape. Based on that, we have the recognition of objects and scenes at the end.
Low-level
pixels, features, edges, etc.

23. A Contemporary View of Vision
Object and Scene
Recognition
High-level
Figure/Ground
Organization
Grouping /
Segmentation
Mid-level
In fact, here is a classical view of visual perception, in which figure/ground organization actually plays a very important role. In this linear architecture, first we have the image on the retina, then pixels in the image are grouped together into regions and contours. After grouping, figure/ground origination takes place, which assigns the ownership of contours and forms the perception of shape. Based on that, we have the recognition of objects and scenes at the end.
But where we
draw this line?
Low-level
pixels, features, edges, etc.

24. Question #1: What (if anything) should be done at the “Low-Level”?
N.B. I have already told you everything that is known. From now on, there aren’t any answers.. Only questions…

25. Who cares? Why not just use pixels?
Pixel differences vs. Perceptual differences

26. Eye is not a photometer!
"Every light is a shade, compared to the higher lights, till you come to the sun; and every shade is a light, compared to the deeper shades, till you come to the night."
— John Ruskin, 1879

27. Cornsweet Illusion

28. Campbell-Robson contrast sensitivity curve
Sine wave
Campbell-Robson contrast sensitivity curve

29. Metamers

30. Question #1: What (if anything) should be done at the “Low-Level”? i.e. What input stimulus should we be invariant to?

31. Invariant to: Brightness / Color changes? low-frequency changes
small brightness / color changes
But one can be too invariant

32. Invariant to: Edge contrast / reversal?
I shouldn’t care what background I am on!
but be careful of exaggerating noise

33. Representation choices
Raw Pixels
Gradients:
Gradient Magnitude:
Thresholded gradients (edge + sign):
Thresholded gradient mag. (edges):

34. Typical filter bank

35. pyramid (e.g. wavelet, stearable, etc)
Filters
Input image

36. What does it capture?
v = F * Patch (where F is filter matrix)

37. Why these filters?

38. Learned filters

39. Spatial invariance Rotation, Translation, Scale
Yes, but not too much…
In brain: complex cells – partial invariance
In Comp. Vision: histogram-binning methods (SIFT, GIST, Shape Context, etc) or, equivalently, blurring (e.g. Geometric Blur) -- will discuss later

40. Many lives of a boundary

41. Often, context-dependent…
input
canny
human
Maybe low-level is never enough?