1. 3D Computer Vision: CSc 83020

2. 3D Computer Vision: CSc 83020 Instructor: Ioannis Stamos
istamos (at) hunter.cuny.edu
Office Hours: Tuesdays 4-6 (at Hunter) or by appoitnment
Office: 1090G Hunter North (69th street bw. Park and Lex.)
Computer Vision Lab: 1090E Hunter North
Course web page:

3. Goals
To familiarize you with basic the techniques and jargon in the field
To enable you to solve computer vision problems
To let you experience (and appreciate!) the difficulties of real-world computer vision
To get you excited!

4. Class Policy You have to Late policy Teaming
Turn in all assignments (60% of grade)
Complete a final project (30% of grade)
Actively participate in class (10% of grade)
Late policy
Six late days (but not for final project)
Teaming
For final project you can work in groups of 2

5. About me 11th year at Hunter and the Graduate Center
Graduated from Columbia in ’01
CS Ph.D.
Research Areas:
Computer Vision
3D Modeling
Computer Graphics

6. Books
Computer Vision: Algorithms and Applications, Richard Szeliski, 2010 (available online for free)
Robot Vision
B. K. P. Horn, The MIT Press (great classic book)
Introductory Techniques for 3-D Computer Vision
Emanuele Trucco and Alessandro Verri, Prentice Hall, 1998 (algorithmic perspective)
Computer Vision A Modern Approach
David A. Forsyth, Jean Ponce, Prentice Hall 2003
An Invitation to 3-D Vision
Yi Ma, Stefano Soatto, Jana Kosecka, S. Shankar Sastry
Springer 2004.
Three-Dimensional Computer Vision: A Geometric Viewpoint Olivier Faugeras
The MIT Press, 1996.

7. Journals/Web International Journal of Computer Vision.
Computer Vision and Image Understanding.
IEEE Trans. on Pattern Analysis and Machine Intelligence.
SIGGRAPH (mostly Graphics)
(CMU’s Robotic Institute)
(The Vision Home Page)
(CV Online)
(Annotated CV Bibliography)

8. Class History Based on class taught at Columbia University
by Prof. Shree Nayar.
New material reflects modern approach.
Taught similar class at Hunter
Taught “3D Photography” class at the Graduate Center of CUNY.
My active research area
Funded by the National Science Foundation

9. Class Schedule Check class website Final project proposals
Due Nov. 7
Design your own or check list of possible projects on class website
Final project presentations and report
May 16 (last class)

10. What is Computer Vision?
Sensors
Images or Video
Illumination
Vision System
Physical 3D World
Scene Description
Measuring Visual Information

11. Computer Graphics Output Image Model Synthetic Camera
(slides courtesy of Michael Cohen)

12. Computer Vision Output Model Real Scene Real Cameras
(slides courtesy of Michael Cohen)

13. Combined Output Image Model Real Scene Synthetic Camera Real Cameras
(slides courtesy of Michael Cohen)

14. Cont. Vision is automating visual processes (Ball & Brown).
Vision is an information processing task (Marr).
Vision is inverting image formation (Horn).
Vision is inverse graphics.
Vision looks easy, but is difficult.
Vision is difficult, but it is fun (Kanade).
Vision is useful.

15. Some Applications
Industrial
Material Handling
Inspection
Assembly

16. Some Applications
Autonomous Navigation

17. Some Applications Vision for Graphics Film Industry Urban Planning
E-commerce
Virtual Reality

18. Some Applications Realistic 3D experience Google Earth
Microsoft Photosynth

19. More Applications! Optical Character Recognition (OCR)
Visual Databases (images or movies)
Searching for image content
Face Recognition (security)
Iris Recognition (security)
Traffic Monitoring Systems
Many more…

20. Vision deals with images

21. Images Look Nice…

22. Images Look Nice…
Ioannis Stamos – CSc Spring 2007

23. ...Essentially a 2D array of numbers

24. Low-Level or “Early” Vision
Considers local properties of an image
“There’s an edge!”
From: Szymon Rusinkiewicz, Princeton.

25. “There’s an object and a background!”
Mid-Level Vision
Grouping and segmentation
“There’s an object and a background!”

26. High-Level Vision
Recognition
“It’s a chair!”

27. Humans
Vision is easy for us.
But how do we do it?

28. Human Vision: Illusions
Fraser’s spiral (Fraser 1908)

29. Illusions Zölner Illusion (1860) Hering Illusion (1861)
Wundt Illusion (1896)

30. Visual Ambiguities
Young-Girl/Old-Woman

31. Visual Ambiguities
From NALWA.

32. Visual Ambiguities

33. Visual Ambiguities
Two lava cones with craters, perceived to be the image of two craters
with mounds. Implicit assumption that lighting is from above.

34. Visual Ambiguities

35. Seeing and Thinking
Kanizsa (1979)

36. Syllabus Overview

37. Image Formation and Optics
Light Source p
Surface normal
CCD Array
Lens P
Object Surface
Projection of 3-D World on a 2-D plane

38. Lenses
Ray of light
Optical Axis

39. Image Sensors/Camera Models
Typical 512x512 CCD array
Imaging Area 262,144 pixels
One Pixel
20μm
20μm
512 (10.25mm)
Convert Optical Images
To Electrical Signals.
512 (10.25mm)

40. Filtering = h g f

41. Image Features Detecting intensity changes in the image
Ioannis Stamos – CSc Spring 2007

42. Grouping image features
Finding continuous lines from edge segments
Ioannis Stamos – CSc Spring 2007

43. Camera Calibration Camera Coordinate Frame Zc Pixel Coordinates Yc
Intrinsic
Parameters Xc
Extrinsic
Parameters
Image Coordinate Frame Zw Yw
World Coordinate Frame Xw

44. Shape from X
Shape from X
Stereo
Motion
Shading
Texture foreshortening

45. Binocular Stereo
depth map

46. Active Sensing Sheet of light CCD array Lens Sources of error:
1) grazing angle,
2) object boundaries.
Sheet of
light
CCD array
Lens

47. Shape from Shading Three-dimensional shape from a single image.
Ioannis Stamos – CSc Spring 2007

48. Motion (optical flow) Determining the movement of scene objects
Ioannis Stamos – CSc Spring 2007

49. Reflectance and Color
Why do these spheres look different?

50. Object Recognition Learning visual appearance.
Real-time object recognition.

51. Template-Based Methods
Cootes et al.

52. Some Vision Systems…

53. Example 2: Structure From Motion
Slide courtesy of
Sebastian Thrun
Stanford

54. Example 2: Structure From Motion
Slide courtesy of
Sebastian Thrun
Stanford

55. Example 2: Structure From Motion
Slide courtesy of
Sebastian Thrun
Stanford

56. Example 2: Structure From Motion
Slide courtesy of
Sebastian Thrun
Stanford

57. Example 2: Structure From Motion
Slide courtesy of
Sebastian Thrun
Stanford

58. Example 4: 3D Modeling Slide courtesy of Sebastian Thrun
Stanford
Drago Anguelov

59. Example 5: Segmentation
Slide courtesy of
Sebastian Thrun
Stanford

60. Example 6: Classification
Slide courtesy of
Sebastian Thrun
Stanford

61. Example 6: Classification
Slide courtesy of
Sebastian Thrun
Stanford

62. Real-world Applications
Osuna et al:

63. Range Scanning Outdoor Structures
Ioannis Stamos – CSc Spring 2007

64. Data Acquisition Spot laser scanner. Time of flight. Max Range: 100m.
Scanning time: minutes for x1000 points.
Accuracy: 6mm.

65. Video

66. Latest Video

67. Inserting models in Google Earth

68. Image sequence (CMU, Virtualized Reality Project)
Dynamic Scenes
Image sequence (CMU, Virtualized Reality Project)

69. Dynamic Scenes
Dynamic 3D model.

70. Dynamic texture-mapped model.
Dynamic Scenes
Dynamic texture-mapped model.

71. Scanning the David Marc Levoy, Stanford
2:30 through closeup of eye
0:40
So here we are scanning our “piece de resistance”
the gantry is extended to nearly its full height
If you’ve looked at our web site
you know the little story about the gantry being too short initially
because we relied on art history books for the height of the David
and they're all wrong – by 3 feet !!
so at the last minute we had to add a piece to the gantry…here
and load another few hundred pounds of ballast to the base…here – to keep the gantry from tipping over on the statue
But as you can see from the photo on the right,
we did finally reach the top of the statue
height of gantry: meters
weight of gantry: kilograms
4:00 through architectural reps
0:45

72. Statistics about the scan
0:20
So here’s a rendering of our computer model of the David
we rolled the gantry, which now weighed about a ton, 480 times
2 billion polygons
you can read the rest
It took 30 days to scan the statue
but we were only allowed to scan at night, when the museum was closed
so it was basically 30 all-nighters in a row
480 individually aimed scans
2 billion polygons
7,000 color images
32 gigabytes
30 nights of scanning
22 people
Ioannis Stamos – CSc Spring 2007
0:30

73. Head of Michelangelo’s David
0:20
We haven’t assembled the David at full-resolution yet
Here’s a 1 mm model of the head
watertight
with color
The photograph at the left was taken with uncalibrated lighting
so the two images don’t match exactly
but hopefully you agree that we’ve basically captured the statue’s appearance
photograph
1.0 mm computer model

74. David’s left eye 0.25 mm model photograph
0:30
Here’s a closeup of David’s left eye
at the maximum resolution of our model – ¼ mm
we can see some interesting things…
These bumps
are holes from Michelangelo’s drill
so that’s real geometry
are artifacts from space carving, the hole-filling part of our range image merging algorithm
smoothing out those artifacts, while preserving the observed surfaces, is future work
Finally, these bumps
are noise from laser subsurface scattering
Let’s zoom in some more
and look at the triangle mesh
noise from laser scatter
holes from Michelangelo’s drill
artifacts from space carving
0.25 mm model
photograph
Ioannis Stamos – CSc Spring 2007
0:45

75. What do you think?