1. ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther
Kawada Industries Inc. has introduced the HRP-2P for Robodex 2002.  This humanoid appears to be very impressive. It is 154 cm (60") tall, weighs 58kg (127 lbs) and has 30 DOF. Here is a news release. Notice the LACK of a battery pack.  Here is a new story about HRP2. 

2. Overview Structured light Plane Sweep Stereo
Shape from X techniques
Structured light
Plane Sweep Stereo
Global optimization methods
Shape from (de)focus
Specularities
Shading, Photometric Stereo
Alternative methods

3. Structured Light Methods
In principle same as multi-view/stereo
Project artificial pattern on the object
Pattern alleviates the correspondence problem
Variants:
Laser Pattern (point, line)
Structured projector pattern (several lines, pattern sequence)
Random projector pattern

4. Structured Light Range Finder
1. Sender (projects plane)
2. Receiver (CCD Camera)
Geometry Z- direction
X- direction
Sensor image

5. 1 plane -> 1 object profile
To get a 3D profile:
Move the object
Scanning Unit for projected plane
Move the Sensor
Object motion by conveyor band:
=> synchronization: measure distance along conveyor
=> y-accuracy determined by distance measurement
Scanning Units (e.g.: rotating mirror) are rare (accurate measurement of mirror motion is hard, small inaccuracy there -> large inaccuracy in geometry
Move the sensor: e.g. railways: sensor in wagon coupled to speed measurement

7. Commercially Available

8. CAESARTM
Civilian American and European Surface Anthropometry Resource Project—CAESARTM
2400 Male/Female Americans
2000 Male/Female Europeans

9. Problems
Occlusions
Sharpness and Contrast
Speckle noise

10. Gleichzeitige Projektion mehrerer Lichtschnitte
Anstatt einer Lichtebene werden mehrere Lichtebenen auf das Objekt projeziert, um
die Anzahl der aufzunehmenden Bilder zu reduzieren.
Entfernungsberechnung:
wie mit einer Lichtebene, jedoch
muß jeder Lichtstreifen im Bild
eindeutig identifizierbar sein.
Problem: Aufgrund von Verdeckungen sind einzelne Streifen teilweise oder gar nicht im Kamerabild sichtbar -> keine eindeutige Identifikation der Lichtstreifen
Anwendung: Glattheitsüberprüfung bei planaren Oberflächen ohne Tiefenwertberechnung.
1) peak at position xo
2) left side und right side of the peal using a threshold Vt
3) center points

11. Pattern projection Camera: IMAG CCD, Res:750x590, f:16 mm
Projected light stripes
Camera: IMAG CCD, Res:750x590, f:16 mm
Projector: Liquid Crystal Display (LCD 640), f: 200mm, Distance to object plane: 120cm
Range Image

12. Line projector (z.b: LCD-640)
Lamp
Lens system
LCD - Shutter
Pattern structure
Line projector (z.b: LCD-640)
Focusing lens (e.g.: 150mm)
Example

13. Tiefenberechnung für Streifenprojektor
Durch die zeitliche Abfolge der Aufnahmen wird es ermöglicht, daß jeder Lichtstreifen im Kamerabild identifiziert wird.
Verschiedene Lichtstreifen sind notwendig, um für jeden Bildpunkt einen zugehörigen Lichtstreifen festzustellen zu können.
1) Unterschiedlich breite Lichtstreifen werden zeitlich aufeinanderfolgend in die Szene projiziert
und von der Kamera aufgenommen.
2) Für jede Aufnahme wird für jeden Bildpunkt festgestellt, ob dieser beleuchtet wird oder nicht.
3) Diese Information wird für jeden Bildpunkt und für jede Aufnahme im sog. Bit-Plane Stack
abgespeichert.
4) Findet man im Bit-Plane Stack für einen Bildpunkt
die Information, daß er bei den Aufnahmen z.B. hell,
dunkel, dunkel, hell war (Code 1001), dann folgt
daraus, daß dieser Bildpunkt vom vierten
Lichtstreifen beleuchtet wird.
5) eindeutige Zuordnung „Lichtstreifen – Bildpunkt“ möglich

14. Coded Light + Phase Shift
Binary code is limited to pixel accuracy (at most).
Increase accuracy by projecting sine wave and measuring phase shift between projected and captured pattern.

15. Joaquim Salvi,
Pattern codification strategies in structured light systems

16. Random Texture Projection

17. Moiré Range Finder 1. Sender (Projektor mit Linien)
Project line structure, observe line structure through a grid
1. Sender (Projektor mit Linien)
2. Receiver (CCD Camera with line filter)
Problem: identification of line ordering possible but hard, unsharp lines => inaccurate results
Moiré Pattern
Moiré Image

18. Moiré Range Finder
Y. Suenaga, 3D Measurements for Computer Animation

19. Plane Sweep Stereo Sweep family of planes through volume
 projective re-sampling of (X,Y,Z)
input image
each plane defines an image  composite homography
Richard Szeliski, IBMR 1998

20. Plane Sweep Stereo For each depth plane
compute composite (mosaic) image — mean
compute error image — variance
convert to confidence and aggregate spatially
Select winning depth at each pixel
Richard Szeliski, IBMR 1998

21. Voxel Coloring

22. Voxel Coloring / Space Carving
S={} /* initial set of colored voxels is empty
for i = 1 to r do /* traverse each of r layers
foreach V in the ith layer of voxels do
project V into all images where V is visible
if sufficient correlation of the pixel colors then
add V to S
Photorealistic Scene Reconstruction by Voxel Coloring S. M. Seitz and C. R. Dyer, Proc. Computer Vision and Pattern Recognition Conf., 1997,

23. Shape From Focus
Recover shape of surfaces from limitied depth of view.
Requires visibly rough surfaces
Typical application: optical microscopy

24. Shape From Focus Visibly Rough Surfaces
Optical roughness: the smallest spatial variations are much larger than the wavelength of incident electromagnetic wave.
Visible roughness: smallest spatial variations are comparable to viewing area of discrete elements (pixels).
Magnification:
Multi-facet level: w1 >> wf, smooth texture
Facet level: w2 ~= wf, rough texture

25. Shape From Focus Focused / Defocused images Focused:
Defocused: object point is mapped to spot with radius
=> defocusing is equivalent to convolution with low pass kernel (pillbox function)

26. Shape From Focus Changing Focus
Displacing the sensor: changes sharp region, magnification and brightness
Moving the lens: changes sharp region, magnification and brightness
Moving the object: changes sharp region only
=> Object is moved in front of static camera

27. Shape From Focus Overview:
At facet level magnification, rough surfaces give texture-rich images
A defocused image is equivalent to a low-passed image
As S moves towards focused plane, its focus increases. When S is best focused,
Challenges:
How to measure focus?
How to find best focus from finite number of measurements?

28. Shape From Focus Focus measure operator Purpose: Possible solution:
respond to high frequency variations in image intensity within a small image area
produce maximum response when image area is perfectly focused
Possible solution:
determine high frequency content using Fourier transform (slow)
Alternative:
Laplacian operator (problem with elimination)
Modified Laplacian

29. Shape From Focus INxM … local intensity function (image window)
Sum Modified Laplacian
Tenengrad Focus Measure
Alternatives: variance of intensities, variance of gradients
INxM … local intensity function (image window)

30. Shape From Focus
Example
Infinite DOF
DEM

31. Shape From Focus Sampling the focus measure function
Consider a single image point (x,y)
Focus measure F is function of depth d: F(d)
Goal find Fpeak from finite number of samples F1…F8

32. Shape from focus Sampling the focus measure function
Possibility1: find highest discrete sample

33. Shape from focus Sampling the focus measure function
Possibility2: Gaussian interpolation
Fit Gauss function to three strongest samples

34. Shape from Specularity
Suitable for highly reflective Surfaces
Specular Reflection map of a single point source forms a sharp peak (Specular model, Phong model)
Reflectance map

35. Shape from Specularity
Principle:
If a reflection is seen by the camera and the position of the point source is known, the surface normal can be determined.
=> use several point sources with known position: structured highlight inspection
Panorama mirror

36. Shape From Shadow Also: Shape from Darkness
Reconstruct Surface Topography from self-occlusion
E.g. Building reconstruction in SAR images, terrain reconstruction in remote sensing

37. Shape From Shadow A static camera C observes a scene.
Light source L travels over the scene x, position of L is given by angle .
L and C are an infinite distance away (orthographic projection).
Shadowgram: binary function f(x, ), stating whether scene point x was shadowed at light position .

38. Photometric Stereo
Multiple images, static camera, different illumination directions
At least three images
Known illumination direction
Known reflection model (Lambert)
Object may be textured

39. REFLECTANCE MODELS n LAMBERTIAN MODEL PHONG MODEL L = E r COS q
L = E (a COS q + b COS a) n
albedo
Diffuse
albedo
Specular
a=0.3, b=0.7, n= a=0.7, b=0.3, n=0.5

40. Photometric Stereo Albedo (reflectivity) Reflection model
3 unknowns per pixel
at least three illumination directions

41. Photometric Stereo
surface normal
albedo
light direction

42. Photometric Stereo: Example
Input
images
From Forsyth & Ponce
Recovered
normals
Recovered
albedo

43. Range Finder principles:
Runtime Range Finder
Triangulation Range Finder
- Sender and receiver with known position, triangulation similar to stereo principle
Optical Range Finder
Ultrasound Range Finder
e.g.:
Spot Projectors
Moiré Range Finder
Structured Light Range Finder
Pattern (stripe) projection
1 image
2 dim + geometry => 3 dim
Depth information
Receiver/Sender position

44. Runtime Range Finder
Determine sensor-object distance by measuring radiation runtime:
1. Sender (coherent light)
2. Scanning Unit
3. Receiver (light detector)
4. Phase detector
Alt. Method: send light pulses => LIDAR (Light Radar), defense industry
Problem: generating pulses, measuring runtime (both very short)
Die Signalphaserelativ zur reference phase ist proportional zur Laufzeit.
Die Signalamplitude ist proportional zur zur Oberflächenreflektion.
=> Lichtintensität und Range Data

45. Ultrasound Range Finder
Used in commercial cameras (Autofocus Spot)
Advantages
Independent of sorrounding light,
Slow speed of ray
Disadvantages
coarse resolution,
Bad accuracy,
Pointwise, scanner necessary
Multiple Reflection/Echoes
Applications:
Obstacle Detection: e.g.: “Car parking radar”
Level Measurement, silos, tanks, …
Underwater ranging (sonar), ...
Typ. specifications:
Range: 5cm to 1-5m
Accuracy: +-3mm