1 Computational Vision CSCI 363, Fall 2012 Lecture 28 Structure from motion

2 2D Motion is just the Beginning 2D image motion contains information about: Relative depth of surfaces 3D motion of objects 3D structure of objects Direction of observer motion Among other things.

3 Structure From Motion Structure from motion originally studied rigorously by Wallach and O'Connell (1953). They studied wire-frame objects and examined peoples ability to judge the structure of the objects when moving. The ability to see a 3D structure from a moving 2D image is known as the Kinetic Depth Effect. Demo:

4 Inherent Ambiguity How do we compute a 3D motion from a 2D image motion? Given the 2D image motion, there are multiple possible 3D motions that could have generated it: v Image plane ? To solve for 3D motion from 2D image information, we must use a constraint. The Rigidity constraint assumes that the object is rigid. Next time: Incremental rigidity scheme.

5 The Rigidity Constraint To solve for 3D motion from 2D image information, we must use a constraint. The Rigidity constraint assumes that the object is rigid. Ullman showed that, for an orthographic projection system, one can compute the 3D structure of an object given: 3 distinct views of 4 non-coplanar points in motion. If there exists a rigid 3D structure consistent with these views, then it is unique. The 2D positions of the points generate a set of equations that can be solved for the 3D structure.

6 The problem with noise For perspective projection, one needs 2 views and 7 points. If working with velocity fields, you need 5 points and 1 view (for perspective projection). Problem: This approach is very sensitive to noise in the velocity estimates. This approach does not allow interpretation of non-rigid motions.

7 Information about Human ability The human visual system takes some time to recover the 3D structure: It is not instantaneous. Humans can cope with significant deviations from rigidity. The visual system integrates multiple sources of information (e.g. static depth cues) to determine 3D structure.

8 Incremental Rigidity Scheme The basic idea: 1)At each instant, generate an estimate of the 3D structure. 2)The recovery process prefers rigid transformations, but rigidity is not required. 3)The scheme can tolerate deviations from rigidity. 4)It should be able to integrate information over an extended viewing period. 5)It should eventually recover the correct 3D structure (or a close approximation). y x (x 1, y 1, z 1 ) -> (x 1 ', y 1 ', ?) (x 2, y 2, z 2 ) -> (x 2 ', y 2 ', ?) (x 3, y 3, z 3 ) -> (x 3 ', y 3 ', ?) Initially set z to zero.

9 Maximizing Rigidity We want to find the 3D model, S'(t), that maximizes the rigidity. Therefore, we want to find new Z values that minimize the change in structure L ij l ij Current model Model update Image Find z i to minimize: The denominator makes sure that distant points count less, as they are less likely to be rigidly connected to each other.

10 Performance The incremental rigidity scheme works fairly well for rigid motions. For a rotating object, it can find the 3D shape within about 2-5% error after a few rotations. Mirror reversals may occur sometimes during the computations. (This may arise from orthographic projection). The performance is similar to human structure from motion. x z 0o0o 90 o 180 o 360 o 720 o 1440 o Results of scheme after various rotations of object.

11 Properties of Incremental Rigidity Scheme 1.Veridicality--Usually get a reasonable approximation of the true structure. 2.Temporal extension--The time required is longer than for the 4 points, 3 views method. 3.There is some residual non-rigidity in computed 3D structure. 4.The improvement over time is non-monotonic. There may be some increase in error at times, before it decreases again. 5.Depth reversals--Sometimes this method exhibits spontaneous depth reversals in the estimated 3D structure. 6.May converge to a local minimum that's not the most rigid interpretation. 7.It can be influenced by static depth cues. 8.It can track moderate amounts of non-rigid motion over time. In many ways, this scheme behaves similarly to the human system.

12 Psychophysics of Structure from Motion Psychophysical results show that people can judge structure-from- motion from motion of points with limited lifetimes. Demo of cylinder: Cylinder DemoCylinder Demo Stimulus: Rotating cylinder composed of dots. Dots have limited lifetimes (e.g. 50, 100, 150 msec) People can see 3D structure of cylinder for point lifetimes as low as 125 msec. This is faster than the incremental rigidity scheme can handle.

13 Hildreth extension of Incremental Rigidity Hildreth and colleagues developed an extension of Ullman's incremental rigidity scheme, that uses velocity estimates instead of measure point positions.

14 Depth Estimate from 2D velocities We can compare the 2D velocities that would be generated by the estimated depths of each point with the measured 2D velocities. => Estimate depths, Z i, and velocities: Minimize:

15 Combine with Incremental rigidity The full model uses the 2D velocity estimates and the rigidity constraint: Minimize: E D + E R Where E D is from previous slide. E R is rigidity error: Temporal integration: Information from multiple frames is combined using Kalman filters. Velocity computed at one instant is used to predict depths at the next instant. This tends to smooth out errors.

16 Results Estimated dot positions Surface reconstruction. Surface reconstruction uses smoothness and position of image points to interpolate a surface between point positions.

17 Cylinder Perceptions Ramachandran showed that people do not see cylinder structure accurately in some conditions: Cylinders with different speeds. Cylinders with different radii. Perception

18 Model results for illusions Illusion 1: Different speeds Illusion 2: Different radii The model generates structures similar to those perceived by people.