CSc83029 – 3-D Computer Vision/ Ioannis Stamos 3-D Computational Vision CSc Optical Flow & Motion The Factorization Method

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow & Motion  Finding the movement of scene objects from time-varying images.  Motion Field  Optical Flow  Computing Optical Flow

CSc83029 – 3-D Computer Vision/ Ioannis Stamos

Computing Time-to-Impact τ L l(t) v f D0D0 D 0 -vt l(t)=f L/D(t)

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Computing Time-to-Impact τ L l(t) v f D0D0 D 0 -vt l(t)=f L/D(t) l(t) / l’(t) = τ Quantities measured from image sequence.

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Sudden change in viewing position/direction: Hard to compute motion field/optical flow.

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Views from a sequence of spatially close viewpoints: Motion Field/ Optical Flow

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Sub problems of Motion Analysis  Correspondence.  Reconstruction.  Segmentation.

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Motion Field 2-D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points. P V dt v dt p f Image plane Scene Point Velocity V=dr o /dt Image Velocity v =dr i /dt roro riri

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Motion Field 2-D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points. P V dt v dt p f Image plane roro riri Perspective projection Velocity of point P as a function of translation and rotation Motion field equations

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Motion Field 2-D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points => SUM of 2 COMPONENTS:

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Motion Field 2-D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points => SUM of 2 COMPONENTS: +

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Special Case 1: Pure Translation 2-D vector field of velocities of image points, induced by relative motion between the viewing camera and the observed points => SUM of 2 COMPONENTS: +

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Pure Translation: Radial Motion Field p0=(x0,y0)

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Pure Translation: Radial Motion Field 1.Tz < 0 : FOCUS OF EXPANSION. 2.Tz > 0: FOCUS OF CONTRACTION. 3.Tz = 0: PARALLEL MOTION FIELD. 4.Vanishing point (epipole) p0. p0=(x0,y0)

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Special Case 2: Moving Plane n P Plane moves: n,d are functions of time. Motion field=?

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Special Case 2: Moving Plane n P Plane moves: n,d are functions of time. Motion field=?

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Special Case 2: Moving Plane n P Plane moves: n,d are functions of time. Motion field=? Motion field: quadratic polynomial in (x,y,f) at any time t. The same motion field can be produced by 2 different planes undergoing 2 different 3-D motions.

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Motion Parallax The relative motion field of two instantaneously coincident points does not depend on the rotational component of the motion… p0

CSc83029 – 3-D Computer Vision/ Ioannis Stamos The notion of Optical Flow Optical Flow: Estimation of the motion field from a sequence of images.

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow Image brightness constancy equation: E(x,y,t)=E(x+uδt, y+vδt, t+δt) or tt+δt (x,y) (x+uδt,y+vδt) u=δx/δt v=δy/δt

CSc83029 – 3-D Computer Vision/ Ioannis Stamos The Aperture Problem Image brightness constancy equation: E(x,y,t)=E(x+uδt, y+vδt, t+δt) Aperture problem 1 constraint 2 unknowns The component of the motion field in the direction orthogonal to the spatial image gradient is not constrained by the image brightness constancy equation.

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow At each point we know dE/dx dE/dy and dE/dt. How can we obtain dx/dt and dy/dt?

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Computing Optical Flow Assumption: The motion field is well approximated by a constant vector field within any small patch of the image plane. Each provides one constraint Solution is

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Computing Optical Flow Assumption: The motion field is well approximated by a constant vector field within any small patch of the image plane. Minimize Q

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Computing Optical Flow Assumption: The motion field is well approximated by a constant vector field within any small patch of the image plane. Minimize Q =>Solve the linear system

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Computing Optical Flow global approach Minimize the error in the image brightness constancy constraint. [Schunck and Horn 81]

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Computing Optical Flow global approach Minimize the error in the image brightness constancy constraint. Minimize the deviation from smoothness of the motion vectors. [Schunck and Horn 81]

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Computing Optical Flow global approach Minimize the error in the image brightness constancy constraint. Minimize the deviation from smoothness of the motion vectors. Find solution by minimizing where lambda weights the smoothness term. [Schunck and Horn 81]

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Optical Flow Solve for the 8 unknowns a, b, c, d, e, f, g and h. If the scene is planar the motion Is described by Using

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Tracking Rigid Bodies B B Random Sampling Algorithm Step 1: Find corners Step 2: Search for correspondence Step 3: Randomly choose small set of matches. Step 4: Estimate F matrix Step 5: Find total number of matches close to epipolar lines Step 6: Go to step 3 Step 7: Choose F with largest number of matches

CSc83029 – 3-D Computer Vision/ Ioannis Stamos

1. Use multiple image stream to compute the information about camera motion and 3D structure of the scene 2. Tracking image features over time Tracked Features Structure and Motion Recovery from Video Original sequence From Jana Kosecka

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Structure and Motion Recovery from Video Computed model 3D coordinates of the feature points Original picture From Jana Kosecka

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Factorization Method FRAMES: i=1…N kiki jiji i

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Factorization Method kiki jiji i Pj FRAMES: i=1…N World Points: j=1…n (x ij,y ij )

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Factorization Method World Reference Frame kiki jiji i Pj FRAMES: i=1…N World Points: j=1…n (x ij,y ij ) Y X Z Ti

CSc83029 – 3-D Computer Vision/ Ioannis Stamos World Reference Frame kiki jiji i Pj FRAMES: i=1…N World Points: j=1…n (x ij,y ij ) Y X Z Ti ASSUMPTIONS: The camera model is orthographic! The positions of n image points have been tracked.

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Measurement Matrix 2*N (Frames) n points per frame 2*N (Frames) n points per frame Registered Measurement Matrix

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Factorization Method World Reference Frame kiki jiji i FRAMES: i=1…N World Points: j=1…n Y X Z Ti Pj 3D Centroid 2D Centroid

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Rank Theorem 2*N (Frames) n points per frame R: “Rotation Matrix” S: Shape Matrix

CSc83029 – 3-D Computer Vision/ Ioannis Stamos Rank Theorem 2*N (Frames) n points per frame R: “Rotation Matrix” S: Shape Matrix Rank of is 3.

CSc83029 – 3-D Computer Vision/ Ioannis Stamos The algorithm 2*N (Frames) n points per frame Decompose into R and S. Is the decomposition unique? Translation estimation?

CSc83029 – 3-D Computer Vision/ Ioannis Stamos