A plane-plus-parallax algorithm Basic Model: When FOV is not very large and the camera motion has a small rotation, the 2D displacement (u,v) of an image point (x,y) in the image plane can be expressed: where

General Framework Estimate the 2D parameters of a single motion Register two successive frames according to the computed 2D parameters. This alignment was proved to cancel the rotation motion for the entire scene, resulting in a new sequence which only contains 3D translation, and looks as if taken from a stabilized platform with no jitter. Further option: Compute FOE (translation) from the residual epipolar displacement field between the two registered frames and recover the 3D rotation parameters from the 2D parameters and 3D translation parameters.

2D parameters estimation: Overview For small motions, assume gray level constancy: - For translation: 2 parameters, u(x,y) = a, v(x,y) = b - For Affine: 6 parameters, u(x,y) = a+bx+cy, v(x,y) = d+ex+fy. - For a pseudo projective transformation: u(x,y)=a+bx+cy+gx^2+hxy, v(x,y)=c+dx+fy+gxy+hy^2 First order Taylor expansion and neglecting all nonlinear terms: where Minimize the error function at Frame t in the region of R: (Its derivatives with respect to the parameters are set to zero.)

2D parameters estimation: Details Multiresolution iterative framework Construct a Gaussian pyramid Start at the lowest resolution level, motion parameters are estimated by solving the set of linear equation to minimize the error function according to the appropriate model. Incremental motion estimation can be performed at each resolution level. Coarse-fine alignment: at each successive iteration, the shift and estimates steps are performed on the next higher resolution pyramid level, that is, the shift estimated at level k+1 of frame t is applied to level k of frame t to form a new level k of frame t, then the residual can be computed between the new level k of frame t and the level k of frame t+1.

Motion Correction Warping two successive frames according to the computed 2D parameters. Theoretically, this step should be able to cancel the rotation motion for the entire scene, resulting in a new sequence which only contains 3D translation, and looks as if taken from a stabilized platform with no jitter. Stabilized GoCart (not so good) Problems: Didn’t work for other test videos ( Got zero gradient, thus NaN …. Need debug and further improvements)