4Problem definition: optical flow How to estimate pixel motion from image H to image I?Solve pixel correspondence problemgiven a pixel in H, look for nearby pixels of the same color in IKey assumptionscolor constancy: a point in H looks the same in IFor grayscale images, this is brightness constancysmall motion: points do not move very farThis is called the optical flow problem
11Lukas-Kanade flow Prob: we have more equations than unknowns Solution: solve least squares problemminimum least squares solution given by solution (in d) of:The summations are over all pixels in the K x K windowThis technique was first proposed by Lukas & Kanade (1981)described in Trucco & Verri reading
15Iterative Refinement Iterative Lukas-Kanade Algorithm Estimate velocity at each pixel by solving Lucas-Kanade equationsWarp H towards I using the estimated flow field- use image warping techniquesRepeat until convergence
18Coarse-to-fine optical flow estimation Gaussian pyramid of image HGaussian pyramid of image Iimage Iimage Hu=10 pixelsu=5 pixelsu=2.5 pixelsu=1.25 pixelsimage Himage I
19Coarse-to-fine optical flow estimation Gaussian pyramid of image HGaussian pyramid of image Iimage Iimage Hrun iterative L-Kwarp & upsamplerun iterative L-K.image Jimage I
20Multi-resolution Lucas Kanade Algorithm Compute Iterative LK at highest levelFor Each Level iTake flow u(i-1), v(i-1) from level i-1Upsample the flow to create u*(i), v*(i) matrices of twice resolution for level i.Multiply u*(i), v*(i) by 2Compute It from a block displaced by u*(i), v*(i)Apply LK to get u’(i), v’(i) (the correction in flow)Add corrections u’(i), v’(i) to obtain the flow u(i), v(i) at ith level, i.e., u(i)=u*(i)+u’(i), v(i)=v*(i)+v’(i)
25Global Flow Dominant Motion in the image Global Motion is caused by Motion of all points in the sceneMotion of most of the points in the sceneA Component of motion of all points in the sceneGlobal Motion is caused byMotion of sensor (Ego Motion)Motion of a rigid sceneEstimation of Global Motion can be used toVideo MosaicsImage Alignment (Registration)Removing Camera JitterTracking (By neglecting camera motion)Video Segmentation etc.
27Global Flow Special Case of General Optical Flow Problem Can be solved by using Lucas Kanade algorithm.Specialized algorithms exist that perform better by further constraining the problem.
28Motion ModelsFirst we look for a parametric form of global flow vector.Global Flow occurs because of 3D rigid motion of either the sensor or the scene.3D Rigid Motion(If is small)Also neglecting the higher order terms
29Iterative Refinement Iterative Algorithm Estimate global flow by solving linear system Aa=BWarp H towards I using the estimated flow- use image warping techniques (to be covered later)Repeat until convergence or a fixed number of iterations
30Coarse-to-fine global flow estimation Gaussian pyramid of image HGaussian pyramid of image Iimage Iimage Hu=10 pixelsu=5 pixelsu=2.5 pixelsu=1.25 pixelsimage Himage I
31Coarse-to-fine global flow estimation Gaussian pyramid of image HGaussian pyramid of image Iimage Iimage HCompute Flow Iterativelywarp & upsampleCompute Flow Iteratively.image Jimage I
37Summary Things to take away from this lecture Assumptions Optical flow problem definitionAperture problem and how it arisesAssumptionsBrightness constancy, small motion, smoothnessDerivation of optical flow constraint equationLukas-Kanade equationConditions for solvabilitymeanings of eigenvalues and eigenvectorsIterative refinementNewton’s methodCoarse-to-fine flow estimationApplicationsImage alignment / Video mosaic