# Computer Vision Optical Flow

## Presentation on theme: "Computer Vision Optical Flow"— Presentation transcript:

Computer Vision Optical Flow
Some slides from K. H. Shafique [ and T. Darrell

Correspondence Which pixel went where? Time: t Time: t + dt
Bahadir K. Gunturk EE Image Analysis II

Motion Field vs. Optical Flow
Scene flow: 3D velocities of scene points. Motion field: 2D projection of scene flow. Optical flow: Approximation of motion field derived from apparent motion of brightness patterns in image. Bahadir K. Gunturk EE Image Analysis II

Motion Field vs. Optical Flow
Consider perfectly uniform sphere rotating in front of camera. Motion field follows surface points. Optical flow is zero since motion is not visible. Now consider stationary sphere with moving light source. Motion field is zero. But optical flow results from changing shading. But, in general, optical flow is a reliable indicator of motion field. Bahadir K. Gunturk EE Image Analysis II

Applications Correct for camera jitter (stabilization)
Object tracking Video compression Structure from motion Segmentation Correct for camera jitter (stabilization) Combining overlapping images (panoramic image construction) Bahadir K. Gunturk EE Image Analysis II

Optical Flow Problem How to estimate pixel motion from one image to another? Bahadir K. Gunturk EE Image Analysis II

Computing Optical Flow
Assumption 1: Brightness is constant. Assumption 2: Motion is small. (from Taylor series expansion) Bahadir K. Gunturk EE Image Analysis II

Computing Optical Flow
Combine In the limit as u and v goes to zero, the equation becomes exact (optical flow equation) Bahadir K. Gunturk EE Image Analysis II

Computing Optical Flow
At each pixel, we have one equation, two unknowns. This means that only the flow component in the gradient direction can be determined. (optical flow equation) The motion is parallel to the edge, and it cannot be determined. This is called the aperture problem. Bahadir K. Gunturk EE Image Analysis II

Computing Optical Flow
We need more constraints. The most commonly used assumption is that optical flow changes smoothly locally. Bahadir K. Gunturk EE Image Analysis II

Computing Optical Flow
One method: The (u,v) is constant within a small neighborhood of a pixel. Optical flow equation: Use a 5x5 window: Two unknowns, 25 equation ! Bahadir K. Gunturk EE Image Analysis II

Computing Optical Flow
Find minimum least squares solution Lucas – Kanade method Bahadir K. Gunturk EE Image Analysis II

Computing Optical Flow
Lucas – Kanade When is This Solvable? ATA should be invertible ATA should not be too small. Other wise noise will be amplified when inverted.) Bahadir K. Gunturk EE Image Analysis II

Computing Optical Flow
What are the potential causes of errors in this procedure? Brightness constancy is not satisfied The motion is not small A point does not move like its neighbors window size is too large Bahadir K. Gunturk EE Image Analysis II

Improving accuracy Recall our small motion assumption
This is not exact To do better, we need to add higher order terms back in: This is a polynomial root finding problem Can solve using Newton’s method Also known as Newton-Raphson method Lucas-Kanade method does one iteration of Newton’s method Better results are obtained via more iterations Bahadir K. Gunturk EE Image Analysis II

Estimate velocity at each pixel by solving Lucas-Kanade equations Warp H towards I using the estimated flow field - use image warping techniques Repeat until convergence Bahadir K. Gunturk EE Image Analysis II

Revisiting the small motion assumption
What can we do when the motion is not small? Bahadir K. Gunturk EE Image Analysis II

Reduce the resolution! Bahadir K. Gunturk EE Image Analysis II

Coarse-to-fine optical flow estimation
Gaussian pyramid of image H Gaussian pyramid of image I image I image H u=10 pixels u=5 pixels u=2.5 pixels u=1.25 pixels image H image I Bahadir K. Gunturk EE Image Analysis II

Coarse-to-fine optical flow estimation
Gaussian pyramid of image H Gaussian pyramid of image I image I image H run iterative L-K warp & upsample run iterative L-K . Bahadir K. Gunturk EE Image Analysis II

Bahadir K. Gunturk EE Image Analysis II

Block-Based Motion Estimation
Bahadir K. Gunturk EE Image Analysis II

Block-Based Motion Estimation
Bahadir K. Gunturk EE Image Analysis II

Block-Based Motion Estimation
Gaussian pyramid of image H Gaussian pyramid of image I image I image H Hierarchical search image H image I Bahadir K. Gunturk EE Image Analysis II

Parametric (Global) Motion
Sometimes few parameters are enough to represent the motion of whole image translation rotation scale Bahadir K. Gunturk EE Image Analysis II

Parametric (Global) Motion
Affine Flow Bahadir K. Gunturk EE Image Analysis II

Parametric (Global) Motion
Affine Flow Bahadir K. Gunturk EE Image Analysis II

Parametric (Global) Motion
Affine Flow – Approach Bahadir K. Gunturk EE Image Analysis II

Iterative Refinement Iterative Estimation
Estimate parameters by solving the linear system. Warp H towards I using the estimated flow field - use image warping techniques Repeat until convergence or a fixed number of iterations Bahadir K. Gunturk EE Image Analysis II

Coarse-to-fine global flow estimation
Gaussian pyramid of image H Gaussian pyramid of image I image I image H Compute Flow Iteratively warp & upsample Compute Flow Iteratively . image J image I Bahadir K. Gunturk EE Image Analysis II

Global Flow Find features Match features Fit parametric model
Application: Mosaic construction Bahadir K. Gunturk EE Image Analysis II

Image Warping Bahadir K. Gunturk EE Image Analysis II

Image Warping Interpolate to find the intensity at (x’,y’)
Pixel values are known in this image Pixel values are to found in this image Bahadir K. Gunturk EE Image Analysis II

Other Parametric Motion Models
Perspective and Pseudo-Perspective: Approximation to perspective. (Planar scene + Perspective projection) Bahadir K. Gunturk EE Image Analysis II