1. MOTION DETAIL PRESERVING OPTICAL FLOW ESTIMATION
Li Xu, Jiaya Jia and Yasuyuki Matsushita

2. CONTENTS INTRODUCTION CONVENTIONAL OPTICAL FLOW OPTICAL FLOW MODEL
ROBUST DATA FUNCTION
EDGE-PRESERVING REGULARIZATION
MEAN FIELD APPROXIMATION
OPTIMIZATION FRAMEWORK
EXTENDED FLOW INITIALIZATION
CONTINUOUS FLOW OPTIMIZATION
OCCLUSION-AWARE REFINEMENT
CONCLUSION
REFERENCE

3. INTRODUCTION
Optical flow is the apparent motion of brightness patterns in the image
Ideally, optical flow would be the same as the motion field
The motion field … is the projection into the image of three-dimensional motion vectors 3

4. CONVENTIONAL OPTICAL FLOW
Dominant Scheme: Coarse-to-Fine Warping Framework
The input image is represented as a tree of regions
The optical flow is estimated by optimizing an energy function
optical flow estimation on the coarser level region-tree is used for defining region-wise finer displacement samplings for finer level region-trees
Middlebury optical flow evaluation

5. MULTI-SCALE PROBLEM IN COARSE-TO-FINE WARPING
(a) (b)
Coarse level (e)
(c) (d)
Fine level (f)
(a)-(b) Two input patches.
(c) Flow estimate using the coarse-to-fine variational setting. (d) Our
flow estimate. (e)-(f) Two consecutive levels in the pyramid. Flow fields
are visualized using the colour code

6. Large displacement optical flow may not be well estimated
Inclination to diminish small motion structures when
spatially significant and abrupt change of the displacement
exists.
Solution
Improve flow initialization to reduce the reliance of the initialization from coarser levels and enables recovering many motion details at each scale

7. Our Work Framework Model Solver
Extended coarse-to-fine motion estimation for both large and small displacement optical flow
Model
A new data term to selectively combine constraints
Solver
Efficient numerical solver for discrete-continous optimization

8. OPTICAL FLOW MODEL
ROBUST DATA FUNCTION
Objective function for development of a new optimization procedure
u denotes the flow field that represents the displacement between frames I1 and I2 ,x represents the 2D coordinates
Data Constraints
Color constraint
D1(u,x)=||I2(x+u)-I1(x)||
Gradient constraint
D I(u,x)=ζ|| I2(x+u)- I1(x)||
Data term
ED (u,x)=∑1/2 D1(u,x)+1/2 D I(u,x)

9. Data cost distributions w.r.t different displacement values
A good model should only use the more fitting constraint, but not both of them
Define a binary weight map α(x):Z―>{0,1} to switch between the two terms. When α(x)=1, the gradient constraint is favored. Otherwise, we select color constraint

10. EDGE-PRESERVING REGULARIZATION
Smoothness term, it maintains motion discontinuity
The final objective function is defined as E(u,α)= ED(u,α)+λEs(u)
where λ is the regularization weight.
MEAN FIELD APPROXIMATION
The effective energy is written as Eeff(u)= EeffD(u)+λEs(u)
The effective data function
β is the inverse temperature
β plays a key role in shaping the data function.

11. Small β makes the distribution close to the original one with α=0.5
A relatively large β yields the distribution approaching the lower envelope of the costs with α=0 and α=1

12. OPTIMIZATION FRAMEWORK
1212
OPTIMIZATION FRAMEWORK
INPUT: A pair of images for optical flow estimation
Construct pyramids for both of he images and set the initial level l=0 and uɭ=0 for all pixels
Propagate uɭ to level l+1
Extended Flow Initialization
3.1. Detect and match SIFT features in level l+1
3.2 Perform patch matching in level l+1
3.3 Generate multiple flow vectors as candidates
3.4 Optimize flow
Continous Flow Optimization
4.1 Compute the ᾱ map
Solve the energy function
5. Occlusion-aware Refinement
If l≠n-1 where n is the total number of levels, l=l+1 and go to step 2
OUTPUT: The optical flow field

13. EXTENDED FLOW INITIALIZATION
Finding multiple extended displacements (denoted as {u0,u1,….,un}) to improve estimation in uc
uc which is the flow field computed in the immediately coarser level.
The steps adopted to obtain the extended displacements.
SIFT Feature Detection
Selection
Expansion
Patch Matching
Matching Field Fusion

14. A)SIFT Feature Detection
SIFT feature detection and matching can efficiently capture large motion for objects undergoing translational and rotational motion
Employ only the sparse matching of discriminative points, which avoids introducing many ambiguous correspondences and outliers.
employ discrete optimization to only select the most credible candidates.

15. B) Selection
The computed displacement vectors by feature matching are denoted as {s0, ,sn}
Robustly screen out the duplicated vectors
Compute the euclidean distance between each si and all uc
If all results are greater than 1 (pixel), we regard si as a new flow candidate.
Repeat this process for all si, and denote the m remaining candidate vectors {sk0, ,skm-1}

16. C)Expansion
The m remaining vectors {sk0, ,skm-1} represent possible missing motion in the present flow field uc
Determine whether or not they are better estimates to replace the original ones

17. D) Patch Matching sometimes it still misses some motion vectors
because small texture-less objects may not have distinct features
SIFT descriptors, the patches on which they operate should at least contain 16x16 samples as suggested.
Compute the matching field un by minimizing energy
Total of five color and gradient channels used.
Noise can be quickly rejected in the following optimization step
with the collection of a set of flow candidates for each pixel.

18. E) Matching Field Fusion
The m+1 new motion fields {u0,..um-1,un} together with the original uc, comprise several motion candidates for each pixel in the present image scale.
Selection of the optimal flow among the m+2 candidates for each pixel is a labeling problem
Solved by discrete optimization efficiently

19. Extended flow initialization.

20. CONTINUOUS FLOW OPTIMIZATION
Refine flow u0 through continuous optimization
We propose decomposing the optimization into three simpler problems, each of which can have the globally optimal solution.
Auxiliary variables p and w, representing the substituted data cost and flow derivatives
The optimal solution is given by the shrinkage
formula

21. INPUT: Images Ik ,initial flow field u0,weights αk
Perform linerization at u0
η= η0
repeat
Compute pk
θ= θ0
Compute w
Compute du
θ=θ/3
until θ=θmin
η= η/3
until η< ηmin
ur=u0+du
OUTPUT: Refined flow field ur

22. fixing them to constants typically results in slow convergence
θ and η are critical parameters that should be small.
fixing them to constants typically results in slow convergence
Initially sets θ and η to large values to allow warm-starting and then decreases them in iterations toward the desired convergence
θ and η minimum values are set to 0.1

23. OCCLUSION-AWARE REFINEMENT
Multiple pixels mapping to the same point in the target image using forward warping are possibly occluded by each other
detect occlusion using the mapping uniqueness criterion expressed as
o(x)=T0.1(f(x+u(x))-1)
f(x+u(x)) is the count of reference pixels mapped to position x+u(x) in the target view using forward warping
measure of the data confidence based on the occlusion detection is expressed as c(x)=max(1-o(x),0.01)

24. CONCLUSION
A unified framework to preserve motion details in both small and large-displacement scenarios.
It include the selective combination of the color and gradient constraints, sparse feature matching, and dense patchmatching to collect appropriate motion candidates
Limitations
--Texture-less motion details
--Large occlusions

25. REFERENCE
L. Alvarez, J. Esclarin, M. Lefebure, and J. Sanchez, “A PDE Model for Computing the Optical Flow”
P. Anandan, “A Computational Framework and an Algorithm for the Measurement of Visual Motion”
S. Baker, D. Scharstein, J. Lewis, S. Roth, M.J. Black, and R. Szeliski, “A Database and Evaluation Methodology for Optical Flow ”
C. Barnes, E. Shechtman, A. Finkelstein, and D.B. Goldman, “Patchmatch: A Randomized Correspondence Algorithm for Structural Image Editing”