MOTION DETAIL PRESERVING OPTICAL FLOW ESTIMATION Li Xu, Jiaya Jia and Yasuyuki Matsushita

CONTENTS INTRODUCTION CONVENTIONAL OPTICAL FLOW OPTICAL FLOW MODEL ROBUST DATA FUNCTION EDGE-PRESERVING REGULARIZATION MEAN FIELD APPROXIMATION OPTIMIZATION FRAMEWORK EXTENDED FLOW INITIALIZATION CONTINUOUS FLOW OPTIMIZATION OCCLUSION-AWARE REFINEMENT CONCLUSION REFERENCE

INTRODUCTION Optical flow is the apparent motion of brightness patterns in the image Ideally, optical flow would be the same as the motion field The motion field … is the projection into the image of three-dimensional motion vectors 3

CONVENTIONAL OPTICAL FLOW Dominant Scheme: Coarse-to-Fine Warping Framework The input image is represented as a tree of regions The optical flow is estimated by optimizing an energy function optical flow estimation on the coarser level region-tree is used for defining region-wise finer displacement samplings for finer level region-trees Middlebury optical flow evaluation

MULTI-SCALE PROBLEM IN COARSE-TO-FINE WARPING (a) (b) Coarse level (e) (c) (d) Fine level (f) (a)-(b) Two input patches. (c) Flow estimate using the coarse-to-fine variational setting. (d) Our flow estimate. (e)-(f) Two consecutive levels in the pyramid. Flow fields are visualized using the colour code

Large displacement optical flow may not be well estimated Inclination to diminish small motion structures when spatially significant and abrupt change of the displacement exists. Solution Improve flow initialization to reduce the reliance of the initialization from coarser levels and enables recovering many motion details at each scale

Our Work Framework Model Solver Extended coarse-to-fine motion estimation for both large and small displacement optical flow Model A new data term to selectively combine constraints Solver Efficient numerical solver for discrete-continous optimization

OPTICAL FLOW MODEL ROBUST DATA FUNCTION Objective function for development of a new optimization procedure u denotes the flow field that represents the displacement between frames I1 and I2 ,x represents the 2D coordinates Data Constraints Color constraint D1(u,x)=||I2(x+u)-I1(x)|| Gradient constraint D I(u,x)=ζ|| I2(x+u)- I1(x)|| Data term ED (u,x)=∑1/2 D1(u,x)+1/2 D I(u,x)

Data cost distributions w.r.t different displacement values A good model should only use the more fitting constraint, but not both of them Define a binary weight map α(x):Z―>{0,1} to switch between the two terms. When α(x)=1, the gradient constraint is favored. Otherwise, we select color constraint

EDGE-PRESERVING REGULARIZATION Smoothness term, it maintains motion discontinuity The final objective function is defined as E(u,α)= ED(u,α)+λEs(u) where λ is the regularization weight. MEAN FIELD APPROXIMATION The effective energy is written as Eeff(u)= EeffD(u)+λEs(u) The effective data function β is the inverse temperature β plays a key role in shaping the data function.

Small β makes the distribution close to the original one with α=0.5 A relatively large β yields the distribution approaching the lower envelope of the costs with α=0 and α=1

OPTIMIZATION FRAMEWORK 1212 OPTIMIZATION FRAMEWORK INPUT: A pair of images for optical flow estimation Construct pyramids for both of he images and set the initial level l=0 and uɭ=0 for all pixels Propagate uɭ to level l+1 Extended Flow Initialization 3.1. Detect and match SIFT features in level l+1 3.2 Perform patch matching in level l+1 3.3 Generate multiple flow vectors as candidates 3.4 Optimize flow Continous Flow Optimization 4.1 Compute the ᾱ map 4.2 Solve the energy function 5. Occlusion-aware Refinement If l≠n-1 where n is the total number of levels, l=l+1 and go to step 2 OUTPUT: The optical flow field

EXTENDED FLOW INITIALIZATION Finding multiple extended displacements (denoted as {u0,u1,….,un}) to improve estimation in uc uc which is the flow field computed in the immediately coarser level. The steps adopted to obtain the extended displacements. SIFT Feature Detection Selection Expansion Patch Matching Matching Field Fusion

A)SIFT Feature Detection SIFT feature detection and matching can efficiently capture large motion for objects undergoing translational and rotational motion Employ only the sparse matching of discriminative points, which avoids introducing many ambiguous correspondences and outliers. employ discrete optimization to only select the most credible candidates.

B) Selection The computed displacement vectors by feature matching are denoted as {s0, . . . ,sn} Robustly screen out the duplicated vectors Compute the euclidean distance between each si and all uc If all results are greater than 1 (pixel), we regard si as a new flow candidate. Repeat this process for all si, and denote the m remaining candidate vectors {sk0, . . . ,skm-1}

C)Expansion The m remaining vectors {sk0, . . . ,skm-1} represent possible missing motion in the present flow field uc Determine whether or not they are better estimates to replace the original ones

D) Patch Matching sometimes it still misses some motion vectors because small texture-less objects may not have distinct features SIFT descriptors, the patches on which they operate should at least contain 16x16 samples as suggested. Compute the matching field un by minimizing energy Total of five color and gradient channels used. Noise can be quickly rejected in the following optimization step with the collection of a set of flow candidates for each pixel.

E) Matching Field Fusion The m+1 new motion fields {u0,..um-1,un} together with the original uc, comprise several motion candidates for each pixel in the present image scale. Selection of the optimal flow among the m+2 candidates for each pixel is a labeling problem Solved by discrete optimization efficiently

Extended flow initialization.

CONTINUOUS FLOW OPTIMIZATION Refine flow u0 through continuous optimization We propose decomposing the optimization into three simpler problems, each of which can have the globally optimal solution. Auxiliary variables p and w, representing the substituted data cost and flow derivatives The optimal solution is given by the shrinkage formula

INPUT: Images Ik ,initial flow field u0,weights αk Perform linerization at u0 η= η0 repeat Compute pk θ= θ0 Compute w Compute du θ=θ/3 until θ=θmin η= η/3 until η< ηmin ur=u0+du OUTPUT: Refined flow field ur

fixing them to constants typically results in slow convergence θ and η are critical parameters that should be small. fixing them to constants typically results in slow convergence Initially sets θ and η to large values to allow warm-starting and then decreases them in iterations toward the desired convergence θ and η minimum values are set to 0.1

OCCLUSION-AWARE REFINEMENT Multiple pixels mapping to the same point in the target image using forward warping are possibly occluded by each other detect occlusion using the mapping uniqueness criterion expressed as o(x)=T0.1(f(x+u(x))-1) f(x+u(x)) is the count of reference pixels mapped to position x+u(x) in the target view using forward warping measure of the data confidence based on the occlusion detection is expressed as c(x)=max(1-o(x),0.01)

CONCLUSION A unified framework to preserve motion details in both small and large-displacement scenarios. It include the selective combination of the color and gradient constraints, sparse feature matching, and dense patchmatching to collect appropriate motion candidates Limitations --Texture-less motion details --Large occlusions

REFERENCE L. Alvarez, J. Esclarin, M. Lefebure, and J. Sanchez, “A PDE Model for Computing the Optical Flow” P. Anandan, “A Computational Framework and an Algorithm for the Measurement of Visual Motion” S. Baker, D. Scharstein, J. Lewis, S. Roth, M.J. Black, and R. Szeliski, “A Database and Evaluation Methodology for Optical Flow ” C. Barnes, E. Shechtman, A. Finkelstein, and D.B. Goldman, “Patchmatch: A Randomized Correspondence Algorithm for Structural Image Editing”