2. Motion Estimation using Markov Random Fields Hrvoje Bogunović Image Processing Group Faculty of Electrical Engineering and Computing University of Zagreb Summer School on Image Processing, Graz 2004

3. Overview Introduction Optical flow Markov Random Fields OF+MRF combined Energy minimization techniques Results

4. Introduction Input: –Sequence of images (Video) Problem –Extract information about motion Applications –Detection, Segmentation, Tracking, Coding

5. Spatio-temporal spectrum φ f

6. Motion – aliasing φ f 1/x 1/t Large area flicker Loss of spatial resolution

7. Large motions - temporal aliasing φ f Temporal aliasing Great loss of spatial resolution

8. Temporal anti-aliasing φ f No more overlaping on the f axis. filtering (anit-aliasing) is performed after sampling, hence the blurring

9. Motion – eye tracking φ f

10. Motion estimation Images are 2-D projections of the 3-D world. Problem is represented as a labeling one. –Assign vector to pixel Vector field  field of movement Low level vision –No interpretation

11. Example Ideal

12. Problems Problem is inherently ill-posed –Solution is not unique Aperture problem –Specific to local methods

13. Optical flow Main assumption: Intensity of the object does not change as it moves –Often violated First solved by Horn & Schunk –Gradient approach Other approaches include –Frequency based –Using corresponding features

14. Image differencing

15. Gradient approach Local by nature. Aperture problem is significant. Image understanding is not required –Very low level

16. Horn & Schunk Intensity stays the same in the direction of movement. I(x,y,t) After derivation

17. Horn & Schunk Spatial gradients I x,I y –e.g. Sobel operator Temporal gradient I t –Image subtraction

18. Regularization Tikhonov regularization for ill-posed problems Add the smoothness term Energy function

19. Result

20. Problems of the H-S method Assumption: There are no discontinuities in the image –Optical flow is over-smoothed. Gradient method. Only the edges which are perpendicular to motion vector contribute Image regions which are uniform do not contribute. Difficulty with large motions (spatial filtering)

21. Optical flow enhancement Optical flow can be piecewise smooth Discontinuities can be incorporated Solution: use the spatial context Problem is posed as a solution of the Bayes classifier. Solution in optimization sense. Search for optimum

22. Bayes classifier Main equation Solution using MAP estimation

23. Markov Random Fields Suitable: Problems posed as a visual labeling problemn with contextual constraints Useful to encode a priori knowledge –required for bayes classifier (smoothness prior) –equvalence to Gibbs random fields (gibbs distribution, exponential like) Neighbourhoods Cliques –pairs,triples of neighbourhood points) –build the energy function

24. MRF Define sites: rectangular lattice Define labels define neighbourhood: 4,8 point Field is MRF: –P(f)>0 –P(f i |f {S-i} )=P(f i |N i )

25. Coupled MRF Field F is an optical flow field Field L is a field of discontinuities –line process Position of the two fields.

26. Context neighbourhoods and cliques

27. Motion estimation equations

28. Energy for MAP estimation Parameters are estimated ad hoc

29. Energy minimization Global minimum –Simulated annealing –Genetic Algorithms Local minimum –Iterated Conditional Modes (ICM) (steepest decent) –Highest Confidence First (HCF) specific site visiting

30. Simulated annealing (1) Find the initial temperature of the system T. (2)Assign initial values of the field to random (3)For every pixel: Assign random value to f(i,j) Calculate the difference in energy before and after If the change is better (diff>0) keep it. Else keep it with the probability exp(diff/T) (4) Repeat (3) N1 times (5) T = f(T) where f decreases monotono (6) Repeat (3-5) N2 times

31. Results (Square) Horn-Schunk OF OF+MRF

32. Taxi

33. Results (Taxi)

34. Line process result (Taxi)

35. Cube

36. Results (cube)

37. Line process result (cube)

38. Q & A