1. O PTICAL F LOW & L AYERED M OTION Prepared by: Lidya Tonkonogi Mu’in Rizik 1

2. I NTRODUCTION – M OTION E STIMATION Motion estimation algorithms calculate the motion vectors that describe the transformation from one image to another 2

3. I NTRODUCTION – M OTION E STIMATION What if… there were no motion estimation algorithms… 3

4. I NTRODUCTION – M OTION E STIMATION Why Do we need Motion Estimation?  Video Compression  Video stabilization: 4

5. I NTRODUCTION – M OTION E STIMATION Old school Registration algorithms(Pre optical flow):  Slow and Costly  Failed in processing images with rotation and other distortions 5

6. O PTICAL FLOW 6

7. D EFINITION OPTICAL FLOW = apparent motion of brightness patterns Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image 7

8. P ROBLEM DEFINITION : OPTICAL FLOW How to estimate pixel motion from image H to image I? Solve pixel correspondence problem –given a pixel in H, look for nearby pixels of the same color in I 8

9. 9 Key assumptions color constancy: a point in H looks the same in I –For grayscale images, this is brightness constancy small motion: points do not move very far This is called the optical flow problem

10. O PTICAL FLOW CONSTRAINTS ( GRAYSCALE IMAGES ) Let’s look at these constraints more closely brightness constancy: Q: what’s the equation? H(x,y)=I(x+u, y+v) 10

11. 11 small motion: (u and v are less than 1 pixel) – suppose we take the Taylor series expansion of I:

12. OPTICAL FLOW EQUATION Combining these two equations In the limit as u and v go to zero, this becomes exact 12

13. O PTICAL FLOW EQUATION Q: how many unknowns and equations per pixel? 2 unknowns, one equation 13 Aperture Problem

14. W HAT ’ S IS THE PROBLEM T HE MOTION FROM LEFT TO RIGHT ??! 14

15. W HAT ’ S IS THE PROBLEM WRONG !!! 15

16. T HE A PERTURE P ROBLEM Different motions – classified as similar source: Ran Eshel 16

17. T HE A PERTURE P ROBLEM Similar motions – classified as different source: Ran Eshel 17

18. T HE A PERTURE P ROBLEM This problem results from the fact that we have only one equation and two unknowns. This equation : (the gradient) Define a line in (u,v) space 18

19. To solve this equation we will calculate the Normal flow We get at most “Normal Flow” – with one point we can only detect movement perpendicular to the brightness gradient. Solution is to take a patch of pixels Around the pixel of interest. T HE A PERTURE P ROBLEM 19

20. A PERTURE P ROBLEM AND N ORMAL F LOW 20

21. O PTICAL F LOW T ECHNIQUES Differential techniques: Local method: Lucas and Kanade Global method: Horn and Schunck 21

22. I NTRODUCTION – M OTION E STIMATION Lucas and Kanade 22

23. Lucas and Kanade proposed an algorithm to solve the aperture problem. We back to the function And they proposed another solution to find u and v. Question : How to get more equations for a pixel? Basic idea: impose additional constraints one method: pretend the pixel’s neighbors have the same (u,v) in other words If we use a kxk window, that gives us equations per pixel! L UCAS -K ANADE ALGORITHM 23

24. For example : Assume that we use a 5x5 window, that gives us 25 equations per pixel! L UCAS -K ANADE - EXAMPLE 24

25. We try to solve this problem by : In the previous example : Minimizing 25

26. L UKAS -K ANADE FLOW Prob:we have more equations than unknowns Solution : solve least squares problem minimum least squares solution given by solution of: 26 K x 22 x 1 K x 1 Minimize :

27. C ONDITIONS FOR SOLVABILITY Optimal (u, v) satisfies Lucas-Kanade equation When is This Solvable? A T A should be invertible A T A should not be too small due to noise –eigenvalues 1 and 2 of A T A should not be too small A T A is solvable when there is no aperture problem 27

28. Hessian Matrix I x = 0 I y = 0 M = 0 Non invertible I x = 0 I y = k M = 0 0 k 2 Non invertible I x = k I y = 0 M = k 2 0 0 Non invertible I x = k 1 I y = k 2 RM = k 2 0 0 Non invertible k 1, k 2 correlated (R = rotation) I x = k 1 I y = k 2 M = k 1 2 0 0 k 2 2 invertible k 1 * k 2 = 0 28

29. E XAMPLE : 29

30. E DGE – large gradients, all the same – large  1, small 2 30

31. L OW TEXTURE REGION – gradients have small magnitude – small  1, small 2 31

32. H IGH TEXTURED REGION – gradients are different, large magnitudes – large  1, large 2 32

33. L UCAS -K ANADE Advantages easy and fast calculation accurate time derivatives Disadvantages errors on boundaries 33

34. E RRORS IN L UKAS -K ANADE What are the potential causes of errors in this procedure? Suppose A T A is easily invertible Suppose there is not much noise in the image When our assumptions are not valid Brightness constancy is not satisfied The motion is not small 34

35. H ORN & S CHUNCK Berthold Horn 35

36. Horn and Schunk proposed an algorithm to solve the aperture problem by using a Global method with smoothness constraint. Minimize a global energy functional with calculus of variations We back to the function And they proposed another solution to find u and v. H ORN -S CHUNCK ALGORITHM 36

37. H ORN -S CHUNCK ALGORITHM They assumed that in most of the pictures, the velocity of the objects do not change radically between adjacent pixels ! Here our new equation and we want to find the minimum : 37

38. H ORN -S CHUNCK ALGORITHM 38

39. H ORN -S CHUNCK ALGORITHM Iterative scheme The superscript k denotes the current iteration The solution depends on the neighboring values of the flow field, it must be repeated once the neighbors have been updated. 39

40. R EVISITING THE SMALL MOTION ASSUMPTION Is this motion small enough? Probably not—it’s much larger than one pixel (2 nd order terms dominate) How might we solve this problem? 40

41. A LIASING Temporal aliasing causes ambiguities in optical flow because images can have many pixels with the same intensity. I.e., how do we know which ‘correspondence’ is correct? nearest match is correct (no aliasing) nearest match is incorrect (aliasing) coarse-to-fine estimation To overcome aliasing: coarse-to-fine estimation. actual shift estimated shift 41

42. R EDUCE THE RESOLUTION ! 42

43. image I image H Gaussian pyramid of image HGaussian pyramid of image I image I image H u=10 pixels u=5 pixels u=2.5 pixels u=1.25 pixels C OARSE - TO - FINE OPTICAL FLOW ESTIMATION 43

44. image I image J Gaussian pyramid of image HGaussian pyramid of image I image I image H C OARSE - TO - FINE OPTICAL FLOW ESTIMATION run iterative L-K warp & upsample...... 44

45. I TERATIVE R EFINEMENT Iterative Lukas-Kanade Algorithm 1.Estimate velocity at each pixel by solving Lucas- Kanade equations 2.Warp I(t-1) towards I(t) using the estimated flow field - use image warping techniques 3.Repeat until convergence 45

46. O PTICAL F LOW R ESULTS 46

47. O PTICAL F LOW R ESULTS 47

48. L AYERED M OTION 48

49. W HAT IS L AYERED M OTION ? A system for representing moving images with sets of overlapping layers 49

50. W HY D O WE NEED IT ? Smooth Edges Reduces blurring – Each layer motion can be smooth 50

51. 51

52. W HY D O WE NEED IT ? Smooth Edges Reduces blurring – Each layer motion can be smooth Used for video segmentation 52

53. V IDEO S EGMENTATION 53

54. H OW D O WE DO IT ? Edward Adelson and John Wang (1994) 54

55. A DELSON AND W ANG High Level – divide the image into three maps: Intensity Map Alpha Map (transparency) Velocity Map 55

56. A DELSON AND W ANG 56

57. ADELSON AND W ANG The garden scene: 57

58. ADELSON AND W ANG Divide the frame into non-overlapping patches 58

59. ADELSON AND W ANG Calculate the velocity with a coarse to fine optical flow algorithm 59

60. ADELSON AND W ANG Find the affine motion estimation, why? We assume that most of the motions in the world we can describe by affine transformation:  Translation  Rotation  Zoom  Shear 60

61. ADELSON AND W ANG 61

62. ADELSON AND W ANG One way to calculate them is by defining matching points between two images and calculating the affine transformation matrix 62

63. ADELSON AND W ANG 63

64. ADELSON AND W ANG 64

65. ADELSON AND W ANG 65

66. ADELSON AND W ANG 66

67. ADELSON AND W ANG 67

68. 68

69. ADELSON AND W ANG The reliability of the hypothesis can be measured by it’s residual error (let’s skip it’s calculation) Patches with a Hypotheses with a residual error greater than a specified threshold are eliminated from the calculation (large residual error usually indicates that the patch contains object boundaries in it) 69

70. ADELSON AND W ANG Patches hypotheses with a residual error smaller than a defined threshold, and similar affine transformations we want to group together 70

71. ADELSON AND W ANG Divide the patches into clusters using k- means (remember the segmentation lecture) 71

72. ADELSON AND W ANG Now we have regions with similar affine transformation We want to verify our affine transformation vs. the actual movement 72

73. ADELSON AND W ANG 73

74. ADELSON AND W ANG 74

75. ADELSON AND W ANG 75

76. ADELSON AND W ANG We iterate to minimize the distortion Conversion is obtained when only a few pixels are reassigned or if we reached the max number of iterations. 76

77. ADELSON AND W ANG The Layers representation: 77

78. ADELSON AND W ANG In the last phase we warp the layers together, using a median filter to overcome slight changes in the intensity 78

79. ADELSON AND W ANG Final Result: 79

80. Good Job Adelson and Wang! But can we do better?? 80

81. L AYERED M OTION O PTIMIZATIONS Optimizations by Weiss and Adelson (1996) Automatically define the optimal number of layers Replace Affine motion model with smooth regularized motion models 81

82. L AYERED M OTION O PTIMIZATIONS Optimizations by Baker, Szeliski and Anandan (1998): Use homographies instead of affine motion estimation 82

83. 83

84. L AYERED M OTION O PTIMIZATIONS Optimizations by Baker, Szeliski and Anandan (1998) – continue: Optimize The layers colors and opacity by minimizing the discrepancies between the re-synthesized and observed motion sequence 84

85. L AYERED M OTION O PTIMIZATIONS Baker, Szeliski and Anandan (1998) – Layers assignment: Adelson and Wang– Layers assignment: 85

86. D EMO TIME ! 86

87. S UMMARY Motions Estimation Algorithms Optical Flow The Aperture Problem Lucas and Kanade Algorithm Horn and Schunck Algorithm Coarse-to-fine Layered Motion Estimation (Adelson and Wang) 87

88. R EFERENCES Books: Computer Vision: Algorithms and Applications, Richard Szeliski Introductory Techniques for 3D Computer Vision, Emanuele Trucco, Alessandro Verri 88

89. R EFERENCES Papers: Layered Representation for Motion analysis, John Y.A. Wang & Edward H. Adelson http://persci.mit.edu/pub_pdfs/wang_tr221.pdf http://en.wikipedia.org/wiki/Horn%E2%80%93Sc hunck_method http://en.wikipedia.org/wiki/Lucas%E2%80%93K anade_method http://robots.stanford.edu/cs223b04/CS%20223- B%20L9%20Optical%20Flow.ppt 89

90. R EFERENCES Youtube: https://www.youtube.com/v/j3uCV0JYMJ4?versio n=3&start=137&end=242&autoplay=0&hl=en_U S&rel=0 https://www.youtube.com/watch?v=sp77AjBdlEc 90