1. MSU Fall 20141 Computing Motion from Images Chapter 9 of S&S plus otherwork.

2. MSU Fall 20142 General topics Low level change detection Region tracking or matching over time Interpretation of motion MPEG compression Interpretation of scene changes in video Understanding human activites

3. MSU Fall 20143 Motion important to human vision

4. MSU Fall 20144 What’s moving: different cases

5. MSU Fall 20145 Image subtraction Simple method to remove unchanging background from moving regions.

6. MSU Fall 20146 Change detection for surveillance

7. MSU Fall 20147 Change detection by image subtraction

8. Closing = dilation+erosion MSU Fall 20148 http://homepages.inf.ed.ac.uk/rbf/HIPR2/close.htm

9. MSU Fall 20149 What to do with regions of change? Discard small regions Discard regions of non interesting features Keep track of regions with interesting features Track in future frames from motion plus component features

10. MSU Fall 201410 Some effects of camera motion that can cause problems

11. MSU Fall 201411 Motion field

12. MSU Fall 201412 FOE and FOC Will return to use the FOE or FOC or detection of panning to determine what the camera is doing in video tapes.

13. MSU Fall 201413 Gaming using a camera to recognize the player’s motion Decathlete game

14. MSU Fall 201414 Decathlete game Cheap camera replaces usual mouse for input Running speed and jumping of the avatar is controlled by detected motion of the player’s hands.

15. MSU Fall 201415 Motion detection input device Running (hands) Jumping (hands)

16. MSU Fall 201416 Motion analysis controls hurdling event (console) Top left shows video frame of player Middle left shows motion vectors from multiple frames Center shows jumping patterns

17. MSU Fall 201417 Related work Motion sensed by crude cameras Person dances/gestures in space Kinect/Leap motion sensorsLeap motion System maps movement into music Creative environment? Good exercise room?

18. MSU Fall 201418 Computing motion vectors from corresponding “points” High energy neighborhoods are used to define points for matching

19. MSU Fall 201419 Match points between frames Such large motions are unusual. Most systems track small motions.

20. MSU Fall 201420 Requirements for interest points Match small neighborhood to small neighborhood. The previous “scene” contains several highly textured neighborhoods.

21. MSU Fall 201421 Interest = minimum directional variance Used by Hans Moravec in his robot stereo vision system. Interest points were used for stereo matching.

22. MSU Fall 201422 Detecting interest points in I1

23. MSU Fall 201423 Match points from I1 in I2

24. MSU Fall 201424 Search for best match of point P1 in nearby window of I2 For both motion and stereo, we have some constraints on where to search for a matching interest point.

25. MSU Fall 201425 Motion vectors clustered to show 3 coherent regions All motion vectors are clustered into 3 groups of similar vectors showing motion of 3 independent objects. (Dina Eldin) Motion coherence: points of same object tend to move in the same way

26. MSU Fall 201426 Two frames of aerial imagery Video frame N and N+1 shows slight movement: most pixels are same, just in different locations.

27. MSU Fall 201427 Can code frame N+d with displacments relative to frame N for each 16 x 16 block in the 2 nd image find a closely matching block in the 1 st image replace the 16x16 intensities by the location in the 1 st image (dX, dY) 256 bytes replaced by 2 bytes! (If blocks differ too much, encode the differences to be added.)

28. MSU Fall 201428 Frame approximation Left is original video frame N+1. Right is set of best image blocks taken from frame N. (Work of Dina Eldin)

29. MSU Fall 201429 Best matching blocks between video frames N+1 to N (motion vectors) The bulk of the vectors show the true motion of the airplane taking the pictures. The long vectors are incorrect motion vectors, but they do work well for compression of image I2! Best matches from 2 nd to first image shown as vectors overlaid on the 2 nd image. (Work by Dina Eldin.)

30. MSU Fall 201430 Motion coherence provides redundancy for compression MPEG “motion compensation” represents motion of 16x16 pixels blocks, NOT objects

31. MSU Fall 201431 MPEG represents blocks that move by the motion vector

32. MSU Fall 201432 MPEG has ‘I’, ‘P’, and ‘B’ frames

33. MSU Fall 201433 Computing Image Flow

34. MSU Fall 201434

35. Motion Field & Optical Flow Field Motion Field = Real world 3D motion Optical Flow Field = Projection of the motion field onto the 2d image 3D motion vector 2D optical flow vector CCD Slides from Lihi Zelnik-Manor

36. MSU Fall 201436 Assumptions

37. When does it break? The screen is stationary yet displays motion Homogeneous objects generate zero optical flow. Fixed sphere. Changing light source. Non-rigid texture motion Slides from Lihi Zelnik-Manor

38. MSU Fall 201438 Image flow equation 1 of 2

39. MSU Fall 201439 Image flow equation 2 of 2

40. Estimating Optical Flow Assume the image intensity is constant Time = t Time = t+dt Slides from Lihi Zelnik-Manor MSU Fall 201440

41. Brightness Constancy Equation First order Taylor Expansion Simplify notations: Divide by dt and denote: Problem I: One equation, two unknowns Slides from Lihi Zelnik-Manor MSU Fall 201441

42. Time t+dt Problem II: “ The Aperture Problem ” Time t ? Time t+dt Where did the yellow point move to? We need additional constraints For points on a line of fixed intensity we can only recover the normal flow Slides from Lihi Zelnik-Manor MSU Fall 201442

43. Use Local Information Sometimes enlarging the aperture can help Slides from Lihi Zelnik-Manor MSU Fall 201443

44. Local smoothness Lucas Kanade (1984) Assume constant (u,v) in small neighborhood Slides from Lihi Zelnik-Manor MSU Fall 201444

45. Lucas Kanade (1984) Goal: Minimize 2x2 2x1 Method: Least-Squares Slides from Lihi Zelnik-Manor MSU Fall 201445

46. Lucas-Kanade Solution We want this matrix to be invertible. i.e., no zero eigenvalues Slides from Lihi Zelnik-Manor MSU Fall 201446

47. Break-downs Brightness constancy is not satisfied A point does not move like its neighbors what is the ideal window size? The motion is not small (Taylor expansion doesn ’ t hold) Correlation based methods Regularization based methodsUse multi-scale estimation Slides from Lihi Zelnik-Manor MSU Fall 201447

48. Multi-Scale Flow Estimation image I t-1 image I Gaussian pyramid of image I t Gaussian pyramid of image I t+1 image I t+1 image I t u=10 pixels u=5 pixels u=2.5 pixels u=1.25 pixels Slides from Lihi Zelnik-Manor MSU Fall 201448

49. MSU Fall 201449 Tracking several objects Use assumptions of physics to compute multiple smooth paths. (work of Sethi and R. Jain)

50. MSU Fall 201450

51. MSU Fall 201451 Tracking in images over time

52. MSU Fall 201452 General constraints from physics

53. MSU Fall 201453 Other possible constraints Background statistics stable Object color/texture/shape might change slowly over frames Might have knowledge of objects under survielance Objects appear/disappear at boundary of the frame

54. MSU Fall 201454

55. MSU Fall 201455 Sethi-Jain algorithm

56. MSU Fall 201456

57. MSU Fall 201457 Total smoothness of m paths

58. MSU Fall 201458 Greedy exchange algorithm

59. MSU Fall 201459 Example data structure Total smoothness for trajectories of Figure 9.14

60. MSU Fall 201460 Example of domain specific tracking (Vera Bakic) Tracking eyes and nose of PC user. System presents menu (top). User moves face to position cursor to a particular box (choice). System tracks face movement and moves cursor accordingly: user gets into feedback- control loop.

61. MSU Fall 201461 Segmentation of videos/movies Segment into scenes, shots, specific actions, etc.

62. MSU Fall 201462 Types of changes in videos

63. MSU Fall 201463 Anchor person scene at left Street scene for news story Scene break From Zhang et al 1993 How do we compute the scene change?

64. MSU Fall 201464 Histograms of frames across the scene change Histograms at left are from anchor person frames, while histogram at bottom right is from the street frame.

65. MSU Fall 201465 Heuristics for ignoring zooms

66. MSU Fall 201466 American sign language example

67. MSU Fall 201467 Example from Yang and Ahuja

68. MSU Fall 201468

69. MSU Fall 201469