Download presentation

Presentation is loading. Please wait.

Published byAlia Beauford Modified over 4 years ago

1
Analysis of Contour Motions Ce Liu William T. Freeman Edward H. Adelson Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology Neural Information Processing Systems 2006

2
Visual Motion Analysis in Computer Vision Motion analysis is essential in –Video processing –Geometry reconstruction –Object tracking, segmentation and recognition –Graphics applications Is motion analysis solved? Do we have good representation for motion analysis? Is it computationally feasible to infer the representation from the raw video data? What is a good representation for motion?

3
Seemingly Simple Examples Kanizsa square From real video

4
Output from the State-of-the-Art Optical Flow Algorithm T. Brox et al. High accuracy optical flow estimation based on a theory for warping. ECCV 2004 Optical flow field Kanizsa square

5
Output from the State-of-the-Art Optical Flow Algorithm T. Brox et al. High accuracy optical flow estimation based on a theory for warping. ECCV 2004 Optical flow field Dancer

6
Optical flow representation: aperture problem Corners Lines Flat regions Spurious junctionsBoundary ownership Illusory boundaries

7
Optical Flow Representation Corners Lines Flat regions Spurious junctionsBoundary ownership Illusory boundaries We need motion representation beyond pixel level!

8
Layer Representation Video is a composite of layers Layer segmentation assumes sufficient textures for each layer to represent motion A true success? J. Wang & E. H. Adelson 1994 Y. Weiss & E. H. Adelson 1994 Achieved with the help of spatial segmentation

9
Layer Representation Video is a composite of layers Layer segmentation assumes sufficient textures for each layer to represent motion A true success? J. Wang & E. H. Adelson 1994 Y. Weiss & E. H. Adelson 1994 Achieved with the help of spatial segmentation Layer representation is good, but the existing layer segmentation algorithms cannot find the right layers for textureless objects

10
Challenge: Textureless Objects under Occlusion Corners are not always trustworthy (junctions) Flat regions do not always move smoothly (discontinuous at illusory boundaries) How about boundaries? –Easy to detect and track for textureless objects –Able to handle junctions with illusory boundaries

11
Analysis of Contour Motions Our approach: simultaneous grouping and motion analysis –Multi-level contour representation –Junctions are appropriated handled –Formulate graphical model that favors good contour and motion criteria –Inference using importance sampling Contribution –An important component in motion analysis toolbox for textureless objects under occlusion

12
Three Levels of Contour Representation –Edgelets : edge particles –Boundary fragments : a chain of edgelets with small curvatures –Contours : a chain of boundary fragments Forming boundary fragments: easy (for textureless objects) Forming contours: hard (the focus of our work)

13
Overview of our system 1. Extract boundary fragments 2. Edgelet tracking with uncertainty. 3. Boundary grouping and illusory boundary 4. Motion estimation based on the grouping

14
Forming Boundary Fragments Boundary fragments extraction in frame 1 –Steerable filters to obtain edge energy for each orientation band –Spatially trace boundary fragments –Boundary fragments: lines or curves with small curvature Temporal edgelet tracking with uncertainties (a)(b) (c)(d) –Frame 1: edgelet (x, y, ) –Frame 2: orientation energy of –A Gaussian pdf is fit with the weight of orientation energy –1D uncertainty of motion (even for T- junctions)

15
Forming Contours: Boundary Fragments Grouping Grouping representation: switch variables (attached to every end of the fragments) –Exclusive : one end connects to at most one other end –Reversible : if end (i,t i ) connects to (j,t j ), then (j,t j ) connects to (i,t i ) Arbitrarily possible connection Reversibility A legal contour grouping Another legal contour grouping 0 1 0 1 1 0

16
Local Spatial-Temporal Cues for Grouping Motion stimulus Illusory boundaries corresponding to the groupings (generated by spline interpolation)

17
Local spatial-temporal cues for grouping: (a) Motion similarity Motion stimulus Velocity space KL( ) < KL( ) The grouping with higher motion similarity is favored

18
Local spatial-temporal cues for grouping: (b) Curve smoothness Motion stimulus The grouping with smoother and shorter illusory boundary is favored

19
Local spatial-temporal cues for grouping: (c) Contrast consistency Motion stimulus The grouping with consistent local contrast is favored

20
The Graphical Model for Grouping Affinity metric terms –(a) Motion similarity –(b) Curve smoothness –(c) Contrast consistency The graphical model for grouping reversibilityaffinityno self-intersection

21
Motion estimation for grouped contours Gaussian MRF (GMRF) within a boundary fragment The motions of two end edgelets are similar if they are grouped together The graphical model of motion: joint Gaussian given the grouping This problem is solved in early work: Y. Weiss, Interpreting images by propagating Bayesian beliefs, NIPS, 1997.

22
Inference Two-step inference –Grouping (switch variables) –Motion based on grouping (easy, least square) Grouping: importance sampling to estimate the marginal of the switch variables –Bidirectional proposal density –Toss the sample if self-intersection is detected Obtain the optimal grouping from the marginal

23
Why bidirectional proposal in sampling?

24
b 1 b 2 : 0.39 b 1 b 3 : 0.01 b 1 b 4 : 0.60 Normalized affinity metrics b 4 b 1 : 0.20 b 4 b 2 : 0.05 b 4 b 3 : 0.85 b 2 b 1 : 0.50 b 2 b 3 : 0.45 b 2 b 4 : 0.05 b 3 b 1 : 0.01 b 3 b 2 : 0.45 b 3 b 4 : 0.54 b 1 b 2 : 0.1750 b 1 b 3 : 0.0001 b 1 b 4 : 0.1200 Affinity metric of the switch variable (darker, thicker means larger affinity) Bidirectional proposal

25
Why bidirectional proposal in sampling? b 1 b 2 : 0.39 b 1 b 3 : 0.01 b 1 b 4 : 0.60 Normalized affinity metrics Bidirectional proposal (Normalized) b 4 b 1 : 0.20 b 4 b 2 : 0.05 b 4 b 3 : 0.85 b 2 b 1 : 0.50 b 2 b 3 : 0.45 b 2 b 4 : 0.05 b 3 b 1 : 0.01 b 3 b 2 : 0.45 b 3 b 4 : 0.54 b 1 b 2 : 0.62 b 1 b 3 : 0.00 b 1 b 4 : 0.38 Bidirectional proposal of the switch variable (darker, thicker means larger affinity)

26
Example of Sampling Motion stimulus Self intersection

27
Example of Sampling Motion stimulus A valid grouping

28
Example of Sampling Motion stimulus More valid groupings

29
Example of Sampling Motion stimulus More valid groupings

30
From Affinity to Marginals Affinity metric of the switch variable (darker, thicker means larger affinity) Motion stimulus

31
From Affinity to Marginals Marginal distribution of the switch variable (darker, thicker means larger affinity) Motion stimulus Greedy algorithm to search for the best grouping based on the marginals

32
Experiments All the results are generated using the same parameter settings Running time depends on the number of boundary fragments, varying from ten seconds to a few minutes in MATLAB

33
Frame 1 Two Moving Bars

34
Frame 2 Two Moving Bars

35
Extracted boundary fragments. The green circles are the boundary fragment end points. Two Moving Bars

36
Optical flow from Lucas-Kanade algorithm. The flow vectors are only plotted at the edgelets Two Moving Bars

37
Estimated motion by our system after grouping Two Moving Bars

38
Boundary grouping and illusory boundaries (frame 1). The fragments belonging to the same contour are plotted in one color. Two Moving Bars

39
Boundary grouping and illusory boundaries (frame 2). The fragments belonging to the same contour are plotted in one color. Two Moving Bars

40
Kanizsa Square

41
Frame 1

42
Frame 2

43
Extracted boundary fragments

44
Optical flow from Lucas-Kanade algorithm

45
Estimated motion by our system, after grouping

46
Boundary grouping and illusory boundaries (frame 1)

47
Boundary grouping and illusory boundaries (frame 2)

48
Dancer

49
Frame 1

50
Frame 2

51
Extracted boundary fragments

52
Optical flow from Lucas-Kanade algorithm

53
Estimated motion by our system, after grouping

54
Lucas-Kanade flow field Estimated motion by our system, after grouping

55
Boundary grouping and illusory boundaries (frame 1)

56
Boundary grouping and illusory boundaries (frame 2)

57
Rotating Chair

58
Frame 1

59
Frame 2

60
Extracted boundary fragments

61
Estimated flow field from Brox et al.

62
Estimated motion by our system, after grouping

63
Boundary grouping and illusory boundaries (frame 1)

64
Boundary grouping and illusory boundaries (frame 2)

65
Conclusion A contour-based representation to estimate motion for textureless objects under occlusion Motion ambiguities are preserved and resolved through appropriate contour grouping An important component in motion analysis toolbox To be combined with the classical motion estimation techniques to analyze complex scenes

66
Thanks! Analysis of Contour Motions Ce Liu William T. Freeman Edward H. Adelson Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology http://people.csail.mit.edu/celiu/contourmotions/

67
Backup Slides

68
Why bidirectional proposal in sampling?

69
b 1 b 2 : 0.39 b 1 b 3 : 0.01 b 1 b 4 : 0.60 Normalized affinity metrics b 4 b 1 : 0.20 b 4 b 2 : 0.05 b 4 b 3 : 0.85 b 2 b 1 : 0.50 b 2 b 3 : 0.45 b 2 b 4 : 0.05 b 3 b 1 : 0.01 b 3 b 2 : 0.45 b 3 b 4 : 0.54 b 1 b 2 : 0.1750 b 1 b 3 : 0.0001 b 1 b 4 : 0.1200 Affinity metric of the switch variable (darker, thicker means larger affinity) Bidirectional proposal

70
Why bidirectional proposal in sampling? b 1 b 2 : 0.39 b 1 b 3 : 0.01 b 1 b 4 : 0.60 Normalized affinity metrics Bidirectional proposal (Normalized) b 4 b 1 : 0.20 b 4 b 2 : 0.05 b 4 b 3 : 0.85 b 2 b 1 : 0.50 b 2 b 3 : 0.45 b 2 b 4 : 0.05 b 3 b 1 : 0.01 b 3 b 2 : 0.45 b 3 b 4 : 0.54 b 1 b 2 : 0.62 b 1 b 3 : 0.00 b 1 b 4 : 0.38 Bidirectional proposal of the switch variable (darker, thicker means larger affinity)

71
Sampling Grouping (Switch Variables) Motion stimulus

72
Lucas-Kanade flow field Estimated motion by our system, after grouping

Similar presentations

© 2020 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google