Presentation is loading. Please wait.

Presentation is loading. Please wait.

University of Toronto Oct. 18, 2004 Modelling Motion Patterns with Video Epitomes Machine Learning Group Meeting University of Toronto Oct. 18, 2004 Vincent.

Similar presentations


Presentation on theme: "University of Toronto Oct. 18, 2004 Modelling Motion Patterns with Video Epitomes Machine Learning Group Meeting University of Toronto Oct. 18, 2004 Vincent."— Presentation transcript:

1 University of Toronto Oct. 18, 2004 Modelling Motion Patterns with Video Epitomes Machine Learning Group Meeting University of Toronto Oct. 18, 2004 Vincent Cheung Probabilistic and Statistical Inference Group Electrical & Computer Engineering University of Toronto Toronto, Ontario, Canada Advisor: Dr. Brendan J. Frey

2 ML Group Meeting, Oct. 18, 2004 Cheung1 / 18 Outline ●Image epitome ► What? ► Why? ► How? ●Implementation computation issues ► Efficiently implementing the learning algorithm ●Video epitome ► Extension to videos ► Filling-in missing information

3 ML Group Meeting, Oct. 18, 2004 Image Learning Video Cheung2 / 18 Image Epitome ●Jojic, N., Frey, B., & Kannan, A. (2003). Epitomic analysis of appearance and shape. In Proc. IEEE ICCV. ●Miniature, condensed version of the image ●Models the image’s shape and textural components. ●Applications ► object detection ► texture segmentation ► image retrieval ► compression

4 ML Group Meeting, Oct. 18, 2004 Image Learning Video Cheung3 / 18 Image Epitome Examples

5 Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung4 / 18 Learning the Image Epitome (1)

6 Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung5 / 18 Learning the Image Epitome (2) Epitome Input image Training Set Sample Patches Unsupervised Learning e Z1Z1 Z2Z2 ZMZM … TMTM T2T2 T1T1 Bayesian network e – epitome T k – mapping Z k – image patch E: M:

7 Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung6 / 18 Shifted Cumulative Sum Algorithm

8 Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung7 / 18 Collecting Sufficient Statistics

9 Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung8 / 18 Extending Epitomes to Videos ●Desire a miniature, condensed version of a video sequence ●Want the epitome to model the basic shape, textural, and motion patterns of the video ●Applications ► optic flow ► segmentation ► texture transfer ► layer separation ► compression ► noise reduction ► fill-in / inpainting

10 Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung9 / 18 Input Video Frame 1 Frame 2 Frame 3 Training Set Sample Patches Video Epitome Unsupervised Learning Video Epitome

11 Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung10 / 18 Video Epitome Example Temporally Compressed Spatially Compressed

12 Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung11 / 18 Video Inpainting (1) ●Fill in missing portions of a video ► damaged films ► occluding objects ●Reconstruct the missing pixels from the video epitome

13 Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung12 / 18 Video Inpainting (2)

14 Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung13 / 18 Filling-in Missing Data Likelihood Joint Variational Approx E-Step M-Step

15 Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung14 / 18 Missing Channels ●Generalization of the video inpainting problem ●Inpainting ► Missing entire pixels ●Missing Channels ► Missing one or more of the red, green, or blue (RGB) components of a given pixel ●Epitome must consolidate multiple patches together to piece together the missing channel information ► No training patch contains all the channel information ► Use the epitome to fill-in the missing data

16 Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung15 / 18 Image Missing Channels Fill-in

17 Image Learning Video ML Group Meeting, Oct. 18, 2004 Cheung16 / 18 Video Missing Channels Fill-in

18 ML Group Meeting, Oct. 18, 2004 Cheung17 / 18 Conclusion ●Improved the efficiency of learning image epitomes ●Extended the concept of epitomes to video sequences ●Demonstrated the ability of video epitomes to model motion patterns through video inpainting

19 ML Group Meeting, Oct. 18, 2004 Cheung18 / 18 Future Work ●Determining the size of the epitome ► Dependent on the complexity of the image / video ■Minimum description length ■Variational Bayesian ●Optimal patch size(s) ► Problem specific ●Additional transformations into the epitome ► Rotation ► Scale ●Additional video epitome applications ► Super-resolution ► Layer separation ► Object recognition


Download ppt "University of Toronto Oct. 18, 2004 Modelling Motion Patterns with Video Epitomes Machine Learning Group Meeting University of Toronto Oct. 18, 2004 Vincent."

Similar presentations


Ads by Google