1. Efficient Spatiotemporal Grouping Using the Nyström Method Charless Fowlkes, U.C. Berkeley Serge Belongie, U.C. San Diego Jitendra Malik, U.C. Berkeley

2. Grouping With Pairwise Affinities Sarkar and Boyer (1996), Shi and Malik (1997), Perona and Freeman (1998), Gdalyahu, Weinshall, and Werman (2000),..... Compute the similarities between pairs of points in the image Find groups of points which have high similarity with each other and low similarity with the rest of the image.

3. Normalized Cuts 2. Find the leading eigenvectors of the Normalized Laplacian 3. Segment the image using the leading eigenvectors 1. Compute matrix K which contains the pairwise similarities Computational Complexity: Need to find eigenvectors of an NxN matrix where N is the number of pixels. Other spectral partitioning techniques have same complexity.

4. Spatiotemporal grouping Adelson and Bergen (1985), Bolles, Baker and Marimont (1987), Shi and Malik (1998)

5. Computational Problem Hard to exploit pairwise clustering techniques since 256x384x30 frames entails ~10 13 pairwise similarities. How can we overcome this problem?

6. Coping with the computational burden 1 Zero out small entries in the affinity matrix Shi and Malik (97,98) 2 Exploit redundancy between rows of the affinity matrix (this talk)

7. Outline Exploiting Redundancy The Nyström approximation Application to segmenting video sequences

8. Exploiting Redundancy

9. Compute Affinity Matrix

10. Exploiting Redundancy Compute Leading Eigenvectors of Normalized Laplacian

11. Exploiting Redundancy Choose sample points

12. Exploiting Redundancy Compute strip of K

13. Exploiting Redundancy

14. Interpolate complete eigenvectors

15. Outline Exploiting Redundancy The Nyström approximation Application to segmenting video sequences

16. Approximating eigenfunctions E. J. Nyström (1929) Baker (1977) Williams and Seeger (2001) Interpolate eigenfunctions using The Nyström Extension: We would like to find numerical solutions to:

17. Matrix Completion Affinity Matrix: Approximate it with: Approximation Error:

18. Approximation Error

19. Extrapolating Eigenvectors Just matrix notation for the Nyström extension Diagonalize approximate K to get complete eigenvectors

20. Nyström-NCuts Algorithm 1 Choose sample points in image 2 Compute similarities for A and B blocks of K 3 Compute row sums to estimate degree 4 Normalize A and B blocks by degree 5 Compute approximate eigenvectors and orthogonalize 6 Cluster the embedded points using k-means

21. Outline Exploiting Redundancy The Nyström approximation Application to segmenting video sequences

22. Affinity Function for Video Pairwise affinity function between pixels in a video sequence makes of three cues - Similarity in color - Proximity in time and space - Common Fate (similarity in optical flow) We use squared-exponential kernel with diagonal weighting

23. Performance Segmenting a 5 frame video sequence at 120x150 resolution (~100,000 pixels) takes less than 1 minute in MATLAB on a PC

41. Conclusions Applied the Nystrom approximation to Normalized Cuts Exploited redundancy in image sequences in order to perform efficient spatiotemporal grouping

45. K-Way Normalized Cuts Find the leading eigenvectors of Normalized Laplacian Embed data and cluster j th pixel i th vector V

46. Sampling the image