1. Computer Vision Segmentation Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Darrel,...

2. Computer Vision 2 Should have been last week ;-) Sequential structure from motion Initialize motion from two images Initialize structure For each additional view –Determine pose of camera –Refine and extend structure Refine structure and motion

3. Computer Vision 3 Initial projective camera motion Choose P and P´compatible with F Reconstruction up to projective ambiguity (reference plane;arbitrary) Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion Same for more views? different projective basis

4. Computer Vision 4 Initializing projective structure Reconstruct matches in projective frame by minimizing the reprojection error Non-iterative optimal solution Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion

5. Computer Vision 5 Projective pose estimation Infere 2D-3D matches from 2D-2D matches Compute pose from (RANSAC,6pts) F X x Inliers: Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion

6. Computer Vision 6 Refining structure Extending structure 2-view triangulation (Iterative linear) Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion Refining and extending structure

7. Computer Vision 7 Refining structure and motion use bundle adjustment Also model radial distortion to avoid bias!

8. Computer Vision 8

9. Computer Vision 9 Application: video augmentation

10. Computer Vision 10 Jan 16/18-Introduction Jan 23/25CamerasRadiometry Jan 30/Feb1Sources & ShadowsColor Feb 6/8Linear filters & edgesTexture Feb 13/15Multi-View GeometryStereo Feb 20/22Optical flowProject proposals Feb27/Mar1Affine SfMProjective SfM Mar 6/8Camera CalibrationSegmentation Mar 13/15Springbreak Mar 20/22FittingProb. Segmentation Mar 27/29Silhouettes and Photoconsistency Linear tracking Apr 3/5Project UpdateNon-linear Tracking Apr 10/12Object Recognition Apr 17/19Range data Apr 24/26Final project Tentative class schedule

11. Computer Vision 11 Segmentation and Grouping Motivation: not information is evidence Obtain a compact representation from an image/motion sequence/set of tokens Should support application Broad theory is absent at present Grouping (or clustering) –collect together tokens that “belong together” Fitting –associate a model with tokens –issues which model? which token goes to which element? how many elements in the model?

12. Computer Vision 12 General ideas tokens –whatever we need to group (pixels, points, surface elements, etc., etc.) top down segmentation –tokens belong together because they lie on the same object bottom up segmentation –tokens belong together because they are locally coherent These two are not mutually exclusive

13. Computer Vision 13 Why do these tokens belong together?

14. Computer Vision 14

15. Computer Vision 15 Basic ideas of grouping in humans Figure-ground discrimination –grouping can be seen in terms of allocating some elements to a figure, some to ground –impoverished theory Gestalt properties –elements in a collection of elements can have properties that result from relationships (Muller-Lyer effect) gestaltqualitat –A series of factors affect whether elements should be grouped together Gestalt factors

16. Computer Vision 16

17. Computer Vision 17

18. Computer Vision 18

19. Computer Vision 19

20. Computer Vision 20

21. Computer Vision 21

22. Computer Vision 22

23. Computer Vision 23 Technique: Shot Boundary Detection Find the shots in a sequence of video –shot boundaries usually result in big differences between succeeding frames Strategy: –compute interframe distances –declare a boundary where these are big Possible distances –frame differences –histogram differences –block comparisons –edge differences Applications: –representation for movies, or video sequences find shot boundaries obtain “most representative” frame –supports search

24. Computer Vision 24 Technique: Background Subtraction If we know what the background looks like, it is easy to identify “interesting bits” Applications –Person in an office –Tracking cars on a road –surveillance Approach: –use a moving average to estimate background image –subtract from current frame –large absolute values are interesting pixels trick: use morphological operations to clean up pixels

25. Computer Vision 25

26. Computer Vision 26

27. Computer Vision 27

28. Computer Vision 28

29. Computer Vision 29

30. Computer Vision 30

31. Computer Vision 31

32. Computer Vision 32

33. Computer Vision 33

34. Computer Vision 34 Segmentation as clustering Cluster together (pixels, tokens, etc.) that belong together Agglomerative clustering –attach closest to cluster it is closest to –repeat Divisive clustering –split cluster along best boundary –repeat Point-Cluster distance –single-link clustering –complete-link clustering –group-average clustering Dendrograms –yield a picture of output as clustering process continues

35. Computer Vision 35 Simple clustering algorithms

36. Computer Vision 36

37. Computer Vision 37 K-Means Choose a fixed number of clusters Choose cluster centers and point-cluster allocations to minimize error can’t do this by search, because there are too many possible allocations.

38. Computer Vision 38 K-means clustering using intensity alone and color alone Image Clusters on intensity Clusters on color

39. Computer Vision 39 K-means using color alone, 11 segments Image Clusters on color

40. Computer Vision 40 K-means using color alone, 11 segments.

41. Computer Vision 41 K-means using colour and position, 20 segments

42. Computer Vision 42 Graph theoretic clustering Represent tokens using a weighted graph. –affinity matrix Cut up this graph to get subgraphs with strong interior links

43. Computer Vision 43

44. Computer Vision 44

45. Computer Vision 45

46. Computer Vision 46

47. Computer Vision 47

48. Computer Vision 48 Measuring Affinity Intensity Texture Distance

49. Computer Vision 49 Scale affects affinity

50. Computer Vision 50 Eigenvectors and cuts Simplest idea: we want a vector a giving the association between each element and a cluster We want elements within this cluster to, on the whole, have strong affinity with one another We could maximize But need the constraint This is an eigenvalue problem - choose the eigenvector of A with largest eigenvalue

51. Computer Vision 51 Example eigenvector points matrix eigenvector

52. Computer Vision 52

53. Computer Vision 53 More than two segments Two options –Recursively split each side to get a tree, continuing till the eigenvalues are too small –Use the other eigenvectors

54. Computer Vision 54 More than two segments

55. Computer Vision 55 Normalized cuts Current criterion evaluates within cluster similarity, but not across cluster difference Instead, we’d like to maximize the within cluster similarity compared to the across cluster difference Write graph as V, one cluster as A and the other as B Maximize i.e. construct A, B such that their within cluster similarity is high compared to their association with the rest of the graph

56. Computer Vision 56

57. Computer Vision 57

58. Computer Vision 58 Normalized cuts Write a vector y whose elements are 1 if item is in A, -b if it’s in B Write the matrix of the graph as W, and the matrix which has the row sums of W on its diagonal as D, 1 is the vector with all ones. Criterion becomes and we have a constraint This is hard to do, because y’s values are quantized

59. Computer Vision 59

60. Computer Vision 60 Normalized cuts Instead, solve the generalized eigenvalue problem which gives Now look for a quantization threshold that maximises the criterion --- i.e all components of y above that threshold go to one, all below go to -b

61. Computer Vision 61 Figure from “Image and video segmentation: the normalised cut framework”, by Shi and Malik, copyright IEEE, 1998

62. Computer Vision 62 Figure from “Normalized cuts and image segmentation,” Shi and Malik, copyright IEEE, 2000

63. Computer Vision 63 Image parsing: Unifying Segmentation, Detection and Recognition Tu, Chen, Yuille & Zhu ICCV’03

64. Computer Vision 64 Image parsing Generative models for each class some random samples for 5 and H some random face samples (PCA model)

65. Computer Vision 65 DDMCMC Data Driven Markov Chain Monte Carlo

66. Computer Vision 66 Image parsing

67. Computer Vision 67 Image parsing

68. Computer Vision 68 Next class: Fitting Reading: Chapter 15