1. LOCUS (Learning Object Classes with Unsupervised Segmentation) A variational approach to learning model- based segmentation. John Winn Microsoft Research Cambridge with Nebojsa Jojic, MSR Redmond 7 th July 2006

2. Overview Learning object models The LOCUS model Experiments & results Extensions to LOCUS

3. Goal Long Term Goal Recognise ~10,000 object classes.

4. Learning from ‘buckets’ of images Horse model Learning algorithm Object Segmentation Object Recognition Object Detection

5. Object segmentation + Horse model LOCUS

6. Related work

7. Constellation models  Weakly supervised  Probabilistic framework  Sparse  No segmentation Object class recognition by unsupervised scale-invariant learning. R. Fergus, P. Perona, and A. Zisserman. CVPR 2003 A Bayesian approach to unsupervised One-Shot learning of Object categories. L. Fei-Fei, R. Fergus, and P. Perona. ICCV 2003

8. Fragment-based Learning to segment. E. Borenstein and S. Ullman. ECCV 2004 Combining top-down and bottom-up segmentation. E. Borenstein, E. Sharon, and S. Ullman. CVPR 2004  Dense model  Supervised  Non-probabilistic  No global shape model

9. Codebook-based Combined object categorization and segmentation with an implicit shape model. B. Leibe, A. Leonardis, and B. Schiele. ECCV ‘04  Probabilistic  Dense model  Supervised  Ad-hoc inference

10. OBJ CUT  Probabilistic  Dense model  Supervised  Requires video

11. LOCUS overview  Weakly supervised learning Buckets of images - no annotation required.  Probabilistic generative model of both object and background.  Dense model All pixels modelled, not just at interest points.  Combines global and local cues Models global shape and local appearance + edges.  Iterative inference process Simultaneous localisation, segmentation, pose estimation.

12. The LOCUS model

13. LOCUS model Deformation field D Position & size T Class shape π Class edge sprite μ o,σ o Edge image e Image Object appearance λ 1 Background appearance λ 0 Mask m Shared between images Different for each image

14. LOCUS model: appearance background object Mask m Background mixture coefficients λ0λ0 Object mixture coefficients λ1λ1 Image z Shared mixture components:

15. LOCUS model: mask background object 8-neighbour Markov Random Field (as used in GrabCut) favours segmentation along contrast edges

16. LOCUS model: shape/position … … TNTN T4T4 T2T2 T3T3 T1T1 Transformation Class shape π

17. Iterative inference … … TNTN T4T4 T2T2 T3T3 T1T1 Class shape π Iteration #1

18. Iterative inference … … TNTN T4T4 T2T2 T3T3 T1T1 Class shape π Iteration #2

19. Iterative inference … … TNTN T4T4 T2T2 T3T3 T1T1 Class shape π Iteration #3

20. Iterative inference … … TNTN T4T4 T2T2 T3T3 T1T1 Class shape π Iteration #5

21. Iterative inference … … TNTN T4T4 T2T2 T3T3 T1T1 Class shape π Iteration #8

22. Iterative inference … … TNTN T4T4 T2T2 T3T3 T1T1 Class shape π Iteration #12

23. Non-rigid objects Class shape π Translation and scale is not enough.

24. LOCUS model: pose Class shape π T Deformation field D 5x5 blocks Prior ensures smoothness

25. LOCUS model: pose Class shape π TD 1 TD 2 TD 3 TD N … …

26. LOCUS model: edge TD 1 TD 2 TD 3 TD N … … Edge images e … Original images Class edge sprite μ o,σ o

27. LOCUS model: overview Deformation field D Position & size T Class shape π Class edge sprite μ o,σ o Edge image e Image Object appearance λ 1 Background appearance λ 0 Mask m Shared between images Different for each image

28. Inference Aim to infer all latent variables, For each image: background appearance λ 0, object appearance λ 1, deformation D, transformation T, mask m, Class variables: shape π, edge sprite μ o, σ o. Bayesian inference is carried out using variational message passing with a fully factorised variational distribution. Optimisation of grid-structured variational free energy terms (relating to the deformation field D and the mask m ) achieved using graph cuts.

29. Experiments & results

30. Experiments LOCUS applied to 8 sets of 20 images each containing objects of the same class. Horses Faces Cars (rear) Cars (side) Motorbikes Aeroplanes Cows Trees For each class, we ran separate experiments for color and texture appearance models.

31. Results: horses

33. Results: cars

35. Results: remaining classes Cars (rear)FacesMotorbikesPlanesCowsTrees

36. Segmentation accuracy HorsesCars (side) LOCUS (color) LOCUS (texture) unannotated training images 93.1% 93.0% 91.4% 94.0% Borenstein et al. hand-segmented training images 93.6%- Each image segmented separately 88.6%82.1% To evaluate segmentation quantitively, we used hand segmentations for horses and cars (side).

37. Object registration Transformation + deformation field registers object outlines (and some internal edges).

38. Object registration

39. Extensions to LOCUS

40. Recognition + segmentation Object recognition using only global shape: Overall: 88% accuracy.

41. Probabilistic Index Maps 2 indices9 indices Each image has a ‘palette’ of appearance models – palette invariance.

42. Probabilistic Index Maps

43. Learning objects from video Object shape Object edge sprite

44. Locumotion Add flow and track constraints to achieve motion segmentation: Tracking/flow estimation by Larry Zitnick

45. Conclusions LOCUS gives unsupervised segmentations of accuracy equivalent to state-of-the-art supervised methods. General-purpose model allows: Object localisation Pose estimation Object segmentation Motion segmentation/object tracking Object recognition/detection (in combination with discriminative model)

46. Questions ?