1. Global and Efficient Self-Similarity for Object Classification and Detection CVPR 2010 Thomas Deselaers and Vittorio Ferrari

2. Conventional Image Descriptors Measure direct image properties gradients colors 2

3. Self-Similarity vs Conventional Descriptors [Shechtman, Irani CVPR 07] Assumption of conventional image descriptors There is a direct visual property shared by images of objects of the same class (e.g. colors, gradients, …). This property can be used to compare images. Self-similarity: Indirect property: geometric layout of repeating patches within an image More general property 3

4. Local Self-Similarity Descriptors 4 [Shechtman, Irani CVPR 07]

5. Using Local Self-Similarity Descriptors Applications: object recognition, image retrieval, action recognition Ensemble matching [Shechtman CVPR 07] Nearest neighbor matching [Boiman CVPR 08] Bag of local self-similarities [Gehler ICCV09, Vedaldi ICCV09, Hörster ACMM08, Lampert CVPR09, Chatfield ICCV09 WS] 1.Compute LSS descriptors for an image 2.Assign the LSS descriptors to a codebook 3.Represent the image as a histogram of LSS descriptors 5

6. Self-Similarity goes Global Capture long-range self-similarities and their spatial arrangement 6

7. Self-Similarity goes Global Capture long-range self-similarities and their spatial arrangement 7

8. compute self-similarity between all pairs of pixels Global Self-Similarity Tensor 4D self-similarity tensor Note: local self-similarities included 8

9. Problems with the GSS Tensor Computation time: Memory requirement: Aim: Reduce both 9 11 ∼ 80GB ∼ 20h

10. Outline Efficient global self-similarity tensor Global self-similarity descriptors – Bag of correlation surfaces – Self-similarity hypercubes Detection with self-similarity hypercubes – Efficient sliding window – Efficient subwindow search Experiments – Global self-similarity better than local self-similarity – Complementary to conventional descriptors – Object detection possible 10

11. Efficient Global Self-Similarity Tensor Find an efficient approximation to Quantize patches according to codebook If two patches are assigned to the same prototype, they are similar Reduces runtime to speedup: 11 750

12. Efficient Global Self-Similarity Two patches are only similar if they are assigned to the same prototype Reduces memory to reduction: 12

13. Patch Prototype Codebooks Remember: Self-similarity encodes image content indirectly Image-specific codebooks can be smaller than conventional ones see paper for more generic codebooks and extensive evaluation 13

14. Self-similarity hypercubes: now Bag of correlation surfaces: only in the paper Global Self-Similarity Descriptors So far: Compact GSS computed efficiently Now: Descriptors that can be used in machine learning classifiers Fixed dimensionality Compact representation 14

15. Self-Similarity Hybercubes SSH of size 15

16. SSHs for Detection Computing SSH naïvely requires operations Sliding windows has to evaluate many windows 16 operations

17. Efficient Computation of SSHs Compute integral self-similarity tensor: operations to compute SSH for an image window 17  ∼ 5000x speedup  160000 can be obtained using 16 lookups in

18. Efficient Subwindow Search for SSH 18 Derive an upper bound on the score of a set of windows Section 5.2 in our paper Similar to [Lampert PAMI09]

19. Experiments: Object classification PASCAL 07 objects – 9608 cropped images of objects from PASCAL 07 – 20 classes Task: Classify each test image into one of 20 classes Model: Linear SVM Train: train+val Test: test 19

20. Classification on the PASCAL 07 objects set + GSS outperform LSS + Self-Similarity is truly complementary to conventional descriptors 20 classification accuracy [%]

21. Experiments: Object detection ETHZ Shape Classes – 255 images – 5 classes (apple logos, bottles, giraffes, mugs, swans) Task: Detect objects in images Detector: Linear SVM, sliding windows 21 e.g. [Ferrari CVPR07, Maji CVPR09]

22. Detection Results + SSH outperforms BOLSS + it is possible to use GSS for detection with good results 22 BoLSSSSH apple logos10.080.0 bottles10.796.4 giraffes23.485.1 mugs6.567.7 swans17.670.6 Average13.680.0 DR at FPPI 0.4 apple logos bottles giraffes mug s swans } } SSH BoLSS FPPI 0.4 Comparison results (avg): [Ferrari CVPR07]: 71.9 [Maji CVPR09]: 93.2 … many more DR at 0.5 PASCAL overlap

23. Runtimes for Computing Descriptors 200x200 image GSS tensor – directly: 5512s ( ∼ 1.5 hours) – using our method: 81s ( ∼ 1.5 minutes) Computing descriptors: few seconds Our method: 70x speedup For Reference: – GIST: 0.4s – BOLSS: 0.7s 23

24. Runtimes for Detection Given the prototype assignment map (80s) (once only) SSH sliding window: 30s/img (once per class) For Comparison – Computing direct GSS tensor for 25000 windows: 4 years/img Speedup: ∼ 1 million ⇒ Using our methods, GSS can be used for object detection For Reference: – Felzenszwalb PAMI 09: 5s. 24 June 2014

25. Global and Efficient Self-Similarity for Object Classification and Detection CVPR 2010 Thomas Deselaers and Vittorio Ferrari Feasible

26. Conclusion self-similarity should be considered globally – Global self-similarity performs better than local self-similarity truly complementary to conventional descriptors global self-similarity is feasible – efficient computation of self-similarity – two descriptors based on self-similarity global self-similarity for detection code will be available soon 26

27. Thank you for your attention