1. Classifying Visual Objects Regardless of Depictive Style Qi Wu, Peter Hall Department of Computer Science University of Bath

2. Summary Conventional Comp.Vis. classifiers do not generalise well across depictive styles. We propose new visual class model, one invariant to depictive style. Experiments validate our model.

3. People can see objects in a wide variety of depictive styles. PhotosArtwork

4. Literature Gap: BoW does not generalise across depictive styles PhotosArtwork 47% (Dense SIFT) PhotosArtwork 51% (Dense SIFT)

5. Our solution: A new Visual Class Model that does generalise across styles. PhotosArtwork 47% (Dense SIFT) PhotosArtwork 51% (Dense SIFT) 64% 67%

6. A New Visual Class Model We assume an object class is characterised by: – the qualitative shape of object parts, – the structural arrangement of those parts. A hierarchical graph model per image: – coarse-to-fine representation (layered), – nodes labelled by primitive shapes, abstracting region shape brings greater robustness. – arcs labelled with displacement vectors Median graph models: – aggregates models from several instances, – single class model.

7. Making a VCM (a): An input collection. (b): Probability maps for each input image, and graph models for each map. (c): The median graph model for the whole class. (d): The refined median graph as the final class model

8. A schematic VCM A hierarchical description Berkeley segmentation Filtering process using cLge A graph Arcs at same level denote touching neighbours. Arcs between layers link parent – children. Nodes label A 6-elements probability vector. The probability that a region belongs to a given prime shape class.

9. Prime Shapes, BMVC 2012

10. Prime Shapes Does a set of elementary planar shapes appear commonly in the world ? Art provides strong anecdotal evidence “yes” – 20th century Western Art --- Cubism

11. Determine Prime Shapes A fully unsupervised framework

12. Determine Prime Shapes

13. Back to our model…

14. Build graphs, one for each image Left: graph model. Right: Object broken in primitive shapes

15. Compute an initial Visual Class Model Median Graph First compute the graph distance between each pair. Using the distance matrix to embed graph into a vector space Compute the Euclidean Median of all the data points. Transfer the median vector back to graph using a state-of-art method proposed in [Ferrer and Valveny, 2008]

16. Refine the Visual Class Model The initial model contains nodes and arcs that derive from visual clutter in back ground of images in the training set Refine the model Match the median back into each training image. Count the number of times a given node in the model appears in the training data. Delete all nodes below a frequency threshold., which is computed via maximising matching score.

17. Some Examples

18. Experiments Compare with other two methods PHOW features (Dense SIFT) [Bosch and Zisserman, ICCV 2007] Local PAS features [Ferrari and Jurie, IJCV 2010] Structure Only [Bai and Song, CVIU 2011]

19. Results

20. Conclusions It’s possible to learn models of objects classes that generalise across depictive styles. Many applications are promised. Just a first step – Simplify the model, still too much nodes and arcs. – Time consuming. – Additional labelling – Move to object localisation.

21. Questions?

22. One application of Prime shapes