1. Object Recognition Based on Shape Similarity Longin Jan Latecki Computer and Information Sciences Dept. Temple Univ., latecki@temple.edulatecki@temple.edu Collaborators: Zygmunt Pizlo, Psychological Sciences Dept., Purdue Univ., Nagesh Adluru, Suzan Köknar-Tezel, Rolf Lakaemper, Thomas Young, Temple Univ., Xiang Bai, Huazhong Univ. of Sci. & Tech. Wuhan, China

2. 2 Object Recognition Process: Source: 2D image of a 3D object Matching: Correspondence of Visual Parts Contour Segmentation Contour Extraction Object Segmentation Contour Cleaning, e.g., Evolution

3. 3 Motivation Once a significant visual part is recognized the whole recognition process is strongly constrained in possible top- down object models. (H1) object recognition is preceded by, and based on recognition of visual parts. (H2): contour extraction is driven by shape similarity to a known shape.

4. 4 What do you see?

5. 5 With grouping constraints we can see (i.e., recognize the object).

6. 6 Object contours Psychophysical and neurophysiological studies provide an abundance of evidence that contours of objects are extracted in early processing stages of human visual perception. Contours play a central role in the Gestalt-theory.

7. 7 Salient visual parts can influence the object recognition (Singh and Hoffman 2001)

8. 8

9. 9

10. 10 Visual parts and shape similarity (H1) object recognition is preceded by, and based on shape recognition of visual parts. (H2): contour extraction is driven by shape similarity to a known shape. becomes: (H2) Contour extraction is based on grouping of contour parts to larger contour parts with grouping assignments driven by shape familiarity.

11. 11 Contour detection is a difficult inverse problem A given image could be produced by infinitely many possible 3D scenes. In order to produce a unique, stable and accurate interpretation, the visual system must use a priori constraints (see Pizlo, 2001 for a review). The solution is obtained by optimizing a cost function which consists of two general terms: 1. how close the solution is to the visual data 2. how well the constraints are satisfied

12. 12 Partial shape similarity

13. 13 Partial shape similarity (1) length problem, (2) scale problem, (3) distortion problem Query ShapeTarget Shape Given only a part (of a shape ), find similar shapes

14. 14 Partial Shape Similarity We reduce partial shape similarity to subsequence matching: This is done by computing a curvature like value at every contour point. We do this for complete contours of known objects in our database and for query contour parts extracted from edge images

15. 15 Subsequence Matching (shape similarity) Database contours Query contours

16. 16 Motivation for subsequence matching The top (red) and bottom (blue) sequences represent parts of contours of two different but very similar bone shapes

17. 17 Motivation(2) Example sequences: a = {1, 2, 8, 6, 8} b = {1, 2, 9, 15, 3, 5, 9}

18. 18 OSB Algorithm Goal: given two real-valued sequences a and b, find subsequences a’ of a and b’ of b such that a’ best matches b’ Possible to skip elements in both a and b The ability to exclude outliers Preserve the order of the elements A one-to-one correspondence

19. 19 OSB Algorithm (2) Create a dissimilarity matrix No restrictions on the distance function d We used d(a i,b j ) = (a i – b j ) 2 To find the optimal correspondence, use a shortest path algorithm on a DAG

20. 20 OSB Algorithm (3) The nodes of the DAG are all the index pairs of the matrix: (i,j){1,…,m}{1,…,n} The edge weights w are defined by C is the jump cost (the penalty for skipping an element)

21. 21 OSB Algorithm (4) The edge cost may be extended to impose a warping window Set a maximal value for k – i – 1 and l – j - 1 This definition of the edge weights is our main contribution

22. 22 A Simple Example a = {1, 2, 8, 6, 8} b = {1, 2, 9, 15, 3, 5, 9} b 12915359 a 1016419641664 210491691949 8 361492591 6 16981919 849361492591 d(a i,b j ) = (a i – b j ) 2 The dissimilarity matrix

23. 23 (indices) elements distance Key (1,1) 1 - 1 0 (1,2) 1 - 2 1 (1,3) 1 - 9 64 (1,4) 1 - 15 196 (1,5) 1 - 3 4 (1,6) 1 - 5 16 (1,7) 1 - 9 64 (2,1) 2 - 1 1 (2,2) 2 - 2 0 (2,3) 2 - 9 49 (2,4) 2 - 15 169 (2,5) 2 - 3 1 (2,6) 2 - 5 9 (2,7) 2 - 9 49 (3,1) 8 - 1 49 (3,2) 8 - 2 36 (3,3) 8 - 9 1 (3,4) 8 - 15 49 (3,5) 8 - 3 25 (3,6) 8 - 5 9 (3,7) 8 - 9 1 (4,1) 6 - 1 25 (4,2) 6 - 2 16 (4,3) 6 - 9 9 (4,4) 6 - 15 81 (4,5) 6 - 3 9 (4,6) 6 - 5 1 (4,7) 6 - 9 9 (5,1) 8 - 1 49 (5,2) 8 - 2 36 (5,3) 8 - 9 1 (5,4) 8 - 15 49 (5,5) 8 - 3 25 (5,6) 8 - 5 9 (5,7) 8 - 9 1.... The DAG

24. 24 Experimental results on MPEG 7 dataset, 1400 targets in 70 classes

25. 25 Humans group contours automatically An adaptive, probabilistic process to perform grouping All shapes contain local symmetry  exploit local symmetry How to find contour parts in images ?

26. 26 Shape model

27. 27

28. 28

29. 29 Play moviemovie

30. 30 Rao-Blackwellized particle filter is an adaptive, probabilistic approach Frequently utilized in SLAM approaches to Robot Mapping Each particle’s successor is its most likely successor Particles are resampled to eliminate poorly performing particles Contour Grouping as Robot Mapping

31. 31 Traversal space generated as discrete “center points”

32. 32 Center Points Center points act as center points for maximal radius disk between the two sample points Full set of center points gives full set of maximal radius disks Entire set of potential skeletal points Want to generate a skeletal path that best groups the segments for a given shape model

33. 33 Center points and particle paths

34. 34 Shape model System needs to utilize reference model Some a priori knowledge to discover the proper shape Model is a sequence of radii at sample skeleton points Position in reference model determined by triangulation

35. 35 Contour Smoothness Smoothness as a criterion for segment selection Smoothness is the measure of the amount of turn and the distance between segments Use least sum of distance to determine both distance and the segment pairing Smoothness as Gaussian mixture of distance and angle

36. 36 SLAM framework Obtain particles by sampling from the maximum posterior probability x is the path traversal m is the contour grouping model z is the observations u is the reference model

37. 37 Particle filter 1)Sampling: The next generation of particles x (i) t is obtained from the current generation x (i) t-1 by sampling from a proposal distribution for

38. 38 2) Importance Weighting: An individual importance weight w (i) is assigned to each particle, according to

39. 39 log(M(c 2 )) – log of pdf that a given pixel is a center point of radius 10

40. 40 log(M(c 2 )) – log of pdf that a given pixel is a center point of radius 10

41. 41 4) Contour Estimating: For each pose sample x (i) t, the corresponding contour estimate m (i) t is computed based on the trajectory and the history of observations according to 3) Resampling: Particles with a low importance weight w (i) are typically replaced by samples with a high weight. Residual resampling was used

42. 42 Results Grouping performed on several pictures Useful groupings on many images Little or no noise grouped Few structural particles missed

43. 43

44. 44

45. 45 Future Work Integration of the shape similarity of parts and the contour grouping Learning good contour parts Further improvements to the contour grouping to make it more robust