1. ter Haar Romeny, ICPR 2010 Introduction to Scale-Space and Deep Structure

2. ter Haar Romeny, ICPR 2010 Importance of Scale Painting by Dali Objects exist at certain ranges of scale. It is not known a priory at what scale to look.

3. ter Haar Romeny, ICPR 2010 At the original scale of a dithered image we cannot calculate a derivative. We need to observe the image at a certain scale. BLUR

4. ter Haar Romeny, ICPR 2010 Solution? Look at all scales simultaneously Scale x y Scale Space

5. ter Haar Romeny, ICPR 2010 Scale Space in Human Vision The human visual system is a multi- scale sampling device The retina contains receptive fields; groups of receptors assembled in such a way that they form a set of apertures of widely varying size.

6. ter Haar Romeny, ICPR 2010 Practical Implementation Convolve the image with a Gaussian Kernel

7. ter Haar Romeny, ICPR 2010 We can calculate derivatives and combinations of them at all scales Gradient Magnitude Laplacian Original Image

8. ter Haar Romeny, ICPR 2010 Main Topic In this presentation we will show how we can exploit the deep structure of images to define invariant interest points and features which can be used for matching problems in computer vision. We consider only grey-value images.

9. ter Haar Romeny, ICPR 2010 Interest Points The locations of particularly characteristic points are called the interest points or key points. These interest points have to be as invariant as possible, but at the same time they have to carry a lot of distinctive information.

10. ter Haar Romeny, ICPR 2010 Why interest points in scale-space? Information in interest points is defined by their neighborhood. But how big should we choose this neighborhood? Let’s take the corners of the mouth as interest points. The red circles are the areas in which the information is gathered. If we make the picture bigger, the size of the neighborhood is too small. The neighborhood should scale with the image

11. ter Haar Romeny, ICPR 2010 When the interest points are detected in scale space they do not only have spatial coordinates x and y, but also a scale . This scale tells us how big the neighborhood should be.   Why interest points in scale-space?

12. ter Haar Romeny, ICPR 2010 Which interest points to use? Our interest points have to be detected in scale space. They also have to… …contain a lot of information …be reproducible …be stable …be well understood

13. ter Haar Romeny, ICPR 2010 We suggest Top-Points The points we introduce have these desired properties.

14. ter Haar Romeny, ICPR 2010 Critical Points, Paths and Top-Points Maxima Minimum Saddles  L=0 Critical Points

15. ter Haar Romeny, ICPR 2010 Critical Points, Paths and Top-Points Maxima Minimum Saddles  L=0 Critical Points Det(H)=0 Top-Points

16. ter Haar Romeny, ICPR 2010 Possible to calculate them for every function of the image L(x,y,  ) OriginalGradient MagnitudeLaplacianDet(H)

17. ter Haar Romeny, ICPR 2010 Detecting critical paths Since for a critical path  L=0, intersection of level surfaces L x =0 with L y =0 will give the critical paths.

18. ter Haar Romeny, ICPR 2010 Detecting Top-Points Since for a top-point both  L=0 and det[H]=L xx L yy -L xy 2 =0, we can find them by intersecting the paths with the level surface Det[H]=0

19. ter Haar Romeny, ICPR 2010

20. Original image Top-points and features Reconstruction

21. ter Haar Romeny, ICPR 2010 Metameric class Original By adjusting boundary and smoothness constraints we can improve the visual performance. For this 300x300 picture 1000 top-points with 6 features were used.

22. ter Haar Romeny, ICPR 2010 Localization of Top-Points For points close to top-points it is possible to calculate a vector pointing towards the position of the top-point. x y  Approximated Top-PointsDisplacement VectorsReal Locations

23. ter Haar Romeny, ICPR 2010 Stability of Top-Points We can calculate the variance of the displacement of top- points under noise. We need 4 th order derivatives in the top-points for that.

24. ter Haar Romeny, ICPR 2010

25. Thresholding on stability Stable Paths Unstable Paths

26. ter Haar Romeny, ICPR 2010 Invariance of top-points Top-points are invariant to certain transformations. By invariant we mean that they move according to the transformation. Allowed Trans.

27. ter Haar Romeny, ICPR 2010 Differential invariants We use the complete set of irreducible 3 rd order differential invariants. These features are rotation and scaling invariant.

28. ter Haar Romeny, ICPR 2010 The task We have a scene and from that scene we want to retrieve the location of the query object.

29. ter Haar Romeny, ICPR 2010 The top-points and differential invariants are calculated for the query object and the scene.

30. ter Haar Romeny, ICPR 2010 Distance between feature vectors A sensible distance between feature vectors is essential. We have used Euclidean distance on ‘normalized’ differential invariants. We tried Mahalanobis distance obtained from a training set.

31. ter Haar Romeny, ICPR 2010 Similarity measure We can calculate the propagation of noise in scale space*. This enables us to calculate a covariance matrix  for each feature vector. The dissimilarity (“distance”) measure is expressed as: *Topological and Geometrical Aspects of Image Structure, Johan Blom

32. ter Haar Romeny, ICPR 2010 We now compare the differential invariant features. compare distance = 0.5distance = 0.2distance = 0.3

33. ter Haar Romeny, ICPR 2010 The vectors with the smallest distance are paired. smallest distance distance = 0.2

34. ter Haar Romeny, ICPR 2010 A set of coordinates is formed from the differences in scale (Log(  o1 )- Log(  s2 )) and in angles (  o1 -  s2 ). (  1,  1 )

35. ter Haar Romeny, ICPR 2010  Dq Important Clusters For these clusters we calculate the mean  and  Clustering ( ,  ) If these coordinates are plotted in a scatter plot clusters can be identified. In this scatter plot we find two dense clusters

36. ter Haar Romeny, ICPR 2010 The stability criterion removes much of the scatter:

37. ter Haar Romeny, ICPR 2010 Rotate and scale according to the cluster means.

38. ter Haar Romeny, ICPR 2010 The translations we find correspond to the location of the objects in the scene.

39. ter Haar Romeny, ICPR 2010 In this example we have two clusters of correctly matched points. C1 C2

40. ter Haar Romeny, ICPR 2010 The transformation of each object in the scene matching to the query object is known from the clustering.

41. ter Haar Romeny, ICPR 2010 We can transform the outline of the query object and project it on the scene image.

42. ter Haar Romeny, ICPR 2010 Video Google?