1. 1 Formation et Analyse d’Images Session 4 Daniela Hall 21 October 2004

2. 2 Course Overview Session 1: –Homogenous coordinates and tensor notation –Image transformations –Camera models Session 2: –Camera models –Reflection models –Color spaces Session 3: –Review color spaces –Pixel based image analysis Session 4: –Gaussian filter operators –Scale Space

3. 3 Course overview Session 5: –Contrast description –Hough transform Session 6: –Kalman filter –Tracking of regions, pixels, and lines Session 7: –Stereo vision –Epipolar geometry Session 8: exam

4. 4 Session Overview 1.Signal derivatives 2.Gaussian derivatives 3.Global and local feature description 4.Scale space 5.Application: object recognition 6.Application: logo detection in sports videos

5. 5 Signal description An image is a 2D signal. It can be approximated by a Taylor serie using the derivatives of the function. The Taylor serie is a projection of the signal on its derivatives.

6. 6 Derivative of a signal For a discrete signal, lim delta t->0 does not exist. How can we compute the derivatives for a signal?

7. 7 Derivatives of a signal How can we compute the derivatives for a signal? –Compute the product of the signal and a kernel function. – is the integral of the product In the case that s(t) is discrete, replace t by nT with sampling step T.

8. 8 Sampling Parameters: interval [-R,R] sampling step T

9. 9 Filters and derivatives The scalar product is a projection of the signal and a function (discrete case). The evaluation at position k of the scalar product of a signal and a function is computed as a convolution.

10. 10 Derivatives of signals To compute a derivative of a signal, you compute the derivative of the filter function (kernel) and then sample the derivative. The derivative of the signal is computed by a convolution of the filter function f’(nT) (T is set to 1 in most cases).

11. 11 Appropriate filter functions The ideal would be a filter function that is invariant to projective transformation. This does not exist. The affine transformations are a subset of the projective transformations. The Gaussian is invariant to affine transformation. Use G(x,y;σ) as filter function for derivative computation.

12. 12 Recall: transformations projective transformation –the most general transformation that transforms any 3 non-collinear points into any other 3 non-colinear points. affine transformation –translation, rotation, scale and shear (non- uniform scaling in some directions). similarity transformation –translation and scaling euclidean transformation –translation and rotation

13. 13 Gaussian function The Gaussian is invariant to affine transformations T aff

14. 14 Gaussian function Gaussian function in 2D (black =0, white=1). General form: d dimensions, mu mean, Cov covariance 2D form with 0 mean and Cov= diag(σ x, σ y ). Gaussian is isotropic (round) when σ x =σ y

15. 15 Gaussian function Affine transformation: translation, rotation, scaling, shear. Invariant to translation: G(x,σ) is local Invariant to rotation: G(x,σ) is isotropic. Invariant to scale: sG(x,σ)=G(sx,sσ) because of the explicit scale parameter σ Invariant to shear: because it is separable and scale invariant G(x,y;σ x,σ y )= G(x,σ x )G(y,σ y ) –separable: first compute 1D Gaussian in 1 direction, then 1D Gaussian in other direction. Result is the same as a 2D Gaussian.

16. 16 Gaussian derivatives approximation of a signal derivative by convolution with a Gaussian derivative

17. 17 Size of the support Window function w R (n) For any signal s(n) that is defined on [-R,R] For other signals, the loss of the convolution with the window function can be computed.

18. 18 Gaussian with finite support If R=2σ: If R=3σ: 2σ2σ 3σ3σ

19. 19 Signal derivatives using Gaussian with finite support By applying the window function, we loose 2% of the information, but enable the computation of the derivative. We know the interval [-R,R] and the sampling step T=1. G(x,1.0) |0.004|0.054|0.242|0.399|0.242|0.054|0.004| G x (x,1.0) |0.013|0.108|0.242|0.00|-0.242|-0.108|-0.013| Example of sampling a 1D Gaussian (sigma=1, R=3) convolution masks

20. 20 Signal derivatives using Gaussian with finite support (2D) Examples:

21. 21 Gaussian derivatives in 2D Impulse responses Gaussian and G x and G y black = 0, yellow = positive, green = negative

22. 22 Gaussian derivatives in 2D Gxx Gxy Gyy

23. 23 Gaussian derivatives in 2D GxxxGxxy Gxyy Gyyy

24. 24 Signal description A signal can be described by a Taylor serie. The Gaussian derivatives can be used to approximate the signal derivatives.

25. 25 Description of a discrete signal The same reasoning is true for a discrete signal. The coefficients (ai) form a signature of the signal. The coefficients can be used for reconstruction.

26. 26 Signature of a signal A signature is a unique representation of the signal in a different space (feature space). Identical signals have the same signature. Similar signals have signatures that are close in feature space. Signatures of different signals have great distance in feature space. Knowing the signature and the basis, the signal can be reconstructed.

27. 27 Reconstruction of a discrete signal Knowing the basis functions (Gaussian derivatives) we can reconstruct the signal from the signature (coefficients (ai)).

28. 28 Session Overview 1.Signal derivatives 2.Gaussian derivatives 3.Global and local feature description 4.Scale space 5.Application: object recognition 6.Application: logo detection in sports videos

29. 29 Global image description An image is represented by a single signature (a single point in feature space) Reconstruction is very bad (in the example we have only 5 coefficients). S= G(i,j,h/3)*I(x0,y0)+Gx(i,j,h/3)*I(x0,y0)+Gy(i,j,h/3)*I(x0,y0) +Gxx(i,j,h/3)*I(x0,y0)+Gxy(i,j,h/3)*I(x0,y0)+Gyy(i,j,h/3)*I(x0,y0) =a0G+a1Gx+a2Gy+a3Gxx+a4Gxy+a5Gyy (a0,a1,a2,a3,a4,a5) is signature of image. global description (x0,y0)

30. 30 Local image feature description Principle: describe a large number of overlapping local image patches (imagettes) Each image patch is a local feature. Typical size of local features 9x9 pixels....

31. 31 Local image feature description We obtain a signature v=(a0,a1,a2,...) T for each imagette S(x k,y k,s). S(x k,y k,s)= G(i,j,s)*I(x k,y k )+Gx(i,j,s)*I(x k,y k )+Gy(i,j,s)*I(x k,y k )+... =a0G(i,j,s)+a1Gx(i,j,s)+a2Gy(i,j,s)+...

32. 32 Example Properties: –Describe image signal according to Taylor series –local, scalable, steerable –Measure basic geometries Seattle

33. 33 Local features Reconstruction is much better –For a patch of 9x9 pixels, perfect reconstruction is obtained using 81 coefficients. –Using 10 coefficients (up to Gyyy) gives good reconstruction. –The low order derivatives measure the basic geometries of the image (lines, bars, edges, corners). High order derivatives add the high frequency details. Gaussian derivative feature space –similar features have signatures that are close in feature space. –Distance in feature space can be used as similarity function.

34. 34 Derivatives of images (sigma=3.0) Gx Gy Gxx Gxy Gyy

35. 35 Gaussian derivative feature space Gaussians have several interesting properties –invariant to affine transformations –2D Gaussian is separable and can be computed efficiently –Gaussian derivatives form an orthogonal basis –Gaussian derivatives can be used to provide a Taylor serie decomposition of the image signal –Low order Gaussian derivatives measure basic geometries –Gaussian derivative operators are called local receptive fields because the receptors of the human visual system are similar to Gaussian derivatives. –Gaussian derivatives are steerable and scalable.

36. 36 Steerablity of Gaussian derivatives Steerability: a function rotated by any angle can be synthesized by a linear combination of the basis functions. This is true for Gaussian derivatives.

37. 37 Example Gx Gy Gtheta, theta=30

38. 38 Orientation invariant description Dominant direction of a neighborhood I(i,j) is computed by generation of receptive field at arbitrary orientation

39. 39 Session Overview 1.Signal derivatives 2.Gaussian derivatives 3.Global and local feature description 4.Scale space 5.Application: object recognition 6.Application: logo detection in sports videos

40. 40 Scalability of Gaussian derivatives Gaussian derivatives can be scaled due to the explicit scale parameter sigma.

41. 41 Example: derivatives at several scales σ=1 σ=2 σ=4 σ=16 σ=8

42. 42 Scale of image features It would be interesting to know which are the interesting scales of an object. Interesting scales: scales at which an important structure is visible. In the example: the interesting scale (if we are interested in the people) is somewhere between 2.0 and 4.0. Scale space theory answers these questions and provides a method to describe local features invariant to scale.

43. 43 Scale of image features Do these image features correspond?

44. 44 Scale space representation L is the scale space representation of I(x,y). L is obtained by smoothing I with Gaussian kernel G(  ). G(  ) is the natural choice for building up a scale space.

45. 45 Scale Space Scale space representation I(x,y) L(x,y,  ) x y  x

46. 46 Intrinsic feature scale Measurement procedure: –At a given spatial point (x,y) we sample the scale direction by applying a normalized derivative of different scales (scale signature f). –Different operators can be used for Laplacian Gradient norm [Lindeberg98]

47. 47 Gradient norm and Laplacian

48. 48 Intrinsic feature scale Intrinsic scale: –The scale at which this normalized derivative assumes a maximum marks a feature containing interesting structure. [Lindeberg98]

49. 49 Scale signature of a feature

50. 50 Intrinsic scale is stable over scales

51. 51 Scale invariant feature description Given a feature with intrinsic scale The description components are normalized for

52. 52 Scale invariant feature description 1.Compute intrinsic feature scale for each imagette. 2.Project imagette on scale normalised Gaussian derivatives. S(x,y,sk)=a0G(i,j,sk)+a1G’(i,j,sk)+a2G’’(i,j,sk)+... 3.Extract scale invariant signature (coefficients and intrinsic scale sk) v(S(x,y,sk))=(a0,a1,a2,...,sk) 4.Store all signatures with image id and position.

53. 53 Scale invariant feature description Do these image features correspond? Yes, because the signatures are close in feature space. dist(v(S(x,y,σ)),v(S(sx,sy,sσ))) <eps S(x,y,σ) S(sx,sy,sσ)

54. 54 Session Overview 1.Signal derivatives 2.Gaussian derivatives 3.Global and local feature description 4.Scale space 5.Application: object recognition 6.Application: logo detection in sports videos

55. 55 Object recognition Traning data: a set of images containing objects. Test data: same objects in different configurations. Goal: given a small number of measurements, recognize which objects are in the image.

56. 56 Training data

57. 57 Test data

58. 58 Recognition systems Learning phase: –sensing the world –describing the measurements –storage of measurement representation in appropriate data structure Recognition phase: –sensing the world –describing the measurements –interpreting the measurements compare with learned measurements decide

59. 59 My recognition system Learning phase: –sensing the world with a digital color camera –discriminant feature description (texture & color) –storage in indexing structure Recognition phase: –sensing the world –describing the measurements –interpreting the measurements search possible matching candidates in indexing structure recognition by vote or hypothesis verification

60. 60 Choice of the feature space The feature space should have few dimensions. The description vector (set of Gaussian derivatives) should be chosen such that they capture the information necessary for recognition The description vector should be invariant to the transformations of the input data. –In case of viewpoint independent recognition, we require invariance to position, orientation, size, and affine transformations. –The choice are scale normalised Gaussian receptive fields oriented to the dominant direction of the imagette.

61. 61 Feature description vector describe local texture up to order 2 and local chrominance up to order 1 description invariant to position, scale and orientation similarity of features is proportional to distance in feature description space

62. 62 Building the model Select representative training images Scale invariant feature description Storage in appropriate indexing structure for efficient retrieval

63. 63 Recognition phase Sensing the world Describing the measurements Interpreting the measurements –search possible matching candidates in the learned data –recognition by vote or hypothesis verification

64. 64 Search matching candidates Matching candidates are found by evaluating the distance in feature space. All elements within a sphere with radius  from the query vector are matching candidates. Tree structures allow efficient nearest neigbor search.

65. 65 Recognize by vote Distance evaluation produces a list of matching candidates. For each measurement we know from which object it came from (information stored during learning). Every matching candidate votes for the object it came from. The object with the higest number of votes is the winner (recognition result).

66. 66 Experiment Image indexing: 8 objects on uniform background. Voting: –Select image features –Search matching candidates –Image for which votes the majority of the matching candidates gets a vote Result: 99% recognition by votes after 9 different measurements

67. 67 Open problems Changes in viewpoint Recognition under natural conditions (changes in lighting, viewing angle, …) Capacity of tree structure is limited

68. 68 Session Overview 1.Signal derivatives 2.Gaussian derivatives 3.Global and local feature description 4.Scale space 5.Application: object recognition 6.Application: logo detection in sports videos

69. 69 Detecting Logos in Video The video producer “sells” air-time of corporate publicity images (logos). How can he measure the value of his product?

70. 70 Detecting logos in video Video publicity statistics –How many times does logo appear? –For each appearance What was the duration? How big was it? How visible was it? How near is it to the focus of attention? Current estimates are obtained off-line by hand –This process is slow, costly and inaccurate.

71. 71 The challenge of logo detection Challenge: –Natural outdoor scenes –Unconstrained view angles –Rapid pan-tilt-zoom of cameras –Occlusions –Similar color/texture –Unknown target position, deformations Goal: real time collection of logo statistics from sports videos

72. 72 Model acquisition Learns model data for detection and identification Data acquisiton: –Hand label example frames –Semi automatic labeling (label first frame, track following frames) Learning phase: –color histograms for detection and tracking –Gaussian derivative histograms for identification Problem: training data must capture operating conditions

73. 73 Model acquisition

74. 74 Detection by color histogram

75. 75 Detection by histograms

76. 76 Identification by Gaussian derivative histograms

77. 77 Logo identification Typical Example