1. (1) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Feature Point Matching Detection, Extraction & Matching Part 2 – Matching via template matching (Previous slides: Feature point detection Moravec detection Harris detection Scale space detection Feature point Extraction ).

2. (2) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Template matching – X-correlation ● Template matching by Cross-correlation  You have already learnt template matching with sum of square differences – SSD: There you were trying to minimize a dissimilarity score.  Here you will learn about template matching by normalized cross-correlation: minimizing a similarity score.

3. (3) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Template matching – X-correlation ● Cross-correlation:  In statistical parlance, the cross-correlation between two random variables is a measure of how closely the two variables vary together.  {A,B} = {1,2},{8,16},{3,6},{-5,-10} ● The magnitude of these variables vary together closely – they have a high positive cross-correlation.  {C,D} = {-1,2},{-8,16},{-3,6},{5,-10} ● The magnitude of these vary oppositely closely – they have a high negative cross-correlation.  {E,F} = {-1,2},{8,0},{-3,6},{-5,-10} ● These don’t vary closely – they have a low cross-correlation.

4. (4) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Template matching – X-correlation ● Cross-correlation: ● Normalized cross-correlation (-1<0<-1):  {A,B} = {1,2},{8,16},{3,6},{-5,-10} ● X(A,B) = 43 C(A,B) = 1.0  {C,D} = {-1,2},{-8,16},{-3,6},{5,-10} ● X(C,D) = -43 C(A,B) = -1.0  {E,F} = {-1,2},{8,0},{-3,6},{-5,-10} ● X(E,F) = 7.4 C(A,B) = 0.25

5. (5) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Template matching – X-correlation ● Lingo alert:  Cross-correlation is usually called Covariance  Normalized cross-correlation is usually called Correlation:

6. (6) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Template matching – X-correlation ● Take an image (I) ● Take a target area to be matched (T) ● For each point in the image (I)  Find the normalized cross-correlation:  Where: and ● Template matching with Cross-correlation

7. (7) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Template matching – X-correlation ● Pick a pixel to be matched (u,v) ● For each (x,y) in the template ● Multiply its value (minus the template average) with the equivalent value in the image (minus the image patch average). Add to running total. ● Then divide the total by the square root of the product of the variances of the image and template. ● How to calculate the cross-correlation function C = 0.86 here

8. (8) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Template matching – X-correlation ● Template matching with Cross-correlation  Q: It looks similar to convolution. How does this relate to convolution?  A: It is indeed very similar. What you learnt as convolution is actually almost the cross-correlation that you learnt here (you just don’t divide by N). But this that is yet another definition of cross-correlation, from signal detection theory…  …so how would you do cross-correlation in matlab? Matlab: help conv2 help corr2 The cross-correlation function at the point is a measure of the degree of match between template and that point in the image.

9. (9) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Template matching – X-correlation ● Template matching with Cross-correlation  But... ● How well would this template match the image? ● Answer: Undefined! Divide by zero! ● It’s the normalization factor’s fault! (Doesn’t affect non-normalized cross-correlation).

10. (10) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Template matching ● Problems with naïve template matching  And... ● How well would this template match the image? ● Answer: -0.16 ● Not rotationally invariant.

11. (11) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Template matching ● Problems with naïve template matching  And... ● Problem: Rotational invariance. ● Possible solution: Match every possible rotation.

12. (12) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Template matching ● Problems with naïve template matching  And... ● Problem: Rotational invariance. ● Possible solution: Align orientations. Direction of fastest change Direction of slowest change  E.g. by gradient direction

13. (13) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Template matching ● Problems with naïve template matching  Also… ● How well would this template match the image? ● Answer: 0.6 ● Needs to handle scale changes better too. ● Could check multiple scales…

14. (14) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Template matching ● Naïve cross-correlation is not the only way to do template matching. ● More sophisticated methods have to be able to handle  Scale changes  Rotation  Affine transformation  Lighting changes  Etc ● More efficient approaches are needed. E.g.  Using pyramidal (down-sampled) images  Frequency domain filtering  Iterative refinement

15. (15) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Alternatives to template matching ● Template matching can be refined to be made very useful. ● But other approaches exist too. ● These approaches generalize template matching by extracting a description (or descriptor) of the target feature point so that the descriptor can be matched. For example:  Training a classifier from local image information  Extracting local image statistics ● E.g. histogram of spatial distribution of local intensity gradient (as per SIFT)  More?  This improvement can be made at the extraction phase.

16. (16) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Summary ● Criteria for a good feature point detector are:  Repeatably detectable under transforms: ● Scale ● Rotation ● Affine transformation ● Illumination ● Etc  Computationally efficient  Accurate at localisation  High detection rate  Robust to noise

17. (17) Feature-point matching by D.J.Duff for CompVis 2007.11.15 Online: http://www.cs.bham.ac.uk/~djd/teaching.php Summary ● Criteria for a good feature point matcher are:  Consistent  Invariant to: ● Scale ● Rotation ● Illumination ● Affine transformation ● Etc  Robust to noise  Accurate