1. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Non-maximum Suppression Using fewer than Two Comparisons per Pixel Tuan Pham Canon Information Systems Research Australia Paper 207 Session: Algorithms and Optimisations 3:10-3:35 PM Wed 15 Dec ACIVS 2010 Advanced Concepts for Intelligent Vision Systems Sydney, Australia, December 2010 Photo: Sally Mayman, Courtesy of Tourism New South Wales

2. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Established in 1990, now has around 300 staff 1.6km from Macquarie University (20-minute walk) Who is CiSRA? You are here We are here to Sydney (15km) Copyright CISRA Slide 2 of 20

3. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Talk outline 1. Motivation Harris corners SIFT keypoints 2. Non-maximum suppression algorithms Straightforward Our improvements: Spiral scan (1987) → Selective spiral scan Block partitioning (2006) → Quarter-block partitioning 3. Evaluation 4. Bonus application Copyright CISRA Slide 3 of 20

4. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Motivation: fast corner detection 1. Cornerness measure Copyright CISRA Slide 4 of 20

5. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Motivation: fast corner detection 2. Peak detection 1. Cornerness measure Copyright CISRA Slide 5 of 20

6. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Motivation: fast corner detection 3. Cornerness threshold 1. Cornerness measure 2. Peak detection Copyright CISRA Slide 6 of 20

7. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Non-maximum suppression in SIFT* 1. Gaussian pyramid 2. Difference of Gaussian (DoG) 3. DoG extrema detection4. SIFT vector formation5. SIFT keypoints … * D.G. Lowe, Distinctive image features from scale-invariant keypoints, IJCV, 60(2): 91-110, 2004. Copyright CISRA Slide 7 of 20

8. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Straightforward method To detect a local maximum over a (2n+1)x(2n+1) neighbourhood Visit each pixel in a raster scan order For each pixel, visit neighbours also in a raster scan order Stop if a larger neighbour is found Advantage Simple Comparisons per pixel O(n 2 ) Copyright CISRA Slide 8 of 20

9. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Spiral scan (F ӧ rstner & Gülch, 1987) To detect a local maximum over a (2n+1)x(2n+1) neighbourhood: Visit each pixel in a raster scan order For each pixel, visit neighbours also in a spiral order Stop if a larger neighbour is found Each candidate must be a local 3x3 maximum before being tested for a 5x5 maximum, and so on… Comparisons per pixel O(n 2 ) but is actually much faster W. F ӧ rstner and E. Gülch, A fast operator for detection and precise locations of distinct points, corners, and centres of circular features, in Proceedings of ISPRS Intercomission workshop, 1987. Copyright CISRA Slide 9 of 20

10. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Our improvement: selective spiral scan The spiral scan algorithm is still inefficient It performs the spiral test for every pixel Solution: Detect 1D maxima along row Perform spiral test on these 1D maxima only 40% faster than F ӧ rstner’s method Extendable to nD Copyright CISRA Slide 10 of 20

11. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Block partitioning (Neubeck & Van Gool, 2006) A (2n+1)×(2n+1) local maximum is a maximum of every n×n neighbourhood around it → efficient algorithm: Partition image into (n+1)×(n+1) blocks Detect the maximum of each block Compare this candidate with its full neighborhood Advantages Prune the number of candidates by a factor of (n+1) 2 → ~ 2.4 comparisons per pixel Disadvantages Most candidates are not local maximum (2n+1) 2 – (n+1) 2 comparisons per candidate Partitioned block (n=3) A. Neubeck and L. Van Gool, Efficient non-maximum suppression, in Proceedings of ICPR, 2006. Copyright CISRA Slide 11 of 20

12. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Disadvantage of Neubeck’s algorithm Block maxima on a slope are not local maxima → lots of comparisons will be wasted on these candidates A. Neubeck and L. Van Gool, Efficient non-maximum suppression, in Proceedings of ICPR, 2006. Copyright CISRA Slide 12 of 20 n=3 3x3 partition blocks Maximum of each block Local maximum over 5x5 neighbourhood

13. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Partition image into blocks of ¼ size of neighborhood Construct an image of block maxima Find local 3x3 maxima of this image → candidates Compare each candidate with its (2n+1) 2 neighborhood Our improvement: quarter block partitioning Copyright CISRA Slide 13 of 20

14. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Both use 1 comparison per pixel (cpp) to find candidates Our algorithm has significantly fewer candidates because they are guaranteed to be local (2m+1)x(2m+1) maxima We requires fewer comparisons per candidate: (2n+1) 2 – (3m) 2 for us versus (2n+1) 2 – (n+1) 2 for Neubeck Overall, we require 60% fewer comparisons (1.5 vs 2.4 cpp) Comparison with Neubeck’s algorithm Neubeck’s method (n=3) Our quarter-block partitioning method (n=3,m=2) Copyright CISRA Slide 14 of 20

15. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Test images Cameraman 256x256 Lena 512x512 Pentagon 1024x1024 Worst-case for straightforward Best-case for straightforward random 256x256 Copyright CISRA Slide 15 of 20

16. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Performance of 3x3 non-maximum suppression Our spiral scan algorithm (Matlab) is 5-time faster than Mathworks’s imregionalmax function (C MEX file, morphology algorithm) Copyright CISRA Slide 16 of 20

17. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Performance of (2n+1)x(2n+1) non-maximum suppression Our spiral scan-line method is best for small neighbourhood (n<5) 60% faster than [F ӧ rstner, 1987] Our quarter-block partitioning method is best for large neighbourhood 40% faster than [Neubeck, 2006] Copyright CISRA Slide 17 of 20

18. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Bonus application: video denoising Copyright CISRA Slide 18 of 20 Use NMS to detect highlight points in video frames Align these points to estimate global shift Average aligned video frames to improve Signal-to-Noise Ratio

19. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Result: denoising from 17-second video Copyright CISRA Slide 19 of 20

20. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Summary Non-maximum suppression is a useful low-level operation Our algorithms requires < 2 comparisons per pixel: Selective spiral scan is 60% faster than [F ӧ rstner, 1987] Quarter-block partition is 40% faster than [Neubeck, 2006] Spiral scan is best for small neighbourhood size (< 11x11) Quarter-block partition is best for larger neighbourhood Fast non-maximum suppression enables new applications Copyright CISRA Slide 20 of 20

21. Australian R&D with global impact T. Pham, Non-maximum suppression using fewer than 2 comparisons per pixel, ACIVS 2010 Thank you tuan.pham@cisra.canon.com.au Copyright CISRA Slide 21 of 20