1. Mixed-Resolution Patch- Matching (MRPM) Harshit Sureka and P.J. Narayanan (ECCV 2012) Presentation by Yaniv Romano 1

2. Problem Definition Patch Matching \ Nearest Neighbor Fields: – For every patch in image A, find the K most similar patches in image B. NNF AB 2

3. Motivation – BM3D Denoising [K. Dabov et al, IEEE-TIP 2007] 3

4. Motivation – Super Resolution [M. Protter and M. Elad, IEEE-TIP 2009] 4 - “known” pixel Up-scaling by a factor of 2

5. Motivation – Deinterlace [M. Protter and M. Elad, IEEE-TIP 2009] 5

6. 6

7. Motivation – Image Editing [Barnes et al, SIGRAPH 2009] 7

8. 8

9. 9

10. Challenges SIMPLE & ELEGANT Short RUN-TIME* High ACCURACY Strong IMAGE PRIOR Supports different SIMILARITY MEASURES ? 10 *very-low extra-memory

11. Today’s talk Problem Definition. Motivation. Challenges. Related works. Pyramid Patch-Matching (PPM). Mixed Resolution Pyramid Patch-Matching (MRPM). Experimental Results. Pros & Cons. Future ideas. 11

12. An iterative randomized algorithm. Based on local propagation and mostly-local search. Heavily relies on the fact that images are generally coherent. Image Space - the PatchMatch algorithm [Barnes et al, SIGRAPH 2009] 12 current match of left neighbor left neighbor candidate

13. An iterative randomized algorithm. Based on local propagation and mostly-local search. Heavily relies on the fact that images are generally coherent. Image Space - the PatchMatch algorithm [Barnes et al, SIGRAPH 2009] 13 current match of top neighbor current match of left neighbor candidates left neighbor top neighbor

14. SIMPLE & ELEGANT “Medium” RUN-TIME* “Medium” ACCURACY “Medium” IMAGE PRIOR Supports different SIMILARITY MEASURES 14 *Compared to the state-of-the-art (CSH) Image Space - the PatchMatch algorithm [Barnes et al, SIGRAPH 2009] *very low extra-memory

15. 1.Hash patches of both images into a table. 2.Find matches, based on: – Appearance. – Coherence. Type 1Type 2 Type 3 Appearance Appearance & CoherenceAppearance Image Space & Appearance Space CSH algorithm [Korman et al, ICCV 2011] PM 15

16. Image Space & Appearance Space CSH algorithm [Korman et al, ICCV 2011] Not SIMPLE at all “Medium-Low” RUN-TIME* “Good” ACCURACY “Good” IMAGE PRIOR Doesn’t Support different SIMILARITY MEASURES (easily) 16 *Compared to previous methods. *medium-low extra-memory

17. 17 Pyramid Patch-Matching

18. Down sampling the input image(s) – Gaussian averaging, area averaging, cubic interpolation, and others. Image Pyramid … 18

19. Pyramid Patch-Matching (PPM) 19

20. Pyramid Patch-Matching (PPM) 20

21. Pyramid Patch-Matching (PPM) 21

22. Pyramid Patch-Matching (PPM) 22

23. Pyramid Patch-Matching (PPM) Source Image Target Image

24. Pyramid Patch-Matching (PPM) Find matches 3 Exhaustive Search Upsampling & Expanding Search Upsampling 12 Source Image Target Image

25. Pyramid Patch-Matching (PPM) Exhaustive Search Upsampling & Expanding Search Find matches Upsampling 12 3 4 Source Image Target Image

26. Pyramid Patch-Matching (PPM) *Possible to use other patch-matching and nearest-neighbor algorithms for search.

27. PPM – Pros Very simple. Fast exhaustive search at the coarsest level. Local searches at finer levels. 27

28. PPM – Cons ×Small regions can disappear. ×The vicinity of a missed low resolution patch may not be searched in higher resolutions. ×Lower resolution versions may lead to:  A smooth distance function.  Poor localization of matches. 28

29. Mixed Resolution Pyramid Patch-Matching Upsampling … … ?

30. Mixed Resolution Pyramid Patch-Matching The algorithm is the same as before with the following major difference: 30 ? ? Use Mixed-Resolution Vectors instead of the traditional patch vectors

31. Pyramid Patch-Matching Exhaustive Search Upsampling & Expanding Search Find matches Upsampling 12 3 4

32. Mixed Resolution Pyramid Patch-Matching Exhaustive Search Upsampling & Expanding Search Upsampling 1 4 2 Find matches 3

33. Mixed Resolution Pyramid Patch-Matching Exhaustive Search Upsampling & Expanding Search Upsampling 1 4 2 Find matches 3

34. Search Range vs. Mixed Resolutions Search Range No. of Resolutions Mixed (L) Values: [Avg. Error* (Time)] L=1 (no mixing)L=2L=3 3X3109.47 (13 sec)96.71 (19 sec)93.93 (24 sec) 5X599.92 (22 sec)90.38 (30 sec)87.74 (35 sec) 7X796.81 (30 sec)88.62 (45 sec)86.18 (53 sec) 9X994.8 (41 sec)87.36 (63 sec)85.03 (73 sec) *Error = Average Euclidean distance between source and matched patches Lower error values are achieved with less cost of time by mixing resolutions Mixing resolutions is more effective than increasing the search range. 34

35. GPU Implementation Mixed resolution patch matching processes each pixel independently. – Perfect for the heavy parallel architecture of GPU. 35

36. GPU Implementation Image SizeSearchCPUGPUSpeed-Up 1920X1080 3X313.46 sec278 msec48X 5X520.45 sec684 msec30X 7X729.68 sec1113 msec26X 256X256Exhaustive356.63 sec4.89 sec72X 36

37. Experimental Results Ground Truth - exhaustive search: – K=10 NN. – patch size = 8X8. Error = Avg. Euclidian distance between the patches and their matches. Mixed pixels from two resolution levels. – 3X3 search range. 37

38. How many of the ground truth matches are found? Higher % implies better accuracy: 38 Best K MatchesCSHMRPM (Proposed)%MPRM – %CSH Best Match92.43%93.57%1.14% Best 5 Matches88.06%91.51%3.45% Best 10 Matches81.72%89.87%8.15% Advantages of the MRPM method are more pronounced when several nearest matches are needed

39. Average Error With a similar computation time to CSH they obtain lower average error: 39

40. Image Reconstruction Find a dense patch matching between A & B: – Reconstruct image A using image B. Reconstruction Mean Square Error, lower is better: 40 Ground TruthProposedCSHPatch Match 5.816.476.297.62 The proposed Image Prior is weaker than CSH (coherency)

41. Bottom line Very SIMPLE & ELEGANT “State-of-the- art” RUN-TIME* “State-of- the-art” ACCURACY “Medium- Good” IMAGE PRIOR “Medium” Support different SIMILARITY MEASURES *Compared to CSH *medium extra-memory 41

42. Mixed Resolution Pyramid Patch-Matching Pros: The algorithm: – Faster than the state-of-the-art. Fast exhaustive search at the coarsest level. Local searches at finer levels. – More accurate than the state-of-the-art. Take into consideration the fine details of the patches. – Very simple and easy to implement. – Parallel property, matching is independent. The article: – Self-contained. – Provides code (both CPU and GPU implementations). – The comparisons to previous works are sufficient. – Builds the story very well.

43. Mixed Resolution Pyramid Patch-Matching

44. Future ideas Future idea #1: – Instead of image pyramid… 44

45. Future ideas Future idea #1: – Use the mixed vectors with “Wavelets”. The mixed vectors contain the whole image details compared to image pyramid

46. Future ideas Future idea #1 – reduce data redundancy: – Use the mixed vectors with “Wavelets”. Future idea #2 – strengthen the image prior: – Use image coherency. Exhaustive Search Upsampling & Expanding Search Upsampling 1 4 2 Find matches 3 46

47. 47

48. We Are Done ! Thank You 48