1. Overview of Texture Synthesis Ganesh Ramanarayanan Cornell Graphics Seminar

2. The Problem small texture ??? big texture

3. A Simple Solution … small texture ??? big texture

4. ??? big texture A Simple Solution … Tiling small texture

5. A Simple Solution … Tiling Tiling is obvious! big texture? small texture

6. Ideal Solution?

7. The Problem … in Words Given texture I, generate a texture J which – Looks like the same texture – Has no obvious copying or tiling from I – Difference between I and J should be the same as the way I “ differs from itself ” [DeBonet97] Things to watch for: – ‘ Looks the same ’ : what is the texture model? – ‘ Obvious copying ’ : how is it avoided? – Underlined text: indicates algorithm parameter

8. Classes of Algorithms Multiresolution pyramids – [HeegerBergen95] Pixel-by-pixel synthesis – [EfrosLeung99] Multiresolution pixel-by-pixel – [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01], [Ashikhmin01] Patch quilting – [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04] Geometric feature matching – [WuYu04], [Liu et.al. 04]

9. Heeger Bergen 1995 Seminal paper that introduced texture synthesis to the graphics community Algorithm: – Initialize J to noise – Create multiresolution pyramids for I and J – Match the histograms of J ’ s pyramid levels with I ’ s pyramid levels – Loop until convergence – Can be generalized to 3D

10. Heeger Bergen 1995 - Algorithm Image pyramids – Gaussian – Laplacian Steerable pyramids [SimoncelliFreeman95] – b): multiple scales of oriented filters – c): a sample image – d): results of filters in b) applied to c)

11. Heeger Bergen 1995 - Results IJ SuccessesFailures IJ

12. Heeger Bergen 1995 - Results

13. Heeger Bergen 1995 - Verdict Texture model: – Histograms of responses to various filters Avoiding copying: – Inherent in algorithm No user intervention required Captures stochastic textures well Does not capture structure – Lack of inter-scale constraints

14. De Bonet 1997 Propagate constraints downwards by matching statistics all the way up the pyramid Feature vector: multiscale collection of filter responses for a given pixel Algorithm: – Initialize J to empty image – Create multiresolution pyramids for I and J – For each pixel in level of J, randomly choose pixel from corresponding level of I that has similar feature vector

15. De Bonet 1997 - Algorithm 6 feature vectors shown Notice how they share parent information

16. De Bonet 1997 - Results

17. De Bonet 1997 - Verdict Texture model: – Feature vector containing multiscale responses to various filters Avoiding copying: – Random choice of pixels with ‘ close ’ feature vectors, but copying still frequent on small scale Individual per-filter thresholds are cumbersome Feature vectors used in later synthesis work

18. Efros Leung 1999 Markov Random Fields: pixels synthesized probabilistically based on causal neighborhood Algorithm: – Initialize J to empty – Synthesize new pixels in a spiral pattern – Select new pixel from I based on conditional probabilities given the current neighborhood in J and similar neighborhoods in I

19. Efros Leung 1999 - Algorithm IJ ‘ causal ’ neighborhood

20. Efros Leung 1999 - Results

21. Efros Leung 1999 – Verdict Texture model: – MRF Avoiding copying: – MRF Neighborhood size = largest feature size Markov model is surprisingly good – “ I spent an interesting evening recently with a grain of salt. ” Search is very slow with large neighborhoods

22. Wei Levoy 2000 Adds tree-structured vector quantization speedup and multiresolution matching Algorithm: – Initialize J to noise – Synthesize new pixels in scanline order – Select new pixel from I that has the closest matching neighborhood to J – In multiresolution case, look instead at neighborhood of feature vectors

23. Wei Levoy 2000 - Algorithm Scanline synthesis with neighborhoods – a) input texture – b) algorithm start – c) midway point – d) ending

24. Wei Levoy 2000 - Results

25. Wei Levoy 2000 - Verdict Texture model: – Multiscale filters and MRF Avoiding copying: – Not explicitly, but in practice inherently avoided; TSVQ adds additional unpredictability Results comparable to [EfrosLeung99] TSVQ speedup comes at a price (quality loss)

26. Ashikhmin 2001 Adds more verbatim image copying and user- constrained synthesis Algorithm: – New array J* stores location in I of J ’ s pixels – Initialize J to empty, J* to random – Synthesize new pixels in scanline order – Select new pixel by copying from patches of I, as determined by locations in J* – In user-constrained case, also include error comparison with user-specified target image T

27. Ashikhmin 2001 - Algorithm J* x=3 y=4 x=4 y=4 x=12 y=11 x=17 y=16 Candidates: -x=4, y=5 -x=18, y=16 -x=11, y=12 x=18 y=16

28. Ashikhmin 2001 - Results

29. Ashikhmin 2001 - Verdict Texture model: – MRF Avoiding copying: – Actually, here it is encouraged on a small scale, but in practice it doesn ’ t occur on a large scale Recognized that copying preserves fine detail Features can get ‘ cut off ’ because of the abrupt jumping from patch to patch

30. Hertzmann et. al. 2001 Framework for solving image analogies: learning an image filter f by example – Given A, f(A), and B, generate f(B) To handle texture synthesis, ignore A, B, and let f(A) = I, f(B) = J Algorithm: – Bring image histograms into correspondence – Combine [WeiLevoy00] and [Ashikhmin01] Each search method is weighted appropriately and then the best choice is used

31. Hertzmann et. al. 2001 - Algorithm A provides set of constraints Histogram matching critical because of color differences between A and B sets of images

32. Hertzmann et. al. 2001 - Results IJIJ

34. Hertzmann et. al. 2001 - Verdict Texture model: – Multiscale filters and MRF Avoiding copying: – See [WeiLevoy00] and [Ashikhmin01] Unique approach to image manipulation and texture synthesis Same fundamental limitations as [WeiLevoy00] and [Ashikhmin01]

35. Efros Freeman 2001 Inspired by chaos mosaics [Xu et. al. 00] Make an ‘ image quilt ’ by pasting together patches of the input and fixing seams Algorithm: – Initialize J to empty – Copy new patches of certain size in scanline order, with fixed overlap width – Randomly select new patch that has error in overlap region less than given threshold – Copy pixels based on minimal error boundary cut

36. Efros Freeman 2001 - Algorithm Input texture block B1 B2 Neighboring blocks constrained by overlap B1B2 Minimal error boundary cut

37. Efros Freeman 2001 - Results

38. Efros Freeman 2001 - Verdict Texture model: – ?? Avoiding copying: – Randomized patch selection, but still noticeable Patch size is a hard parameter to understand Results are surprisingly good given algorithm Multiscale goes on a brief hiatus

39. Kwatra et. al. 2003 Generalizes seam computation in overlap regions as a graph cut problem – Based on [Boykov et. al. 99] (with Ramin Zabih) Algorithm: – Initialize J to empty – Copy pieces of I to J using a variety of methods – Formulate flow graph in overlap region based on errors and compute minimum cut – Copy sink-side pixels to J – Variety of strategies to further hide seams

40. Kwatra et. al. 2003 - Algorithm (assume cut region is 3x3 for simplicity)

41. Kwatra et. al. 2003 - Results

43. Kwatra et. al. 2003 - Verdict Texture model: – MRF Avoiding copying: – Even with a multitude of patch selection methods, still noticeable when it happens repeatedly Paper presents a bag of synthesis tricks without much intuition for when to use what Graph cut formalization is useful and powerful

44. Wu Yu 2004 Synthesizes a feature image J* in parallel Tries to match features in overlap region Algorithm: – Compute feature image I* – Initialize J* to empty – Synthesize J and J* in parallel – When searching for new patch, add in weighted feature matching error – Prior to copying, warp selected patch to match features exactly

45. Wu Yu 2004 - Algorithm Match feature pixels based on – Tangent orientation – Distance – Bijectivity Example: – L 1 out matches L 1 in – L 2 out matches L 3 in – Warping will move matching features closer to each other Feature correspondence

46. Wu Yu 2004: Results

47. Wu Yu 2004: Verdict Texture model: – Feature lines and MRF Avoiding copying: – No; in fact, mirroring is often very noticeable Unclear how to combine this metric with others Insight to use warping is valuable, although the next paper does a much better job with it

48. Liu et. al. 2004 Regular textures: can be tiled Near-regular textures: statistical distortion of a regular texture Algorithm (simplified): – Mark lattice points in I – Transform I into regular texture R – Express transformation as deformation texture I* – Synthesize larger deformation J* – Synthesize J by combining J* with R

49. Liu et. al. 2004 - Algorithm RI Deformation Texture I*

50. Liu et. al. 2004 - Results

51. original texture transfer

52. Liu et. al. 2004 - Verdict Texture model: – Regular texture with modulated geometry and color Avoiding copying: – Depends on the synthesis algorithm Proposes specific model to handle specific types of textures Interestingly, this method depends on previous texture synthesis methods

53. Classes of Algorithms - Recap Multiresolution pyramids – [HeegerBergen95] Pixel-by-pixel synthesis – [EfrosLeung99] Multiresolution pixel-by-pixel – [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01], [Ashikhmin01] Patch quilting – [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04] Geometric feature matching – [WuYu04], [Liu et.al. 04]

54. Evolution of Texture Model Responses to image filters / decompositions Markov Random Fields Copies of small pieces of image Feature data Various combinations of the above

55. What ’ s Next? Texture synthesis is really hard Rather than creating a silver bullet, focus on algorithms for more specific problems Superresolution: + = ???

56. What ’ s Next? Texture synthesis is really hard Rather than creating a silver bullet, focus on algorithms for more specific problems Superresolution: + = [Hertzmann et. al. 01]