1. Texture Synthesis Tiantian Liu

2. Definition Texture – Texture refers to the properties held and sensations caused by the external surface of objects received through the sense of touch. Texture is sometimes used to describe the feel of non-tactile sensations. Texture can also be termed as a pattern that has been scaled down (especially in case of two dimensional non-tactile textures) where the individual elements that go on to make the pattern not distinguishable. Texture Synthesis

3. Definition Texture – Local Each pixel of a texture image is characterized by a small set of spatially neighboring pixels – Stationary characterization is the same for all pixels Texture Synthesis

4. Definition Texture Synthesis Texture

5. Definition Texture Synthesis Texture Animal Skin StoneWater Hair FabricBricks

6. Definition Texture Synthesis – Texture synthesis is the process of algorithmically constructing a large digital image from a small digital sample image by taking advantage of its structural content. It is object of research to computer graphics and is used in many fields, amongst others digital image editing, 3D computer graphics and post-production of films. Texture Synthesis

7. Definition Texture Synthesis Input Result Synthesis Algorithm

8. Application Surface Texture producing Texture Synthesis

9. Application Image Editing Texture Synthesis

10. Application Image Editing Texture Synthesis

11. Application Flow-controlled 2D texture Texture Synthesis

12. Papers Fast Texture Synthesis using Tree-structured Vector Quantization – Wei, L.-Y., and Levoy, M. 2000. Texture Optimization for Example-based Synthesis – Kwatra V, Essa I, Bobick A, Kwatra N. 2005 Accelerated Parallel Texture Optimization – Huang HD, Tong X, Wang WC. 2007 Texture Synthesis

13. Papers Fast Texture Synthesis using Tree-structured Vector Quantization – Wei, L.-Y., and Levoy, M. 2000. Texture Optimization for Example-based Synthesis – Kwatra V, Essa I, Bobick A, Kwatra N. 2005 Accelerated Parallel Texture Optimization – Huang HD, Tong X, Wang WC. 2007 Texture Synthesis

14. Fast Texture Synthesis Texture Synthesis

15. Fast Texture Synthesis Texture Synthesis

16. Fast Texture Synthesis Texture Synthesis search copy input output Basic idea

17. Fast Texture Synthesis Texture Synthesis Basic idea How large should the neighbor be?

18. Fast Texture Synthesis Texture Synthesis Basic idea What if the neighbor is too large or too small?

19. Fast Texture Synthesis Texture Synthesis Multiple resolutions noise Search Copy input output

20. Fast Texture Synthesis Texture Synthesis Multi-resolution allows the use of smaller neighborhoods 1 level 5  5 3 levels 5  5 1 level 11  11

21. Fast Texture Synthesis Texture Synthesis A major issue with pixel-based algorithms introduced so far – they are very slow For input with N pixels and output with M pixels – O(MN) time complexity – Make it O(cMN) for neighborhood size c A fixed neighborhood allows us to do it faster – O( c log(N) M ) – TSVQ

22. Fast Texture Synthesis Texture Synthesis Acceleration - TSVQ Search per input pixelSearch tree

23. Fast Texture Synthesis Texture Synthesis Pseudo Code

24. Fast Texture Synthesis Texture Synthesis Results Random Oriented RegularSemi-regular

25. Fast Texture Synthesis Texture Synthesis Results

26. Fast Texture Synthesis Texture Synthesis Problems – Possibly get stuck on details. – Still slow.

27. Fast Texture Synthesis Texture Synthesis Problems – Possibly get stuck on details. – Still slow.

28. Papers Fast Texture Synthesis using Tree-structured Vector Quantization – Wei, L.-Y., and Levoy, M. 2000. Texture Optimization for Example-based Synthesis – Kwatra V, Essa I, Bobick A, Kwatra N. 2005 Accelerated Parallel Texture Optimization – Huang HD, Tong X, Wang WC. 2007 Texture Synthesis

29. Texture Optimization Texture Synthesis

30. Texture Optimization Texture Synthesis

31. Texture Optimization Texture Similarity and Synthesis Shape, size, orientation of texture elements similar to source Texture Synthesis

32. Texture Optimization Texture Synthesis Texture Energy – Tricky! Pixel-by-pixel comparison of source and target not possible – Compare texture “elements” Local pixel neighborhoods – Want each target neighborhood to be similar to some source neighborhood

33. Texture Optimization Texture Energy Texture Synthesis X (target frame) Z (source)

34. Texture Optimization Texture Energy Texture Synthesis p (pixel) Z (source) X (target frame)

35. Texture Optimization Texture Energy Texture Synthesis X p (neighborhood) Z (source) X (target frame)

36. Texture Optimization Texture Energy Texture Synthesis X p (neighborhood) X (target frame) Z (source)

37. Texture Optimization Texture Energy Texture Synthesis (nearest neighbor) Z p X p (neighborhood) Z (source) X (target frame)

38. Texture Optimization Texture Energy Texture Synthesis X Z (nearest neighbor) Z p X p (neighborhood) Texture Energy (single neighborhood)

39. Texture Optimization Texture Energy Texture Synthesis X Z

40. Texture Optimization Texture Energy Texture Synthesis X Z  individual neighborhood energy

41. Texture Optimization Texture Synthesis Flow-guided Texture Animation – Flow consistency Perceived motion similar to flow – Texture similarity Shape, size, orientation of texture elements similar to source += TextureFlow FieldOutput

42. Texture Optimization Texture Synthesis Target Flow Flowing Target Source Texture Texture Similarity Flow Consistency Naïve way: Ignore Texture Similarity Warp each frame via flow field X1X1 XnXn Target Frames X0X0 Warp

43. Texture Optimization Texture Synthesis Warp XnXn X0X0 X1X1 Target Frames

44. Texture Optimization Texture Synthesis Simply Warp – Problem: Texture structure not maintained

45. Texture Optimization Texture Synthesis Warp + Correct – Problem: Texture structure not maintained Target Frames XX– Warp W Correct Warped Frame Synthesized Frame

46. Texture Optimization Texture Synthesis Link to original goals X Target Frames X– W Correct Source Flow Consistency Texture Similarity Warp

47. Texture Optimization Texture Synthesis Energy = Flow Energy + Texture Energy X Target Frames X– W Correct Source Flow Consistency Texture Similarity Warp

48. Texture Optimization Texture Synthesis Energy = Flow Energy + Texture Energy Flow Energy X Target Frames X– Warp W

49. Texture Optimization Texture Synthesis Optimize Total Energy of target frame: Initialize: X  W (target frame  warped frame) Iteratively improve target frame (E-M algorithm!)

50. Texture Optimization Texture Synthesis Two variables we need to optimize over

51. Texture Optimization Texture Synthesis Two variables we need to optimize over Target Frame

52. Texture Optimization Texture Synthesis Two variables we need to optimize over Source Neighborhoods

53. Texture Optimization Texture Synthesis Iterative Algorithm – Step 1 – Minimize E( X ) w.r.t {Z p } – Find Nearest Source Neighborhoods

54. Texture Optimization Texture Synthesis Iterative Algorithm – Step 2 – Minimize E( X ) w.r.t X – Set  Solve Linear System

55. Texture Optimization Texture Synthesis Pseudo code

56. Texture Optimization Texture Synthesis Multi-level synthesis – Random Initialization – Multiple Resolution Levels Neighborhood Sizes – Progressively refined output [KEBK05]

57. Texture Optimization Texture Synthesis Multi-level synthesis – Random Initialization – Multiple Resolution Levels Neighborhood Sizes – Progressively refined output [KEBK05]

58. Texture Optimization Texture Synthesis Results [KEBK05]

59. Texture Optimization Texture Synthesis Results [KEBK05]

60. Texture Optimization Texture Synthesis Results [KEBK05]

61. Texture Optimization Texture Synthesis Results [KEBK05]

62. Texture Optimization Texture Synthesis Problems – Linear Solve may cause blurriness Solution: Discrete Optimization – Can get stuck in local minima Several solutions, but not work well – Very very very slow (especially on surfaces) Solution: Exploit parallel hardware Solution: Nearest neighbor queries can be made in parallel

63. Papers Fast Texture Synthesis using Tree-structured Vector Quantization – Wei, L.-Y., and Levoy, M. 2000. Texture Optimization for Example-based Synthesis – Kwatra V, Essa I, Bobick A, Kwatra N. 2005 Accelerated Parallel Texture Optimization – Huang HD, Tong X, Wang WC. 2007 Texture Synthesis

64. Accelerated Parallel Texture Optimization Texture Synthesis

65. Accelerated Parallel Texture Optimization Texture Synthesis GPU-Based Texture Optimization – K-coherence search – Principal Components Analysis(PCA)

66. Accelerated Parallel Texture Optimization Texture Synthesis K-coherence search on GPU

67. Accelerated Parallel Texture Optimization Texture Synthesis K-coherence search on GPU

68. Accelerated Parallel Texture Optimization Texture Synthesis K-coherence search on GPU (pre-computing) For each input pixel, find k others with similar neighborhoods input 1 1A 1B 2 2A 2B k = 2 111 11 111 111 11 111 111 11 111

69. Accelerated Parallel Texture Optimization Texture Synthesis K-coherence search on GPU (searching) inputoutput 123 45 678 1 1A 1B 2 2A 2B 11A1B 22A2B

70. Accelerated Parallel Texture Optimization Texture Synthesis PCA – reducing the dimension mathematically defined as an orthogonal linear transformation that transforms the data to a new coordinate system such that the greatest variance by any projection of the data comes to lie on the first coordinate (called the first principal component), the second greatest variance on the second coordinate, and so on. theoretically the optimum transform for given data in least square terms

71. Accelerated Parallel Texture Optimization Texture Synthesis Pseudo Code

72. Accelerated Parallel Texture Optimization Texture Synthesis GPU implementation details: Divide parallel scheme into 3 fragment programs. – E-step program – M-step program – Up sampling program

73. Accelerated Parallel Texture Optimization Texture Synthesis GPU implementation details: Divide parallel scheme into 3 fragment programs. – E-step program » Input: neighborhood texture Y(storing {z p }}) » Output: pixel texture X(storing x) – M-step program – Up sampling program

74. Accelerated Parallel Texture Optimization Texture Synthesis GPU implementation details: Divide parallel scheme into 3 fragment programs. – E-step program – M-step program » Input: X and related pre-computed data » Output: new neighborhood Y – Up sampling program

75. Accelerated Parallel Texture Optimization Texture Synthesis GPU implementation details: Divide parallel scheme into 3 fragment programs. – E-step program – M-step program – Up sampling program » Needed in multi-level synthesis.

76. Accelerated Parallel Texture Optimization Texture Synthesis Results [HTW 2007]

77. Accelerated Parallel Texture Optimization Texture Synthesis Comparison [HTW 2007] CPU GPU

78. Sum up Texture Synthesis Texture synthesis has many applications: – Surface texture procedure, texture editing, 2D flow controlled texture, spatio-temporal textures, lossy compression etc. Current CPU implementation methods are mature: – Pixel based, patch based, texturing optimization, user controlled, flow controlled etc.

79. Sum up (cont.) Texture Synthesis Time consuming is still a problem of texture synthesis. Parallel algorithms that can be run on GPUs are few.

80. Thank You … Texture Synthesis