2D Texture Synthesis Instructor: Yizhou Yu

Texture synthesis Goal: increase texture resolution yet keep local texture variation

Synthesis by global statistics Idea: Obtain statistics of input image, match with output image Histograms: normalized graph of intensity frequencies

Synthesis by global statistics Consider extreme case: Histogram matching can generate two images:

Using higher-order global stats Histogram matching does not take into account spatial information. Need additional info: For every pixel, examine local neighborhood Gives local but not global features (e.g. – veins) Use higher order stats: Correlation average of the product of the intensities = tells how interdependent pixel values are

General procedure 1.) Define error metric: What do you want to match? 2.) Match statistics Histogram matching: Get the density value of a pixel in the output image. Map this density to a pixel intensity in the input image  Overwrite the pixel in the output image with the mapped pixel intensity in the input image

Pyramid-based Texture Analysis/Synthesis Paper by David J. Heeger and James R. Bergen from SIGGRAPH rints/heeger-siggraph95.pdf

Pyramid-based Synthesis Downsample image several times and keep track of only differences in a feature image To reconstruct, upsample small images and add differences Pixels in feature images are close to zero Provides multiple scales with each feature image representing different feature sizes

Laplacian Pyramid 1-D filter: (1/16) ( ) To generalize to 2D, use tensor product Normalize by dividing by 256

Upsampling To upsample, copy pixels to every other pixel in larger image Fill the rest with zeroes, run low pass filter to fill in values

Oriented Filters Gaussian filters are symmetric, thus edges and contours are not detected Use multiple oriented filters to catch non-symmetric features Left top series: Oriented filtersRight image: Texture Left bottom series: Filtered textures

Pyramids with Oriented Filters Each oriented filter creates its own feature image Thus, for each downsampled image, keep one image for each oriented filter Each oriented filter captures one orientation of lines

Matching Image Features Input parameters: noise: initial noisy texture texture: texture to be matched Output texture is stored in noise

Matching Image Features Helper functions: MatchHistogram(noise, texture) Matches histogram, using method described last time MakePyramid(texture) Create pyramid images (base and feature images) CollapsePyramid(pyramid) Constructs high resolution image from base and feature images

Matching Image Features MatchTexture(noise, texture) { MatchHistogram(noise, texture) // first-order matching analysis_pyr = MakePyramid(texture) // create pyramid from input texture for several iterations synthesis_pyr = MakePyramid(noise) // create pyramid from noise for each feature image, f i of analysis_pyr for each feature image, f j of synthesis_pyr of same orientation MatchHistogram(f i, f j ) end for noise = CollapsePyramid(synthesis_pyr) MatchHistogram(noise, texture) end for }

Matching Image Features MatchTexture matches histograms of feature images, not pixel values, thus providing a much better matching Feature images already consider local neighborhoods, resulting in better approximations Good for randomized textures Textures with large scale features or thin and long features not matched well e.g. stripes in wood, lines in coral

Texture Synthesis Using Local Neighborhoods Main goal is to keep local spatial coherence, but not global stats. Randomized method Pick pixel and copy it along with its neighborhood to random parts in synthesized image If two neighborhoods overlap, just blend This can result in features getting cut off if larger than local neighborhood

Neighborhood-Based Texture Synthesis Patch-Based Patch-based sampling achieves real-time speed. [Liang et. al. 2001] Image quilting: high-quality results and simple implementation [Efros & Freeman 2001] Graph cut provides a powerful and refinable scheme. [Kwantra et. al. 2003] Pixel-Based [Efros & Leung 99], [Wei & Levoy 2000], [Ashikhmin 2001], [Hertzmann et. al. 2001], [Zhang et. al. 2003]

Pixel-wise Synthesis Grow pixel by pixel Start from an existing patch of the texture (as opposed to noise texture like Pyramid-based Synthesis) Look for regions in input texture most similar to current region in new texture Copy pixels next to best-match region to expand new texture

Following Raster Order Blue region: already set by algorithm Green region: compare this region to input image Yellow region: closest match Red region: replace these pixels with magenta region to maintain local integrity

Hierarchical Synthesis Build a multi-resolution pyramid for the example texture Generate a synthesized pyramid for the output texture At each level, follow the raster order

Randomizing Synthesis Instead of picking pixels from closest matching region, use threshold to pick a few candidate matches Randomly pick one of the candidates Refer to “Texture Synthesis by Non- parametric Sampling” h/synthesis.html

Results

Patch-Based Synthesis Search in a sample texture for neighborhoods most similar to a context region Merge a patch with the partially synthesized output texture

The Seam Problem Feature discontinuities may appear in patch-based synthesis.

The Second Reason Didn’t find the smoothest transition between the incoming patch and context region. Solutions: use dynamic programming [Efros and Freeman 2001] or graph cut [Kwatra et al. 2003] to find an optimal cut. OriginalWarped

The First Reason Rigid template matching (SSD) often employed in the first step does NOT consider geometric similarity OriginalWarped

Feature Map Guided Texture Synthesis Basic Steps [Wu and Yu 2004] Maintain an input and output feature map Match and align curvilinear features Integrate feature maps into template-based texture synthesis

Comparisons SampleFeature MapGraphcutQuiltingTexton Mask

Acceleration Schemes Fourier Transform Acceleration techniques for nearest-neighbor search Tree-structured vector quantization Kd-trees Minimizing the candidate set Coherent synthesis, [Ashikhmin 2001] Precomputing candidate set K-coherent search, [Tong et al. 2002] Jump Map, [Zelinka and Garland 2002] GPU with parallel synthesis [Lefebvre and Hoppe 2005] OriginalWarped