Texture Synthesis by Image Quilting CS766 Class Project Fall 2004 Eric Robinson

Today we will answer these questions: What is texture synthesis? What is texture synthesis? Can it be used in the real world? Can it be used in the real world? What were the earlier approaches to texture synthesis? What were the earlier approaches to texture synthesis? What is image quilting? What is image quilting? How does image quilting perform? How does image quilting perform? What can we say about image quilting? What can we say about image quilting? Where can we go from here? Where can we go from here?

What is texture synthesis? Create large texture image from small one Create large texture image from small one Hopefully capture texture properties Hopefully capture texture properties Given this: Produce something like this:

Can it be used in the real world? Reconstruct world Time consuming Time consuming Computationally expensive Computationally expensive Probably overkill Probably overkill Example task: Image based rendering Sample, synthesize, map Fast Fast Limited input required Limited input required Requires ability to synthesize (and transfer) large amounts of image data Requires ability to synthesize (and transfer) large amounts of image data

What were the earlier approaches? “Statistical property” matching “Statistical property” matching Determine relevant texture properties (histograms, filter responses), synthesize image with matching properties Determine relevant texture properties (histograms, filter responses), synthesize image with matching properties Tiling Tiling Place together random blocks - captures local properties, many boundary errors Place together random blocks - captures local properties, many boundary errors Pixel-at-a-time estimation from neighbors Pixel-at-a-time estimation from neighbors Use conditional probability of pixel values to synthesize new image – requires search of entire input for each output pixel! Use conditional probability of pixel values to synthesize new image – requires search of entire input for each output pixel!

What is Image Quilting? Similar to tiling approach Similar to tiling approach Reduce boundary errors by: Reduce boundary errors by: Find good fit of overlapping tiles and Find good fit of overlapping tiles and Allow ragged edges between tiles to minimize boundary error Allow ragged edges between tiles to minimize boundary error B1B2 Illustration based on that used in Efros & Freeman paper “Image Quilting for Texture Synthesis and Transfer” B1B2B1B2

How does Image Quilting perform ?

What can we say about Image Quilting? Strengths Excellent results on structured patterns Excellent results on structured patterns Relatively fast Relatively fast Intuitive approach Intuitive approachWeaknesses Boundaries (still!) Boundaries (still!) Excessive repetition Excessive repetition Parameter setting Parameter setting

Where can we go from here? Fully-structured textures: Fully-structured textures: Results are quite good Results are quite good Semi-structured textures: Semi-structured textures: Boundary problem Boundary problem Allow rectangular patches Allow rectangular patches Allow variable overlap size Allow variable overlap size Repetition problem Repetition problem Flip or rotate patches Flip or rotate patches

What does patch size affect? 24 pixels 6 pixels 10 pixels 42 pixels 6 pixels 24 pixels

What does overlap size affect? 1 pixel7 pixels21 pixels 8 pixels24 pixels36 pixels