Graphcut Textures:Image and Video Synthesis Using Graph Cuts Vivek Kwatra ,Arno Schodl ,Irfan Essa , Greg Turk ,Aaron Bobick

Outline Texture Synthesis Graphcut Texture Video Textures What is texture? How to synthesis? Graphcut Texture Main idea, Contribution Patch placement and matching techniques Patch fitting Refinements and extensions Video Textures

Texture Generate a large image from smaller image or longer video from smaller one How to do it? Copy patches (or pixels ) from input to output Problems Artifacts ( e.g., boundaries of patches)

Solution Copy patches from input to output with overlap

Definitions Where to position the input texture called Offset Which part of the input texture to transfer called Seam

M(s,t,A,B) = ||A(s) – B(s)|| + || A(t) – B(t)|| Cutting the graph cut 8 1 4 7 8 cut existing pixels A new patch B new patch corresponding graph old pixels 8 8 2 5 8 8 8 3 6 9 overlap Use Max Flow/ Min Cut algorithm to find the best cut where the cost between two adjacent nodes s and t : M(s,t,A,B) = ||A(s) – B(s)|| + || A(t) – B(t)|| A(s) and B(s) are pixel colors in position s in the old and new patch respectively

Cutting the graph 8 8 8 8 8 8 8 8 old pixels existing pixels 1 4 7 corresponding graph cut new patch 2 5 8 8 3 6 9 8 new patch overlap 8 8 old cut existing pixels 1 5 9 13 cut overlap 2 6 10 14 old pixels new patch 3 7 11 15 corresponding graph new cut 4 8 12 16 8

Seam nodes Problem: What should we do if we lay down a patch in a spot that already has existing seams? How should we take preexisting seams into account when making a cut? old cut new cut old cut new cut 1 4 7 8 8 existing pixels A new patch B 8 8 new patch corresponding graph old pixels 2 5 8 8 8 overlap 3 6 9

Seam nodes Solution: add a seam node between two pixels and connect the seam node with an arc to the new patch. The weight of the arc will be the old cost between the pixels. old cut new cut old cut new cut 1 4 7 8 S1 8 existing pixels A new patch B S2 corresponding graph old pixels 8 8 new patch 2 5 8 S3 8 8 overlap 3 6 9 S4 Then add arcs from pixels to seam node with new costs M(1,2, A, B) M(1,2, B, A) B A 1 S1 4 7 8 8

The Synthesis Process Step1:Patch placement and Matching ( Choose candidate patches or offset) Random Placement Entire Patch Matching Sub-patch Matching Step2:Patch Fitting ( Choose optimal portion or seam) Only those pixels are copied that are chosen by graph-cut algorithm Cost of graph-cut is a measure of similarity

Stemp1. Patch Placement Random Placement Entire input image is translated to random location in the output image Good results for random textures Patch Matching ( Entire or Sub ) Used when we already have some patches in the output image (refinement). Every seam has a cost (min graph cut cost) Uses error region

Sub-Patch Matching

Entire Patch Matching

Error Region Error = seam cost Choose a pixel with largest error Sum of costs along minimum cut patch Choose a pixel with largest error Select a region around that pixel, called error region Patch Matching ( entire or sub ) will select those patches that completely cover our error region

Patch Placement Entire Patch Matching Matching criteria: Search for translated input versions and choose that gives best match Matching criteria: Normalized SSD: C(t): The smaller, the better (means similar)

Patch Placement Compute cost for all possible offsets. Cost is inversely proportional to similarity Choose the cost that has the highest probability of resulting in a similar region A low value of k leads to picking of only those patch locations that have a very good match with the output whereas a high value of k leads to more random patch selection. Good results for structured and semi-structured texture

Patch Placement Sub-Patch Matching Matching criteria: Pick a small sub-patch from output Search for output-patch in input texture or look for translations of input sub-patch Matching criteria: Use the same probability function Best results for unstructured regions or video textures ( fire, waves, smoke, etc.)

Patch Fitting Make graph for overlap region Every pixel is a node Edge weights: Associate weight with each edge Find minimum graph cut

Refinements Modified Matching cost function

Refinements Search across all translations is costly (use FFT) speed up 150x100x30 video sequence Search for a new patch reduces from around 10 minutes (using naïve search) to 5 seconds (using FFT-based search)

Extensions Translation to Transformation Rotation, scaling, affine or projection Interactive merging and blending Many source images User specifies position & constraints pixels Algorithm finds best seam SIGGRAPH banner

Interactive merging and blending

Interactive merging and blending

Extensions Goal is to loop the video forever Video texture One way is to find the pair of similar looking frames and use them to repeat the video Video synthesis using graphcut Find time of transition on pixel-wise basis

Video Texture – GraphCut Finding Seams for Video Transitions Find good transition between pair of images Take a window around transition (60 frames) Construct the graph by connecting a pixel to its neighbors in space and time Min cut will give you time of transition on per pixel basis Use translation in time and space both

Video Synthesis

Video Synthesis Temporal Constraints for Video To loop the video, add k frames in start and end of video (10 frames), constraint these frames to stay the same, graph is generated and best seam is found, then k frames are removed.

Video Synthesis We fix the first k and last k frames of the output sequence to be the same k frames of the input. The pixels in these frames are now constrained to stay the same. This is ensured by adding links of infinite cost between these pixels and the patches they are constrained to copy from, during graph construction Use graphcut to find the best seam given that these pixels don’t change. Once the output has been generated, we remove the first k frames from it. This ensures a video loop since the kth frame of the output is the same as its last frame before this removal operation