Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Junhwan Kim and Fabio Pellacini Cornell University

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini GoalGoal Allow tile deformation Arbitrarily-shaped container and tiles Allow tile deformation Arbitrarily-shaped container and tiles 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Algorithm Criteria Visually pleasing result Acceptable computational cost Little user effort Algorithm Criteria Visually pleasing result Acceptable computational cost Little user effort GoalGoal 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Motivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusionsMotivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusions Outline 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Motivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusionsMotivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusions Outline 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Giuseppe Arcimboldo Motivation Fruit Face 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Motivation Fruit Face

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Motivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusionsMotivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusions Outline 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Related works 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions Mosaicing algorithms Photomosaics [Silvers,R and Hawley,M] (1997) (1997) Simulated Decorative Mosaics [Hausner] (siggraph2001) (siggraph2001) Escherization [Kaplan and Salesin] (siggraph2000) (siggraph2000) Packing algorithms Translational Polygon Containment and Minimal Enclosure using Mathematical Programming- [Milenkovic] (1999) Translational Polygon Containment and Minimal Enclosure using Mathematical Programming- [Milenkovic] (1999) Mosaicing algorithms Photomosaics [Silvers,R and Hawley,M] (1997) (1997) Simulated Decorative Mosaics [Hausner] (siggraph2001) (siggraph2001) Escherization [Kaplan and Salesin] (siggraph2000) (siggraph2000) Packing algorithms Translational Polygon Containment and Minimal Enclosure using Mathematical Programming- [Milenkovic] (1999) Translational Polygon Containment and Minimal Enclosure using Mathematical Programming- [Milenkovic] (1999)

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Related works

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Motivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusionsMotivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusions Outline 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Preparing Inputs Input=(container image + a set of tiles) of arbitrary shape Input=(container image + a set of tiles) of arbitrary shape Since we require a fairly high number of tiles, we need to be able to extract the shape of the tiles directly from the images themselves. We do so using active contours (Snake). Active contours are also used to extract the container shape.

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Preparing Inputs Following Hausner (Simulated Decorative Mosaics ‘01) approach, we segment the input image to generate a set of disjoint arbitrarily-shaped containers. Since we preserve the edge of each container, the final composite will preserve the important edges on the input image.

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Motivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusionsMotivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusions Outline 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini FrameworkFramework Energy minimization wCECwCECwCECwCEC wCECwCECwCECwCEC 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions Trying 3 rd tile…Trying 4 th tile… Container Container to be filled Container filled Tiles Legend Trying 5 th tile…Trying 2 nd tile… … high gap energy  discard Gap region …high overlap energy  discard Overlap region …high deformation energy  discard Shape mismatch …lowest energy  Accept Accepted tile Trying 1 st tile… Color mismatch …high color energy  discard + w G E G E = + w O E O + w D E D

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Framework (Energy terms evaluation) Framework (Energy terms evaluation) Ec : Which is estimated by taking the average of L^2 difference of the colors of the final image and the input container at random locations on the surface of the container. Ec : Which is estimated by taking the average of L^2 difference of the colors of the final image and the input container at random locations on the surface of the container. E G,E O : Each vertex of a tile is attached with a spring to the nearest edge of the other tiles or the container. If the signed distance d between the vertex and the anchor is positive (a gap between them), we add d^2/2 to E G. On the other hand, if d is negative, we add d^2/2 to Eo. E G,E O : Each vertex of a tile is attached with a spring to the nearest edge of the other tiles or the container. If the signed distance d between the vertex and the anchor is positive (a gap between them), we add d^2/2 to E G. On the other hand, if d is negative, we add d^2/2 to Eo. E D : E D :

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Advantages of Energy minimization Easy to control Easy to extend Photomosaic Photomosaic Simulated Decorative Mosaic Little user effort Advantages of Energy minimization Easy to control Easy to extend Photomosaic Photomosaic Simulated Decorative Mosaic Little user effort FrameworkFramework 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Motivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusionsMotivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusions Outline 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Basic Algorithm Overview 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions a) Initial container image b) Tile contours after tile placement c) Tile contours after tile refinement d) Final Jigsaw Image Mosaic Phase 3: Adjusting images Phase 1: Placing tiles Phase 2: Refining tiles In the third phase, we assemble the final mosaic by placing each tile in its position and warping each image to its final deformed shape using the image warping technique presented in (image warping by RBF,1994, Arad,N)

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini E = w C E C + w G E G + w O E O + w D E D PackingPacking Basic Algorithm Packing TileContainer with placed tile Container for next iteration abc Initial container 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Basic Algorithm Packing ContainerAvailable Tiles Place 1 st tileCannot place next Try again 1 st tile Backtrack Place next Done ab c 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Placing a tile When placing a tile in a container of arbitrary shape, it would be prohibitive to try every possible location. In order to reduce the branching overhead, we try those locations that are most likely to make the container shape easier to fill after updating. Unfortunately this depends on which tile we place. Nevertheless, we can guess how the container would look after we put an “average” tile. A container will be easier to fill if it does not have a protrusion and is as convex as possible.

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Placing a tile Before placing a new tile, we construct a CVD (centroidal voronoi diagram), where each site has an area roughly equal to the average size of tiles. We then select a random site among the ones that have the least number of neighbors, thus making the container as easy as possible to fill.

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini The refinement phase of our approach solves this issue by deforming the tiles obtained from the packing stage, while balancing between maintaining the original tile shape as closely as possible and minimizing the full energy function. E = w C E C + w G E G + w O E O PackingPacking RefinementRefinement Basic Algorithm Refinement + w D E D 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Basic Algorithm Refinement a) Initial contours b) Intermediate contours c) Converged contours 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Motivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusionsMotivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusions Outline 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Algorithm Optimization Algorithm Optimization Placing a tile Branch-bound with Look-ahead Geometric hashing Placing a tile Branch-bound with Look-ahead Geometric hashing 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions V tile( V container) : the number of vertices per tile (container) N tile : the number of tiles in the database N tileInContainer : the number of tiles in the container b is the overhead due to branching in the search tree (backtracking).

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Algorithm Optimization Placing a tile 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Algorithm Optimization Look ahead (Russell and Norvig 1994) Algorithm Optimization Look ahead (Russell and Norvig 1994) Every time we cannot find a suitable tile to fill a container, we need to backtrack to the configuration that has minimal energy so far. To reduce branching overhead, we use a look-ahead technique. When placing a new tile, we penalize tiles that will make it harder to fill the container in the next iteration. To do this we add a term to the energy formulation that takes into account how the container will look after tile placement. The container shape term advocates for containers with a small area and short circumference.

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini E = w C E C + w G E G + w O E O PackingPacking E LA = w A area Look ahead Algorithm Optimization Look ahead (Russell and Norvig 1994) + w LA E LA + (1 - w A ) length 2 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions Where area is the container’s area, length is its boundary length, and W A controls the weight of area in relation to the weight of length.

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Algorithm Optimization Container cleanup Algorithm Optimization Container cleanup After we place a tile in a container, we update the container by subtracting the tile from the container. However, the new container can be very jagged, or even have disjoint regions. If these fragments are shallower than the shallowest tile, we know it can never be filled with any existing tile. In that case, it is safe to separate those fragments and consider them as a gap. This cleanup process reduces the running time by cutting the number of vertices in the container. It also reduces the branching overhead, since it prevents the algorithm from wasting time attempting to fill unpromising fragments of the container.

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Algorithm Optimization Geometric Hashing Algorithm Optimization Geometric Hashing We use geometric hashing to select a few tiles that will suit to a particular position in the container. We then evaluate the energy term for them and pick the best fitting one. Intuitively we use geometric hashing as a pruning technique to reject bad tiles. We create a grid of squares in the plane in a preprocessing phase. Each square corresponds to a hash table entry. If a shape boundary crosses a square, we will record the tile ID and its orientation as an entry in the list attached to that hash table entry.

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Algorithm Optimization Geometric Hashing Algorithm Optimization Geometric Hashing Every time we need to place a new tile in a specific position in the container during the packing stage we register the container boundary segment to the hash table and access the hash table entries of the squares that the container passes through; for every entry found there, we cast a vote for the (tile ID, tile orientation) pair.

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Algorithm Optimization Geometric Hashing [Wolfson and Rigoutsos, 1997] 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Motivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusionsMotivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusions Outline 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini ResultsResults 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini ResultsResults a) Base case b) Lower color weight c) Lower overlap weight d) Tiles of finer scales 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini ResultsResults

ResultsResults

ResultsResults

ResultsResults

ResultsResults

ResultsResults

Motivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusionsMotivation Related Work Preparing Input Framework Basic Mosaic Algorithm Algorithm Optimization ResultsConclusions Outline 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini Contributions New kind of mosaic General framework for mosaics Effective algorithm Future work Faster algorithm Extension to 3D Contributions New kind of mosaic General framework for mosaics Effective algorithm Future work Faster algorithm Extension to 3D ConclusionsConclusions 1.Motivation 2. Related Work 3. Framework 4. Algorithm 5.Results 6.Conclusions

Jigsaw Image Mosaics - Junhwan Kim and Fabio Pellacini 報告完畢