Presentation is loading. Please wait.

Presentation is loading. Please wait.

Image and Video Upscaling from Local Self Examples

Similar presentations

Presentation on theme: "Image and Video Upscaling from Local Self Examples"— Presentation transcript:

1 Image and Video Upscaling from Local Self Examples
Gilad Freedman Raanan Fattal Hebrew University of Jerusalem

2 Background and overview Algorithm description Local self similarity
Non-dyadic filter bank Filter design Results Let’s begin

3 Single image upscaling
1. large 2. realistic 3. faithful 4. fast Our work concerns with enlarging the resolution of an image given no additional data beyond the input image pixels. The goal is to produce a larger image with native-resolution looking edges. The output image should be faithful to the input image and the algorithms should be fast.

4 parametric image model
Previous work parametric image model example based Sun et al. 2008 Shan et al 2008 Fattal 2007 Freeman et al. 2002 noisy result Glasner et al. 2009 I will start with a quick review of recent image upscaling works. Of course there are always the linear interpolation methods found in every image editing program, but they produce blurry and low quality enlargements. The more recent methods can be roughly divided into two categories. [click] The first are the parametric methods. In these methods some parametric edge formation model is used to generate the output image. In the second category are the example based methods which fill-in the missing high-frequency content using pairs of example patches taken from an external source. Both approaches have their drawbacks. The parametric methods tend to create “generic” looking edges which contribute to an overall artificial look of the output. The example based methods tend to be noisy and involve time-consuming searching. Recently, Glasner et al combined the ideas of the example based methods with classical multi-frame super-resolution and use examples found from the image itself at various scales. Removing the need for external image database somewhat lowers the search cost. generic looking edges

5 New approach: locale example based
local self similarity small upscaling ratios local self similarity local search speed corner step edge line step edge non smooth shading 1/2 4/5 new non-dyadic filter bank Increase exemplar quality and size maintain search locality novel components: local self similarity non-dyadic filter bank locality too few examples Our work extends the example based framework in two ways. Unlike existing methods we do not search for example patches in large external databases. Also, we do not search the input image at multiple scales. We show that very small regions in a downscaled version of the image is the best alternative. The basis of our approach is the observation that natural images show self similarity not only globally between different features, but even at localized regions where the same feature resembles itself---a property we call Local Self-Similarity. [click] As you can see here, various types of singularities, such as edges, remain invariant at different scales. We exploit this property to extract example patches from very restricted regions within a scaled-down version of the input image. This allows us to considerably reduce the search efforts needed to predict the high-frequencies. Not good – the presentation says: locality -> few examples. This assumption becomes less accurate as the scaling ration grows. To overcome this limitation we use small upscaling ratios such that more examples of a greater relevance are found at the scaled-down image. To achieve this, we derive a new family of non-dyadic filters that allow fast and local computations. Using these filters we are able to increase the exemplar size and quality while maintaining the locality of the search To Summarize our main contributions are: the notion of local self-similarity, and the derivation of Non-Dyadic filters

6 Background and overview Algorithm description Local self similarity
Non-dyadic filter bank Filter design Results Before I describe and justify the local self-similarity and the non-dyadic filter bank in a greater depth, here is an overview of our algorithm.

7 Local self-examples upscaling
frequency content interpolated image original image low pass high pass The algorithm upscales the image in multiple steps, each step upscales the image by a small factor, until the desired size is reached. Each step works as follows: Given the input low-resolution image, which has content across all frequencies, as shown by this green bar, we start by interpolating it using our custom filters, to a higher resolution. The interpolated image miss the high frequency content as we can see in the upper bar [click] Then, we decompose the input image into low- and high-frequency components which we use as the exemplar. Note that the low-passed image has a frequency content that is compatible with the interpolated image.

8 Local self-examples upscaling
For each patch: frequency content Add to interpolated image interpolated image Take corresponding patch Search a local area for best example low pass high pass [click] We then search for each patch in the interpolated image, for its closest match in the low-passed image, and copy the corresponding high-pass values to the higher resolution image.

9 Local self-examples upscaling
Repeat for all patches, to fill the high frequencies frequency content interpolated image low pass high pass We repeat this process to all the high-resolution patches, to get a larger image with full frequency content. This sums the entire the algorithm!

10 Algorithm description Local self similarity Non-dyadic filter bank
Overview Algorithm description Local self similarity Non-dyadic filter bank Filter design Results No we will go into explaining and justifying the algorithm components. And first, let us explain and justify the local self similarity.

11 Local self similarity cropped ≈ downscaled
When we take a small patch of pixels downscale it and then compare it with a cropped version of it, we observe a striking similarity.

12 Local self similarity Patches in original image can matched locally with ones in downscaled version This applies to many types of patches which are abundant in natural images such as step edges, corner edges and other types of singularities as shown here

13 Local examples are enough
query db image local image database full image best matches We have preformed an experiment that compares the different options for examples extraction in order to check the validity of the local self similarity assumption. Given query patches from the image shown here at the center, we search for the closest match at three different sources: [click] An external database shown at the left, which is double the number of pixels Freeman et al use. [click] The entire down-scaled input image [click] only small localized regions within the down-scaled input image. As you can see – the results form search in the whole input image are better than searching the external database, even though it is about 10 times larger. The localized search, in many cases, retrieves the same patches as the entire image case despite doing much less work.

14 Visual assessment – external, exact NN, local
Large external example database external database global search local search Searching the entire image Searching local regions in image Here’s a visual comparison of the results of upscaling an image by some of these strategies. While the image constructed from the external DB shows slight artifacts, upon a close inspection we could not see any difference between the result of the local and global searches. Once again, this is despite the fact that the local search performs a fraction of less than 1% work compared to the global search. [click] By zooming-in, one can observe mild artifacts along the glasses frame in the image produced by the external DB. These artifacts are not seen in the local and global image searches.

15 Comparison of example search methods
Here is a graph summarizing the error versus cost of the different approaches. We see our localized approach outperforms the rest in terms of its cost-effectiveness.

16 Background and overview Algorithm description Local self similarity
Non-dyadic filter bank Filter design Results Having justified the use of local self similarity we will now go over to explain non dyadic filter bank, their need and their use in our algorithm.

17 Need for non-dyadic scalings
large ratios small ratios mixed ratios Here we compare the results we get by exploiting self-similarity when scaling the image in steps of various scaling ratios. [click] When using large scaling ratios the local-self similarity does not held well and noticeable artifacts arise. Small ratios show much better results as the local-sel similarity holds better. A tradeoff where we start by small ratios and then increase them gives the best cost effective choice.

18 full frequency content
Dyadic filters higher half full frequency content 1:2 lower half dyadic filter bank Ideally, regular dyadic scaling by a factor of two splits the spectral content by half.

19 Non-dyadic filter bank
1:2 4:5 higher part full frequency content lower part small scaling ratios better examples non-dyadic filter bank Non-dyadic scaling, which scale the image by arbitrary ratios smaller than 1, split the spectral content by the same amount. When using these smaller ratios the self similarity assumption holds much better.

20 Non-dyadic filters: downscaling
example for the 2:3 ratio: dyadic case: 1. convolve with 2 filters 1. convolve with one filter 2. subsample by 2 2. subsample each by 3 How do we downscale an image using these filters? In the dyadic case, the downscaled image is produced by applying a single smoothing filter followed by subsampling the signal at every second pixels. [click] In non-dyadic 2:3 scaling ratio, we have two smoothing filters shown here And we subsample each one of them every third pixel to get a total of two-thirds the number of pixels.

21 Non-dyadic filters: upscaling
example for the 2:3 ratio: dyadic case: 1. zero upsample by 1 2. convolve with 1 filter 1. zero upsample by 2 The analog takes place in the upsampling transformation: Every other value is upsampled by a factor of three-by inserting two zeros between every pixel, [click] Followed by a two separate and different filtering steps. Finally, the two are summed to a single upscaled image. 2. convolve with 2 filters 3. sum

22 Use of the filters in upscaling
Upscaling using inverse scaling filters Smoothing by downscaling and upscaling These non dyadic filters come to use in our algorithm in the two first steps. [click] First, the initial interpolation is done using the inverse scaling filters, as we just explained in the last slide. Second the exemplar is created by smoothing the input which is done by downscaling and upscaling it again using our filters.

23 Background and overview Algorithm description Local self similarity
Non-dyadic filter bank Filter design Results There are several filter design principles that guide our construction of non-dyadic filters and are needed for image upscaling.

24 1. Uniform stretch When interpolating, smooth areas come from input Uniformly spaced grids should remain uniform 255 brightness Our first requirement is to achieve a uniform scaling. This can be measured on a linear ramp image that equates grid coordinates with brightness. We want the linear ramp to remain a linear ramp after scaling the image. In the paper we show that is analogous to requiring vanishing moments in wavelet construction. This property allows us to keep constant or very smooth regions in the image to remain that way in the upscaled output. grid coordinates

25 The interpolated image, if downscaled should be equal to the input.
2. Consistency The interpolated image, if downscaled should be equal to the input. Formally, ↓ ↑ 𝑰 =𝑰 upsample downsample Previous methods achieve consistency by solving large linear systems to achieve this property A second requirement is consistency between the input and the output images. More specifically, we want that the upscaled image [click] After being downscaled to the input resolution, will result in the input image. In the paper we show that this requirement is a biorthogonality constrain on the filters. Note that previous methods either did not achieve this property or achieved it by solving a large system of linear equations

26 3. PSF modeling Difference between point spread functions
Large image - small camera point spread function Small image - large camera point spread function Picture acquisition is modeled by blurring an subsampling. Images at different zoom levels differ by the size of their blur kernel. [click] This difference in blur levels is must be accounted for when modeling the relation between the input and the upscaled images. Therefore, the downscaling filters we design must be low-pass filters.

27 4. Low frequency span frequency
When upsampling don’t add new frequencies Upsampling filter should be low-pass Similar demand is posed on the upscaling filters. When we uspcale an image we do not want to add new high frequencies, And thus the upscaling filter should also be low-pass original interpolated

28 5. Singularities preservation
similar amount of blur blurred Image interpolated image Finally, we want the initial interpolated image that we compute and the example smoothed image used for finding example patches to have similar looking edges. We do this by optimizing the response of both downsamping and upsampling filters to be the same on step edges.

29 Real time video upsampling on GPU
main GPU memory Search and filter-banks are both local operations GPU cores NTSC to full 24 fps The localized search we described as well as the filtering we use, both operate locally on the image. This makes our algorithm trivially parallel In fact we implemented it on a GPU by dividing the input image into sub-images and upscaled each part individually. All the required data in? for each thread is kept in the core local memory or cached, minimizing the need to access the slow global memory. [click] This allowed us to achieve 24fps for upscaling NTSC frames to full HD on an Nvidia Geforce 480 while displaying the result using OpenGL. We did not made any change in our algorithm when upscaling video sequence. Temporal consistency is a result of consistency with the input and the consistency between the output and input frames. Since the input is time-coherent sequence of frame, and our upscales images are consistent with the input, then we get temporal consistency between frame even if we process them individually.

30 Background and overview Algorithm description Local self similarity
Non-dyadic filter bank Filter design Results And now for some results

31 Bicubic x3 (zoomed in) Ours X3 (zoomed in)
Here is an image upscaled by a factor of thee using bicubic interpolation and our method. Note that all the edges are much better reconstructed. Ours X3 (zoomed in)

32 Here is another example
Bicubic x3 Ours X3

33 Genuine Fractals™ x4 Ours X4
Here is a comparison with the most popular commercial upscaling product, called genuine fractals. Genuine Fractals™ x4 Ours X4

34 Here we compare our method with the recent method of Glasner et al
Here we compare our method with the recent method of Glasner et al. While their method achieve sharper edges it also introduces more ringing and staircasing artifacts. Glasner et al x4 Ours X4

35 Finally here is some video footage upsampled by a factor of 2
Finally here is some video footage upsampled by a factor of 2.6 to this projector full 1280 resolution – from common smartphone portrait to HD. This upsampling can be done in real time on GPU.

36 And another one.

37 Thank you! Thank you. Paper is available on my web page, Paper & additional results can be found at:

Download ppt "Image and Video Upscaling from Local Self Examples"

Similar presentations

Ads by Google