1. Image and Video Upscaling from Local Self Examples
Gilad Freedman
Raanan Fattal
Hebrew University of Jerusalem

2. Background and overview Algorithm description Local self similarity
Non-dyadic filter bank
Filter design
Results
Let’s begin

3. Single image upscaling
1. large
2. realistic
3. faithful
4. fast
Our work concerns with enlarging the resolution of an image given no additional data beyond the input image pixels. The goal is to produce a larger image with native-resolution looking edges. The output image should be faithful to the input image and the algorithms should be fast.

4. parametric image model
Previous work
parametric image model
example based
Sun et al. 2008
Shan et al 2008
Fattal 2007
Freeman et al. 2002
noisy
result
Glasner et al. 2009
I will start with a quick review of recent image upscaling works.
Of course there are always the linear interpolation methods found in every image editing program, but they produce blurry and low quality enlargements.
The more recent methods can be roughly divided into two categories.
[click]
The first are the parametric methods. In these methods some parametric edge formation model is used to generate the output image.
In the second category are the example based methods which fill-in the missing high-frequency content using pairs of example patches taken from an external source.
Both approaches have their drawbacks. The parametric methods tend to create “generic” looking edges which contribute to an overall artificial look of the output. The example based methods tend to be noisy and involve time-consuming searching.
Recently, Glasner et al combined the ideas of the example based methods with classical multi-frame super-resolution and use examples found from the image itself at various scales. Removing the need for external image database somewhat lowers the search cost.
generic looking edges

5. New approach: locale example based
local self similarity
small upscaling ratios
local self similarity
local search
speed
corner step edge
line step edge
non smooth shading
1/2
4/5
new non-dyadic filter bank
Increase exemplar quality and size
maintain search locality
novel components:
local self similarity
non-dyadic filter bank
locality
too few examples
Our work extends the example based framework in two ways. Unlike existing methods we do not search for example patches in large external databases. Also, we do not search the input image at multiple scales. We show that very small regions in a downscaled version of the image is the best alternative.
The basis of our approach is the observation that natural images show self similarity not only globally between different features, but even at localized regions where the same feature resembles itself---a property we call Local Self-Similarity.
[click]
As you can see here, various types of singularities, such as edges, remain invariant at different scales.
We exploit this property to extract example patches from very restricted regions within a scaled-down version of the input image.
This allows us to considerably reduce the search efforts needed to predict the high-frequencies.
Not good – the presentation says: locality -> few examples.
This assumption becomes less accurate as the scaling ration grows.
To overcome this limitation we use small upscaling ratios such that more examples of a greater relevance are found at the scaled-down image.
To achieve this, we derive a new family of non-dyadic filters that allow fast and local computations.
Using these filters we are able to increase the exemplar size and quality while maintaining the locality of the search
To Summarize our main contributions are: the notion of local self-similarity, and the derivation of Non-Dyadic filters

6. Background and overview Algorithm description Local self similarity
Non-dyadic filter bank
Filter design
Results
Before I describe and justify the local self-similarity and the non-dyadic filter bank in a greater depth, here is an overview of our algorithm.

7. Local self-examples upscaling
frequency
content
interpolated image
original image
low pass
high pass
The algorithm upscales the image in multiple steps, each step upscales the image by a small factor, until the desired size is reached. Each step works as follows:
Given the input low-resolution image, which has content across all frequencies, as shown by this green bar, we start by interpolating it using our custom filters, to a higher resolution. The interpolated image miss the high frequency content as we can see in the upper bar
[click]
Then, we decompose the input image into low- and high-frequency components which we use as the exemplar. Note that the low-passed image has a frequency content that is compatible with the interpolated image.

8. Local self-examples upscaling
For each patch:
frequency
content
Add to interpolated image
interpolated image
Take corresponding patch
Search a local area for best example
low pass
high pass
[click] We then search for each patch in the interpolated image, for its closest match in the low-passed image, and copy the corresponding high-pass values to the higher resolution image.

9. Local self-examples upscaling
Repeat for all patches, to fill the high frequencies
frequency
content
interpolated image
low pass
high pass
We repeat this process to all the high-resolution patches, to get a larger image with full frequency content.
This sums the entire the algorithm!

10. Algorithm description Local self similarity Non-dyadic filter bank
Overview
Algorithm description
Local self similarity
Non-dyadic filter bank
Filter design
Results
No we will go into explaining and justifying the algorithm components.
And first, let us explain and justify the local self similarity.

11. Local self similarity cropped ≈ downscaled
When we take a small patch of pixels downscale it and then compare it with a cropped version of it, we observe a striking similarity.

12. Local self similarity
Patches in original image can matched locally with ones in downscaled version
This applies to many types of patches which are abundant in natural images such as step edges, corner edges and other types of singularities as shown here

13. Local examples are enough
query db image local
image database
full image
best matches
We have preformed an experiment that compares the different options for examples extraction in order to check the validity of the local self similarity assumption.
Given query patches from the image shown here at the center, we search for the closest match at three different sources:
[click] An external database shown at the left, which is double the number of pixels Freeman et al use.
[click] The entire down-scaled input image
[click] only small localized regions within the down-scaled input image.
As you can see – the results form search in the whole input image are better than searching the external database, even though it is about 10 times larger.
The localized search, in many cases, retrieves the same patches as the entire image case despite doing much less work.

14. Visual assessment – external, exact NN, local
Large external
example database
external database
global search
local search
Searching the
entire image
Searching local
regions in image
Here’s a visual comparison of the results of upscaling an image by some of these strategies.
While the image constructed from the external DB shows slight artifacts, upon a close inspection we could not see any difference between the result of the local and global searches. Once again, this is despite the fact that the local search performs a fraction of less than 1% work compared to the global search.
[click]
By zooming-in, one can observe mild artifacts along the glasses frame in the image produced by the external DB. These artifacts are not seen in the local and global image searches.

15. Comparison of example search methods
Here is a graph summarizing the error versus cost of the different approaches. We see our localized approach outperforms the rest in terms of its cost-effectiveness.

16. Background and overview Algorithm description Local self similarity
Non-dyadic filter bank
Filter design
Results
Having justified the use of local self similarity we will now go over to explain non dyadic filter bank, their need and their use in our algorithm.

17. Need for non-dyadic scalings
large ratios
small ratios
mixed ratios
Here we compare the results we get by exploiting self-similarity when scaling the image in steps of various scaling ratios.
[click]
When using large scaling ratios the local-self similarity does not held well and noticeable artifacts arise.
Small ratios show much better results as the local-sel similarity holds better.
A tradeoff where we start by small ratios and then increase them gives the best cost effective choice.

18. full frequency content
Dyadic filters
higher half
full frequency content
1:2
lower half
dyadic filter bank
Ideally, regular dyadic scaling by a factor of two splits the spectral content by half.

19. Non-dyadic filter bank
1:2
4:5
higher part
full frequency content
lower part
small scaling ratios
better examples
non-dyadic filter bank
Non-dyadic scaling, which scale the image by arbitrary ratios smaller than 1, split the spectral content by the same amount.
When using these smaller ratios the self similarity assumption holds much better.

20. Non-dyadic filters: downscaling
example for the 2:3 ratio:
dyadic case:
1. convolve with 2 filters
1. convolve with one filter
2. subsample by 2
2. subsample each by 3
How do we downscale an image using these filters?
In the dyadic case, the downscaled image is produced by applying a single smoothing filter followed by subsampling the signal at every second pixels.
[click]
In non-dyadic 2:3 scaling ratio,
we have two smoothing filters shown here
And we subsample each one of them every third pixel to get a total of two-thirds the number of pixels.

21. Non-dyadic filters: upscaling
example for the 2:3 ratio:
dyadic case:
1. zero upsample by 1
2. convolve with 1 filter
1. zero upsample by 2
The analog takes place in the upsampling transformation:
Every other value is upsampled by a factor of three-by inserting two zeros between every pixel,
[click]
Followed by a two separate and different filtering steps.
Finally, the two are summed to a single upscaled image.
2. convolve with 2 filters
3. sum

22. Use of the filters in upscaling
Upscaling using inverse scaling filters
Smoothing by downscaling and upscaling
These non dyadic filters come to use in our algorithm in the two first steps.
[click]
First, the initial interpolation is done using the inverse scaling filters, as we just explained in the last slide.
Second the exemplar is created by smoothing the input which is done by downscaling and upscaling it again using our filters.

23. Background and overview Algorithm description Local self similarity
Non-dyadic filter bank
Filter design
Results
There are several filter design principles that guide our construction of non-dyadic filters and are needed for image upscaling.

24. 1. Uniform stretch
When interpolating, smooth areas come from input Uniformly spaced grids should remain uniform
255
brightness
Our first requirement is to achieve a uniform scaling. This can be measured on a linear ramp image that equates grid coordinates with brightness.
We want the linear ramp to remain a linear ramp after scaling the image. In the paper we show that is analogous to requiring vanishing moments in wavelet construction.
This property allows us to keep constant or very smooth regions in the image to remain that way in the upscaled output.
grid coordinates

25. The interpolated image, if downscaled should be equal to the input.
2. Consistency
The interpolated image, if downscaled should be equal to the input.
Formally,
↓ ↑ 𝑰 =𝑰
upsample
downsample
Previous methods achieve consistency by solving large linear systems to achieve this property
A second requirement is consistency between the input and the output images.
More specifically, we want that the upscaled image
[click]
After being downscaled to the input resolution, will result in the input image.
In the paper we show that this requirement is a biorthogonality constrain on the filters.
Note that previous methods either did not achieve this property or achieved it by solving a large system of linear equations

26. 3. PSF modeling Difference between point spread functions
Large image - small camera point spread function
Small image - large camera point spread function
Picture acquisition is modeled by blurring an subsampling. Images at different zoom levels differ by the size of their blur kernel.
[click]
This difference in blur levels is must be accounted for when modeling the relation between the input and the upscaled images.
Therefore, the downscaling filters we design must be low-pass filters.

27. 4. Low frequency span frequency
When upsampling don’t add new frequencies
Upsampling filter should be low-pass
Similar demand is posed on the upscaling filters.
When we uspcale an image we do not want to add new high frequencies,
And thus the upscaling filter should also be low-pass
original
interpolated

28. 5. Singularities preservation≈
similar
amount of blur
blurred Image
interpolated image
Finally, we want the initial interpolated image that we compute and the example smoothed image used for finding example patches to have similar looking edges.
We do this by optimizing the response of both downsamping and upsampling filters to be the same on step edges.

29. Real time video upsampling on GPU
main GPU memory
Search and filter-banks
are both local operations
GPU cores
NTSC to full 24 fps
The localized search we described as well as the filtering we use, both operate locally on the image. This makes our algorithm trivially parallel
In fact we implemented it on a GPU by dividing the input image into sub-images and upscaled each part individually.
All the required data in? for each thread is kept in the core local memory or cached, minimizing the need to access the slow global memory.
[click]
This allowed us to achieve 24fps for upscaling NTSC frames to full HD on an Nvidia Geforce 480 while displaying the result using OpenGL.
We did not made any change in our algorithm when upscaling video sequence. Temporal consistency is a result of consistency with the input and the consistency between the output and input frames.
Since the input is time-coherent sequence of frame, and our upscales images are consistent with the input, then we get temporal consistency between frame even if we process them individually.

30. Background and overview Algorithm description Local self similarity
Non-dyadic filter bank
Filter design
Results
And now for some results

31. Bicubic x3 (zoomed in) Ours X3 (zoomed in)
Here is an image upscaled by a factor of thee using bicubic interpolation and our method. Note that all the edges are much better reconstructed.
Ours X3
(zoomed in)

32. Here is another example
Bicubic x3
Ours X3

33. Genuine Fractals™ x4 Ours X4
Here is a comparison with the most popular commercial upscaling product, called genuine fractals.
Genuine Fractals™ x4
Ours X4

34. Here we compare our method with the recent method of Glasner et al
Here we compare our method with the recent method of Glasner et al. While their method achieve sharper edges it also introduces more ringing and staircasing artifacts.
Glasner et al x4
Ours X4

35. Finally here is some video footage upsampled by a factor of 2
Finally here is some video footage upsampled by a factor of 2.6 to this projector full 1280 resolution – from common smartphone portrait to HD.
This upsampling can be done in real time on GPU.

36. And another one.

37. Thank you!
Thank you.
Paper is available on my web page,
Paper & additional results can be found at: