Presentation on theme: "Image and Video Upscaling from Local Self Examples"— Presentation transcript:
1Image and Video Upscaling from Local Self Examples Gilad FreedmanRaanan FattalHebrew University of Jerusalem
2Background and overview Algorithm description Local self similarity Non-dyadic filter bankFilter designResultsLet’s begin
3Single image upscaling 1. large2. realistic3. faithful4. fastOur work concerns with enlarging the resolution of an image given no additional data beyond the input image pixels. The goal is to produce a larger image with native-resolution looking edges. The output image should be faithful to the input image and the algorithms should be fast.
4parametric image model Previous workparametric image modelexample basedSun et al. 2008Shan et al 2008Fattal 2007Freeman et al. 2002noisyresultGlasner et al. 2009I will start with a quick review of recent image upscaling works.Of course there are always the linear interpolation methods found in every image editing program, but they produce blurry and low quality enlargements.The more recent methods can be roughly divided into two categories.[click]The first are the parametric methods. In these methods some parametric edge formation model is used to generate the output image.In the second category are the example based methods which fill-in the missing high-frequency content using pairs of example patches taken from an external source.Both approaches have their drawbacks. The parametric methods tend to create “generic” looking edges which contribute to an overall artificial look of the output. The example based methods tend to be noisy and involve time-consuming searching.Recently, Glasner et al combined the ideas of the example based methods with classical multi-frame super-resolution and use examples found from the image itself at various scales. Removing the need for external image database somewhat lowers the search cost.generic looking edges
5New approach: locale example based local self similaritysmall upscaling ratioslocal self similaritylocal searchspeedcorner step edgeline step edgenon smooth shading1/24/5new non-dyadic filter bankIncrease exemplar quality and sizemaintain search localitynovel components:local self similaritynon-dyadic filter banklocalitytoo few examplesOur work extends the example based framework in two ways. Unlike existing methods we do not search for example patches in large external databases. Also, we do not search the input image at multiple scales. We show that very small regions in a downscaled version of the image is the best alternative.The basis of our approach is the observation that natural images show self similarity not only globally between different features, but even at localized regions where the same feature resembles itself---a property we call Local Self-Similarity.[click]As you can see here, various types of singularities, such as edges, remain invariant at different scales.We exploit this property to extract example patches from very restricted regions within a scaled-down version of the input image.This allows us to considerably reduce the search efforts needed to predict the high-frequencies.Not good – the presentation says: locality -> few examples.This assumption becomes less accurate as the scaling ration grows.To overcome this limitation we use small upscaling ratios such that more examples of a greater relevance are found at the scaled-down image.To achieve this, we derive a new family of non-dyadic filters that allow fast and local computations.Using these filters we are able to increase the exemplar size and quality while maintaining the locality of the searchTo Summarize our main contributions are: the notion of local self-similarity, and the derivation of Non-Dyadic filters
6Background and overview Algorithm description Local self similarity Non-dyadic filter bankFilter designResultsBefore I describe and justify the local self-similarity and the non-dyadic filter bank in a greater depth, here is an overview of our algorithm.
7Local self-examples upscaling frequencycontentinterpolated imageoriginal imagelow passhigh passThe algorithm upscales the image in multiple steps, each step upscales the image by a small factor, until the desired size is reached. Each step works as follows:Given the input low-resolution image, which has content across all frequencies, as shown by this green bar, we start by interpolating it using our custom filters, to a higher resolution. The interpolated image miss the high frequency content as we can see in the upper bar[click]Then, we decompose the input image into low- and high-frequency components which we use as the exemplar. Note that the low-passed image has a frequency content that is compatible with the interpolated image.
8Local self-examples upscaling For each patch:frequencycontentAdd to interpolated imageinterpolated imageTake corresponding patchSearch a local area for best examplelow passhigh pass[click] We then search for each patch in the interpolated image, for its closest match in the low-passed image, and copy the corresponding high-pass values to the higher resolution image.
9Local self-examples upscaling Repeat for all patches, to fill the high frequenciesfrequencycontentinterpolated imagelow passhigh passWe repeat this process to all the high-resolution patches, to get a larger image with full frequency content.This sums the entire the algorithm!
10Algorithm description Local self similarity Non-dyadic filter bank OverviewAlgorithm descriptionLocal self similarityNon-dyadic filter bankFilter designResultsNo we will go into explaining and justifying the algorithm components.And first, let us explain and justify the local self similarity.
11Local self similarity cropped ≈ downscaled When we take a small patch of pixels downscale it and then compare it with a cropped version of it, we observe a striking similarity.
12Local self similarityPatches in original image can matched locally with ones in downscaled versionThis applies to many types of patches which are abundant in natural images such as step edges, corner edges and other types of singularities as shown here
13Local examples are enough query db image localimage databasefull imagebest matchesWe have preformed an experiment that compares the different options for examples extraction in order to check the validity of the local self similarity assumption.Given query patches from the image shown here at the center, we search for the closest match at three different sources:[click] An external database shown at the left, which is double the number of pixels Freeman et al use.[click] The entire down-scaled input image[click] only small localized regions within the down-scaled input image.As you can see – the results form search in the whole input image are better than searching the external database, even though it is about 10 times larger.The localized search, in many cases, retrieves the same patches as the entire image case despite doing much less work.
14Visual assessment – external, exact NN, local Large externalexample databaseexternal databaseglobal searchlocal searchSearching theentire imageSearching localregions in imageHere’s a visual comparison of the results of upscaling an image by some of these strategies.While the image constructed from the external DB shows slight artifacts, upon a close inspection we could not see any difference between the result of the local and global searches. Once again, this is despite the fact that the local search performs a fraction of less than 1% work compared to the global search.[click]By zooming-in, one can observe mild artifacts along the glasses frame in the image produced by the external DB. These artifacts are not seen in the local and global image searches.
15Comparison of example search methods Here is a graph summarizing the error versus cost of the different approaches. We see our localized approach outperforms the rest in terms of its cost-effectiveness.
16Background and overview Algorithm description Local self similarity Non-dyadic filter bankFilter designResultsHaving justified the use of local self similarity we will now go over to explain non dyadic filter bank, their need and their use in our algorithm.
17Need for non-dyadic scalings large ratiossmall ratiosmixed ratiosHere we compare the results we get by exploiting self-similarity when scaling the image in steps of various scaling ratios.[click]When using large scaling ratios the local-self similarity does not held well and noticeable artifacts arise.Small ratios show much better results as the local-sel similarity holds better.A tradeoff where we start by small ratios and then increase them gives the best cost effective choice.
18full frequency content Dyadic filtershigher halffull frequency content1:2lower halfdyadic filter bankIdeally, regular dyadic scaling by a factor of two splits the spectral content by half.
19Non-dyadic filter bank 1:24:5higher partfull frequency contentlower partsmall scaling ratiosbetter examplesnon-dyadic filter bankNon-dyadic scaling, which scale the image by arbitrary ratios smaller than 1, split the spectral content by the same amount.When using these smaller ratios the self similarity assumption holds much better.
20Non-dyadic filters: downscaling example for the 2:3 ratio:dyadic case:1. convolve with 2 filters1. convolve with one filter2. subsample by 22. subsample each by 3How do we downscale an image using these filters?In the dyadic case, the downscaled image is produced by applying a single smoothing filter followed by subsampling the signal at every second pixels.[click]In non-dyadic 2:3 scaling ratio,we have two smoothing filters shown hereAnd we subsample each one of them every third pixel to get a total of two-thirds the number of pixels.
21Non-dyadic filters: upscaling example for the 2:3 ratio:dyadic case:1. zero upsample by 12. convolve with 1 filter1. zero upsample by 2The analog takes place in the upsampling transformation:Every other value is upsampled by a factor of three-by inserting two zeros between every pixel,[click]Followed by a two separate and different filtering steps.Finally, the two are summed to a single upscaled image.2. convolve with 2 filters3. sum
22Use of the filters in upscaling Upscaling using inverse scaling filtersSmoothing by downscaling and upscalingThese non dyadic filters come to use in our algorithm in the two first steps.[click]First, the initial interpolation is done using the inverse scaling filters, as we just explained in the last slide.Second the exemplar is created by smoothing the input which is done by downscaling and upscaling it again using our filters.
23Background and overview Algorithm description Local self similarity Non-dyadic filter bankFilter designResultsThere are several filter design principles that guide our construction of non-dyadic filters and are needed for image upscaling.
241. Uniform stretchWhen interpolating, smooth areas come from input Uniformly spaced grids should remain uniform255brightnessOur first requirement is to achieve a uniform scaling. This can be measured on a linear ramp image that equates grid coordinates with brightness.We want the linear ramp to remain a linear ramp after scaling the image. In the paper we show that is analogous to requiring vanishing moments in wavelet construction.This property allows us to keep constant or very smooth regions in the image to remain that way in the upscaled output.grid coordinates
25The interpolated image, if downscaled should be equal to the input. 2. ConsistencyThe interpolated image, if downscaled should be equal to the input.Formally,↓ ↑ 𝑰 =𝑰upsampledownsamplePrevious methods achieve consistency by solving large linear systems to achieve this propertyA second requirement is consistency between the input and the output images.More specifically, we want that the upscaled image[click]After being downscaled to the input resolution, will result in the input image.In the paper we show that this requirement is a biorthogonality constrain on the filters.Note that previous methods either did not achieve this property or achieved it by solving a large system of linear equations
263. PSF modeling Difference between point spread functions Large image - small camera point spread functionSmall image - large camera point spread functionPicture acquisition is modeled by blurring an subsampling. Images at different zoom levels differ by the size of their blur kernel.[click]This difference in blur levels is must be accounted for when modeling the relation between the input and the upscaled images.Therefore, the downscaling filters we design must be low-pass filters.
274. Low frequency span frequency When upsampling don’t add new frequenciesUpsampling filter should be low-passSimilar demand is posed on the upscaling filters.When we uspcale an image we do not want to add new high frequencies,And thus the upscaling filter should also be low-passoriginalinterpolated
285. Singularities preservation ≈similaramount of blurblurred Imageinterpolated imageFinally, we want the initial interpolated image that we compute and the example smoothed image used for finding example patches to have similar looking edges.We do this by optimizing the response of both downsamping and upsampling filters to be the same on step edges.
29Real time video upsampling on GPU main GPU memorySearch and filter-banksare both local operationsGPU coresNTSC to full 24 fpsThe localized search we described as well as the filtering we use, both operate locally on the image. This makes our algorithm trivially parallelIn fact we implemented it on a GPU by dividing the input image into sub-images and upscaled each part individually.All the required data in? for each thread is kept in the core local memory or cached, minimizing the need to access the slow global memory.[click]This allowed us to achieve 24fps for upscaling NTSC frames to full HD on an Nvidia Geforce 480 while displaying the result using OpenGL.We did not made any change in our algorithm when upscaling video sequence. Temporal consistency is a result of consistency with the input and the consistency between the output and input frames.Since the input is time-coherent sequence of frame, and our upscales images are consistent with the input, then we get temporal consistency between frame even if we process them individually.
30Background and overview Algorithm description Local self similarity Non-dyadic filter bankFilter designResultsAnd now for some results
31Bicubic x3 (zoomed in) Ours X3 (zoomed in) Here is an image upscaled by a factor of thee using bicubic interpolation and our method. Note that all the edges are much better reconstructed.Ours X3(zoomed in)
33Genuine Fractals™ x4 Ours X4 Here is a comparison with the most popular commercial upscaling product, called genuine fractals.Genuine Fractals™ x4Ours X4
34Here we compare our method with the recent method of Glasner et al Here we compare our method with the recent method of Glasner et al. While their method achieve sharper edges it also introduces more ringing and staircasing artifacts.Glasner et al x4Ours X4
35Finally here is some video footage upsampled by a factor of 2 Finally here is some video footage upsampled by a factor of 2.6 to this projector full 1280 resolution – from common smartphone portrait to HD.This upsampling can be done in real time on GPU.