Presentation is loading. Please wait.

Presentation is loading. Please wait.

Optimizing and Learning for Super-resolution Lyndsey C. Pickup, Stephen J. Roberts & Andrew Zisserman Robotics Research Group, University of Oxford.

Similar presentations


Presentation on theme: "Optimizing and Learning for Super-resolution Lyndsey C. Pickup, Stephen J. Roberts & Andrew Zisserman Robotics Research Group, University of Oxford."— Presentation transcript:

1 Optimizing and Learning for Super-resolution Lyndsey C. Pickup, Stephen J. Roberts & Andrew Zisserman Robotics Research Group, University of Oxford

2 The Super-resolution Problem Given a number of low-resolution images differing in:  geometric transformations  lighting (photometric) transformations  camera blur (point-spread function)  image quantization and noise. Estimate a high-resolution image:

3 Low-resolution image 1

4 Low-resolution image 2

5 Low-resolution image 3

6 Low-resolution image 4

7 Low-resolution image 5

8 Low-resolution image 6

9 Low-resolution image 7

10 Low-resolution image 8

11 Low-resolution image 9

12 Low-resolution image 10

13 Super-Resolution Image

14 Generative Model Registrations, lighting and blur. High-resolution image, x. y1y1 y2y2 y3y3 y4y4 Low-resolution images W4W4 W3W3 W2W2 W1W1

15 Generative Model Geometric registrations Point-spread function Photometric registrations We don’t have: We have: Set of low-resolution input images, y.

16 Maximum a Posteriori (MAP) Solution Standard method: 1.Compute registrations from low-res images. 2.Solve for SR image, x, using gradient descent. y1y2y3y4 W4 W3W2 W1 x [Irani & Peleg ‘90, Capel ’01, Baker & Kanade ’02, Borman ‘04]

17 What’s new 1.We solve for registrations and SR image jointly. 2.We also find appropriate values for parameters in the prior term at the same time.  Hardie et al. ’97: adjust registration while optimizing super-resolution estimate. Exhaustive search limits them to translation only. Simple smoothness prior softens image edges. i.e. given the low-res images, y, we solve for the SR image x and the mappings, W simultaneously. y1y2y3y4 W4 W3W2 W1 x

18 Overview of rest of talk Simultaneous Approach –Model details –Initialisation technique –Optimization loop Learning values for the prior’s parameters Results on real images

19 Maximum a Posteriori (MAP) Solution Image x. Corrupt with additive Gaussian noise. Warp, with parameters Φ. Blur by point- spread function. Decimate by zoom factor. y1y2y3y4 W4 W3W2 W1 x y

20 Details of Huber Prior Huber function is quadratic in the middle, and linear in the tails. Probability distribution is like a heavy-tailed Gaussian. ρ (z,α)p (z|α,v) Red: large α Blue: small α This is applied to image gradients in the SR image estimate.

21 Details of Huber Prior Ground Truth α=0.1 v=0.4 Too little smoothingToo much smoothing α=0.05 v=0.05α=0.01 v=0.01α=0.01 v=0.005 Edges are sharper Advantages: simple, edge-preserving, leads to convex form for MAP equations. Solutions as α and v vary:

22 Advantages of Simultaneous Approach  Learn from lessons of Bundle Adjustment: improve results by optimizing the scene estimate and the registration together.  Registration can be guided by the super- resolution model, not by errors measured between warped, noisy low-resolution images.  Use a non-Gaussian prior which helps to preserve edges in the super-resolution image.

23 Overview of Simultaneous Approach 1.Start from a feature-based RANSAC -like registration between low-res frames. 2.Select blur kernel, then use average image method to initialise registrations and SR image. 3.Iterative loop:  Update Prior Values  Update SR estimate  Update registration estimate

24  Use average image as an estimate of the super-resolution image (see paper).  Minimize the error between the average image and the low-resolution image set.  Use an early-stopped iterative ML estimate of the SR image to sharpen up this initial estimate. Initialisation Average image ML-sharpened estimate

25 1.Update prior’s parameter values (next section) 2.Update estimate of SR image 3.Update estimate of registration and lighting values, which parameterize W  Repeat till converged. Optimization Loop

26 Joint MAP Results Decreasing prior strength Registration FixedJoint MAP

27 Learning Prior Parameters α, ν  Split the low-res images into two sets: Use first set to obtain an SR image. Find error on validation set.

28 Learning Prior Parameters α, ν  Split the low-res images into two sets: Use first set to obtain an SR image. Find error on validation set.  But what if one of the validation images is mis-registered?

29 Learning Prior Parameters α, ν  Instead, we select pixels from across all images, choosing differently at each iteration.  We evaluate an SR estimate using the unmarked pixels, then use the forward model to compare the estimate to the red pixels.

30 Learning Prior Parameters α, ν  Instead, we select pixels from across all images, choosing differently at each iteration.  We evaluate an SR estimate using the unmarked pixels, then use the forward model to compare the estimate to the red pixels.

31 Learning Prior Parameters α, ν  To update the prior parameters: 1.Re-select a cross-validation pixel set. 2.Run the super-resolution image MAP solver for a small number of iterations, starting from the current SR estimate. 3.Predict the low-resolution pixels of the validation set, and measure error. 4.Use gradient descent to minimise the error with respect to the prior parameters.

32 Results: Eye Chart MAP version: fixing registrations then super-resolving Joint MAP version with adaptation of prior’s parameter values

33 Results: Groundhog Day

34  The blur estimate can still be altered to change the SR result. More ringing and artefacts appear in the regular MAP version. Results: Groundhog Day Blur radius = 1 Blur radius = 1.4Blur radius = 1.8 Regular MAP Simultaneous

35 Lola Rennt

36 Real Data: Lola Rentt

37

38

39

40 Conclusions Joint MAP solution: better results by optimizing SR image and registration parameters simultaneously. Learning prior values: preserve image edges without having to estimate image statistics in advance. DVDs: Automatically zoom in on regions with a registrations up to a projective transform and with an affine lighting model. Further work: marginalize over the registration – see NIPS 2006.


Download ppt "Optimizing and Learning for Super-resolution Lyndsey C. Pickup, Stephen J. Roberts & Andrew Zisserman Robotics Research Group, University of Oxford."

Similar presentations


Ads by Google