1. Linear View Synthesis Using a Dimensionality Gap Light Field Prior
Anat Levin and Fredo Durand
Weizmann Institute of Science & MIT CSAIL 1

2. Light fields 2
Light field: the set of rays emitted from a scene in all possible directions

3. (Animation by Marc Levoy)
Light fields 3
Novel view rendering
(Animation by Marc Levoy)

4. (Animation by Marc Levoy)
Light fields 4
Novel view rendering
(Animation by Marc Levoy)

5. (Animation by Marc Levoy)
Light fields 5
Novel view rendering
Synthetic refocusing
(Animation by Marc Levoy)

6. 4D light field 6 u v
The set of light rays hitting the camera aperture plane is 4D:
Ray hitting point- 2D
Ray orientation- 2D
(In general: a 7D plenoptic space, including time and wavelength dimensions)

7. Light field acquisition schemes and priors7
Very different approaches to light field acquisition and manipulations exist in the literature.
The inherent difference between them is a different prior model on the light field space

8. Light field acquisition schemes and priors8
4D:
The light field is smooth, but involves 4 degrees of freedom
-Capture: 4D data (e.g. camera array)
-Inference: linear .

9. Light field acquisition schemes and priors9
4D:
-Capture: 4D data (e.g. camera array)
-Inference: linear
2D:
For Lambertian scenes all rays emerging from one point have same color If depth is known, only 2 degrees of freedom
-Capture: 2D data (e.g. stereo camera)
-Inference: non linear depth estimation

10. In this talk: 3D light field prior10
4D:
-Capture: 4D data (e.g. camera array
-Inference: linear
2D:
-Capture: 2D data (e.g. stereo camera)
-Inference: non linear depth estimation
3D:
Depth is a 1D variable, hence the union of images at any depth covers no more than a 3D subset Show that in the frequency domain there is only a 3D manifold of non zero entries.
-Capture: 3D data (e.g. focal stack)

11. Outline Linear view synthesis from a focal stack sequence11
Linear view synthesis from a focal stack sequence
The 3D light field prior
Frequency derivation of synthesis algorithm
Other applications of the 3D prior

12. Linear view synthesis with 3D prior12
Input: Focal stack (3D data)
1D set of 2D images focused at different depth
Output: Novel viewpoints (4D data)
2D Images x 2D set of novel viewpoints
Linear image processing

13. Linear view synthesis algorithm13
No depth estimation!
Shift focal stack images by disparity of desired view
Average shifted images
Depth invariant deconvolution 1 2 3

14. Shift invariant convolution~ focus sweep camera14
Average shifted images
Depth invariant blur kernel
Ideal pinhole image
Inspiration: The focus sweep camera Hausler 72, Nagahara et al. 08
Captures a single image, average over all focus depths during exposure, provides EDOF image from a single view

15. Linear view synthesis results15
Video animation here

16. Disclaimers Novel viewpoints limited to the aperture area16
Novel viewpoints limited to the aperture area
Convolution model breaks at occlusion boundaries
Assume scene is Lambertian- in practice holds within the narrow range of angles of the aperture

17. Outline Linear view synthesis from a focal stack sequence17
Linear view synthesis from a focal stack sequence
The 3D light field prior
Frequency derivation of synthesis algorithm
Other applications of the 3D prior

18. 4D light field v y v x u u (x,y,u,v)18 u v u v y x y x
(x,y,u,v)
The set of light rays hitting the lens is 4D

19. 4D light field v y v x u u (?,?,u0,0) (x,y,u,v)19 u v u v y x y x
(?,?,u0,0)
(x,y,u,v)
The set of light rays hitting the lens is 4D

20. 4D light field v y v x u u (?,?,0,v0) (x,y,u,v)20 u v u v y x y x
(?,?,0,v0)
(x,y,u,v)
The set of light rays hitting the lens is 4D

21. 4D light field v y v x u u (x,y,u,v)21 u v u v y x y x
(x,y,u,v)
The set of light rays hitting the lens is 4D

22. 4D light field spectrum y v x u (x,y,u,v) L( , , , )22 v y x u
4D Fourier Transform
(x,y,u,v)
The set of light rays hitting the lens is 4D
Study the 4D Fourier domain
L( , , , )

23. 4D light field spectrum y v x u L( ,0,?,?) (x,y,u,v) L( , , , )23 v y x u
4D Fourier Transform
L( ,0,?,?)
(x,y,u,v)
The set of light rays hitting the lens is 4D
Study the 4D Fourier domain
L( , , , )

24. 4D light field spectrum y v x u24 v y x u
4D Fourier Transform
Frequency content only along 1D segments

25. Energy portion away from focal segments
4D light field spectrum
Scene
4D Light field spectrum
Energy portion away from focal segments

26. The slicing theorem y v x u 4D Fourier Transform26 v y x u
4D Fourier Transform
2D focused images at varying depths
2D Fourier Transform

27. The dimensionality gap27 v y x u
far
4D Fourier Transform
depth color coding
near
Light field spectrum: 4D
Image spectrum: 2D
Depth: 1D
→ Dimensionality gap (Ng 05, Levin et al. 09)
Only the 3D manifold corresponding to physical focusing distance is useful 3D

28. 3D Gaussian light field prior28
Gaussian prior: assigns non zero variance only to 3D set of entries on the focal segments
Gaussian=> inference simple and linear
Focal stack directly samples the manifold with non zero variance

29. Outline Linear view synthesis from a focal stack sequence29
Linear view synthesis from a focal stack sequence
The 3D light field prior
Frequency derivation of synthesis algorithm
Other applications of the 3D prior

30. View synthesis in the frequency domain30
4D spectrum of constant depth scene
Average focal stack spectra
Spectra of correct depth
Sample density
Spectra of focal stack images
Deconvolution (frequency domain)

31. Outline Linear view synthesis from a focal stack sequence31
Linear view synthesis from a focal stack sequence
The 3D light field prior
Frequency derivation of synthesis algorithm
Other applications of the 3D prior

32. Prior to infer light field from partial samples32
In many other light field acquisition schemes we capture only a partial information on the light field- limited resolution, aliasing and each.
However, we capture linear measurements
On the other hand, we have a Gaussian prior, and we know the light field actually occupies only a low dimensional manifold of the 4D space.
Use the prior to “invert the rank deficient projection” and interpolate the measurements to get a light field with higher resolution, less aliasing.

33. Improved viewpoints sample33
4D Light field acquisition systems sample a 2D set of view points
Can we do with sparser sample and 3D Gaussian prior for interpolation?
How many samples needed? What is the right spacing?
Shall we distribute samples on a grid? Better arrangement?
Grid: Standard sampling pattern
Circle: Sampling pattern with improved reconstruction using 3D prior

34. Superesolution of plenoptic camera measurements34
Plenoptic camera measurements are aliased
Replicas off the focal segments are high frequencies which we can re-bin and restore high frequency information

35. Superesolution of plenoptic camera measurements35
Bicubic interpolation
Lumsdaine and Georgiev: applies for a single known depth
Our result: applies for all depths simultaneously, no depth estimation

36. Summary 36
Light field acquisition and synthesis strongly depends on light field prior
Existing priors:
Linear view synthesis from the focal stack
Other applications of 3D prior:
- viewpoints sample pattern
- depth invariant superesolution of plenoptic camera data
4D prior: capture- 4D data (e.g. camera array), inference- linear
2D prior: capture- 2D data (e.g. stereo), inference- non linear
Our new prior:
3D prior: capture- 3D data (e.g. focal stuck), inference linear