1. Computational Photography
Light Fields
Computational Photography
Connelly Barnes
Slides from Alexei A. Efros, James Hays, Marc Levoy, and others

2. What is light?
Electromagnetic radiation (EMR) moving along rays in space
R(l) is EMR, measured in units of power (watts)
l is wavelength
Useful things:
Light travels in straight lines
In vacuum, radiance emitted = radiance arriving
i.e. there is no transmission loss

3. What do we see?
3D world
2D image
Figures © Stephen E. Palmer, 2002

4. What do we see?
3D world
2D image
Painted
backdrop

5. On Simulating the Visual Experience
Just feed the eyes the right data
No one will know the difference!
Philosophy:
Ancient question: “Does the world really exist?”
Science fiction:
Many, many, many books on the subject, e.g. slowglass from “Light of Other Days”
Latest take: The Matrix
Physics:
Light can be stopped for up to a minute: Scientists stop light
Computer Science:
Virtual Reality
To simulate we need to know:
What does a person see?

6. The Plenoptic Function
Figure by Leonard McMillan
Q: What is the set of all things that we can ever see?
A: The Plenoptic Function (Adelson & Bergen)
Let’s start with a stationary person and try to parameterize everything that he can see…

7. Grayscale snapshot P(q,f) is intensity of light
Seen from a single view point
At a single time
Averaged over the wavelengths of the visible spectrum
(can also do P(x,y), but spherical coordinate are nicer)

8. Color snapshot P(q,f,l) is intensity of light
Seen from a single view point
At a single time
As a function of wavelength

9. A movie P(q,f,l,t) is intensity of light Seen from a single view point
Over time
As a function of wavelength

10. Holographic movie P(q,f,l,t,VX,VY,VZ) is intensity of light
Seen from ANY viewpoint
Over time
As a function of wavelength

11. The Plenoptic Function
P(q,f,l,t,VX,VY,VZ)
Can reconstruct every possible view, at every moment, from every position, at every wavelength
Contains every photograph, every movie, everything that anyone has ever seen! It completely captures our visual reality! Not bad for a function…

12. Sampling Plenoptic Function (top view)
Just lookup – Google Street View

13. Model geometry or just capture images?

14. How would you recreate this scene?

15. Ray P(q,f,VX,VY,VZ) Let’s not worry about time and color: 5D
3D position
2D direction
P(q,f,VX,VY,VZ)
Slide by Rick Szeliski and Michael Cohen

16. How can we use this?
Lighting
No Change in
Radiance
Surface
Camera

17. Ray Reuse Infinite line 4D Assume light is constant (vacuum)
2D direction
2D position
non-dispersive medium
Slide by Rick Szeliski and Michael Cohen

18. Only need plenoptic surface

19. Synthesizing novel views
Slide by Rick Szeliski and Michael Cohen

20. Lumigraph / Lightfield
Outside convex space 4D
Empty
Stuff
Slide by Rick Szeliski and Michael Cohen

21. Lumigraph - Organization
2D position
2D direction s q
Slide by Rick Szeliski and Michael Cohen

22. u s Lumigraph - Organization 2D position 2 plane parameterization
Slide by Rick Szeliski and Michael Cohen

23. s,t u,v t s,t v u,v u s Lumigraph - Organization 2D position
2 plane parameterization
s,t
u,v t v u s
Slide by Rick Szeliski and Michael Cohen

24. s,t u,v Lumigraph - Organization Hold s,t constant Let u,v vary
An image
s,t
u,v
Slide by Rick Szeliski and Michael Cohen

25. Lumigraph / Lightfield
The stanford and washington groups reverse their notation with regard to u,v and s,t

26. s,t u,v Lumigraph - Capture Idea 1
Move camera carefully over s,t plane
Gantry
see Lightfield paper
s,t
u,v
Slide by Rick Szeliski and Michael Cohen

27. s,t u,v Lumigraph - Capture Idea 2 Move camera anywhere
Find camera pose from markers
Rebinning
see Lumigraph paper
s,t
u,v
Slide by Rick Szeliski and Michael Cohen

28. Lumigraph - Rendering s u For each output pixel determine s,t,u,v
either
use closest discrete RGB
interpolate near values
Slide by Rick Szeliski and Michael Cohen

29. s u Lumigraph - Rendering Nearest Blend 16 nearest closest s closest u
draw it
Blend 16 nearest
quadrilinear interpolation
Slide by Rick Szeliski and Michael Cohen

30. Lumigraph Rendering
Use rough depth information to improve rendering quality

31. Lumigraph Rendering
Use rough depth information to improve rendering quality

32. Lumigraph Rendering
Without using geometry
Using approximate
geometry

33. Light Field Results
Marc Levoy Results
Light Field Viewer

34. Stanford multi-camera array
640 × 480 pixels × 30 fps × 128 cameras
synchronized timing
continuous streaming
flexible arrangement

35. Light field photography using a handheld plenoptic camera Commercialized as Lytro
Ren Ng, Marc Levoy, Mathieu Brédif,
Gene Duval, Mark Horowitz and Pat Hanrahan

36. Conventional versus light field camera

37. Conventional versus light field camera
uv-plane
st-plane

38. Prototype camera
medium format cameras – sensor twice as large as 35mm full frame, quite expensive (10k+)
Contax medium format camera
Kodak 16-megapixel sensor
Adaptive Optics microlens array
125μ square-sided microlenses
4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens

40. Digitally stopping-down aperture (i. e
Digitally stopping-down aperture (i.e. decrease aperture size, increase DOF) Σ Σ
stopping down = summing only the central portion of each microlens

41. Digital refocusing Σ Σ
refocusing = summing windows extracted from several microlenses

42. Example of digital refocusing

43. Digitally moving the observerΣ Σ
moving the observer = moving the window we extract from the microlenses

44. Example of moving the observer

45. Unstructured Lumigraph Rendering
Further enhancement of lumigraphs: do not use two-plane parameterization
Store original pictures: no resampling
Hand-held camera, moved around an environment

46. Unstructured Lumigraph Rendering
To reconstruct views, assign penalty to each original ray
Distance to desired ray, using approximate geometry
Resolution
Feather near edges of image
Construct “camera blending field”
Render using texture mapping

47. Unstructured Lumigraph Rendering
Blending field
Rendering

48. Unstructured Lumigraph Rendering
Unstructured Lumigraph Rendering Results

49. 3D Lumigraph
One row of s,t plane
i.e., hold t constant
s,t
u,v

50. s u,v 3D Lumigraph One row of s,t plane i.e., hold t constant
thus s,u,v
a “row of images” s
u,v

51. P(x,t)
by David Dewey

52. 2D: Image What is an image? All rays through a point Panorama?
Slide by Rick Szeliski and Michael Cohen

53. Image
Image plane 2D
position

54. Spherical Panorama All light rays through a point form a panorama
See also: 2003 New Years Eve
All light rays through a point form a panorama
Totally captured in a 2D array -- P(q,f)
Where is the geometry???

55. Other ways to sample Plenoptic Function
Moving in time:
Spatio-temporal volume: P(q,f,t)
Useful to study temporal changes
Long an interest of artists:
Claude Monet, Haystacks studies

56. Time-Lapse Spatio-temporal volume: P(q,f,t) Factored Time Lapse Video
Google/Time Magazine Time Lapse

57. Space-time images
Other ways to slice the
plenoptic function… t y x

58. Other Lightfield Acquisition Devices
Spherical motion of camera around an object
Samples space of directions uniformly
Second arm to move light source – measure BRDF

59. The “Theatre Workshop” Metaphor
(Adelson & Pentland,1996)
desired image
Sheet-metal
worker
Painter
Lighting Designer

60. Painter (images)

61. Lighting Designer (environment maps)
Show Naimark SF MOMA video

62. Sheet-metal Worker (geometry)
Let surface normals do all the work!

63. … working together Want to minimize cost
clever Italians
Want to minimize cost
Each one does what’s easiest for him
Geometry – big things
Images – detail
Lighting – illumination effects