1. Light field photography and microscopy Marc Levoy Computer Science Department Stanford University

2.  Marc Levoy The light field [Gershun 1936] Radiance as a function of position and direction for general scenes –the “plenoptic function” –five-dimensional –L ( x, y, z,  ) (w/m 2 sr) in free space –four-dimensional –L ( u, v, s, t ) two-plane parameterization [Levoy and Hanrahan 1996]

3.  Marc Levoy Devices for recording light fields small baseline big baseline handheld camera [Buehler 2001] camera gantry [Stanford 2002] array of cameras [Wilburn 2005] plenoptic camera [Ng 2005] light field microscope [Levoy 2006] (using geometrical optics)

4.  Marc Levoy Light fields at micron scales wave optics must be considered –diffraction limits the spatial × angular resolution most objects are no longer opaque –each pixel is a line integral through the object »of attenuation »or emission –can reconstruct 3D structure from these integrals »tomography »3D deconvolution

5. High performance imaging using large camera arrays Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez, Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005)

6.  Marc Levoy Stanford multi-camera array 640 × 480 pixels × 30 fps × 128 cameras synchronized timing continuous streaming flexible arrangement

7.  Marc Levoy Ways to use large camera arrays widely spaced light field capture Manex’s bullet time array

8.  Marc Levoy Ways to use large camera arrays widely spaced light field capture tightly packed high-performance imaging

9.  Marc Levoy Ways to use large camera arrays widely spaced light field capture tightly packed high-performance imaging intermediate spacing synthetic aperture photography

10.  Marc Levoy Synthetic aperture photography

11.  Marc Levoy Synthetic aperture photography

12.  Marc Levoy Synthetic aperture photography

13.  Marc Levoy Synthetic aperture photography 

14.  Marc Levoy Synthetic aperture photography 

15.  Marc Levoy Synthetic aperture photography 

16.  Marc Levoy Example using 45 cameras [Vaish CVPR 2004]

18. Light field photography using a handheld plenoptic camera Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz and Pat Hanrahan (Proc. SIGGRAPH 2005 and TR 2005-02)

19.  Marc Levoy Conventional versus plenoptic camera

20.  Marc Levoy Conventional versus plenoptic camera uv-planest-plane

21. Prototype camera 4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens Contax medium format cameraKodak 16-megapixel sensor Adaptive Optics microlens array125μ square-sided microlenses

23.  Marc Levoy Digital refocusing refocusing = summing windows extracted from several microlenses Σ Σ

24.  Marc Levoy A digital refocusing theorem an f / N light field camera, with P × P pixels under each microlens, can produce views as sharp as an f / (N × P) conventional camera – or – it can produce views with a shallow depth of field ( f / N ) focused anywhere within the depth of field of an f / (N × P) camera

25.  Marc Levoy Example of digital refocusing

26.  Marc Levoy Refocusing portraits

27.  Marc Levoy Action photography

28. Extending the depth of field conventional photograph, main lens at f / 22 conventional photograph, main lens at f / 4 light field, main lens at f / 4, after all-focus algorithm [Agarwala 2004]

29.  Marc Levoy Macrophotography

30.  Marc Levoy Digitally moving the observer moving the observer = moving the window we extract from the microlenses Σ Σ

31.  Marc Levoy Example of moving the observer

32.  Marc Levoy Moving backward and forward

33.  Marc Levoy Implications cuts the unwanted link between exposure (due to the aperture) and depth of field trades off (excess) spatial resolution for ability to refocus and adjust the perspective sensor pixels should be made even smaller, subject to the diffraction limit 36mm × 24mm ÷ 2μ pixels = 216 megapixels 18K × 12K pixels 1800 × 1200 pixels × 10 × 10 rays per pixel

34. Light Field Microscopy Marc Levoy, Ren Ng, Andrew Adams, Matthew Footer, and Mark Horowitz (Proc. SIGGRAPH 2006)

35.  Marc Levoy A traditional microscope objective specimen intermediate image plane eyepiece

36.  Marc Levoy A light field microscope (LFM) 40x / 0.95NA objective ↓ 0.26μ spot on specimen × 40x = 10.4μ on sensor ↓ 2400 spots over 25mm field 125 2 -micron microlenses ↓ 200 × 200 microlenses with 12 × 12 spots per microlens objective specimen intermediate image plane eyepiece sensor → reduced lateral resolution on specimen = 0.26μ × 12 spots = 3.1μ

37.  Marc Levoy A light field microscope (LFM) sensor

38.  Marc Levoy Example light field micrograph orange fluorescent crayon mercury-arc source + blue dichroic filter 16x / 0.5NA (dry) objective f/20 microlens array 65mm f/2.8 macro lens at 1:1 Canon 20D digital camera ordinary microscope light field microscope

39.  Marc Levoy The geometry of the light field in a microscope microscopes make orthographic views translating the stage in X or Y provides no parallax on the specimen out-of-plane features don’t shift position when they come into focus front lens element size = aperture width + field width PSF for 3D deconvolution microscopy is shift-invariant (i.e. doesn’t change across the field of view) f objective lenses are telecentric

40.  Marc Levoy Panning and focusing panning sequencefocal stack

41.  Marc Levoy Real-time viewer

42.  Marc Levoy Other examples fern spore (60x, autofluorescence) mouse oocyte (40x, DIC) Golgi-stained neurons (40x, transmitted light)

43.  Marc Levoy Extensions digital correction of aberrations –by capturing and resampling the light field Nikon 40x 0.95NA (dry) Plan-Apo eyepiece

44.  Marc Levoy Extensions digital correction of aberrations –by capturing and resampling the light field

45.  Marc Levoy Extensions digital correction of aberrations –by capturing and resampling the light field correcting for aberrations caused by imaging through thick specimens whose index of refraction doesn’t match that of the immersion medium

46.  Marc Levoy Extensions digital correction of aberrations –by capturing and resampling the light field multiplexing of variables other than angle –by placing gradient filters at the aperture plane, such as neutral density, spectral, or polarization neutral density

47.  Marc Levoy Extensions digital correction of aberrations –by capturing and resampling the light field multiplexing of variables other than angle –by placing gradient filters at the aperture plane, such as neutral density, spectral, or polarization neutral density wavelength

48.  Marc Levoy Extensions digital correction of aberrations –by capturing and resampling the light field multiplexing of variables other than angle –by placing gradient filters at the aperture plane, such as neutral density, spectral, or polarization neutral density wavelength... or polarization direction... or ??? gives up digital refocusing?

49.  Marc Levoy Extensions digital correction of aberrations –by capturing and resampling the light field multiplexing of variables other than angle –by placing gradient filters at the aperture plane, such as neutral density, spectral, or polarization microscope scatterometry –by controlling the incident light field using a micromirror array + microlens array

50.  Marc Levoy Programmable incident light field light source + micromirror array + microlens array 800 × 800 pixels = 40 × 40 tiles × 20 × 20 directions driven by image from PC graphics card

51.  Marc Levoy Other applications of light field illumination: “4D designer lighting”

52.  Marc Levoy http://graphics.stanford.edu