1. Fourier Slice Photography
Ren Ng
Stanford University

2. Conventional Photograph
Okay, so here’s the conventional photograph that we would have gotten. Notice that it’s focused on Matt, and that Jacquie and Matt are out of focus.

3. Light Field Photography
Capture the light field inside the camera body
Okay, the first thing that I want to do is define this “light field” term that I’ve been using. The light field is just a representation for the light traveling along all rays in free space. Inside the camera, we can parameterize all the rays by where they originate on the lens plane, and where they terminate on the sensor.
So the light traveling from (u,v) on the lens to (x,y) on the sensor is given by L(u,v,x,y). Note that this is a four dimensional function – the space of rays is 4D.
The concept of the light field is the second most cited idea in computer graphics. It was introduced in 1996 by Marc and Pat at Stanford, and by Steve Gortler and colleagues at Harvard and Microsoft Research.

4. Hand-Held Light Field Camera
Medium format digital camera
Camera in-use
16 megapixel sensor
Microlens array

5. Light Field in a Single Exposure

6. Light Field in a Single Exposure

7. Light Field Inside the Camera Body
Okay, the first thing that I want to do is define this “light field” term that I’ve been using. The light field is just a representation for the light traveling along all rays in free space. Inside the camera, we can parameterize all the rays by where they originate on the lens plane, and where they terminate on the sensor.
So the light traveling from (u,v) on the lens to (x,y) on the sensor is given by L(u,v,x,y). Note that this is a four dimensional function – the space of rays is 4D.
The concept of the light field is the second most cited idea in computer graphics. It was introduced in 1996 by Marc and Pat at Stanford, and by Steve Gortler and colleagues at Harvard and Microsoft Research.

8. Digital Refocusing
Okay, so here’s the conventional photograph that we would have gotten. Notice that it’s focused on Matt, and that Jacquie and Matt are out of focus.

9. Digital Refocusing
But by simulating the light light… etc…

10. Questions About Digital Refocusing
What is the computational complexity? Are there efficient algorithms?
What are the limits on refocusing? How far can we move the focal plane?

11. Overview Fourier Slice Photography Theorem
Fourier Refocusing Algorithm
Theoretical Limits of Refocusing
Here’s an overview for what I’ll be talking about in this section.
First, I’d like to go over the derivation of the theorem at the heart of this whole analytic approach. The theorem says that in the Fourier domain, photographs are just 2D slice in the 4D light field. That’s much simpler than in the spatial-domain, where photographs are integral projections of the light field.
Then I want to show you how the theorem can be applied to make theoretical analysis of our camera easier, and how it gives us a fast algorithm for digital refocusing.

12. Previous Work Integral photography Lippmann 1908, Ives 1930
Lots of variants, especially in 3D TV Okoshi 1976, Javidi & Okano 2002
Closest variant is plenoptic camera Adelson & Wang 1992
Fourier analysis of light fields
Chai et al. 2000
Refocusing from light fields
Isaksen et al. 2000, Stewart et al. 2003

13. Fourier Slice Photography Theorem
In the Fourier domain, a photograph is a 2D slice in the 4D light field. Photographs focused at different depths correspond to 2D slices at different trajectories.

14. Digital Refocusing by Ray-Tracingx
Lens
Sensor

15. Digital Refocusing by Ray-Tracingx
Imaginary film
Lens
Sensor

16. Digital Refocusing by Ray-Tracingx
Imaginary film
Lens
Sensor

17. Digital Refocusing by Ray-Tracingx
Imaginary film
Lens
Sensor

18. Digital Refocusing by Ray-Tracingx
Imaginary film
Lens
Sensor

19. Refocusing as Integral Projectionx u u x
Imaginary film
Lens
Sensor

20. Refocusing as Integral Projectionx u u x
Imaginary film
Lens
Sensor

21. Refocusing as Integral Projectionx u u x
Imaginary film
Lens
Sensor

22. Refocusing as Integral Projectionx u u x
Imaginary film
Lens
Sensor

23. Classical Fourier Slice Theorem
Integral Projection
2D Fourier Transform
1D Fourier Transform
Okay so here’s the diagram for the Fourier Slice Photography Theorem again. So all existing approaches compute the refocused photograph from the light field by working in the spatial domain. The idea for a faster algorithm is to work in the FOurier domain.
Slicing

24. Classical Fourier Slice Theorem
Integral Projection
2D Fourier Transform
1D Fourier Transform
Okay so here’s the diagram for the Fourier Slice Photography Theorem again. So all existing approaches compute the refocused photograph from the light field by working in the spatial domain. The idea for a faster algorithm is to work in the FOurier domain.
Slicing

25. Classical Fourier Slice Theorem
Integral Projection
2D Fourier Transform
1D Fourier Transform
Okay so here’s the diagram for the Fourier Slice Photography Theorem again. So all existing approaches compute the refocused photograph from the light field by working in the spatial domain. The idea for a faster algorithm is to work in the FOurier domain.
Slicing

26. Classical Fourier Slice Theorem
Integral Projection
Spatial Domain
Fourier Domain
Okay so here’s the diagram for the Fourier Slice Photography Theorem again. So all existing approaches compute the refocused photograph from the light field by working in the spatial domain. The idea for a faster algorithm is to work in the FOurier domain.
Slicing

27. Classical Fourier Slice Theorem
Integral Projection
Spatial Domain
Fourier Domain
Okay so here’s the diagram for the Fourier Slice Photography Theorem again. So all existing approaches compute the refocused photograph from the light field by working in the spatial domain. The idea for a faster algorithm is to work in the FOurier domain.
Slicing

28. Fourier Slice Photography Theorem
Integral Projection
Spatial Domain
Fourier Domain
Okay so here’s the diagram for the Fourier Slice Photography Theorem again. So all existing approaches compute the refocused photograph from the light field by working in the spatial domain. The idea for a faster algorithm is to work in the FOurier domain.
Slicing

29. Fourier Slice Photography Theorem
Integral Projection
4D Fourier Transform
Okay so here’s the diagram for the Fourier Slice Photography Theorem again. So all existing approaches compute the refocused photograph from the light field by working in the spatial domain. The idea for a faster algorithm is to work in the FOurier domain.
Slicing

30. Fourier Slice Photography Theorem
Integral Projection
4D Fourier Transform
2D Fourier Transform
Okay so here’s the diagram for the Fourier Slice Photography Theorem again. So all existing approaches compute the refocused photograph from the light field by working in the spatial domain. The idea for a faster algorithm is to work in the FOurier domain.
Slicing

31. Fourier Slice Photography Theorem
Integral Projection
4D Fourier Transform
2D Fourier Transform
Okay so here’s the diagram for the Fourier Slice Photography Theorem again. So all existing approaches compute the refocused photograph from the light field by working in the spatial domain. The idea for a faster algorithm is to work in the FOurier domain.
Slicing

32. Fourier Slice Photography Theorem
Integral Projection
4D Fourier Transform
2D Fourier Transform
Okay so here’s the diagram for the Fourier Slice Photography Theorem again. So all existing approaches compute the refocused photograph from the light field by working in the spatial domain. The idea for a faster algorithm is to work in the FOurier domain.
Slicing

33. Photographic Imaging Equations
Spatial-Domain Integral Projection
Fourier-Domain Slicing
So here are the definitions of these operators. We already saw in the spatial domain that imaging is this double integral.
And in the Fourier domain, here’s the definition of the slicing operator, which shows that the value at any point in the photograph is given by the value at a corresponding point in the light field.
So I think this makes really clear why this theorem is useful. On the theoretical side, if you have to do any math, it’s much nicer to work with the Fourier definition where you don’t have to deal with the integral symbols. And on the practical side, if you have to do any computation, it might be better to work in the Fourier domain, where you can just look up an array element rather than summing a bunch of array elements.
10 minutes to here

34. Photographic Imaging Equations
Spatial-Domain Integral Projection
Fourier-Domain Slicing
So here are the definitions of these operators. We already saw in the spatial domain that imaging is this double integral.
And in the Fourier domain, here’s the definition of the slicing operator, which shows that the value at any point in the photograph is given by the value at a corresponding point in the light field.
So I think this makes really clear why this theorem is useful. On the theoretical side, if you have to do any math, it’s much nicer to work with the Fourier definition where you don’t have to deal with the integral symbols. And on the practical side, if you have to do any computation, it might be better to work in the Fourier domain, where you can just look up an array element rather than summing a bunch of array elements.
10 minutes to here

35. Photographic Imaging Equations
Spatial-Domain Integral Projection
Fourier-Domain Slicing
So here are the definitions of these operators. We already saw in the spatial domain that imaging is this double integral.
And in the Fourier domain, here’s the definition of the slicing operator, which shows that the value at any point in the photograph is given by the value at a corresponding point in the light field.
So I think this makes really clear why this theorem is useful. On the theoretical side, if you have to do any math, it’s much nicer to work with the Fourier definition where you don’t have to deal with the integral symbols. And on the practical side, if you have to do any computation, it might be better to work in the Fourier domain, where you can just look up an array element rather than summing a bunch of array elements.
10 minutes to here

36. Theorem Limitations Film parallel to lens
Everyday camera, not view camera
Aperture fully open
Closing aperture requires spatial mask

37. Overview Fourier Slice Photography Theorem
Fourier Refocusing Algorithm
Theoretical Limits of Refocusing
Here’s an overview for what I’ll be talking about in this section.
First, I’d like to go over the derivation of the theorem at the heart of this whole analytic approach. The theorem says that in the Fourier domain, photographs are just 2D slice in the 4D light field. That’s much simpler than in the spatial-domain, where photographs are integral projections of the light field.
Then I want to show you how the theorem can be applied to make theoretical analysis of our camera easier, and how it gives us a fast algorithm for digital refocusing.

38. Existing Refocusing Algorithms
Existing refocusing algorithms are expensive
O(N4) where light field has N samples in each dimension
All are variants on integral projection
Isaksen et al
Vaish et al
Levoy et al
Ng et al
The point I want to make here is that all existing digital refocusing approaches are variants on spatial domain integration, from the original paper demonstrating refocusing by Isaksen, McMillan and Gortler in 2000, through work on synthetic photography at Stanford i the last two years.
The point here is the digital refocusing is expensive – O(N^4) if we have N samples in each dimension – because to compute any photograph we have to project and integrate the entire light field.

39. Refocusing in Spatial Domain
Integral Projection
4D Fourier Transform
2D Fourier Transform
Okay so here’s the diagram for the Fourier Slice Photography Theorem again. So all existing approaches compute the refocused photograph from the light field by working in the spatial domain. The idea for a faster algorithm is to work in the FOurier domain.
Slicing

40. Refocusing in Fourier Domain
Integral Projection
Inverse 2D Fourier Transform
4D Fourier Transform
Okay so here’s the diagram for the Fourier Slice Photography Theorem again. So all existing approaches compute the refocused photograph from the light field by working in the spatial domain. The idea for a faster algorithm is to work in the FOurier domain.
Slicing

41. Refocusing in Fourier Domain
Integral Projection
Inverse 2D Fourier Transform
4D Fourier Transform
Okay so here’s the diagram for the Fourier Slice Photography Theorem again. So all existing approaches compute the refocused photograph from the light field by working in the spatial domain. The idea for a faster algorithm is to work in the FOurier domain.
Slicing

42. Asymptotic Performance
Fourier-domain slicing algorithm
Pre-process: O(N4 log N)
Refocusing: O(N2 log N)
Spatial-domain integration algorithm
Refocusing: O(N4)
So the asymptotic performance in the Fourier domain is O(N4 log N) for the pre-process, and O(N2 log N), dominated by the inverse 2D transform.
And of course that’s to be compared against the spatial domain algorithm, which is O(N4) for each refocusing step.
I also want to mention that practically, the absolute performance of the two algorithms is about the same for resolutions captured with our prototype. At directional resolutions twice as high, Fourier algorithm is order of magnitude faster.

43. Resampling Filter Choice
Kaiser-Bessel filter (width 2.5)
Gold standard (spatial integration)
Triangle filter (quadrilinear)

44. Overview Fourier Slice Photography Theorem
Fourier Refocusing Algorithm
Theoretical Limits of Refocusing
Here’s an overview for what I’ll be talking about in this section.
First, I’d like to go over the derivation of the theorem at the heart of this whole analytic approach. The theorem says that in the Fourier domain, photographs are just 2D slice in the 4D light field. That’s much simpler than in the spatial-domain, where photographs are integral projections of the light field.
Then I want to show you how the theorem can be applied to make theoretical analysis of our camera easier, and how it gives us a fast algorithm for digital refocusing.

45. Problem Statement Assume a light field camera with An f /A lens
N x N pixels under each microlens
If we compute refocused photographs from these light fields, over what range can we move the focal plane?
Analytical assumption
Assume band-limited light fields

46. Band-Limited Analysis
Okay, so if what do we get with this band-limited assumption? Well, here is the continuous situation. This shows the Fourier transform of the continuous light field in 2D, which is in general unbounded, and here is the optical photograph that forms at a particular focal depth: it’s just a slice in the light field.
The conventional camera band-limits this continuous photograph by cutting off its high frequencies, and if we focus at a different depth, getting a different slice, the band-limit is the same.
In the plenoptic camera, the band-limit isn’t directly on the photo, but rather on the light field. So we cut off the high frequencies in the light field.
Now, when we produce a photograph with digital refocusing, we extract the slice here, but the slice is clipped to the bounds of the light field. If we refocus at a different depth, the width of the bandwidth changes.
One thing to note is that the refocused photographs is just a band-limited version of the continuous photograph. By comparing the bandwidths, we can tell how close digital refocusing is getting us to the ideal result.

47. Band-Limited Analysis
Band-width of measured light field
Okay, so if what do we get with this band-limited assumption? Well, here is the continuous situation. This shows the Fourier transform of the continuous light field in 2D, which is in general unbounded, and here is the optical photograph that forms at a particular focal depth: it’s just a slice in the light field.
The conventional camera band-limits this continuous photograph by cutting off its high frequencies, and if we focus at a different depth, getting a different slice, the band-limit is the same.
In the plenoptic camera, the band-limit isn’t directly on the photo, but rather on the light field. So we cut off the high frequencies in the light field.
Now, when we produce a photograph with digital refocusing, we extract the slice here, but the slice is clipped to the bounds of the light field. If we refocus at a different depth, the width of the bandwidth changes.
One thing to note is that the refocused photographs is just a band-limited version of the continuous photograph. By comparing the bandwidths, we can tell how close digital refocusing is getting us to the ideal result.
Light field shot with camera

48. Band-Limited Analysis
Okay, so what do we get with this band-limited assumption? Well, here is the continuous situation. This shows the Fourier transform of the continuous light field in 2D, which is in general unbounded, and here is the optical photograph that forms at a particular focal depth: it’s just a slice in the light field.
The conventional camera band-limits this continuous photograph by cutting off its high frequencies, and if we focus at a different depth, getting a different slice, the band-limit is the same.
In the plenoptic camera, the band-limit isn’t directly on the photo, but rather on the light field. So we cut off the high frequencies in the light field.
Now, when we produce a photograph with digital refocusing, we extract the slice here, but the slice is clipped to the bounds of the light field. If we refocus at a different depth, the width of the bandwidth changes.
One thing to note is that the refocused photographs is just a band-limited version of the continuous photograph. By comparing the bandwidths, we can tell how close digital refocusing is getting us to the ideal result.

49. Band-Limited Analysis
Okay, so if what do we get with this band-limited assumption? Well, here is the continuous situation. This shows the Fourier transform of the continuous light field in 2D, which is in general unbounded, and here is the optical photograph that forms at a particular focal depth: it’s just a slice in the light field.
The conventional camera band-limits this continuous photograph by cutting off its high frequencies, and if we focus at a different depth, getting a different slice, the band-limit is the same.
In the plenoptic camera, the band-limit isn’t directly on the photo, but rather on the light field. So we cut off the high frequencies in the light field.
Now, when we produce a photograph with digital refocusing, we extract the slice here, but the slice is clipped to the bounds of the light field. If we refocus at a different depth, the width of the bandwidth changes.
One thing to note is that the refocused photographs is just a band-limited version of the continuous photograph. By comparing the bandwidths, we can tell how close digital refocusing is getting us to the ideal result.

50. Band-Limited Analysis
Okay, so if what do we get with this band-limited assumption? Well, here is the continuous situation. This shows the Fourier transform of the continuous light field in 2D, which is in general unbounded, and here is the optical photograph that forms at a particular focal depth: it’s just a slice in the light field.
The conventional camera band-limits this continuous photograph by cutting off its high frequencies, and if we focus at a different depth, getting a different slice, the band-limit is the same.
In the plenoptic camera, the band-limit isn’t directly on the photo, but rather on the light field. So we cut off the high frequencies in the light field.
Now, when we produce a photograph with digital refocusing, we extract the slice here, but the slice is clipped to the bounds of the light field. If we refocus at a different depth, the width of the bandwidth changes.
One thing to note is that the refocused photographs is just a band-limited version of the continuous photograph. By comparing the bandwidths, we can tell how close digital refocusing is getting us to the ideal result.

51. Photographic Imaging Equations
Spatial-Domain Integral Projection
Fourier-Domain Slicing
So here are the definitions of these operators. We already saw in the spatial domain that imaging is this double integral.
And in the Fourier domain, here’s the definition of the slicing operator, which shows that the value at any point in the photograph is given by the value at a corresponding point in the light field.
So I think this makes really clear why this theorem is useful. On the theoretical side, if you have to do any math, it’s much nicer to work with the Fourier definition where you don’t have to deal with the integral symbols. And on the practical side, if you have to do any computation, it might be better to work in the Fourier domain, where you can just look up an array element rather than summing a bunch of array elements.
10 minutes to here

52. Results of Band-Limited Analysis
Assume a light field camera with
An f /A lens
N x N pixels under each microlens
From its light fields we can
Refocus exactly within depth of field of an f /(A  N) lens
In our prototype camera
Lens is f /4
12 x 12 pixels under each microlens
Theoretically refocus within depth of field of an f/48 lens

53. Light Field Photo Gallery
Okay, now some more photographic results.

54. Stanford Quad

55. Rodin’s Burghers of Calais

56. Palace of Fine Arts, San Francisco

57. Palace of Fine Arts, San Francisco

58. Waiting to Race

59. Start of the Race

60. Summary of Main Contributions
Formal theorem about relationship between light fields and photographs
Computational application gives asymptotically fast refocusing algorithm
Theoretical application gives analytic solution for limits of refocusing

61. Future Work Apply general signal-processing techniques
Cross-fertilization with medical imaging

62. Thanks and Acknowledgments
Collaborators on camera tech report
Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz and Pat Hanrahan
Readers and listeners
Ravi Ramamoorthi, Brian Curless, Kayvon Fatahalian, Dwight Nishimura, Brad Osgood, Mike Cammarano, Vaibhav Vaish, Billy Chen, Gaurav Garg, Jeff Klingner
Anonymous SIGGRAPH reviewers
Funding sources
NSF, Microsoft Research Fellowship, Stanford Birdseed Grant

63. Questions?
“Start of the race”, Stanford University Avery Pool, July 2005