ECEN 4616/5616 Optoelectronic Design Class website with past lectures, various files, and assignments: (The first assignment will be posted here on 1/22) To view video recordings of past lectures, go to: and select “course login” from the upper right corner of the page. Lecture #25: 3/12/14

LightField Photography “Shoot first, Focus later”: “LightField” photography refers to cameras able to capture enough information about an object that the resulting image can be ‘re- focused’ later. The information captured is called the “LightField”.

LightField Photography The lightfield is a geometrical optics concept, whose elements are rays. Consider a lens interacting with rays coming from several objects, all at different distances from the lens: Z In a conventional camera, a detector may be placed at some distance, Z, behind the lens. What the detector records is a 2D record of where the rays strike the detector. If the detector happens to be at the conjugate distance from some object, then an image of that object will be recorded. If the detector and object are not at conjugate distances (that is, related by the imaging equation: then only a blur will be recorded. The only thing we can deduce about the rays from the image is where they ended up.

LightField Photography Suppose that we are able to record, for each ray, not only where it arrives on the detector, but where it was previously: Ray Measurement Plane #1 Ray Measurement Plane #2 We now have 4 measurements for each ray: its 2D position at each of two planes. This is called the “4D Light Field”. Since rays travel in straight lines, we also have all the information we need to calculate where each ray would be at any position, and hence the image that would be captured if a detector were at that position: Calculated image positions Measured ray positions:

LightField Photography Somewhat surprisingly, the concept of the lightfield is over 100 years old. Perhaps the first clear description of the concept was by Gabriel Lippmann, who called it “Integral Photography”: How is this a measure of the 4D light field? Assume that the object, A, is far away compared to the size of the lenslets. Then, each lenslet records 4 things about each ray (at the Z-position of the lenslet array): 1.Its X-Y position at the lenslet plane. (To the precision of the lenslet size.) 2.Its direction of travel, encoded in the x-y position of the recorded position of the ray behind the individual lenslet. Note that knowing the position and direction of a ray is equivalent to knowing it at two positions. Either set of data uniquely determines a straight line in 3D space, and hence uniquely determines a ray.

Lenticular Printing Lipmann planned to display the developed photograph behind the same lenslet array, so that users would see a different perspective, depending on their position and direction of view. (The computational effort to re-focus the light field at another position would have been prohibitive in 1908, as it would have to have been done entirely by hand.) The Lippmann work led to the development of “Lenticular Printing” – the method of producing 3D (and now, short videos) on pieces of paper, using a micro-array of cylindrical lenslets over a set of prepared image slices: Each color represents a different image, separated into slices. Array of cylindrical lenslets (seen edge on). Each lenslet projects each images in different directions: Different images are visible from different directions of view.

Lenticular Printing Here’s another illustration of how each eye (or each viewer position) can see a different picture. Today, you can buy sheets of lenticular material and make your own ‘light- field’ displays using public license software and a printer. Lenticular prints project a 3D ‘slice’ of the 4D lightfield – since they don’t change if the view moves along the long axis of the lenslets.

Lenticular Printing Lenticular printing has developed many uses, such as greeting cards, movie posters, and other advertisements: Here is a lenticular print used to demonstrate a high zoom ratio lens via a table-top poster: Movie Posters:

LightField Photography Today, however, we have the computational capability to fully manipulate the 4D light field: Re-focusing to different planes (or even focusing onto curved surfaces): Curved object 4D Lightfield detector Calculated image plane This flexibility might be used in a microscope, say, for imaging the surface of a cell’s nucleus. There is no other way that such an image could be made.

LightField Photography There have been many ideas (and experiments) on how to measure the light field, but the currently commercially successful ideas are variations on Lippmann’s original design, sometimes called a “Plenoptic” camera instead of a Lightfield camera: Camera Lens Lenslet array Pixel array Exit Pupil Layout of a lightfield detector: 1.Each lenslet in the array images the exit pupil onto the detector. 2.Each image of the pupil (on the detector) represents one pixel in the final image. The resolution of the final image is that of the lenslet array, not the pixel array – the 4D lightfield needs 1 -2 orders of magnitude more information to define than a single image. Each pixel of the image at the detector plane is simply the sum of all the pixels in the exit pupil image for each lenslet.

LightField Photography Re-Focusing the Lightfield: To re-create the image at a different location, extend each ray in the detected lightfield to the Z-location of the virtual detector, and add its value to the (virtual) pixel (lenslet) reached. Real Detector Location Virtual Detector Location RAYS 2 3 Lenslets A B C Detector Arrays Z 1 Exit Pupil Plane a b c A’ B’B’ C’C’ a’ b’ c’ The 4D lightfield can be recreated at the virtual location as well, by adding the rays to the corresponding pixels in the virtual detector: a’, b’, and c’. The image pixel, B’, in the virtual detector is made from rays 1, 2, and 3, whose values are the detector pixels a, b, and c, behind image pixels (lenslets) A, B, and C. Object:

LightField Photography A B C ABC Raw Light Field Image: Light Field Photography with a Hand-held Plenoptic Camera Stanford Tech Report CTSR Sub-images, showing that each ‘pixel’ of the image is made of an image of the exit pupil of the optics: (This paper is an excellent introduction to the optical and algorithmic design issues involved in lightfield photography.)

LightField Photography There are several companies now marketing LightField Cameras.

LightField Photography ‘Natural’ image from summing the pupil images of the raw light-field photograph: “Re-Focused” images produced by combining different pixels from the pupil images: (This is the image that would have resulted if the lenslet array had been replaced by a detector array of the same resolution.)

LightField Photography Re-Focused Image Raw Image: It is not necessary for the pixel arrays to be focused on the exit pupil. One company has marketed a lightfield camera in which the pixel arrays reproduce images of segments of the object.

LightField Photography Re-Focused Image Raw Image:

LightField Photography We’ve been talking about light field detectors made by putting lenslet arrays over higher-resolution digital detectors. From our discussion of pinhole photography, however, it’s evident that the lenslet array could be replaced by a simple pinhole array and also make a lightfield detector, at some cost in sensitivity: Camera Lens Pinhole array Pixel array Exit Pupil

LightField Photography Here is a website which demonstrates how to convert a digital camera to a lightfield camera at low cost (~$5) using a printed pinhole array: Building your own Light Field Camera

Light Field Microscope The light field capture concept would seem to be of great interest to microscopists, as it would allow post-capture exploration of the depth details of the object. The extreme loss of resolution, however, has prevented the technique from being used as more than just a demonstration. Here is a schematic of a confocal scanning microscope: Object is scanned in x-y Scanning Light Field Microscope: Pixelated Detector Instead of an image a one depth, the entire light field is captured over the scan.