camera surface reference images desired ray ‘closest’ ray focal surface ‘closest’ camera Light Field Parameterization We take a non-traditional approach to computer graphics modeling and rendering, in which a scene is represented by a collection of images rather than the geometry and surface properties used in typical computer graphics. Essentially, we treat a collection of images as a database of rays. New views can be constructed from this database on a ray-by-ray basis by selecting the closest ray to each desired ray. Image-Based Synthetic Aperture Rendering Interface Solution (PCI) Interface Solution (PCI) Sensor Pod - A BAB C A C D B D C A C D B D Motherboard FIFO Address Data Mux/ Logic Motherboard Interleaving Address Sensor Pod - A Sensor Pod - B Data 32 bit Sensor Pod CMOS Sensor (on board A/D) Control FPGA Logic SDRAM Address Data Address Data Address Project Goals Approach Real-time Acquisition Low Cost Acquisition Display Technology MIT PIs: Prof. Leonard McMillan (MIT LCS), Prof. Julie Dorsey (MIT LCS), and Dr. Hiroshi Murase (NTT) The primary focus of our research effort is to develop technology to create virtual experiences that will approach the fidelity of the real world. In the future, such technologies will have a dramatic impact on the way we work and play. They will enable new forms of commerce, bring together individuals separated by large distances, and provide us with new forms of entertainment. Our dynamically reparameterized light field representation allows us to synthesize images with photographic effects such as variable focus and depth-of-field. Depth- of-field effects are created by varying the extent of the reconstruction filters used on the camera surface. A variable focal length can be simulated by varying the focal plane used in the reconstruction process. In a synthetic aperture camera both the aperture and focal- length settings can be varied from pixel to pixel. The allows effects that are impossible with a traditional camera. Ultimately we intend to create a device for capturing and processing dynamically reparameterized light fields in real-time. We call this device a synthetic aperture camera array. It is composed of a two-dimensional array of randomly accessible image sensors that memory-mapped in the address space of a host processor. Such a system will allow images to be synthesized from a wide range of virtual camera positions in real-time. We plan to support multiple simultaneous video streams to support stereoscopic display as well as multiple viewers. A high-level block diagram of our proposed system is shown above. The camera’s host interface will be an industry standard personal computer bus. The camera array will be constructed from modular sensor units mounted on a common motherboard. The addressing of sensor modules will be interleaved in order to maximize the communication bandwidth between the image sensors and the host computer. Each sensor pod contains a CMOS image sensor, buffer memory, and glue logic. The multi- frame buffer memory is used for two functions. It is used to store information for noise cancellation, and it allows the host to access image rays asynch- ronous to the image scanning process. This modular design approach will allow us to upgrade to higher resolution sensors as they become available. We have also prototyped two low- cost devices for acquiring light fields. We have developed two acquisition systems for acquiring light fields of static scenes. The first uses a robotic XY-platform to move a digital camera. This system allows us to explore the trade-offs between camera spacing and resolution in order to estimate the per- formance of our camera array. This system uses a precision image sensor, precision optics, and a motion platform with a travel distance of approximately one meter squares. It can acquire a 16 by 16 image light field in under 20 minutes, and it cost approximately $10,000 US to construct. Our second system is based on an off-the-shelf flat bed scanner, and an array of plastic lenses. We have modified the scanner to operate off of battery power so that this system can be taken out into the field to acquire images. Additional processing is required to correct for shortcoming in the image sensor and low cost optics. None the less, the system can acquire an 8 by 12 image light field in under 3 minutes, and cost under $100. We have also developed techniques for direct auto- stereoscopic viewing of our light fields. These methods are similar to various lenticular techniques for viewing stereo images. Our synthetic aperture generation ap- proach provides much greater flexibility than tradition optical approaches. In particular it can overcome many limitations such a focus control and skewed frustums. We have demonstrated viewers with true parallax (both horizontal and vertical), and variable controlled focus. Our displays have nearly all of the desired properties of holograms, yet they are true color and viewable under normal lights. Furthermore, the technology is easily adaptable to the display of dynamic 3-D images. Currently we are only limited by the resolution of flat panel displays. The image on the left, when viewed through a hexagonal lens array, can be seen as a three- dimensional image of a flower. It can be simul- taneously seen by multiple viewers. It was computed from a dy- namically re- parameterized light field, which allows us to precisely control the focus at all viewing angles. The inset provides a magnified view of the image.