Course 3: Computational Photography Ramesh Raskar Mitsubishi Electric Research Labs Jack Tumblin Northwestern University Course WebPage :

Course 3: Computational Photography Course WebPage Course Evaluation /courses_evaluation

Welcome Understanding Film-like PhotographyUnderstanding Film-like Photography –Parameters, Nonlinearities, Ray-based concepts Image Processing and Reconstruction ToolsImage Processing and Reconstruction Tools –Multi-image Fusion, Gradient domain, Graph Cuts Improving Camera PerformanceImproving Camera Performance –Better dynamic range, focus, frame rate, resolution Future DirectionsFuture Directions –HDR cameras, Gradient sensing, Smart optics/lighting

Goals –Review of 30+ recent papers –Understand computational aspects of cameras Discuss issues in traditional camerasDiscuss issues in traditional cameras Explore alternative imaging methodsExplore alternative imaging methods Learn vision and optics techniquesLearn vision and optics techniques –Discuss image processing and reconstruction tools –What we will not cover Film Cameras, Novel view rendering (IBR), Color issues, Traditional image processing/editingFilm Cameras, Novel view rendering (IBR), Color issues, Traditional image processing/editing

Speaker: Jack Tumblin Jack Tumblin is an Assistant Professor of Computer Science at Northwestern University. His interests include novel photographic sensors to assist museum curators in historical preservation, computer graphics and visual appearance, and image-based modeling and rendering. During his doctoral studies at Georgia Tech and post-doc at Cornell, he investigated tone-mapping methods to depict high-contrast scenes. His MS in Electrical Engineering (December 1990) and BSEE (1978), also from Georgia Tech, bracketed his work as co-founder of IVEX Corp., (>45 people as of 1990) where his flight simulator design work was granted 5 US Patents. He is an Associate Editor of ACM Transactions on Graphics, was a member of the SIGGRAPH Papers Committee (2003, 2004), and in 2001 was a Guest Editor of IEEE Computer Graphics and Applications.

Speaker: Ramesh Raskar Ramesh Raskar is a Senior Research Scientist at MERL. His research interests include projector-based graphics, computational photography and non-photorealistic rendering. He has published several articles on imaging and photography including multi-flash photography for depth edge detection, image fusion, gradient-domain imaging and projector-camera systems. His papers have appeared in SIGGRAPH, EuroGraphics, IEEE Visualization, CVPR and many other graphics and vision conferences. He was a course organizer at Siggraph 2002, 2003 and He is a panel organizer at the Symposium on Computational Photography and Video in Cambridge, MA in May He is a member of the ACM and IEEE.

Opportunities –Unlocking Photography How to expand camera capabilitiesHow to expand camera capabilities Digital photography that goes beyond film-like photographyDigital photography that goes beyond film-like photography –Opportunities Computing corrects for lens, sensor and lighting limitationsComputing corrects for lens, sensor and lighting limitations Computing merges results from multiple imagesComputing merges results from multiple images Computing reconstructs from coded image samplesComputing reconstructs from coded image samples Cameras benefit from computerized light sourcesCameras benefit from computerized light sources –Think beyond post-capture image processing Computation well before image processing and editingComputation well before image processing and editing –Learn how to build your own camera-toys

Traditional Photography Lens Detector Pixels Image

Computational Photography: Optics, Sensors and Computations Generalized Sensor Generalized Optics Computations Picture 4D Ray Bender Upto 4D Ray Sampler Ray Reconstruction

Computational Photography Novel Cameras Generalized Sensor Generalized Optics Processing

Computational Photography Novel Illumination Novel Cameras Generalized Sensor Generalized Optics Processing Light Sources

Computational Photography Novel Illumination Novel Cameras Scene : 8D Ray Modulator Generalized Sensor Generalized Optics Processing Light Sources

Computational Photography Novel Illumination Novel Cameras Scene : 8D Ray Modulator Display Generalized Sensor Generalized Optics Processing Recreate 4D Lightfield Light Sources

Computational Photography Novel Illumination Novel Cameras Scene : 8D Ray Modulator Display Generalized Sensor Generalized Optics Processing 4D Ray Bender Upto 4D Ray Sampler Ray Reconstruction Generalized Optics Recreate 4D Lightfield Light Sources Modulators 4D Incident Lighting 4D Light Field

A Teaser: Dual Photography Scene Photocell Projector

A Teaser: Dual Photography Scene Photocell Projector

A Teaser: Dual Photography Scene Photocell Projector

A Teaser: Dual Photography Scene Photocell Projector Camera

camera The 4D transport matrix: Contribution of each projector pixel to each camera pixel scene projector

camera The 4D transport matrix: Contribution of each projector pixel to each camera pixel scene projector Sen et al, Siggraph 2005

camera The 4D transport matrix: Which projector pixel contribute to each camera pixel scene projector Sen et al, Siggraph 2005 ?

Dual photography from diffuse reflections the camera’s view Sen et al, Siggraph 2005

Camera Obscura, Gemma Frisius, A Brief History of Images

Lens Based Camera Obscura, A Brief History of Images

Still Life, Louis Jaques Mande Daguerre, A Brief History of Images

Silicon Image Detector, A Brief History of Images

A Brief History of Images Digital Cameras

Dream of A New Photography Old New People and Time ~Cheap Precious Each photo Precious Free Lighting Critical Automated* External Sensors No Yes ‘Stills / Video’ Disjoint Merged Exposure Settings Pre-select Post-Process Exposure Time Pre-select Post-Process Resolution/noise Pre-select Post-Process ‘HDR’ range Pre-select Post-Process

Survey How many of you are photographers ?How many of you are photographers ? How many of you are photo-artists ?How many of you are photo-artists ? How many of you are involved in camera design/development ?How many of you are involved in camera design/development ? How many do active programming ?How many do active programming ? Field of work: Academics? Industry ? Research ? Art ?Field of work: Academics? Industry ? Research ? Art ?

8:30 Introduction (Raskar) 8:40 Photographic Signal & Film-like Photography (Tumblin) 9:10 Image Processing Tools (Raskar) 9:40 Improving Film-like Photography (Tumblin) 10:15 Break 10:30 Image Reconstruction Techniques (Raskar) 11:15 Smart Lights and Beyond Photography (Tumblin) 11:45 Smart Optics and Sensors (Raskar) 12:05 Discussion Schedule Course Page :