Extended Depth of Field For Long Distance Biometrics Shree K. Nayar Columbia University ONR MURI Annual Meeting October 28, 2011 Graduate Students: Oliver Cossairt, Daniel Miau, Sophia Li

Depth Effects: Magnification and Blur Depth Range = 100 m Sensor Lens 200 mm 5 mm FL = 2 m Object Distance = 300 m Captured Image

Extending Depth of Field: Approaches Focal Sweep Cameras [Hausler ‘72] [Nagahara et al. ’08] Sensor Lens Focal Plane Diffusion Coding [Cossairt et al. ‘10] Lens Sensor Diffuser

Approach1: Focal Sweep Lens Scene Sensor Final PSF Instantaneous PSF [Hausler ’72] [Nagahara et al. ’08] Lens Scene Sensor Final PSF Instantaneous PSF t = 1 + + + + + + = One way of extending DOF that was explored here in the lab is to utilize sensor motion during exposure The idea is to move the sensor during exposure so that every object is in focus at least one instant. The result is a PSF that is depth-invariant and preserves high frequencies. t = 2 t = 3 t = 4 t = 5 t = 6 t = 7

Focal Sweep PSF: Depth Invariance PSFs for Different Depths

Focal Sweep Experiments with Telescope Meade LX200 Schmidt-Cassegrain 8’’ Telescope System 75 m 50 m Conventional image 75 m 50 m Focus sweep

Focal Sweep without Moving Parts? With motion x x We can achieve the same effect without motion if we can use an optical device to distribute the ray energy in exactly the same way

Focal Sweep without Moving Parts? Can we achieve the performance of focal sweep without moving parts? With motion x x Lens We can achieve the same effect without motion if we can use an optical device to distribute the ray energy in exactly the same way Optical Device Without motion x x ? Lens

Approach 2: Diffusion Coding Conventional Diffuser w Sensor Diffuser Light ray x Radially Symmetric Diffuser Lens Sensor These optical devices exist – they are called diffusers. Diffusers are typically random surfaces whose statistics are chosen to give the desired scattering profile In our case, we were interested in a design that only scatters light along radial lines, so we designed a radially symmetric diffuser

Diffusion Coding: Performance Deblurring Error vs. Depth Focus Sweep Diffusion Coding Deblurring Error noise Depth Similar performance to focal sweep without moving parts

Diffusion Coding: Implementation Diffuser surface profile Diffuser scatter function r (mm) RPC Photonics [www.rpcphotonics.com] Diffuser height map

Diffusion Coding: Experiments Experimental Setup Fabricated Diffuser Canon 50mm EF lens Canon 450D Sensor Measured PSFs depth Without diffuser With diffuser

Face Detection with Diffusion Coding Conventional Camera (F:2.0) Diffusion Coding Camera (F:2.0)

Towards a Diffusion Coded Telescope Diffusion Coding for Long Range EDOF Imaging ?

Diffusion Coded Telescope: Optical Design Diffuser Annular Aperture Mirror 2 Mirror 1 Sensor 8” dia 80” Focal Length

Work in Progress Telescope Diffuser Diffuser Design Issues 8” dia. Telescope Diffuser Diffuser Design Issues Focal Length = 2000 mm Diffusion Angle < 50 μradians Fabrication Issues 8” Diameter Optical Element Surface Thickness < 30 μm Precision Optical Alignment

Depth of Field and Biometrics Conventional Camera Captured Image Sensor Lens Depth 1 Depth 2 Depth 3 Depth 4 Depth 5 Image Database Verification Classifier Same or Different? Brad Pitt

Verification Through Attributes Attribute-Based Face Verification Verification Through Attributes Images Low-level features Attributes Verification RGB HOG LBP SIFT … + - Dark hair Male Asian Round Jaw Different RGB HOG LBP SIFT … + - [Kumar et al. 2009]

EDOF Face Verification: Experiment Sensor Ground Truth Simulated Image Depth 1 Depth 2 Depth 3 Depth 4 Depth 5 Sensor Lens Conventional Camera Simulated Image Depth 1 Depth 2 Depth 3 Depth 4 Depth 5 Sensor Pinhole Pinhole Camera Simulated Image Depth 1 Depth 2 Depth 3 Depth 4 Depth 5 Lens Focal Sweep Camera Simulated Image Sensor Depth 1 Depth 2 Depth 3 Depth 4 Depth 5

EDOF Face Verification: Experiment For each camera type, simulate 5 different depth locations for all 200 people in pubfig database For each person, generate 20 positive matches and 200 negative matches Test verification accuracy using attribute classifier trained on LFW database [Kumar et al. ‘09]

EDOF Face Verification: Results Ground Truth (800mm EFL F/10, No Blur, Zero Noise) 1 0.75 True Positive Rate Depth = 100 m Depth = 136 m Depth = 163 m Depth = 190 m Depth = 218 m 0.5 0.25 0.25 0.5 0.75 1 False Positive Rate

EDOF Face Verification: Results Conventional Camera (800mm EFL F/10) conventional 1 0.75 True Positive Rate Depth = 100 m Depth = 136 m Depth = 163 m Depth = 190 m Depth = 218 m 0.5 0.25 0.25 0.5 0.75 1 False Positive Rate

EDOF Face Verification: Results Pinhole Camera (800mm EFL F/10) focalsweep 1 0.75 True Positive Rate Depth = 100 m Depth = 136 m Depth = 163 m Depth = 190 m Depth = 218 m 0.5 0.25 0.25 0.5 0.75 1 False Positive Rate

EDOF Face Verification: Results Focal Sweep Camera (800mm EFL F/10) 0.25 0.5 0.75 1 conventional True Positive Rate Depth = 100 m Depth = 136 m Depth = 163 m Depth = 190 m Depth = 218 m False Positive Rate

The Problem of Magnification Lens Focal Sweep Camera Sensor Depth 1 Depth 2 Depth 3 Depth 4 Depth 5

The Problem of Magnification Lens Focal Sweep Camera Sensor depth 1 depth 2 depth 3 depth 4 depth 5 Depth 1 Depth 2 Depth 3 Depth 4 Depth 5 Decreasing Performance

Focal sweep produces a depth-invariant PSF Input Focal sweep produces a depth-invariant PSF

Generalized Focal Sweep Depth 1 Depth 2 Depth 3 Depth 4 Depth 5 PSF Input Generalized focal sweep preserves energy at smaller magnifications

Generalized Focal Sweep: Motion Trajectories Sinusoid Wave Triangle Wave 5 Hz: Position Time Time Sinusoid Wave Triangle Wave 25 Hz: Position Time Time Blue: input trajectory signal; Red: monitored trajectory signal

Generalized Focal Sweep: Telephoto System BEI Kimco Voice Coil Motor Elmo Motion Control Driver Lumenera 1/3’’ CMOS Sensor Power Supply Positioning Stage Canon 800mm EFL Lens

Future Work Diffusion Coded Telescope Fabricate 8” diameter Diffuser Design Experiments: Long Range EDOF Videos Generalized Focal Sweep Develop Generalized Focal Sweep Theory Implement Generalized Focal Sweep Camera System EDOF Face Verification Experiments Optimize Motion Trajectory for Face Verification