1. Lensless Imaging Richard Baraniuk Rice University Ashok Veeraraghavan
Aswin Sankaranarayanan
CMU
John Rogers
UIUC
Richard Baraniuk
Rice University

2. Re-Imagining Imaging Conventional camera design
based on human visual system model
objective lens: directs a cone of light into the camera
1-to-1 correspondence between scene points and camera pixels makes imaging easy
Inspired by the discovery of light sensing opsin in the skin of cephalopods, we are developing new kind of camera architectures based on distributed light sensing.
Such new cameras promise exciting new form factors (eg: flat, flexible)

3. Re-Imagining Imaging Conventional camera design
based on human visual system model
objective lens: directs a cone of light into the camera
1-to-1 correspondence between scene points and camera pixels makes imaging easy
Goal: A large, potentially flexible, imaging platform capable of distributed acquisition of light fields
inspired by distributed light sensing in cephalopod skin
Approach: Lensless imaging
leverage recent progress in coded aperture and compressive sensing
exciting opportunities for flat and flexible cameras
Inspired by the discovery of light sensing opsin in the skin of cephalopods, we are developing new kind of camera architectures based on distributed light sensing.
Such new cameras promise exciting new form factors (eg: flat, flexible)

4. Problem With incoherent light, no phase information
all photo-detectors measure roughly the same information, the average light level of the scene
sensor
scene
photo-detector 1
photo-detector 2

5. Solution Add a mask in front of the sensor/photo-detector
attenuates certain rays of light
How does this help?
same scene point is attenuated differently at different photo-detectors
each photo-detector sees a different linear combination of the scene points
can design mask(s) such that we can recover a high-resolution version of the scene (compressive sensing)
mask
sensor
scene
photo-detector 1
photo-detector 2

6. Mask Design Random mask
rich theory and algorithms available from compressive sensing
provable recovery bounds
sparse signal
measurements
nonzero
entries
super sub-Nyquist measurement

7. Mask Design Random mask
rich theory and algorithms available from compressive sensing
provable recovery bounds
major impact in a variety of DOD and industrial sensing systems: medical, radar, sonar, hyperspectral, IR, THz imaging, …

8. f( ) = X Simulations 10 units 1/10 unit mask(s) sensor scene X
sensor measurements
mask = f( ) X
image
noiseless system
PSNR = 19dB
noisy system
PSNR = 15dB

9. Planar Prototype Sensor: Flea3 Point Grey camera,
1024x1280 pixels (5.3µm each)
Mask: Random binary mask (1) 135x135 features (85µm each) 10% of mask “pixels” transparent
Target projected on screen 15cm from camera (target height/width 24cm)
sensor-mask assembly
walls to limit FOV
mask
target on monitor
gap (0.5mm)
mount
sensor
sensor image

10. Planar Prototype Results 1
target on screen
recovered image
Sensor: Flea3 Point Grey camera,
1024x1280 pixels (5.3µm each)
Mask: Random binary mask (1) 135x135 features (85µm each) 10% of mask “pixels” transparent
Target projected on screen 15cm from camera (target height/width 24cm)
Reconstruction: Least-squares
Ongoing: High-resolution reconstruction leveraging sparsity and priors on scene structure

11. Planar Prototype Results 2
target on screen
recovered image
Sensor: Flea3 Point Grey camera,
1024x1280 pixels (5.3µm each)
Mask: Random binary mask (1) 270x270 features (42µm each) 10% of mask “pixels” transparent
Target projected on screen 15cm from camera (target height/width 24cm)
Reconstruction: Least-squares
Ongoing: High-resolution reconstruction leveraging sparsity and priors on scene structure

12. Color Images

13. From Challenge … With a planar sensor, must limit the field of view
Outside field of view, image recovery becomes increasingly ill-posed
scene X
sensor
mask(s)

14. … To Opportunity
Without the need for a lens, we can make the mask and sensor curved
Example: (Hemi)spherical camera
180/360 degree field of view
no field-of-view ill-posedness!
mask(s)
sensor array
with John Rogers

15. … To Opportunity
Without the need for a lens, we can make the mask and sensor curved
Example: (Hemi)spherical camera
180/360 degree field of view
no field-of-view ill-posedness!
spherical
sensor array
scene projected on sphere
un-warped recovered image

16. Simulations additive noise noiseless scene 𝜎=0.1 𝜎=0.5 𝜎=1 512
256
256
256
128
128
128
512
Number of pixels in reconstruction

17. Spherical Prototype
Sensor: White paint on a spherical shell acts as diffuser
Mask: random binary mask 68x68 features (170µm each) 10% of mask “pixels” transparent
Target projected on screen 18cm away (target height/width 24cm)
plastic shell with diffuser inner surface
(proxy for a spherical sensor)
current prototype uses a Grasshopper point grey camera to capture image formed on a 1.2 cm2 area (400x400 pixels)
planar mask in front of shell
(flexible PDMS masks on shell)

18. Spherical Prototype Results
Sensor: White paint on a spherical shell acts as diffuser
Mask: random binary mask 68x68 features (170µm each) 10% of mask “pixels” transparent
Target projected on screen 18cm away (target height/width 24cm)
Reconstruction: Least-squares
Ongoing: High-resolution reconstruction leveraging sparsity and priors on scene structure Spherical photo-detector to improve SNR
targets on screen
recovered images
32x32 recovered images

19. Planned Research flat, flexible, (hemi)spherical cameras
Radically new kinds of cameras
flat, flexible, (hemi)spherical cameras
beyond visible (IR, THz, …)
numerous potential DOD and industrial applications
New theory and algorithms
mask design (light throughput versus invertibility)
dynamic masks
new recovery algorithms needed for 0/1 masks
Can perform exploitation directly on compressive measurements (detection/classification, etc.) without numerical scene reconstruction
Sensing light fields instead of images