Jitter Camera: High Resolution Video from a Low Resolution Detector Moshe Ben-Ezra, Assaf Zomet and Shree K. Nayar IEEE CVPR Conference June 2004, Washington DC, USA

Video Resolution MiniDV Camera. Resolution: 720x480 Digital Camera. Resolution: 2592x1944 Plasma Display Resolution 1366x768

Fundamental Resolution Tradeoff Spatial resolution (pixels) Temporal resolution (fps) K 720x480 Conventional video camera M 2048x1536 Hi-resolution still Camera

Super-Resolution Sequence taken by a moving camera High-Resolution computed image Super-Resolution Shechtman, Caspi, and Irani ECCV2002 Zomet and S. Peleg. ICIP2000 Baker and Kanade. CVPR2000 Chiang and Boult, IVC2000 Capeland, Zisserman ICPR2000 Elad and Feuer IP1997 Irani and Peleg GMIP1996

What is Super Resolution?

What is Super Resolution S 11 S 12 S 21 S 22 S = S 11 + S 12 + S 21 + S 22

Super Resolution All Sampled Images y = (D G)x + z Decimation Blurring Op. Hi Res. Image Noise

Motion Blur Hurts Us Again!

Capture Images without Motion Blur

Effect of Motion Blur on Super-Resolution Input: No Motion BlurSuper-Resolution Result Input : With Motion Blur (known)Super-Resolution Result

Quantifying The Affect of Motion Blur Empirical tests: RMS error. Volume of Solutions (Linear Model): High-Resolution Image Noise (Quantization) Input Images Volume of Solutions 1/det(A ) Blur & Decimation Baker and Kanade

How Bad is Motion Blur for Super-Resolution? Space of Super-Resolution Solutions Motion blur in pixels RMS Error After Super-Resolution Motion blur in pixels

Avoid Motion Blur using Jitter Sampling Conventional Sampling Time Space Spatial Jitter Sampling Time Space

The Jitter Camera LensDetector Micro-actuator

The Jitter Camera LensDetector Micro-actuator Detector is a light weight device! Jitter is instantaneous and synchronous

Computer Controlled X Micro-Actuator Computer Controlled Y Micro-Actuator Board Camera Lens

1μm1μm X Pixels Y Pixels Jitter Mechanism Accuracy Desired locations. Actual locations.

Result: Resolution Chart Super-Resolution Image Four Images from the Jitter Camera

De-Mosaic Artifacts around edges

Result: Color DeMosaicing and Super-Resolution 1 (out of 4) Jitter camera Image Super-Resolution

Jitter Video (Stabilized) How can we handle dynamic scenes?

Adaptive Super-Resolution for Dynamic Scenes Static blocks: 4 frames used. Occlusions: 1 frame used. Moving object: frames used

I-3 I-2 I-1 I I+1 I+2 I+3 Adaptive Super-Resolution Algorithm 1. Estimate the aliasing error ‘  ’ (stdv) for each block I k in I. 2. Compute robust block matching between all pairs {I}{I  1,2,3}. Use ‘  ’ as a scale factor for an M-Estimator error function. 3. For each block I k try to find 3 matching blocks {I  x} k, s.t. : a) SSD(I k, {I  x} k ) -0.5 < 3  b) {I  x} k are temporally closest to I k (smallest x) 4. Apply super-resolution to the selected blocks. The algorithm degrades gradually from 4-frames super- resolution to a single frame interpolation and deblurring.

Scale Estimate Mean 6.4, Stdv 14 Mean 10.5, Stdv 27 Mean 7.5, Stdv 16 Mean 15.2, Stdv 30 Mean 8.6, Stdv 17 Mean 17.7, Stdv 33 Low Res - Hi-Res Aliasing Error (Simulated) Low Res 2 nd derivative (Simulated)