1. Jitter Camera: High Resolution Video from a Low Resolution Detector Moshe Ben-Ezra, Assaf Zomet and Shree K. Nayar IEEE CVPR Conference June 2004, Washington DC, USA

2. Video Resolution MiniDV Camera. Resolution: 720x480 Digital Camera. Resolution: 2592x1944 Plasma Display Resolution 1366x768

3. Fundamental Resolution Tradeoff Spatial resolution (pixels) Temporal resolution (fps) 30 330K 720x480 Conventional video camera 130 3 3M 2048x1536 Hi-resolution still Camera

4. Super-Resolution Sequence taken by a moving camera High-Resolution computed image Super-Resolution Shechtman, Caspi, and Irani ECCV2002 Zomet and S. Peleg. ICIP2000 Baker and Kanade. CVPR2000 Chiang and Boult, IVC2000 Capeland, Zisserman ICPR2000 Elad and Feuer IP1997 Irani and Peleg GMIP1996

5. What is Super Resolution?

6. What is Super Resolution S 11 S 12 S 21 S 22 S = S 11 + S 12 + S 21 + S 22

7. Super Resolution All Sampled Images y = (D G)x + z Decimation Blurring Op. Hi Res. Image Noise

8. Motion Blur Hurts Us Again!

9. Capture Images without Motion Blur

10. Effect of Motion Blur on Super-Resolution Input: No Motion BlurSuper-Resolution Result Input : With Motion Blur (known)Super-Resolution Result

11. Quantifying The Affect of Motion Blur Empirical tests: RMS error. Volume of Solutions (Linear Model): High-Resolution Image Noise (Quantization) Input Images Volume of Solutions 1/det(A ) Blur & Decimation Baker and Kanade

12. How Bad is Motion Blur for Super-Resolution? Space of Super-Resolution Solutions 0 1 2 3 4 5 Motion blur in pixels RMS Error After Super-Resolution 0 1 2 3 4 5 18 Motion blur in pixels

13. Avoid Motion Blur using Jitter Sampling Conventional Sampling Time Space Spatial Jitter Sampling Time Space

14. The Jitter Camera LensDetector Micro-actuator

15. The Jitter Camera LensDetector Micro-actuator Detector is a light weight device! Jitter is instantaneous and synchronous

16. Computer Controlled X Micro-Actuator Computer Controlled Y Micro-Actuator Board Camera Lens

18. 1μm1μm X Pixels Y Pixels Jitter Mechanism Accuracy Desired locations. Actual locations.

19. Result: Resolution Chart Super-Resolution Image Four Images from the Jitter Camera

20. De-Mosaic Artifacts around edges

21. Result: Color DeMosaicing and Super-Resolution 1 (out of 4) Jitter camera Image Super-Resolution

22. Jitter Video (Stabilized) How can we handle dynamic scenes?

23. Adaptive Super-Resolution for Dynamic Scenes Static blocks: 4 frames used. Occlusions: 1 frame used. Moving object: 2 - 4 frames used

25. I-3 I-2 I-1 I I+1 I+2 I+3 Adaptive Super-Resolution Algorithm 1. Estimate the aliasing error ‘  ’ (stdv) for each block I k in I. 2. Compute robust block matching between all pairs {I}{I  1,2,3}. Use ‘  ’ as a scale factor for an M-Estimator error function. 3. For each block I k try to find 3 matching blocks {I  x} k, s.t. : a) SSD(I k, {I  x} k ) -0.5 < 3  b) {I  x} k are temporally closest to I k (smallest x) 4. Apply super-resolution to the selected blocks. The algorithm degrades gradually from 4-frames super- resolution to a single frame interpolation and deblurring.

26. Scale Estimate Mean 6.4, Stdv 14 Mean 10.5, Stdv 27 Mean 7.5, Stdv 16 Mean 15.2, Stdv 30 Mean 8.6, Stdv 17 Mean 17.7, Stdv 33 Low Res - Hi-Res Aliasing Error (Simulated) Low Res 2 nd derivative (Simulated)