1 Remote Engineered Super Resolved Imaging Zeev Zalevsky Faculty of Engineering, Bar-Ilan University, Ramat-Gan, Israel

2 Outline Introduction The “ SW Adaptation ” Process Diffractive type Super Resolution Geometrical type Super Resolution Hearing with light Conclusions

3 Outline Introduction The “ SW Adaptation ” Process Diffractive type Super Resolution Geometrical type Super Resolution Hearing with light Conclusions

4 Introduction-Diffraction Limitation What is Resolution?  Resolution is finest spatial feature that an imaging system can resolve.  Resolution of optical systems is restricted by diffraction (Lord Rayleigh, Abbe), by the geometry of the detector and by the noise equivalence of its pixels.

5 Introduction-Diffraction Limitation Diffraction limitation of resolution is proportional to the F number of the imaging optics.

6 Introduction- Geometrical Limitation Geometrical resolution is limited by the number of detector ’ s pixels and their size.

7 Introduction- Noise Equivalent Resolution Noise equivalent resolution is originated by the internal noises existing within each pixel of the detector (electronic noises, shot noises etc).

8 Outline Introduction The “ SW Adaptation ” Process Diffractive type Super Resolution Geometrical type Super Resolution Hearing with light Conclusions

9 SW Adaptation Process If not resolved, is it hopeless? Types of a priori information: A single dimensional object Polarization restricted information Temporally restricted signal Wavelength restricted signal Object shape No, if A Priori information on the object is available!!!

10 SW Adaptation Process- cont. The Suggested Solution: Having a priori knowledge of the signal may lead to super resolution using an SW (space-bandwidth) adaptation process: Adapt the SW of the signal to the acceptance SW of the system acceptance SW of the system

11 Outline Introduction The “ SW Adaptation ” Process Diffractive type Super Resolution Geometrical type Super Resolution Hearing with light Conclusions

12 Diff. SR- Time Multiplexing Time Multiplexing: Conversion of temporal degrees of freedom to spatial domain (diffraction) Obj. Img. Ap. G1 G2 Synchron. moving gratings The structure of the rotated grating (for 2-D S.R. effect) Recent improvements: Automatic synchronization (one grating, transmitted twice) 2-D objects, 2-D gratings Dammann gratings

Diff. SR- Time Multiplexing

14 With clear aperture Without time multiplexing With time multiplexing Diff. SR- Time Multiplexing

15 Remote Diff. SR Projected grating Open aperture Closed aperture Reconstruction

16 Diff. SR- Speckle Projection Closed aperture Reconstruction Coherent Incoherent

17 Remote Diff. SR- via Background Open apertureClosed apertureReconstruction

18 Remote Diff. SR- via background, cont. Closed aperture Reconstruction Background Open aperture Reconstruction Closed aperture sequence

19 Remote Diff. SR- from satellite Numerical simulations Experimental results

20 Remote Diff. SR- via rain/droplets Open apertureClosed aperture Reconstruction

21 Spatial DLP based SR Two possible experimental setups. FOV SR.

22 FOV improvement 3X3 Sensor size image Raw images (different DMD positions)

23 FOV improvement 3X3 Two types of algorithms (LSQR, L1) N = 8 DMDs positions Original FOV = middle square only LSQR N=8 L1 N=8

24 Spatial DLP based SR The experimental setup. Experimental results: (a). Image captured with open iris. (b). Image captured with semi closed iris. (c). Reconstructed super resolved image (demonstrating optical zooming of 3x). (a).(b). (c).

25 Optical SAR- I Left: Schematic sketch of the proposed configuration. Right: The proposed iterative algorithm.

26 Optical SAR- II, Simulations Numerical simulations. (a). Original image. (b). Its reconstruction. (c). The type of reconstruction that is obtained when the phase is wrongly reconstructed. (a).(b).(c). The unwrapped phase of the Fourier of the original image and the reconstructed phase. Left: Original object. Middle: Original object Fourier transform. Right: Blurred image.

27 Optical SAR- III, Experimental results

28 Outline Introduction The “ SW Adaptation ” Process Diffractive type Super Resolution Geometrical type Super Resolution Hearing with light Conclusions

29 Geometrical SR- Intermediate plane mask Reconstruction: (a). Field of view border condition. (b). High resolution mask in the intermediate image plane. (c). Low resolution mask in the intermediate image plane. Mask+sensor are shifted for the micro scanning process (a). High resolution ref. (b). Low resolution image without SR (a).(b). (a). (b).(c).

30 Geometrical SR- Improved Technique (a) A random mask with size of. (b) Optical configuration including the binary mask. The mask has to be in an intermediate imaging position. The mask can be moved only in one direction to get different images each time. (a). (b).

31 Geometrical SR- Improved Technique Improvements: The mask is the only part being shifted (instead of mask+sensor). The SR is achieved, without any spatial loss of information. The reconstruction has a reduced sensitivity to noise. Although the movement of the mask is in 1-D, the obtainable SR is 2-D. The movement of the mask does not have to be in sub pixel steps. The recovery time is improved (the reconstruction process each pixel can be treated separately & simultaneously).

32 Geometrical SR- Simulations vs. SR factor Simulation results depicting the algorithm dependence on the super resolution factor. Image Size: 256  256, Number of images taken during the process = 2×SR factor, percentage of the image covered by the random mask = 50%, Noise variance = (a). Reference image. Low resolution images ((b), (d), (f), (h)) and their corresponding high resolution reconstructions ((c), (e), (g), (i)) for SR factor of 4X4, 8X8, 12X12 and 16x16 respectively. Dependence of the algorithm and the runtime on the super resolution factor. Number of images taken during the process = 2× SR factor, the size of the ring image is 256×256 pixels. It has been obtained by performing matrix inversions. The size of each matrix is 2×SR factor rows, and SR factor columns.

33 Geometrical SR- Experimental results Auto Collimator Folding Mirror Binary Mask Relay Lens Spherica l Mirror Apertur e Stop Detector Auto Collimator Manual micromete r The experimental setup

34 Geometrical SR- Experimental results Upper row: The left image: Central part of a low resolution image. The right image: Resulting reconstructed higher resolution image. Lower row: The cross sections.

35 Outline Introduction The “ SW Adaptation ” Process Diffractive type Super Resolution Geometrical type Super Resolution Hearing with light Conclusions

36 Hearing with Light: Features The ultimate voice recognition system compatible to “ hear ” human speech from any point of view (even from behind). There is no restriction on the position of the system in regards to the position of the sound source. Capable of hearing heart beats and knowing physical conditions without physical contact for measuring. Opto-Phone: Hearing with Light

37 Features- cont. Works clearly in noisy surroundings and even through vacuum. Allows separation between plurality of speakers and sounds sources. Works through glass window. Simple and robust system ( does not include interferometer in the detection phase). Opto-Phone: Hearing with Light

38 Opto-Phone: Hearing with Light

39 Let ’ s listen … from 80m Heart beat pulse taken from a throat Cell phone Counting…1,2,3,4,5,6 Face (profile) Counting…5,6 Back part of neck Counting…5,6,7 All recordings were done in a very noisy constriction site at distance of more than 80m.

40 Results: Detection of occluded objects I (a). Camouflaged object. (b). Camouflage without the object. (c). The object (upper left part) and the low resolution camouflaged scenery. (a).(b).(c). (d). The spectrogram of the camouflaged object with its engine turned on. (e). The spectrogram of the object with its engine turned on and without the camouflage. (f). The spectrogram of the camouflaged object without turning on its engine. (d). (e). (f).

41 Results: Detection of occluded objects II (a). The scenario of the experiment. (b). Experimental results: upper recording is of the camouflaged subject. Lower recording is the same subject without the camouflage. (a).(b).

42 Outline Introduction The “ SW Adaptation ” Process Diffractive type Super Resolution Geometrical type Super Resolution Hearing with light Conclusions

43 Conclusions: Resolution of optical system is restricted by various terms. SW Adaptation process is a useful tool for designing super-resolution systems. A generalization for handling more types of resolution restrictions was introduced for large variety of applications. Examples of achieving super resolution effects were viewed. New approach for “hearing” with light was demonstrated.