Real-Time Phase-Stamp Range Finder with Improved Accuracy Akira Kimachi Osaka Electro-Communication University Neyagawa, Osaka , Japan 1August 2, 2009Optical Engineering + Applications, San Diego Convention Center

Outline Introduction Phase-stamp range finder (PSRF) –Time-domain correlation image sensor (CIS) –Phase-stamp imaging (PSI) –Problem of artifacts Accuracy improvement by calibrating the CIS for PSI –Experiment #1: CIS output behavior in PSI –CIS output model for PSI –Compensation for phase stamp errors –Experiment #2: accuracy evaluation Conclusions 2

Real-time range imaging Demands –Assembly/inspection of industrial products –Environment recognition for robots/vehicles –Observation of mobile objects –Assistance in human workspace Active vs. passive range finders –Active methods are preferable in terms of accuracy/reliability 3 # of sensors Active illumination Accuracy /reliability Methods Active1YesHigh Light sectioning Structured light Time-of-flight Time-stamp Phase-stamp Passive> 1NoLow Binocular stereo Multiple-camera

Real-time active range finders Robustness vs. depth resolution –Difficult to establish both 4 MethodSensor Depth resolution Robustness to disturbance Ambient illumination Surface texture Temporal variation Spatial variation Light sectioning High-speed camera < 1 mmLow VLSI sensor< 1 mmLowHighLow Structured light High-speed camera < 1 mmLowHigh Time-of-flight (TOF) VLSI sensor> 1 mmHigh Axi-Vision Camera > 1 mmHigh Time-stampVLSI sensor< 1 mmLow Phase-stamp Correlation image sensor < 1 mmHigh

flat 400 mm Objective Phase-stamp range finder (PSRF) Kimachi and Ando, Electronic Imaging (2007) –Time-domain correlation image sensor (CIS) –Frame-rate 3D capture based on “phase stamp” imaging (PSI) Problem of artifacts Solution –Analyze the behavior of CIS outputs in PSI –Model the CIS outputs for PSI –Calibrate the PSRF by compensating for CIS output errors 5 undulation ~ 3.5 mm rms random noise pattern ~ 1 mm rms

Correlation image sensor (CIS) 6 Ando and Kimachi, Trans. IEEE ED (2003) Output images Average intensity 200x200-pixel CMOS camera Temporal correlation : frame integral

frame Phase-stamp imaging (PSI) 7 Light pulse energy image Phase stamp image Correlation images uncorrelated noise Higher temporal resolution than the frame period

Phase-stamp range finder (PSRF) 8 SOL energy ambient illumination surface reflectance (SOL) Incident light intensity Reference signals SOL angle Phase stamp Correlation images Range image : pixel spacing in x One SOL scan in one frame Based on PSI Kimachi and Ando, Electronic Imaging (2007) Range image from single frame Ambient illumination removed Surface reflectane canceled

Experimental PSRF system 9 CIS200x200-pixel Frame rate12.5 fps Reference signal frequency 50 Hz Mirror scanning rate 25 Hz Laser DPSS ， 40 mW ， 658 nm Camera lens25 mm F1.4

PSRF artifacts Artifacts in range images –Undulation –Random noise pattern Considered to be caused by errors in detected phase stamps Temporal correlations may not follow the ideal characteristics with respect to SOL incidence time 10 phase stamprange mapaverage intensitySOL energy (flat 400 mm) 0   /2  /2

Investigating CIS outputs in PSI Capture a sequence of correlation images while shifting pulse occurrence time 11 Pulse occurrence time0~20 ms (1 ms step) Pulse height0~8, 9 levels Pulse width0.2~2 ms (0.2 ms step) Reference signal amplitude0.05~0.5 V (0.05 step)

Results: image average behavior 12 Distortions in from sine functions become severer as increases Image averages of temporal correlations increase monotonically with,, and Pulse width varied Pulse height variedAmplitude varied Undulations in the phase stamp become severer as increases

Results: pixel-wise deviation behavior 13 0   /2  /2 Pulse width varied Pulse height variedAmplitude varied Pixel-wise deviations of temporal correlations increase monotonically as,, and Pixel-wise deviations of computed phase stamps oscillate with oscillate with in arbitrary waveforms, randomly with respect to pixel and channel

Ideal characteristic of CIS temporal correlation outputs Experiment-based CIS output model for PSI –Undulation/distortion → Harmonics,, –Pixel-/channel-wise random deviation → Coefficients,,,, –Dependence on light pulse energy → Multiplication by Coefficients are estimated from a sequence of PSI images by least-squares fitting CIS output model for PSI 14 noise (for fixed )

Compensation for phase stamp errors For a single-frame set of temporal correlation images,, and, 1.Compute the phase stamps 2.Approximate the PSI CIS model by regarding as containing a small error 3.Estimate and pixel-wise by least-squares fitting to the model with and the pre- estimated coefficients,,,, 4.Obtain the phase stamp estimates 15

Results: CIS calibration for PSI (1) 16 test data = calibration data Fitting — over a sequence of Compensation — on a single frame of

Results: CIS calibration for PSI (2) 17 test data = calibration datatest data ≠ calibration data

Results: CIS calibration in PSI (3) 18 test data ≠ calibration data Before compensation After compensation

Results: artifacts removal in PSRF 19 SOL intensityphase 400 mm compensated uncompensated flat board paper box can bottle CompensationBeforeAfter Offset phase [rad]− range [mm] Undulation (rms) phase [rad] range [mm] Random noise pattern (rms) phase [rad] range [mm]

Conclusions Artifacts in PSRF outputs have been removed –CIS outputs were modeled based on PSI experiments –A method for compensating for phase stamp errors in CIS outputs was proposed –Confirmed in experiments Accuracy improved on a PSRF system –Undulation — 3.45 mm → 0.48 mm –Random noise pattern — 1.09 mm → 0.55 mm 20