General Engineering Research Institute A Real-Time Multi-Sensor 3D Shape Surface Measurement System Using Fringe Analysis By Mohammad Al Sa’d www.megurath.org

Introduction General background Functional requirements of the system Stages of the 3D surface reconstruction process Specifications of the system Hardware design of the system Software design of the system Results

Background (1/2) Fringe Pattern Profilometry Projector Object Optical non-contact 3D surface method Fringes generation Laser Interference structured light projection Measurement precision: from 1μm Depends on the optical resolution of the fringes Fringes width and their optical quality (depth of field, camera resolution and display resolution) Light Wavelength Applications Inspection of components during the production process (turbine blades and circuit boards) Reverse engineering (CAD data from existing objects) Documenting objects of cultural heritage Medical applications: live measuring of human body shape Camera Projected Pattern Image plane patterns are generated by computer and displayed by a projector

Background (2/2) Metrology Guided Radiotherapy Radiation therapy is used since about more than 100 years for the treatment of cancer. The goal is to destruct the cancer cells with minimal radiation damage to the surrounding healthy cells. Pre-treatment stages: 3D planning models are created (CT, MR or others) to accurately guide treatment. Radiation treatment sessions are planned and radiation doses are calculated (dosimetry). Treatment stages: Radiation beam is shaped to precisely hit the target (site of tumour). Radiation is delivered from multiple angles, using the controlled gantry and patient table. Treatment is repeated over multiple sessions. Any small movement (like breathing) or patient’s body changes during successive sessions affect the goal of the treatment. Rotating Gantry The goal is to destruct the cancer cells with minimal radiation damage to the surrounding healthy cells

Functional Requirements 400mm Z Y X Field of View At least 400mm × 400mm × 400mm Spatial Resolution At least 100 x,y points Measurement Error (Accuracy) Not to exceed ±1 mm (according to the tolerance of the dosimteric models used in radiotherapy planning). Dynamic Real-Time Measurement At least FIVE measurements per second (to detect small movements, like breathing) Patients are asked to be static as possible, however movement in their bodies come from breathing

3D Surface Reconstruction Stages (1/5) Fringe Profilometry Analyses Phase Unwrapping Calibration

3D Surface Reconstruction Stages (2/5) Fringe Profilometry Analyses Spatial Fringe Analysis Techniques (modulation phase is generated from a single input image): Fourier Profilometry: Windowed Fourier Profilometry: Processing window passes through the image to find the phase at the centre pixel using forward and inverse FFT. Wavelet Profilometry: Generating phase from the wavelets of the image line- by-line Temporal Fringe Analysis Techniques (modulation phase is generated from multiple input images – at least three images): Phase-Stepping Profilometry: Using least square method to extract the phase. Forward FFT Filter out the fringes Inverse FFT Calculated Phase 1 3 2

3D Surface Reconstruction Stages (3/5) Phase Unwrapping To remove the phase ambiguity (2 modules). Types: Path-Dependent Unwrappers Schafer Unwrapping Algorithm Path-Independent Unwrappers Goldstein's Branch Cut Algorithm Quality-Guided Path Following Algorithm Flynn's Minimum Discontinuity Algorithm Preconditioned Conjugate Gradient (PCG) Algorithm Lp-Norm Algorithm Reliability Ordering Algorithm Synthesis Algorithm Differ in speed and robustness.

3D Surface Reconstruction Stages (4/5) Absolute Height Calibration Unwrapped phase map is converted to real world heights Height calibration process: Triangulation spot (embedded inside the fringe pattern) is detected. Unwrapped phase value at the spot (x,y) location is subtracted from the unwrapped phase map (to generate a relative phase map). Unwrapped phase map is linked to the real world heights via interpolation (using the height calibration volume). To compensate the geometric distortions by optics and perspective. X,Y world coordinates are generated for a number of height steps to generate the traversal calibration volumes X,Y world coordinates are retrieved depending on the correspondent height value of each pixel and using interpolation. Traversal (XY) Calibration

Specifications of the System (1/3) Deliverables (so far!) Speed: 8Hz (using Fourier Profilometry and Goldstein's unwrapper) 5Hz (using Fourier Profilometry and Reliability ordering unwrapper) Field-of-view: (X,Y,Z) = 400mm × 500mm × 400mm Spatial resolution: 262,144 x,y points Multiple sensors: Coverage area around 270° Measurement error: Accuracy around ±0.5 mm Pre-processing Techniques: Noise Reduction and gamma Correction Catalogue of measurement techniques: Ability to select different algorithms User interaction and multi-user modes: GUI to interact with the user Normal and advanced modes (for both metrology experts and normal users) Various operating modes: Online: for real-time measurements Offline: for pre-saved images and videos 3D visualisation and 2D plotting. Various image saving choices. Treatment coach Sensor2 Sensor3 Sensor1

Specifications of the System (2/3) Program snapshots

Specifications of the System (3/3) Program snapshots

Hardware Design of the System (1/3) Hardware Configuration Main Control & Processing Unit Synchronisation Unit Sensor Processing Unit Sensor2 Sensor3 Sensor1 Sensor Processing Unit Sensor Processing Unit

Hardware Design of the System (2/3) Sensor Components: Projector Canon XEED SX60 Conventional LCD projector LCoS projector

Hardware Design of the System (3/3) Sensor Components: Camera Gige Broadcasting Phase-Stepping Triangulation Fourier Controller Gige Camera Prosilica GE1380 GigE technology: Progressive scan CCD 20 fps @ 1360 × 1024 35 fps @ 512 x 512 Direct image registration to the system memory via a compatible Gigabit port Upto 100 meter cable length

Software Design of the System Software Configuration: Processing Core Multithreaded processing framework: Input Thread: Project and grab frames Processing thread: Apply measurement, unwrapping and calibration techniques Output thread: Stream/save/display results Grabbed Images from the Camera Input Thread Input Buffer Processing Thread Output Buffer Output Thread Display and/or store according to the user preferences

Results (1/3) Static Object Measurement – One Sensor

Results (2/3) Static Object Measurement – Multi-Sensor Sensor1 Sensor2 Treatment coach Sensor2 Sensor3 Sensor1

Results (3/3) Moving Object Measurement – One Sensor

Thank You!