By: David Gelbendorf, Hila Ben-Moshe Supervisor : Alon Zvirin

- GIP sensor: Today we are capable to capture 3D images - We are able to print in 3D, with the right printer - Everyone has a passport photo on he’s I.D card - Why not printing a face in 3D?

 First Stage:  Developing a tool for processing a facial video and printing a 3D face automatically  Input: a 3D record of a frontal face  Output: a 3D printed model of the face  Second Stage:  Wrap the tool in a easy-to-use GUI

- Printing a 3D object requires: - A good 3D image, which contains all face features (eyes, nose and mouth) and with high density

- Printing a 3D object requires: - Smoothness and fullness - The 3D frames are not complete. Many pixels has NAN values and missing data (non-Continuity) - Choosing bounding box

- Printing a 3D object requires: - Transformation of the 3D GIP file to a 3D printer standard format (for instance obj, stl)

Recoding a video Choosing Best Frame Processing frame (creating model) Converting to STL Model ready to print

- How to choose a good frame? - Solution: Use Viola Jones face detection algorithm: - Use VJ for searching facial features - Use VJ for detecting movement of facial features Choosing Best Frame

Profile Face (no detections) True positives + False Positives (double detection of mouth and left eye) Semi-Profile Face with low detection results Choosing Best Frame

Double detection of mouth, no detection of left/right eye No detection of pair of eyes Perfect match (one match for each detector) Choosing Best Frame

Facial features detection: - Scanning each frame for a face, pair of eyes, left eye, right eye, nose and mouth - Sorted by importance : -2: Face and pair of eyes: -1: left eye, right eye, *nose -0.5: Mouth -*0.75: if found more than one nose, there will be penalty -*0.5: if found more than 2 pair of eyes - Grade: normalized to [0..1] Choosing Best Frame

Movement Detection: -Examining difference of VJ detections in adjacent frames (previous and next frames) : -Pair of eyes, left eye, mouth, nose -Calculating the distance of adjacent detections frames (Euclidean metric) -Calculating a grade according to the results. A perfect grade (1.00) will be achieved only if perfect match is found between previous and next frames in all examined features. Choosing Best Frame

- Grades examples: Choosing Best Frame

 The chosen image is a raw 3d image, contains noise, holes and discontinuities.  Goal: automatic filtering of the image in order to create a 3D complete, continuous model Choosing Best Frame Processing Frame

 Challenge: choosing a right bounding box for the desired model (for instance, without background)  Solution: choosing the bounding box automatically with optimization Choosing Best Frame Processing Frame

 Bad Bounding Box (Noise is cut. Also, in the back of the head there is too much information) Choosing Best Frame Processing Frame

 Good Bounding Box: Choosing Best Frame Processing Frame

 Optimization of bounding box:  Specially important for Z axis (depth)  Iterative “exhaustive search” for optimal max Z:  Starting from a min size of max Z, and progressing with max of Z axis  Dividing the min – max range into levels, increasing the max Z value in each iteration  Calculating a grade for each max Z value  Searching for minimum grade  Second option: run constrained optimization to find optimal min and max value for x, y, z (6 variables) Choosing Best Frame Processing Frame

 Searching for minimum Weighted grade according to:  Length of Contour 2D: 0.2  Length of Contour 3D: 0.2  Number of pixels: 0.3  Quadratic Shape of Contour: 0.7 Choosing Best Frame Processing Frame

Choosing Best Frame Processing Frame

 Filtering:  3D filter:  A chosen filter (median, average, raw, beltrami)  Clean Margins  Remove pixels far from the median of their neighbors  Remove pixels with few neighbors  Choose K largest connectivity components  Filling Holes:  cubic interpolation of each 2D connected component Choosing Best Frame Processing Frame

 Problem:  Dark eyebrows: risk for missing data for building the model  Solution:  Detect and fix holes in eyebrows:  Detect eyebrows area with Viola Jones algorithm (by detecting pair of eyes position).  If holes detected, try to fix by moderate filtering. Choosing Best Frame Processing Frame

Choosing Best Frame Processing Frame

 Fixing Eyebrows – Examples (Detect):  Example 1 : eyebrows detection returns true negative (0% missing pixels):  Example 2: eyebrows detection returns true positive (28% missing pixels): Choosing Best Frame Processing Frame

 Fixing Eyebrows –Repair:  Trying to moderate image filtering parameters:  Changing threshold of minimal neighbors from 4 to 2.  Iteratively compute the distance threshold between a pixel and it’s median with is neighbors. Increasing threshold until fraction of missing data <8% or after reaching maximum threshold value. Choosing Best Frame Processing Frame

 Fixing Eyebrows – Examples ( Detect and fix ):  Example 1 : eyebrows detection returns true negative (0% missing pixels):  Example 2: eyebrows detection returns true positive (28% missing pixels):  After fixing, missing pixels fraction decreases to 5.7% Choosing Best Frame Processing Frame

 Fixing Eyebrows – Examples:  Example 3: Final Model with and without fixing Choosing Best Frame Processing Frame

 Fixing Eyebrows – drawbacks:  If Viola Jones algorithm fails to detect pair of eyes detection, eyebrows detection will fail as well.  Moderate filtering may result with a distorted model. Choosing Best Frame Processing Frame

 After bounding box optimization and filtering:  The 3D model is prepared  The Contour of the model is prepared Choosing Best Frame Processing Frame

 Face model is ready, we can transform the model into 3D STL format which is a known standard for 3D models  In the writing process, the 3D model’s points are re- arranged in triangles and their corresponding normals. Choosing Best Frame Processing FrameWriting STL file

 Automatic bounding box may result in a cut face  Automatic filtering process (which is not configured to each face individually) may result with a less accurate model, compared to a model which was filtered manually.  Run-time of the all automatic modules may take a couple of minutes

 Improving run-time of bottlenecks :  Run-time for calculating grade for each frame  Run-time for calculating bounding box  Run-time for writing STL modules  Improving filtering by adjusting parameters for each frame (change some parameters to be dynamic)  Building a better model from several different frames

 All option can be easily operated from the GUI:  Choosing a video  First Option:  Getting a 3D model to print automatically  Second Option:  Edit parameters for each module:  Choose best frame  Choose bounding box (option for manual or automatic bounding box)  Choose file format and save the model to print  Advanced menu to change several parameters

 Viola Jones in Matlab:  on.cascadeobjectdetector-class.html