Face Recognition and Retrieval in Video Basic concept of Face Recog. & retrieval And their basic methods. C.S.E. Kwon Min Hyuk

True? False? Q1 : in recently, face recognition researches focus on video-based rather than still image-based (O / X) Q2 : There is three approaches; (O/ X) ◦ 1. key-frame based ◦ 2. Temporal model based ◦ 3. image set based Face variation and expression make face recognition difficult. (O/X)

Answers All of statement is true.

Intro Q1 : Why do we need Face recognition system? ◦ Increasing request to search specific people related video contents ◦ Can be applied at security, human-computer inter action etc.

Intro Q2: What is recent trend of approach to Face recog.? ◦ Traditionally, focused on Still image-based appr. ◦ Recently, focused on Video-based appr. ◦ We can extract more information from video than that of still image.

General steps for Face recognition Where is face located in video frame? ◦ We should look for which part of the frame is face.  Face detecting and Tracking. Recognizing face ◦ There is some basic approach for Face Recog.  Key fame-based approach  Temporal Model-based approach  Image set-based approach

Face detection Using statistical geometric model ◦ From the frame Extract appearance features such as edge, intensity, color(histogram) To evolve the face detector by using machine learning tech. ◦ Adaboost ◦ Neural Network ◦ Support Vector Machine

Face tracking Face detection’s limit. ◦ It detect only frontal or near frontal view.  Tracking face is needed to handle large head motions. Face tracking ◦ Difficulties ◦ Method to solve

Face tracking Difficulty 1 : There is ◦ Face appearance variation ◦ 3D motion ◦ Background change Method to solve: face online boosting ◦ Using tracked images in previous frames. ◦ Applying current result to tracking seq. for next frame. (real-time updating feedback) example in next slide 

Example for online boosting algorithm. So-called “adaptive tracker.”

Face tracking(con’d) Difficulty 2: The adaptive tracker can adapt to non-targets. Method to solve : add basal appearance of target ◦ Teaching the tracker about some basal appearance of the target. ◦ Basal appearance : Image set of various target condition (face expression, pose etc)

After face detecting and tracking We can determine the part of frame where face is located.  Now, we can get to face recognition.

Face Recognition Basic steps for Face Recog. ◦ 1. get weak evidence in individual frame. ◦ 2. collect that evidence over time. ◦ 3. lead(determine) reliable result. Three approaches ◦ 1. key-frame approach ◦ 2. temporal model-based approach ◦ 3. image set-based approach

Key-frame based approach Treat each video as a collection of images. Basic steps of the approach. ◦ 1. input data(still images, video) ◦ 2. from data, extract images of the target.  Extracted images are called key-frames or examplars. ◦ 3. matching them with all or subset of other video sequence(where the target is).

Key-frame based approach(con’d) How can we get some ‘good’ key-frame from input data? ◦ By image-based recognition ◦ In each frame, probe the nose and eyes’ triangular structure. If it is in the frame, then face recognition is performed. And key-frame is extracted.

Key-frame based approach(con’d) ◦ Applying K-Means clustering  Cluster means a group whose elements have some common property.  This algorithm is grouping some data observations into one of cluster which has nearest mean.

Key-frame based approach(con’d) Other algorithms ◦ Isomap algorithm ◦ Combination of majority and probabilistic voting. ◦ And so on. (I’ll skip the details.) Finally, all or subset of video sequence will be compared(matched) with extracted ‘good’ key-frame to determine recognition.

Temporal Model Based approach To handle face dynamics ◦ Ex: face expression(non-rigid) or head movement(rigid) Using temporal sequence(continuous coherent) ◦ Ex> Using whole sequence of changing face dynamics as a image set.

Temporal Model Based approach(con’d) Basic methods ◦ Matching the face Trajectory.  Trajectory means the moving face’s path(orbit) through in surfaces.  Two(model and object) trajectory distance accumulates recognition evidence over time.

Temporal Model Based approach(con’d) Other method ◦ Trained statistical face model  Using density estimation. ◦ Probabilistic approach  Using time-series state space variables ◦ Hidden Markov Model  Fusing pose and person-discriminant features. I’ll skip all of details.

Image set based approach This approach uses ◦ both image collected over consecutive time (similar with temporal image set) ◦ And independent still image set (similar with key-frame) Combination of both temporal and key- frame based approaches. Two major approaches. ◦ Statistical modal-based ◦ Mutual subspace-based

Image set based approach(con’d) Image set classification ◦ Non-parametric sample based  Compare representative images of each image sets ◦ Parametric model-based  In terms of probabilistic, compare two distributions of each image set.

Image set based approach(con’d) Statistical Model-based ◦ To determine recognition, consider similarity of two manifolds  manifold is large group (more than cluster) which contains several cluster. Drawback ◦ Need to solve the difficult parameter estimation problem.

Image set based approach(con’d) Mutual Subspace-Based Model(MSM) ◦ To determine Similarity between image sets  measure by the smallest principal angles between subspaces. CMSM is expansion of MSM ◦ Assume more constraints.

Face Retrieval It is difficult to recognize face in the uncontrolled condition like face dynamics, light intensity, hair styles ◦ There is two applications 1. Person Retrieval 2. Cast listing

Person Retrieval Face recognition tech. are applied. Basic method ◦ Using head model (with multiple texture map) ◦ Step1. rendering(extract or generate) face images ◦ Step2. identifying target face. ◦ Step3. updating the texture map of the model.

Cast listing (cast : actors or characters in film) Automatic cast listing is interesting problem. Based on face recognition ◦ Because face is repeatable cue in the film Using image clustering method For accuracy, Treat clothing appearance additional cues for clustering.

Challenges and Future direction Databases. ◦ Constructed in lab. enviro. Not a real world. ◦ Limited face appearance of variation. Low-quality Video data ◦ Exist lots of noise hard to filter out. Computational Cost ◦ Face recognition requires quite high power devices.

Conclusion Face recog. can be applied in various area. Face detecting and Tracking. Three general Methods ◦ Key-frame based ◦ Temporal model based ◦ Image set based Person Retrieval and Cast listing Challenges to evolve Face recog. system.