I mage and M edia U nderstanding L aboratory for Performance Evaluation of Vision-based Real-time Motion Capture Naoto Date, Hiromasa Yoshimoto, Daisaku Arita, Satoshi Yonemoto, Rin-ichiro Taniguchi Kyushu University, Japan

Laboratory for Image and Media Understanding Background of Research Motion Capture System – Interaction of human and machine in a virtual space – Remote control of humanoid robots – Creating character actions in 3D animations or video games Sensor-based Motion Capture System – Using Special Sensors (Magnetic type, Infrared type etc.) – User ’ s action is restricted by attachment of sensors Vision-based Motion Capture System – No sensor attachments – Multiple cameras and PC cluster

Laboratory for Image and Media Understanding Key Issue Available features acquired by vision process is limited. – Head, faces and feet can be detected robustly. How to estimate human postures from the limited visual features – Three kinds of estimation algorithms – Comparative study of them

Laboratory for Image and Media Understanding System Overview 人物 2 CG model PC camera

Laboratory for Image and Media Understanding System Overview 人物 2 CG model PC camera PC Using 10 cameras for robust motion capture

Laboratory for Image and Media Understanding System Overview 人物 2 CG model 1 top-view camera on the ceiling PC camera

Laboratory for Image and Media Understanding System Overview 人物 2 CG model 9 side-view cameras around the user PC camera

Laboratory for Image and Media Understanding System Overview 人物 2 CG model Using PC cluster for real-time feature PC camera

Laboratory for Image and Media Understanding System Overview 人物 2 CG model First, take images with each camera PC camera

Laboratory for Image and Media Understanding System Overview 人物 2 CG model Extract image-features on the first stage PCs PC camera

Laboratory for Image and Media Understanding System Overview 人物 2 CG model PC camera Reconstruct human CG model by feature parameters in each image

Laboratory for Image and Media Understanding System Overview 人物 2 CG model Synchronous IEEE1394 cameras: 15fps PC camera

Laboratory for Image and Media Understanding System Overview 人物 2 CG model CPU : Pentium Ⅲ 700MHz ｘ 2 OS : Linux Network ： Gigabit LAN Myrinet camera PC

Laboratory for Image and Media Understanding Top-view camera process Background subtraction Opening operation Inertia principal axis Detect body direction and transfer it

Laboratory for Image and Media Understanding Top-view camera process Background subtraction Opening operation Inertia principal axis Detect body direction and transfer it

Laboratory for Image and Media Understanding Top-view camera process Background subtraction Opening operation Inertia principal axis Detect body direction and transfer it

Laboratory for Image and Media Understanding Top-view camera process Background subtraction Opening operation Feature extraction – Inertia principal axis – Body direction

Laboratory for Image and Media Understanding Side-view camera process Background subtraction Calculate centroids of skin-color blobs

Laboratory for Image and Media Understanding Side-view camera process Background subtraction Calculate centroids of skin-color blobs

Laboratory for Image and Media Understanding Side-view camera process Background subtraction Calculate centroids of skin-color blobs

Laboratory for Image and Media Understanding From all the combination of cameras and blob centroids, we select all possible pairs of lines of sight. Then we calculate an intersection point of each line pair. Unless the distance of the two lines is smaller than a threshold, we decide there is no intersection point. Estimate 3D position of skin-color blob

Laboratory for Image and Media Understanding Estimate 3D position of skin-color blob The calculated points are clustered according to distances from the feature points (head, hands, feet) of the previous frame. Select points where feature points are dense as the 3D positions of the true feature points.

Laboratory for Image and Media Understanding Estimate 3D position of torso L1 L2 head right shoulder V: V is the vector which intersects perpendicularly with a body axis and with a body direction. V torso ・A method based on simple body model Center point

Laboratory for Image and Media Understanding Performance evaluation of right hand position estimation

Laboratory for Image and Media Understanding Estimate 3D positions of elbows and knees 3 estimation methods – Inverse Kinematics (IK) – Search by Reverse Projection (SRP) – Estimation with Physical Restrictions (EPR)

Laboratory for Image and Media Understanding Estimate 3D positions of elbows and knees IK –  assumed to be a constant

Laboratory for Image and Media Understanding Estimate 3D positions of elbows and knees SRP

Laboratory for Image and Media Understanding EPR  An arm is assumed to be the connected two spring model.  The both ends of a spring are fixed to the position of the shoulder, and the position of a hand.  The position of an elbow is converged to the position where a spring becomes natural length. (the natural length of springs is the length of the bottom arm and the upper arm which acquired beforehand.) Estimate 3D positions of elbows and knees

Laboratory for Image and Media Understanding Accuracy of estimating right elbow position

Laboratory for Image and Media Understanding Accuracy of posture parameters

Laboratory for Image and Media Understanding Visual comparison of 3 methods

Laboratory for Image and Media Understanding Computation time required in each algorithm Top-view camera processing: 50ms Side-view camera processing: 26ms 3D blob calculation : 2ms IK calculation : 9ms SRP calculation : 34ms EPR calculation : 22ms

Laboratory for Image and Media Understanding Online demo movie (EPR)

Laboratory for Image and Media Understanding We have constructed a Vision-based Real-time Motion Capture System and evaluated its performance Future works – Improvement of posture estimation algorithm – Construction of various applications  Man and machine interaction in a virtual space  Humanoid robot remote control system Conclusions