Human Computer Interface based on Hand Tracking P. Achanccaray, C. Muñoz, L. Rojas and R. Rodríguez 4 th International Symposium on Mutlibody Systems and Mechatronics Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Outline  Introduction  Processing  Background Subtraction  Dynamic Model  Non-Gaussian Models  Measurement Model  Condensation algorithm  Results  Applications  Future works  Conclusions

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Introduction  Computer Vision Systems  Application Fields  Medical diagnosis  Controlling processes  Military  Gaming Industry  Etc…

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Processing

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Background Subtraction I  Skin Color Classifier It was developed to isolate the hand silhouette, in the YCrCb color space.

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Background Subtraction II  Noise Reduction and Edge detection Morphological operations were used to reduce the noise in the filter. Then, edges was detected by Canny filter.

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Hand Contour Model  The following parameters were used to describe the hand silhouette. It was modeled by Catmull Rom’s curves from 50 control points. X and Y: Coordinates of the hand centroid. : hand scale  : rotation angle of the hand palm.  0,  1,  2 and  3: angles formed by each finger with hand palm. l0, l1, l2 and l3: lengths of each fingers.  4,  5: angles of the thumb.

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Dynamic Model Where Wk is the gaussian noise, Xk is the sample in each image.

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Non-Gaussian Models  The task of hand tracking is a non-Gaussian model because the probability density for Xk, P(x), at time tk, is multi-modal. For that reason, the Kalman filter is not applicable.  P(x): Probability density

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Non-Gaussian Models

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Measurement Model

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Condensation Algorithm set of ‘n’ samples at time tk: Weight of the samples ‘n’ at time tk.

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Results  Computer facilities:  Hardware: Dual Core 2.20 GHz laptop  Standard webcam  Software: Windows 7, Visual.Net  Additional Libraries: OpenCV, OpenGL

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Results  Efficiency Calculation. The following equation was used to calculate the efficiency: Where: : is the maximum difference of pixels between the hypothetic contour and the real contour. : are the coordinates of the hypothetic particle. : are the coordinates of the real particle, which comes from the camera.

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Results  Rotation Tracking Efficiency

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Results  Traslation Tracking Efficiency

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Application  Simulation of helicopter’s movement control

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Helicopter’s control HandHelicopter Turn right Turn left Move to the frontTurn up Move to the backTurn down UpMove to the front DownMove to the back

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Conclusions  The proposed interface is suitable for different kind of applications, as it was demonstrated with the sample object (an helicopter), because the program developed could be changed depending of the required application.  The tracking efficiency is around 81% for the different movements that made the silhouette of the user’s hand. It indicates that the tracking is fast and efficient for applications such as simulation in virtual environments and control by communication protocols.  As it was expected, the algorithm is robust because it works properly in different backgrounds.

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Future works  Improve the helicopter’s control using a close loop, to avoid sudden movements.  Increase the degrees of freedom (DOF) of the system. A tracking for each finger would be implemented, so it would allow users to simulate tactil a surface and the interaction will be easier and natural.  Increase the processing speed using CUDA (GPU). We can implement a

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica References  A. Doucet, Nando de Freitas, and N. Gordon. Sequential Monte Carlo Methods in Practice. Springer, USA,  M.S. Arulampalam, S. Maskell, and N. Gordon. A tutorial on particle filters for online/non-Gaussian Bayesian Tracking. Signal Processing, vol. 50, nº 2, ,  P. Menezes, L. Brethes, F. Lerasle, P. Danes, and J. Dias. Visual Tracking of Silhouettes for Humand-Robot Interaction Proceedings of the International Conference on Advanced Robotics (ICAR01), vol. 2, Coimbra,  A. Blake and M. Isard. Active Contours. Springer,  M. Tosas. Visual Articulated hand tracking for Interactive Surfaces. PhD. Thesis, University of Nothingham,  E. Catmull and J. Clark. A class of local interpolating splines. Computer Aided Geometric Design, , Academic Press, New York.  M. Isard and A. Blake. Condensation, conditional density propagation for visual tracking Proceedings of the European Conference on Computer Vision,  J. Christophe and T. Shigeru. Comparative performance of different chrominance spaces for color segmentation and detection of human faces in complex scene images Proceedings of the 12 th Conference on Vision Interface, 1999.

Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica Acknowledgments  First of all, to the National University of Engineering (UNI), and ProUNI (Patronato UNI) for the financial support to participate in this symposium.  Second, to our adviser, M. Sc. Ricardo Rodriguez, for his constant support during the project.  Finally, to our families, for their patience and encourage us to go ahead.

Human Computer Interface based on Hand Tracking P. Achanccaray, C. Muñoz, L. Rojas and R. Rodríguez 4 th International Symposium on Mutlibody Systems and Mechatronics Universidad Nacional de Ingeniería Facultad de Ingeniería Mecánica