Francois de Sorbier Hiroyuki Shiino Hideo Saito

I. Introduction II. Overview of our system III. Violin extraction and 3D registration IV. Virtual advising V. Conclusion Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 2

 Violin is a beautiful instrument…  … but one of the most complicated ◦ No fret on the fingerboard ◦ No help for the position the bow on strings Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 3 I.Introduction

 Music Jacket ◦ Vibro-tactile feedback ◦ Guide the bowing arm  Guitar playing support ◦ Tracking with marker ◦ Guide with virtual hand Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 4 Y. Motokawa, H. Saito. “Support system for guitar playing using augmented reality display”. In Proceedings of the 5th IEEE and ACM ISMAR, , 2006 van der Linden, J., Schoonderwaldt, E. and Bird, J. “Good Vibrations: Guiding Body Movements with Vibrotactile Feedback”. Proceedings of the Third International Workshop on Physicality, 13-18, 2009 I.Introduction

 Overlay virtual information on the violin ◦ Virtual frets ◦ Guides for the bow and fingers ◦ Sound analysis Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 5 I.Introduction

 No intrusive device  No marker  Real time feedback Marker-free violin tracking using a RGBD camera Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 6 I.Introduction

 Tracking of the violin  Feedback displayed on the screen ◦ Constant view of the violin ◦ Virtual information Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 7 II.Overview of our system

 Features detection and extraction ◦ Many occlusions caused by the player ◦ The surface has a poor texture ◦ The material is highly specular  Difficult to use features in this context Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 8 II.Overview of our system

 Use Kinect for tracking the violin ◦ Depth values for the pose estimation Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 9 II.Overview of our system

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 10 II.Overview of our system Color Depth Segmentation Violin detection Violin extraction Database Registration Virtual information displayed

 Detect the brown color in the image  Remove noise  Many parts are missing ◦ Occlusions ◦ Specular material ◦ Strings and fingerboard  Not enough for tracking Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 11 III.Violin extraction and 3D registration

 Get the 3D points from extracted color  Compute the corresponding plane equation ◦ Optimized with RANSAC  Align a 3D volume to the plane ◦ Typical dimensions of a violin  Keep the 3D points in it Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 12 III.Violin extraction and 3D registration

 Iterative Closest Point algorithm ◦ Compare input points with a model ◦ Slow if too many points ◦ Inaccurate if not enough  Our proposed solution ◦ Increase the number of models ◦ Reduce the number of points per model ◦ Fast retrieval with a plane equation comparison Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 13 III.Violin extraction and 3D registration

 Offline phase  25 models ◦ Compare the plane equations ◦ Store candidate if the difference is big enough ◦ Store also the plane equation Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 14 III.Violin extraction and 3D registration

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 15 III.Violin extraction and 3D registration

 Real time (21 milliseconds)  Pose also estimated using markers Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 16 III.Violin extraction and 3D registration

 Use the pose estimation  Location manually defined during the capture of the models Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 17 IV.Virtual advising

 Emphasize the string to be pressed  Display a red dot at the junction of the string and the fret  Define where the finger has to press the string Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 18 IV.Virtual advising

 Suggest the violinist to play a given note  Display the fret/string to be pressed  Analyze the sound obtained ◦ If fingering is considered correct ◦ Advice about the position of the bow given the difference of pitch (OK / LOW / HIGH) Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 19 IV.Virtual advising

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 20 IV.Virtual advising

 Performed by confirmed player ◦ Bowing is correct  Compute the difference of pitch on each fret with the expected one Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 21 IV.Virtual advising

 Original marker-free method for virtual advising on a violin  Method based on several pre-computed models ◦ Real-time ◦ Accurate  Display virtual guides on the fingerboard  Analyze the note played for further advices Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 22 IV.Virtual advising

 Perform a user based analysis ◦ Validate or improve our approach  Use different kind of display ◦ See-through HMD ◦ Spatial augmented reality  Apply to other similar instruments ◦ Japanese shamisen Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 23

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier 24 Thank you for your attention