Relative Hidden Markov Models Qiang Zhang, Baoxin Li Arizona State University

Introduction Understanding human motion is an important task in many fields: –sports, rehabilitation, surgery, computer animation and dance; One key problem in such applications is the analysis of skills associated with body motion. Many computational methods have been developed for this purpose: –A popular choice is HMM based.

Motivation One practical difficulty: they require the skill labels for the training data; Labeling the skill of a trainee is currently done by senior surgeons; –a costly practice; –subjective and less quantifiable; Sufficient and consistent skill label for a large amount of data—difficult, if not impossible.

Relative Label Instead of Absolute Label It is hard to say whether (b) is smiling or not. But it is easy to find (b) is less smiling than (a) but more than (c). We use similar idea in our motion analysis: given two videos, we only need to know which one is better.

Proposed Method

Proposed Method Cont’d

Measuring the Skills

Proposed Method

Proposed Method Cont’d

Update the Model

Update the Model Cont’d

Sub-problem 2

Algorithms

Relationship to Latent SVM Latent Variable State Path Pair as Latent Var.

Relationship to Latent SVM

Synthetic Experiment Randomly generate 6 HMMs and order them randomly; 200 sequences from each HMM: –50 for training and 150 for testing; From 50x6 sequences, 1000 pairs are randomly selected.

Convergence Behavior

Performances with # Training Pairs

Log Likelihood of Data

Parameter Selection C

Experiment Videos captured from FLS trainer box: –546 in total from 18 subjects; –Duration of four weeks, 3 sessions/week; Assumption: the skills of the subject get improved during training process –the scores of videos from the last session are better than those at the first session for each subject. Result: MethodHMMBaselineImproved # pairs Accuracy79.39%77.54%87.25%

Experiments: Skill Curve

Experiments: Learned Models

Experiment: Emotion Recognition Recognizing the emotional state of the speakers is very important; –Human computer interaction Existing methods try to classify the audio to predefined labels or levels: –Labeled training data is required; We can leverage the power of pairwise comparison via the proposed method;

Emotion Recognition with RHMM Extract MFCC Bag of Words RHMM Models Pairwise Rank Training Data 991 audios, 6 emotions at 7 levels, half for training and 1000 randomly selected pair for input.

Experiment Results DimensionImprovedBaselineHMM Pleasantness77.30%57.96%75.05% Arousal86.95%55.74%69.55% Dominance87.95%63.04%77.32% Credibility76.68%55.11%71.74% Interest81.90%62.56%78.07% Positivity74.99%67.84%70.36% Average81.28%53.14%73.72%

Future Work Theoretic analysis of the learned model; Allowing more types of observation models; Modeling multiple relative attributes jointly via multi-task learning framework; Modeling multiple attributes jointly can be also made possible by utilizing hierarchical Dirichlet Process.

Related Publications Qiang Zhang and Baoxin Li, Relative Hidden Markov Models for Evaluating Motion Skills, IEEE Computer Vision and Pattern Recognition (CVPR) 2013, Portland, OR Lin Chen, Qiongjie Tian, Qiang Zhang and Baoxin Li. Learning Skill-Defining Latent Space in Video-Based Analysis of Surgical Expertise: A Multi-Stream Fusion Approach. NextMed/MMVR20. San Diego, CA, Qiongjie Tian, Lin Chen, Qiang Zhang and Baoxin Li. Enhancing Fundamentals of Laparoscopic Surgery Trainer Box via Designing A Multi-Sensor Feedback System. NextMed/MMVR20. San Diego, CA, Qiang Zhang, Lin Chen, Qiongjie Tian and Baoxin Li. Video-based analysis of motion skills in simulation-based surgical training. SPIE Multimedia Content Access: Algorithms and Systems VII. San Francisco, CA, Qiang Zhang and Baoxin Li. Video-based motion expertise analysis in simulationbased surgical training using hierarchical dirichlet process hidden markov model. In Proceedings of the 2011 international ACM workshop on Medical multimedia analysis and retrieval (MMAR ’11). ACM, New York, NY, USA, Zhang, Qiang and Li, Baoxin, Towards Computational Understanding of Skill Levels in Simulation-Based Surgical Training via Automatic Video Analysis, International Symposium on Visual Computing (ISVC) 2010, Las egas, NV Qiang Zhang, Baoxin Li, “Relative Hidden Markov Models for Video-based Evaluation of Motion Skills in Surgical Training,” Pattern Analysis and Machine Intelligence, IEEE Transactions on [under review]