1. Towards a real-time, configurable, and affordable system for inducing sensory conflicts in a virtual environment for post-stroke mobility rehabilitation: vision-based categorization of motion impairments
Babak Taati, Jennifer Campos, Jeremy Griffiths, Mona Gridseth, Alex Mihailidis
Sept 11, 2012
ICDVRAT

2. Outline Motivation Background Problem Definition Tracking Technologies
Method
Experimental Results
Conclusions & Future Work

3. Post-Stroke Rehabilitation
Motivation
Post-Stroke Rehabilitation
15 million people suffer stroke each year
65% of stroke survivors have difficulty using their upper limbs
The economic cost of stroke is ~$6.3 billion a year
Training exercises that provide patients with visual feedback on the movement of the affected limb facilitate rehabilitation

4. Background
Mirror Box Illusion
(Ramachandran et al, Nature,1995)
View of the affected arm is replaced by the mirror reflection of the healthy arm
Simultaneous motion, or attempt of motion of both arms results in artificial visual feedback which reinforces neurorehabilitation
Mirror box therapy is effective in restoring mobility

5. Background
Rubber Hand Illusion
(Botvinick and Cohen, Nature, 1998)
Simultaneous brushing of the rubber hand and the real hand generates a sensory conflict in the brain between the location of the visual vs. proprioceptive signal
Continued brushing typically results in a sensory takeover of proprioception and the person would feel ownership of visually perceived plastic arm

6. Objective Reproduce and combine the Mirror Box and the Rubber Hand
Problem Def.
Objective
I felt my arm was here …
… but I see it here!
Reproduce and combine the Mirror Box and the Rubber Hand
illusions in a virtual environment
Computer vision: markerless skeleton tracking
Computer graphics: visualization + pose augmentation
Machine learning: classification + latent space projection
affordable
Neuro rehabilitation
post-stroke patients
upper-limb mobility and function

7. Milestones Duplicate the Rubber Hand Effect
Problem Def.
Milestones
Duplicate the Rubber Hand Effect
Visual skeleton tracking & visualization
Synchronized tactile stimulation
Combine the Rubber Hand with the Mirror Box Effect
Categorize motion impairments
Apply a movement gain (normalize / exaggerate)

8. Tracking Tech.
Human Pose Tracking
[Kanaujia, 2011] 8

9. Human Pose Tracking (cont’d)
Tracking Tech.
Human Pose Tracking (cont’d)
Marker-based
Elaborate setup
Markerless (vision-based)
Accuracy vs. computational efficiency
[Flickr: beratus]
[CMU Motion Capture Database]
Model-based,
Discriminative, generative
Lee & Nevatia: 5 minutes / frame
[Lee & Nevatia, 2009] 9

10. Consumer Depth Cameras
Tracking Tech.
Consumer Depth Cameras
KinectTM (Microsoft)
WAVI Xtion (ASUS / PrimeSense)
Real-time color & depth sensing
Real-time skeleton tracking software libraries
Microsoft SDK or NITE 10

11. Challenges Noise / inaccuracies / systematic bias Missing information
Tracking Tech.
Challenges
Noise / inaccuracies / systematic bias
Missing information
Ground truth annotation
Evaluation of categorical time series prediction
True Labels
Predicted Labels 11

12. Dataset 7 healthy adults
Method
Dataset
7 healthy adults
Simulated 2 of the most common upper limb flexion synergies present in the post-stroke population
Elbow flexion, shoulder flexion, shoulder abduction
Stereotypical post-stroke movement synergy: Rotating the Trunk
Elbow flexion
Stereotypical post-stroke movement synergy: Elevating the Shoulder
10x normally, 10x simulating an impaired motion
The stereotypical post-stroke movement synergy
associated with stroke

13. Algorithms Cross validation Features Multi-class categorization
Method
Algorithms
Features
Displacement Vectors
Principal Components Analysis (PCA)
Multi-class categorization
Logistic Regression
Support Vector Machines (SVM)
Random Forest (RF)
Cross validation
7-fold leave-one-subject-out

14. Dominant Modes of Motion 1
Results
Dominant Modes of Motion 1
Reaching Across
(Elbow flexion, shoulder flexion, shoulder abduction)
“Normal”
With shoulder elevation

15. Dominant Modes of Motion 2
Results
Dominant Modes of Motion 2
Reaching Up
(Elbow flexion)
“Normal”
With trunk rotation

16. Classification Accuracy
Results
Classification Accuracy
# of PCA components
Classifier
Action N
Logistic Reg.
SVM RF
Reach Across 1
54.3
97.1
97.9 3
61.4
85.7
Elbow Flexion
64.3
95.7
90.0
59.3
85.0
92.9

17. Conclusions
Summary
Inherent biases were observed in the tracking of an elevated shoulder
Despite accuracy and tracking bias issues, it was possible to separate impaired motions
A single most dominant mode of motion, as identified by PCA, was sufficiently discriminative
SVM classifier obtained consistent detection accuracies >95%

18. Future Work Exaggerate / normalize motions
Conclusions
Future Work
Exaggerate / normalize motions
Latent space projection
Combine the Rubber Hand and Mirror Box illusions
Real data (real post-stroke patients)
Transfer learning

19. Acknowledgements

20. Toronto Rehab (iDAPT research facilities)
Home Environment Laboratory (HEL)
Challenging Environment Assessment Laboratory (CEAL)