1. KINECT REHABILITATION
Stroke Therapy Research
Kathryn LaBelle
kathryn labelle
kinect stroke therapy research with prof streigel
started spring 2011 for honors thesis
research supported by a contribution by tom meurer

2. RESEARCH TOPIC
Can the Kinect’s joint-tracking capability be used in clinical and in-home stroke rehabilitation tools?

3. OUTLINE Background Potential of Kinect in Rehabilitation
Stroke Therapy
Kinect
Potential of Kinect in Rehabilitation
Research Questions
Software
Data Gathering
Data Analysis
Conclusions

4. STROKE THERAPY Stroke survivors can experience:
restricted movement
loss of sense of balance
decreased strength
Regained through physical therapy
balance exercises
range of motion activities
coordination practice

5. MICROSOFT KINECT Developed for the Xbox 360 gaming console
Tracks your movements: you are the controller
Sensors
Depth Camera and Sensors
RGB Camera
Microphone array
Motorized base
Released November 2010
Depth Camera and Sensors
infra red projector
monochrome CMOS sensor
creates depth image
RGB Camera
on screen display
facial identification
Microphone array
four microphones
speech recognition
acoustic source localization
Motorized base
tilts to adjust view vertically

6. DEPTH IMAGING
Infra-red projector shines grid of light on the scene, encoded with data.
Light bounces off objects in the scene.
Kinect light sensors receive reflected light.
By analyzing time of flight and distoritions in the encoded data, the Kinect makes a depth map of the scene.

7. JOINT TRACKING ALGORITHM
Input: depth map
Machine learning algorithm
Collected recordings of people using the Kinect
Joint positions marked by hand
Algorithm was fed this “training” data and learned how to correctly identify joints from a depth image
Output: x, y, z joint positions
This is Microsoft’s implementation

8. JOINT TRACKING AND STROKE REHABILITATION
Clinical applications:
assess patients’ performance
track patients’ progress
pinpoint areas for improvement
At-home exercise aids:
provides constructive feedback to patients
give encourgement and motivation
generate summary reports for doctors

9. RESEARCH QUESTIONS
What SDKs and drivers are available for use with a PC?
What type of information can be obtained?
What is the quality of the joint data obtained from the Kinect?
Sampling rates
Consistency
How resilient is the Kinect’s joint data and performance to variation in testing conditions?
What functionality could be provided in a stroke therapy application that uses the Kinect?
What SDKs and drivers are available for using the Kinect with a PC? What capabilities do these provide and which of them will be most useful for stroke therapy?
What type of data and information can be obtained from the Kinect using these SDKs and APIs?
What is the quality of the joint data obtained from the Kinect? What is the sampling rate and consistency of this information?
How resilient is the Kinect’s joint data and performance to variables such as distance, body type, clothing, number of subjects, and amount of movement?
What functionality could be provided in a stroke therapy application that uses the Kinect, and what are the limitations of such a program?

10. SDK COMPARISON OpenNI Microsoft Raw depth and image data Yes
Joint position tracking
Save raw data stream to disk No
Joint tracking without calibration
Easy installation
Number of joints available 15 20
Quality of documentation
Adequate
Excellent
Even before Microsoft released their SDK in June 2011, there were many open source sdks available.
openni suited our needs best so i developed some software using that
later microsoft’s sdk came out and i used that too

11. SOFTWARE DEVELOPED Display depth video and skeleton
Joint positions and instantaneous frames per second written to file
Balance board integration
Record depth stream to file
Obtain joint positions from recording

12. DATA GATHERING
to gather data for analysis, used six subjects and did three variations of sit to stand exercises
explain sit to stand
subject sat in a chair directly facing the kinect with their feet on a balance board. the tv displayed the running program so they could see the depth map and skeleton on the screen

13. DATA ANALYSIS Sampling rates of joint position data
Identifying phases of movement from joint positions
Consistency and stability of joint positions

14. SAMPLING RATE OpenNI Microsoft Average Frame Rate (fps) 25.0 19.6
Std Deviation
(between trials)
5.8
2.3
Minimum
9.8
14.1
Maximum
30.0
23.7
OpenNI a little faster
Microsoft more consistent
Both more than adequate even at their worst

15. IDENTIFYING PHASES OF MOVEMENT
sit to stand exercise
easy to pick out the parts of the exercise
note head bob

16. Standard Deviation of Joint Positions while Subject is Motionless
DATA STABILITY
Standard Deviation of Joint Positions
while Subject is Motionless
Joint
OpenNI (cm)
Microsoft (cm)
Head
0.34
1.8
Hip
0.42
1.2
Knee
0.70
1.5

17. DATA STABILITY: Assisted Tests
Clinical therapy often involves an assistant supporting a patient while he performs exercises
Test procedure:
subject begins by sitting alone
assistant joins, putting hands on subject’s shoulders
subject stands up
conducted tests with many variations like type of exercise and distance, but for the sake of time i’ll focus on the effects of adding a person in the scene

18. DATA STABILITY: Assisted Tests
one type of irregularity observed occurred as assistant entered the scene
probably because the algorithm was determining which parts of the depth image belonged to which person
not a huge problem because they could be easily identified as errors and corrected: it is clear where the true value lies

19. DATA STABILITY: Assisted Tests
another type of variation observed was skeleton merging
after the patient stood, the algorithm identified the people as one and produced a strange-looking skeleton (ex: identify assistant’s head as if it were the patient’s shoulder)
incorrect positions, and data was less stable
more of a problem because it is not certain where the true value of the person’s head position lies

20. DATA STABILITY: Assisted Tests
It is at least possible to distinguish between normal deviation and deviation with skeleton merging
More investigation needs to be done to iron out this obstacle since it could become a real problem in a clinical setting

21. CONCLUSIONS
OpenNI Framework and Microsoft SDK for Windows are best tools to use
Can provide significant functionality in a joint-tracking application
track and record joint positions in three dimensions
display image of tracked joints in real time
integrate Kinect with the Wii balance board
Sampling rate exceeds acceptable level
Phases of movement are easily identifiable from graphs of joint positions
Joint position stability is more than adequate with one subject in view
Skeleton merging could pose a problem for clinical use of Kinect

22. FUTURE WORK Deeper investigation into assisted exercises
Different types of exercises
Position the assistant differently
Determine conditions causing skeleton merging
Further development of software
Investigate applications in other fields of physical therapy
-conditions causing skeleton merging: relative heights of assistant and patient
-further development of software: example -- guided in-home exercises, with constructive feedback, variation in difficulty levels, performance reports for physicians
-other physical therapy: athletic injuries, chronic pain, neurological diseases (parkinsons, multiple sclerosis, cerebral palsy)

23. Questions?