Presentation on theme: "KINECT REHABILITATION"— Presentation transcript:
1 KINECT REHABILITATION Stroke Therapy ResearchKathryn LaBellekathryn labellekinect stroke therapy research with prof streigelstarted spring 2011 for honors thesisresearch supported by a contribution by tom meurer
2 RESEARCH TOPICCan the Kinect’s joint-tracking capability be used in clinical and in-home stroke rehabilitation tools?
3 OUTLINE Background Potential of Kinect in Rehabilitation Stroke TherapyKinectPotential of Kinect in RehabilitationResearch QuestionsSoftwareData GatheringData AnalysisConclusions
4 STROKE THERAPY Stroke survivors can experience: restricted movementloss of sense of balancedecreased strengthRegained through physical therapybalance exercisesrange of motion activitiescoordination practice
5 MICROSOFT KINECT Developed for the Xbox 360 gaming console Tracks your movements: you are the controllerSensorsDepth Camera and SensorsRGB CameraMicrophone arrayMotorized baseReleased November 2010Depth Camera and Sensorsinfra red projectormonochrome CMOS sensorcreates depth imageRGB Cameraon screen displayfacial identificationMicrophone arrayfour microphonesspeech recognitionacoustic source localizationMotorized basetilts to adjust view vertically
6 DEPTH IMAGINGInfra-red projector shines grid of light on the scene, encoded with data.Light bounces off objects in the scene.Kinect light sensors receive reflected light.By analyzing time of flight and distoritions in the encoded data, the Kinect makes a depth map of the scene.
7 JOINT TRACKING ALGORITHM Input: depth mapMachine learning algorithmCollected recordings of people using the KinectJoint positions marked by handAlgorithm was fed this “training” data and learned how to correctly identify joints from a depth imageOutput: x, y, z joint positionsThis is Microsoft’s implementation
8 JOINT TRACKING AND STROKE REHABILITATION Clinical applications:assess patients’ performancetrack patients’ progresspinpoint areas for improvementAt-home exercise aids:provides constructive feedback to patientsgive encourgement and motivationgenerate summary reports for doctors
9 RESEARCH QUESTIONSWhat SDKs and drivers are available for use with a PC?What type of information can be obtained?What is the quality of the joint data obtained from the Kinect?Sampling ratesConsistencyHow resilient is the Kinect’s joint data and performance to variation in testing conditions?What functionality could be provided in a stroke therapy application that uses the Kinect?What SDKs and drivers are available for using the Kinect with a PC? What capabilities do these provide and which of them will be most useful for stroke therapy?What type of data and information can be obtained from the Kinect using these SDKs and APIs?What is the quality of the joint data obtained from the Kinect? What is the sampling rate and consistency of this information?How resilient is the Kinect’s joint data and performance to variables such as distance, body type, clothing, number of subjects, and amount of movement?What functionality could be provided in a stroke therapy application that uses the Kinect, and what are the limitations of such a program?
10 SDK COMPARISON OpenNI Microsoft Raw depth and image data Yes Joint position trackingSave raw data stream to diskNoJoint tracking without calibrationEasy installationNumber of joints available1520Quality of documentationAdequateExcellentEven before Microsoft released their SDK in June 2011, there were many open source sdks available.openni suited our needs best so i developed some software using thatlater microsoft’s sdk came out and i used that too
11 SOFTWARE DEVELOPED Display depth video and skeleton Joint positions and instantaneous frames per second written to fileBalance board integrationRecord depth stream to fileObtain joint positions from recording
12 DATA GATHERINGto gather data for analysis, used six subjects and did three variations of sit to stand exercisesexplain sit to standsubject sat in a chair directly facing the kinect with their feet on a balance board. the tv displayed the running program so they could see the depth map and skeleton on the screen
13 DATA ANALYSIS Sampling rates of joint position data Identifying phases of movement from joint positionsConsistency and stability of joint positions
14 SAMPLING RATE OpenNI Microsoft Average Frame Rate (fps) 25.0 19.6 Std Deviation(between trials)5.82.3Minimum9.814.1Maximum30.023.7OpenNI a little fasterMicrosoft more consistentBoth more than adequate even at their worst
15 IDENTIFYING PHASES OF MOVEMENT sit to stand exerciseeasy to pick out the parts of the exercisenote head bob
16 Standard Deviation of Joint Positions while Subject is Motionless DATA STABILITYStandard Deviation of Joint Positionswhile Subject is MotionlessJointOpenNI (cm)Microsoft (cm)Head0.341.8Hip0.421.2Knee0.701.5
17 DATA STABILITY: Assisted Tests Clinical therapy often involves an assistant supporting a patient while he performs exercisesTest procedure:subject begins by sitting aloneassistant joins, putting hands on subject’s shoulderssubject stands upconducted tests with many variations like type of exercise and distance, but for the sake of time i’ll focus on the effects of adding a person in the scene
18 DATA STABILITY: Assisted Tests one type of irregularity observed occurred as assistant entered the sceneprobably because the algorithm was determining which parts of the depth image belonged to which personnot a huge problem because they could be easily identified as errors and corrected: it is clear where the true value lies
19 DATA STABILITY: Assisted Tests another type of variation observed was skeleton mergingafter the patient stood, the algorithm identified the people as one and produced a strange-looking skeleton (ex: identify assistant’s head as if it were the patient’s shoulder)incorrect positions, and data was less stablemore of a problem because it is not certain where the true value of the person’s head position lies
20 DATA STABILITY: Assisted Tests It is at least possible to distinguish between normal deviation and deviation with skeleton mergingMore investigation needs to be done to iron out this obstacle since it could become a real problem in a clinical setting
21 CONCLUSIONSOpenNI Framework and Microsoft SDK for Windows are best tools to useCan provide significant functionality in a joint-tracking applicationtrack and record joint positions in three dimensionsdisplay image of tracked joints in real timeintegrate Kinect with the Wii balance boardSampling rate exceeds acceptable levelPhases of movement are easily identifiable from graphs of joint positionsJoint position stability is more than adequate with one subject in viewSkeleton merging could pose a problem for clinical use of Kinect
22 FUTURE WORK Deeper investigation into assisted exercises Different types of exercisesPosition the assistant differentlyDetermine conditions causing skeleton mergingFurther development of softwareInvestigate applications in other fields of physical therapy-conditions causing skeleton merging: relative heights of assistant and patient-further development of software: example -- guided in-home exercises, with constructive feedback, variation in difficulty levels, performance reports for physicians-other physical therapy: athletic injuries, chronic pain, neurological diseases (parkinsons, multiple sclerosis, cerebral palsy)