Repetition Counting With Microsoft Kinect Presented by: Jonathan Gurary Dai Jun

Proposal Design a Kinect-based system for physical therapy activities  Usable by the end-user in their home  No professional staff or facility needed  Simple UI anyone can use  Similar in performance to expensive 8- camera setups Identify and count a variety of motions

Implementation Hardware: Microsoft Kinect Software: OpenNI API, executed through Processing  Designed for visual software, communicating with vision sensors

OpenNI Cross-platform support -Linux, Mac, Windows Older mod community -Less updates = more working samples -Issues are better documented And there's books -First Kinect SDK books just hit shelves -OpenNI has a couple books already

Limiting Kinect Problems Bad guesses  Lost a limb? It must be straight at your side! Mismatched limbs  Crossed legs or arms can “swap” Jitter  Shaky and unstable reads These can happen with no effect on Confidence

Effects on Repetition Counting Angles and positions will be collected in error  Almost certainly out-of-bounds for the exercise  Best case: Happens during non-critical portion of motion, no effect  Worst case: miss flag, false count, reset the motion

Our Solution to Erroneous data Make sure the user knows what the Kinect is seeing  Jerky skeletons, mismatched or lost limbs, all very obvious. Just stop and try again. Stop looking at data when confidence drops  Allows for the split-second guessing errors common in some motions. Use a tolerance  Mitigate the effects of jittery reads

Flag-Based Motion Identification Each motion or exercise is broken down and split into five intermediate steps Example: waving your hand  Change in elbow angle of about 50 degrees  Step 1: elbow angle reduces 10 degrees  Step 2: elbow angle reduces 10 more degrees

Flag-Based Motion Identification Every step becomes a flag to be passed  Must be passed in order  All flags reset (exercise failed) if a body part goes out of bounds  If every flag is passed correctly, count the action

How Are Flags Determined Flags are manually input by the coder Can use angles of all major limbs, height of torso, and distance of hands and feet  Potentially (future feature) scale to user's height

Recording and Playback Main program records user's entire session automatically  Saved during execution to a.onni file Separate program available for playback of session.  Skeleton filtered out  Can pause, fast forward, slow down, or seek through recording

User Interface Design User Friendly, easy to use -no programming knowledge required -motion icons to choose -works like a button, but beautiful! Replay mode -control bar -works like a real media player!

User Interface Design

Dynamic Flag Setup Identifying flags by examining Kinect output is tedious Dynamic program allows the user to “input” flags with their body  Allows any almost any motion to be input on the fly  Data is output to a text file, allowing coder to turn the input into a featured exercise easily

Dynamic Flag Limitations Impossible to tell what tolerance will be needed Whole body is captured  All limbs must be visible and not guessed.  Exercise is limited to the position (distance wise) that it was started at  Counting isn't always accurate, especially for small or complex motions

Demo Presentation

Summary Developed a relatively accurate, user friendly physical therapy program Count repetitions of exercises Exercises can be set by a professional, added to the code  Dynamic flag setter to identify motion, then potentially added to exercises by coder User can view raw footage of exercises

Future Features Allow the dynamic flag set to ignore some parts of the body Adjust for big height differences between users Allow a health professional to easily add a dynamically created exercise to code