2. Lynne Grewe, Steven Magaña-Zook CSUEB, lynne.grewe@csueastbay.edu A cyber-physical system for senior collapse detection

3. Seniors Falling Over 1/3rd of seniors above 65 fall each year Lead to serious injury and even death Falls account for 25% of all hospital admissions, and 40% of all nursing home admissions 40% of those admitted do not return to independent living; 25% die within a year. Fast medical attention can make a difference Many falls do not result in injuries, yet a large percentage of non-injured fallers (47%) cannot get up without assistance.

4. Cost of Falling? 2005, CDC study – Cost for Falls leading to fatality

5. Goal create a “smart home” system to predict and detect the falling of senior/geriatric participants in home environments More seniors living at home autonomously

6. SCD: Senior Collapse Detection Overview

7. SCD: uses Kinect Sensor Inexpensive, commercial, well tested, good API support Modalityexample 2D 3D Audio

8. Feature Extraction Perform Skeleton Tracking Ideal – fall indicators often involve joint locations and range of motion Good Resolution – 21 joints

9. Skeleton Tracking Has Noise Degrading performance with occlusion General Twitching Also degrades as more occlusion from being on floor << not bad << notice rear leg position problem from self occlusion

10. Noise Reduction: Physical Therapy Skeleton Model Use Physical Therapy Model data to determine normal range of motion and joint distances. Calculate joint certainty metric = f(joint angles, joint distances, physical therapy skeleton model) = 1 if within limits of model <1 non-linear function of deviation from model Currently use 1 model based on maximum ranges Future = model for different demographics (age, height, weight), or learned from user. Concept = Can use Joint Reliability to determine if a joint should be used in Fall Detection OR can use in determination of confidence of a Fall detected

11. What is a Fall? How can we detect it? SCD defines fall as “loss of control resulting in downward motion ending with body on floor” Previous work: Wearable devices: Accelerometers, gyroscopes, movement sensors Autonomous: 2D with mixed results 3D beginning work Detection Ideas Quick movement (acceleration) – whole or what part of body? Body Orientation – parallel to floor Location – little but, some looking at general location

12. SCD Fall Detectors Currently 3 based on all ideas (location, orientation and acceleration). Currently operate independently – any can trigger fall detection event

13. Location –need Floor Detection Uses 3D floor plane detected by Kinect Sensor One for each skeleton calculated Good News- Gamers want this accurate Ax + By + Cz + D = 0

14. SCD: Head Movement Detector Falling Detector / Idea: quick movement indicates falling Measure: both head joint velocity and trajectory (downward) and the head ends up near the floor. Buffer 1 second of data (30 frames / second) Trajectory – 2 slopes Empirically chosen Thresholds velocity>1ft/second Last frame of 1 second head position within 1.5 ft of floor Trajectory toward floor

15. SCD: Head Movement Detector – Reliability and Confidence Reliability: function (number tracked joints, number inferred joints) Confidence: function (velocity)

16. SCD: Horizontal Ratio Detector Fall Detector / Idea: senior lands on the floor in horizontal- parallel to floor orientation Concept = 3D bounding box 2 Ratios = Width/Height and Depth/Height Empirically chosen Threshold: 1.5 for either Ratio = elongated, parallel to floor Head Height Ratio FALL

17. SCD: On Floor Detector Fall Detector / Idea: senior lands on the floor Hip near floor Minimum number of joints near floor Empirically Chosen Thresholds Minimum 1 hip joint (out of 3 possible) Minimum 8 joints “near” floor “near” = 1.5 ft Reliability = #tracked / (#tracked + #inferred) = 0.25 threshold

18. How Many Falls? Some of our detectors are “Fallen” detectors Don’t want too many triggers for same fall Minimum time between fall events is set currently at 15 seconds. No data but, seemed fastest time between different falls Example: http://www.youtube.com/watch?v=Tm_fsp5puVk

19. Emergency Response Configure Emergency Contact(s) Email Phone – sms text

20. SCD: Speech Processing Use Microsoft SDK Text-To-Speech Use Microsoft SDK Speech Recognition Kinect has microphone array.

21. Fall Detection Event and Emergency Response System Senior Hears Audio Prompts from System – asking if assistance is needed. If Yes or No Response the predetermined emergency response is triggered Here you see both the Diagnostics GUI and an illustration of the final Audio

22. Examining Test case Head Motion Detector: FALL Trajectory = slope average was -1.258 Head Position Last Frame = 1.37ft from floor Velocity = 1.003ft/sec On Floor Detector: FALL 9 joints near floor All 3 hip joints on floor Horizontal Ratio Detector: FALL W/H = 1.7, D/H = 0.89 Head Distance to Floor = 1.37ft from floor

23. Both Live and Semi-Automated Testing Have ability to cycle through sets of pre-recorded data Output to HTML results

24. SCD: RESULTS OnFloor Performs best 100% On Floor Detector Horizontal Detector Head Movement Detector True Positive Rate 24 out of 2421 out of 2423 out of 24 False Positive Count 022 True Negative Rate 14 out of 1412 out of 14 False Negative Count 031

25. Issues, Problems and Future work

26. Limitations with Kinect Limited depth range(solution: multiple Kinect) Occlusion (solution: multiple Kinect or use tilt feature of Kinect)

27. Issues Skeleton engine needs some number of frames to recognize when user enters frame. This is unavoidable with current concept of skeleton tracking Processing – on common commercial home use laptops and desktops ($400-700) we experience a lag time when all diagnostics are being displayed from 1 to 20 seconds worst case to process frame leading to detection. Typical (little data) around 0.5-5 seconds.

28. Future Work More Testing Combine Decisions? Learn Formulation? Fine Tune/ Learn Thresholds Improve Performance Speeds Other modules Fall prediction = gait tracking Post Fall detection = rolling, vocalizations Learning Individual Physical Model Multi-Kinect System calibration, sensor inference, coordinated communication and decision making Kinect 1 improvement in resolution.