1. Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided in part by IISI and AFRL/IF

2. Making Sense of Sensors or … Climbing the Data Interpretation Food-Chain

3. The Ubiquitous Future Rapidly declining size and cost of sensing and networking technology makes it practical to rapidly deploy systems that monitor large environments in great detail –factories, airports, hospitals, homes –oceanic regions, cities, countryside Problem: it is easier to collect data than make to sense of it!

4. Data Fusion Traditional work in data-fusion attacks problem of recovering specific physical phenomena from the readings of homogeneous networks of noisy sensors E.g.: given readings from underwater microphone array, determine the position of a submarine

5. Current Trends Heterogeneous sensors –Instrumented environment: motion detectors, weight detectors, video, audio, … –Instrumented personnel: smart badges, GPS phones, metabolic sensors. … Goal: high-level understanding –What actions are being performed? –What are the goals of the subjects? –Do we need to intervene?

6. Example: Security System monitors activity in a post office Tracks common tasks performed by individuals –Mailing packages –Getting mail from PO boxes –Buying stamps Alerts operator when abnormalities noted –Person leaves package on floor and exits –Loitering (but not waiting in line!)

7. Example: Guiding Activity Compass: GPS system that –Learns daily patterns of travel –Understands walking, car, bus, bike –Integrates external information Real-time bus data Predicts problems –Will user miss appointment? –Is user on the wrong bus? Offer proactive help –E.g., suggest alternative travel plan

8. Triple-Use Technology Plan-Aware Computing Military surveillance augmented cognition Commercial Software intelligent user interfaces Patient Care aging in place assisted cognition

9. Key Issue How to go from noisy and incomplete sensor measurements to A meaningful description of what a person is doing “Waiting to mail package” “Trying to get home” A decision by the system about whether or not to intervene … in a principled and scalable manner!

10. Data Interpretation Food Chain Movement IntentionsBehaviorInterventions

11. Model-Based Interpretation General approach: build a probabilistic model of –Common user goals –Plans (complex behaviors) that achieve those goals Feasibility constraints Temporal constraints Failure (abnormality) modes – How simple behaviors are sensed Run model “backwards” to interpret sensed data

12. Million-Mile View In principal we know how to estimate the state of the system under observation: To make this practical, we must take advantage of the regular structure of the domain state at time t observation at time t system dynamics

13. Technical Foundations Hidden Markov models –Mathematical framework for describing processes with hidden state that must be inferred from observations Hierarchical plan networks –Represents how a task can be broken down into subtasks Hierarchical hidden Markov models* –Key to climbing food-chain! * Precisely speaking: factorial hierarchical hidden semi-Markov models

14. VideoDoor SensorMotion Location Example Enter PO Wait in line Let go package Pay cashier Exit PO Mail Package

15. VideoDoor SensorMotion Location Enter PO Go to PO boxes Open PO box Pick up mail Exit PO Retrieve Mail Example

16. VideoDoor SensorMotion Location Mail Package PO Patron Retrieve Mail Outside PO Example

17. Inexplicable Observations Enter PO Wait in line Let go package Pay cashier Exit PO Mail Package Enter PO Go to PO boxes Open PO box Pick up mail Exit PO Retrieve Mail Enter PO Let go package Exit PO

18. Absolute Timing Constraints Mail Package active [9 am – 4 pm] Enter PO Retrieve Mail active [6 am – 8 pm] Enter PO

19. Relative Timing Constraints Go to PO boxes Open PO box Retrieve Mail Timeout seconds Forgot combo? Safecracking?

20. Summary Commonsense knowledge base of “significant” behaviors –Hierarchically organized –Probabilistic at all levels –Many parallel ongoing activities possible –Absolute and relative timing constraints –Probabilities “tuned” by machine learning techniques for individual users –Inexplicable observations and failure modes – points of possible intervention

21. Interventions Framework allows system to predict when an anomalous situation is likely Different anomalies have different costs –Confused patron –Deliberate loitering –Planting bomb Must avoid:

22. Deciding When to Intervene (Horvitz 98) G = prediction that help is needed

23. Common Architecture

25. Activity Compass Palm-based wireless GPS –No explicit programming – learns pattern of transportation plans –Accesses user’s calendar, real-time bus information –Constantly tries to predict where user will go next, and whether problems will arise –Proactive help: “Walk faster or you’ll miss the 9:15 bus!” “Green St bus is late, suggest you take Elm St bus instead”

26. Substeps Cleaning up GPS data –3 meter accuracy –frequent signal loss –determine most likely path Infer mode of transportation Predict when and where transitions in mode of travel will occur Predict points of possible failure indoors walk bus bike car

27. Gathering Data

28. On Foot: Across Campus

29. By Bus: Across Seattle

30. Transition Prediction Training Data: –20,000 GPS readings gathered over 3 weeks Inferring current mode –Input: current location, time, velocity –98% accuracy (10 FCV) Predicting next transition –Input: current mode, location, time, velocity –97% accuracy (10 FCV)* * Don is a very organized guy. Your accuracy may vary.

31. Predicting Transition Location

32. User Interface

33. Assisted Cognition “Plan aware” systems to help people with cognitive disabilities New project based at University of Washington –Computer Science & Engineering –UW Medical Center, ADRC –Collaborators: Intel, OGI, Elite Care http://assistcog.cs.washington.edu/

34. Summary Potential of widespread sensor networks just beginning to be tapped Key issue: interpreting data in terms of human behavior, plans, and goals Researchers in data fusion, AI, and “ubicomp” coming together around a core set of representations and algorithms