Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided in part by IISI and AFRL/IF

Making Sense of Sensors or … Climbing the Data Interpretation Food-Chain

The Ubiquitous Future Rapidly declining size and cost of sensing and networking technology makes it practical to rapidly deploy systems that monitor large environments in great detail –factories, airports, hospitals, homes –oceanic regions, cities, countryside Problem: it is easier to collect data than make to sense of it!

Data Fusion Traditional work in data-fusion attacks problem of recovering specific physical phenomena from the readings of homogeneous networks of noisy sensors E.g.: given readings from underwater microphone array, determine the position of a submarine

Current Trends Heterogeneous sensors –Instrumented environment: motion detectors, weight detectors, video, audio, … –Instrumented personnel: smart badges, GPS phones, metabolic sensors. … Goal: high-level understanding –What actions are being performed? –What are the goals of the subjects? –Do we need to intervene?

Example: Security System monitors activity in a post office Tracks common tasks performed by individuals –Mailing packages –Getting mail from PO boxes –Buying stamps Alerts operator when abnormalities noted –Person leaves package on floor and exits –Loitering (but not waiting in line!)

Example: Guiding Activity Compass: GPS system that –Learns daily patterns of travel –Understands walking, car, bus, bike –Integrates external information Real-time bus data Predicts problems –Will user miss appointment? –Is user on the wrong bus? Offer proactive help –E.g., suggest alternative travel plan

Triple-Use Technology Plan-Aware Computing Military surveillance augmented cognition Commercial Software intelligent user interfaces Patient Care aging in place assisted cognition

Key Issue How to go from noisy and incomplete sensor measurements to A meaningful description of what a person is doing “Waiting to mail package” “Trying to get home” A decision by the system about whether or not to intervene … in a principled and scalable manner!

Data Interpretation Food Chain Movement IntentionsBehaviorInterventions

Model-Based Interpretation General approach: build a probabilistic model of –Common user goals –Plans (complex behaviors) that achieve those goals Feasibility constraints Temporal constraints Failure (abnormality) modes – How simple behaviors are sensed Run model “backwards” to interpret sensed data

Million-Mile View In principal we know how to estimate the state of the system under observation: To make this practical, we must take advantage of the regular structure of the domain state at time t observation at time t system dynamics

Technical Foundations Hidden Markov models –Mathematical framework for describing processes with hidden state that must be inferred from observations Hierarchical plan networks –Represents how a task can be broken down into subtasks Hierarchical hidden Markov models* –Key to climbing food-chain! * Precisely speaking: factorial hierarchical hidden semi-Markov models

VideoDoor SensorMotion Location Example Enter PO Wait in line Let go package Pay cashier Exit PO Mail Package

VideoDoor SensorMotion Location Enter PO Go to PO boxes Open PO box Pick up mail Exit PO Retrieve Mail Example

VideoDoor SensorMotion Location Mail Package PO Patron Retrieve Mail Outside PO Example

Inexplicable Observations Enter PO Wait in line Let go package Pay cashier Exit PO Mail Package Enter PO Go to PO boxes Open PO box Pick up mail Exit PO Retrieve Mail Enter PO Let go package Exit PO

Absolute Timing Constraints Mail Package active [9 am – 4 pm] Enter PO Retrieve Mail active [6 am – 8 pm] Enter PO

Relative Timing Constraints Go to PO boxes Open PO box Retrieve Mail Timeout seconds Forgot combo? Safecracking?

Summary Commonsense knowledge base of “significant” behaviors –Hierarchically organized –Probabilistic at all levels –Many parallel ongoing activities possible –Absolute and relative timing constraints –Probabilities “tuned” by machine learning techniques for individual users –Inexplicable observations and failure modes – points of possible intervention

Interventions Framework allows system to predict when an anomalous situation is likely Different anomalies have different costs –Confused patron –Deliberate loitering –Planting bomb Must avoid:

Deciding When to Intervene (Horvitz 98) G = prediction that help is needed

Common Architecture

Activity Compass Palm-based wireless GPS –No explicit programming – learns pattern of transportation plans –Accesses user’s calendar, real-time bus information –Constantly tries to predict where user will go next, and whether problems will arise –Proactive help: “Walk faster or you’ll miss the 9:15 bus!” “Green St bus is late, suggest you take Elm St bus instead”

Substeps Cleaning up GPS data –3 meter accuracy –frequent signal loss –determine most likely path Infer mode of transportation Predict when and where transitions in mode of travel will occur Predict points of possible failure indoors walk bus bike car

Gathering Data

On Foot: Across Campus

By Bus: Across Seattle

Transition Prediction Training Data: –20,000 GPS readings gathered over 3 weeks Inferring current mode –Input: current location, time, velocity –98% accuracy (10 FCV) Predicting next transition –Input: current mode, location, time, velocity –97% accuracy (10 FCV)* * Don is a very organized guy. Your accuracy may vary.

Predicting Transition Location

User Interface

Assisted Cognition “Plan aware” systems to help people with cognitive disabilities New project based at University of Washington –Computer Science & Engineering –UW Medical Center, ADRC –Collaborators: Intel, OGI, Elite Care

Summary Potential of widespread sensor networks just beginning to be tapped Key issue: interpreting data in terms of human behavior, plans, and goals Researchers in data fusion, AI, and “ubicomp” coming together around a core set of representations and algorithms