1. Context-Aware Computing
An Overview of
Context-Aware Computing
Jason I. Hong
Group for User Interface Research
University of California at Berkeley

2. Motivation Modern computers are divorced from reality
Unaware of who, where, and what around them
Leads to mismatch
Computers have extremely limited input
Aware of explicit input only
Can take a lot of effort to do simple things
Context-Aware Computing
Making computers more aware of the physical and social worlds we live in
Breaking computers out of the box
Mismatch between how people do things and what today's computers can offer
Screen saver activates at wrong times
Cell phones interrupt us at bad times
Apr

3. Why Context-Aware Computing?
Existing Examples
Context Types
Human Concern
Room Activity
Auto Lights On / Off
Convenience
Personal Identity
& Time
File Systems
Finding Info
Time
Calendar Reminders
Memory
Context-Awareness isn't new, it's fundamental to how we build things (tools and applications)
Room Activity
Smoke Alarm
Safety
Object Identity
Barcode Scanners
Efficiency
Apr

4. Technology Trends Sensors Recognition algorithms Wireless technologies
GPS, Active Badges, Active Bats
Smart Dust
Cameras and microphones
Recognition algorithms
MSR Radar location from
Smart Floor footstep force
Wireless technologies
Bluetooth, , cell phones
Three trends are changing computer-based context-awareness
Picture of BATS
Other recognition algorithms
Biometrics - Speaker identification
Extracting location from Wireless LAN
Vision recognition (object identification, face recognition)
Apr

5. Why Context-Aware Computing?
Potential Examples
Existing Examples
Context Types
Human Concern
Auto Cell Phone
Off In Meetings
Auto Lights On / Off
Activity
Identity
Time
Location
Proximity
Activity
History …
Convenience
Tag Photos
File Systems
Activity
Finding Info
Calendar Reminders
Proximal Reminders
Identity
Memory
Here are some potential context-aware applications enabled by these new technologies
Other concerns
Rapid decision making
Health for elderly and for young
Multimodal interaction and disambiguation
Efficiency
Safety
Health Alert
Smoke Alarm
Identity & Time
Safety
Barcode Scanners
Service Fleet
Dispatching
Time
Efficiency
Apr

6. Outline Background and Motivation Sample Context-Aware Apps
Building Context-Aware Apps
Open Research Challenges
Summary
Apr

7. Sample Context-Aware Apps1 2
Active Badges
ParcTabs
Cyberguide
Enhanced PDAs 3 4
Apr

8. Sample Context-Aware Apps Active Badges
Badges emit infrared signals
Gives rough location + ID
Teleport
Redirect screen output from "home" computer to nearby computer
Phone forwarding
Automatically forward phone calls to nearest phone
Active Badge
Olivetti / AT&T
Hopper, Harter, et al
Apr

9. Sample Context-Aware Apps ParcTabs
Active badge + wireless
Rough location + ID
Proximate selection
Interfaces for nearby objects
Auto-diaries
People, places, and time
Triggers
Alerts on preset events
Reconfiguration
Bind device to room
ParcTabs
Xerox PARC
Want, Schilit, et al
Apr

10. Sample Context-Aware Apps ParcTabs
Active badge + wireless
Rough location + ID
Proximate selection
Interfaces for nearby objects
Auto-diaries
People, places, and time
Triggers
Alerts on preset events
Reconfiguration
Bind device to room
ParcTabs
Xerox PARC
Want, Schilit, et al
Apr

11. Sample Context-Aware Apps ParcTabs
Active badge + wireless
Rough location + ID
Proximate selection
Interfaces for nearby objects
Auto-diaries
People, places, and time
Triggers
Alerts on preset events
Reconfiguration
Bind device to room
ParcTabs
Xerox PARC
Want, Schilit, et al
Apr

12. Sample Context-Aware Apps ParcTabs
Active badge + wireless
Rough location + ID
Proximate selection
Interfaces for nearby objects
Auto-diaries
People, places, and time
Triggers
Alerts on preset events
Reconfiguration
Bind device to room
ParcTabs
Xerox PARC
Want, Schilit, et al
Apr

13. Forget-me-not system
Apr

14. Apr

15. Apr

16. Sample Context-Aware Apps ParcTabs
Active badge + wireless
Rough location + ID
Proximate selection
Interfaces for nearby objects
Auto-diaries
People, places, and time
Triggers
Alerts on preset events
Reconfiguration
Bind device to room
ParcTabs
Xerox PARC
Want, Schilit, et al
Apr

17. "Like living in a rule-based expert system!"
Context-Aware Computing Applications
Bill Schilit, Norman Adams, Roy Want
"Like living in a rule-based expert system!"
Apr

18. Sample Context-Aware Apps ParcTabs
Active badge + wireless
Rough location + ID
Proximate selection
Interfaces for nearby objects
Auto-diaries
People, places, and time
Triggers
Alerts on preset events
Reconfiguration
Bind device to room
ParcTabs
Xerox PARC
Want, Schilit, et al
Apr

19. Sample Context-Aware Apps Tour Guides
GPS or infrared tracking
Fairly precise location
Display location on screen
Predefined points of interest
Automatically pop up if nearby
Travel journal
Keep log of places seen and photographs taken
The most often created context-aware application
Cyberguide
Georgia Tech
Abowd et al
Apr

20. Sample Context-Aware Apps Tour Guides
Cyberguide
Georgia Tech
Abowd et al
Apr

21. Sample Context-Aware Apps Tour Guides
Cyberguide
Georgia Tech
Abowd et al
Apr

22. Sample Context-Aware Apps Enhanced PDAs
Microsoft Research
Hinckley et al
Apr

23. Sample Context-Aware Apps Enhanced PDAs
Voice memo recording
Hold like phone near mouth to start recording
Portrait / Landscape mode
Just physically rotate screen
Tilt scrolling
Tilt instead of scrollbars
Power management
Turn on if being held and tilted
Microsoft Research
Hinckley et al
Apr

24. Some Issues in Context-Aware Computing
Sensor ambiguity
Sensors not 100% reliable
Precision / Accuracy / Granularity
Sensor Fusion
Merging different sensor inputs together
Self-contained vs. distributed systems
PDA doesn't need location sensors if it can ask nearby sensors to approximate
Requires lots of knowledge and effort to build
Sensors, recognition algorithms, devices, application
Few kinds of context beyond location + ID used
There haven't been many rigorous evaluations of utility
Not Precise
Not Accurate
Precise
Rough Granularity
Fine Granularity
Apr

25. Some Issues in Context-Aware Computing
Sensor ambiguity
Sensors not 100% reliable
Precision / Accuracy / Granularity
Sensor Fusion
Merging different sensor inputs together
Self-contained vs. distributed systems
PDA doesn't need location sensors if it can ask nearby sensors to approximate
Requires lots of knowledge and effort to build
Sensors, recognition algorithms, devices, application
Few kinds of context beyond location + ID used
There haven't been many rigorous evaluations of utility
Apr

26. Outline Background and Motivation Sample Context-Aware Apps
Building Context-Aware Apps
Open Research Challenges
Summary
Apr

27. Building Context-Aware Apps
Describe support at app-level
ParcTab System
Context Toolkit
MUSE
Cooltown
Mobisaic
Stick-E Notes
Context Fabric
Try to make it easier to build a certain class of context-aware apps
Apr

28. A Rough Taxonomy of Context-Aware Apps
Triggers
Metadata Tagging
Reconfiguration and Streamlining
Input specification
Presentation
Apr

29. A Rough Taxonomy of Context-Aware Apps
Triggers
On X do Y
"Notify doctor and nearby ambulances if serious health problem detected"
"Remind me to talk to Chris about user studies next time I see him"
Metadata Tagging
"Where was this picture taken?"
"Find all notes taken while Mae was talking"
Memory prosthesis
"Who is that person sitting next to Uncle Ralph?"
"Where on earth was that picture taken?"
"Was this a photo from my second wedding? Or my third?"
Apr

30. A Rough Taxonomy of Context-Aware Apps
Reconfiguration and Streamlining
Telephone forwarding and Teleport
Turn off cell phone in theaters
Automatically adjust brightness / volume
Automatic file pre-caching
Select modes in multimodal interaction
Multimedia / Bandwidth adaptation
Apr

31. A Rough Taxonomy of Context-Aware Apps
Input specification
Send mail only to people in building now
Print to nearest printer
"Find gas stations nearest me"
Presentation
Current location
Idle?
Currently in?
Contextual info about objects
Proximate selection
Apr

32. ParcTab System (Schilit 1995)
Three key abstractions
Device Agents, User Agents, Active Maps
Device agents for maintaining the status of devices
User agents for managing user preferences
Active maps for maintaining the location of devices and users
Apr

33. ParcTab System (Schilit 1995)
Queries
Check context state in Agents or Active Maps
{Room LID:35-2-*}
{{Printer *} {model lw}}
Events
Standing queries
Notified on changes in context state
Apr

34. Context Toolkit (Dey, Salber, Abowd 2001)
Toolkit for distributed context-aware apps
Framework for acquiring and handling context
Standard components
Three key abstractions
Widgets, Interpreters, and Aggregators
Apr

35. Context Toolkit (Dey, Salber, Abowd 2001)
Widgets abstract out sensors
App
App
Location
Widget
Active
Badge
Active
Badge
GPS
Cell Phone
Location
Apr

36. Context Toolkit (Dey, Salber, Abowd 2001)
Interpreters transform context data
App
Location
to Room
Interpreter
Location
to Street
Interpreter
Location
Widget
Apr

37. Context Toolkit (Dey, Salber, Abowd 2001)
Aggregators group related widgets together
App
App
Location
to Room
Interpreter
Person
Aggregator
Location
Widget
Activity
Widget
Affect
Widget
Apr

38. Context Toolkit (Dey, Salber, Abowd 2001)
In / Out Board
Apr

39. Context Toolkit (Dey, Salber, Abowd 2001)
Dummbo
Apr

40. MUSE (Castro and Muntz 2000)
Sensor management issues
Sensor discovery (currently via Sun's Jini)
Specify performance goals and resource constraints
"Find a printer but minimize power consumption"
Probabilistically modeling sensor data
Can measure service cost
Sensor fusion (via Bayesian nets and HMMs)
Apr

41. CoolTown (Hewlett-Packard 2000)
Since web is universal, link context awareness to web
Literally everything has a URL
People, places, things
Infrared beacons, bar codes, etc
Literally everything has a web page
Current status, contact info, services offered, etc
Apr

42. CoolTown (Hewlett-Packard 2000)
SF Exploratorium
Get extra info
Create bookmarks for later use
Apr

43. CoolTown (Hewlett-Packard 2000)
Buses equipped with GPS and net
In bus
Show location
Show nearby points of interest
Waiting for bus
Show wait time
Apr

44. Mobisaic (Voelker and Bershad 1994)
Context-awareness + web page authoring
Dynamic URLs
Variables processed by modified web browser
Active Documents
Extensions to HTML
<!- subscribe to $(Location), reload ->
Apr

45. Stick-E Notes (Pascoe 1997)
Authoring tool for using context
<note>
<required>
<at> (1,4) .. (3,5)
<facing>
<during> December
<body>
The large floodlit building at the
bottom of the hill is the cathedral.
Apr

46. Context Fabric (Hong and Landay 2001)
Distributed sensors, services, and devices
Won't be designed a priori for interoperability
Shift burden of context-awareness from individual devices onto infrastructure
Provide basic services usable by any device
Provide a high-level way of specifying context needs
Dynamically coordinate sensors and services
Apr

47. Context Fabric (Hong and Landay 2001)
Push burden of context-awareness from individual devices onto network
Share computational power, data, sensors
Support incremental evolution
Support richer set of devices
Provide core set of context services
Basic services that exist everywhere
Uniform level of abstraction hiding nasty details
Apr

48. Automatic Path Creation
A way of assembling small services into larger, more interesting ones
Operators
Paths
ZIP
Weather
GPS
Cell
GPS
ZIP
ZIP
Theaters
GPS
ZIP
ZIP
Weather
Apr

49. Automatic Path Creation
Sensor Input
ZIP
Theaters
GPS
What are the nearby movie theaters?
Cell
Context Output
Movie
Suppose we had a way to specify context needs
Suppose we had a way to specify service semantics
Automatically assemble paths based on resources
Don't have to know specific sensors and services
Apr

50. Context Fabric Architecture
App
Application
Layer
Context
Specification
Language
Context
Event
Service
Context
Query
Service
Services
Context
Layer
Automatic Path Creation
Logical Sensor
Layer
Sensor Management Service
Physical Sensor
Layer
Apr

51. Outline Background and Motivation Sample Context-Aware Apps
Building Context-Aware Apps
Open Research Challenges
Summary
Apr

52. Open Research Challenges Systems Issues
Programming model
Programming the physical world
Unreliable sensors, recognition algorithms, plus standard distributed computing issues
Interoperability
Sensors, services, and devices
Useless if everyone has proprietary / custom systems
Need standard data formats, protocols, and frameworks
Varying capabilities of sensors, services, and devices
Apr

53. Open Research Challenges Systems Issues
Need clearer definitions of context
What is and isn't context?
Temperature? The monitor I'm looking at? Personal history?
Have to avoid the AI tarpit (ie does it matter?)
Apr

54. Open Research Challenges People Issues
Avoiding embarrassing situations
Active Badges + bathrooms
Inconvenient phone forwarding
Avoiding dangerous situations
Need to take into consideration cost of mistake
Smoke alarms when cooking
Lights that turn off when you're still there
Woman locked in "smart toilet stall"
Will adding more context really help here?
Apr

55. Open Research Challenges People Issues
Making it predictable and understandable
Setting preferences
"I want my cell phone to ring except in theaters and when I'm in a meeting unless…"
Why the heck did it do that?
Privacy
What does the computer know about me?
What do others know about me?
What do I gain? What do I lose?
Apr

56. Take Home Ideas
Strides in sensors, recognition, and wireless are enabling new context apps
Still need to lower barriers to entry
Lots of systems issues, even more people issues
Lots of potential here for new kinds of interaction and applications
Apr

57. The End Group for User Interface Research
University of California at Berkeley
Context Fabric Link
Apr

58. Backup Slides
Apr

59. Proposed Solution App Agents Context Specification Language Context
Application
Layer
Device Agents
Context
Specification
Language
User Agents
Context
Event
Service
Context
Query
Service
Services
Context
Layer
Automatic Path Creation
Logical Sensor
Layer
Sensor Management Service
Physical Sensor
Layer
Apr

60. Context Specification Language
Need to express context about and relationships between People, Places, Things
Predicates
Identity (Who is…? What is…? Is with…?)
Location (Near? Nearest? Distance? Path?)
Activity (Is busy? Is in meeting? Current task?)
Time (In past? In present? In future? On date?)
Some of this vocabulary done by Schilit
Implicitly encoded in his APIs
One goal is to extend his work in spec language
Another is to make it extensible for future context types
Apr

61. Context Specification Language
Common parameters
Max number of results wanted
Return name
Return data type (e.g. String, List, Table)
Minimal probability of correctness desired
Relevant sensor input requestor has
Event parameters
Event rate (e.g. at most 1 event per second)
Event callback (e.g. RPC, socket port)
Max number of events desired
Granularity of change (e.g. 1 meter)
Apr

62. What is Context? Situated Action (Suchman) Activity Theory (Nardi)
Responsiveness, improvisation to situation
Actions are moment-by-moment, highly fluid on situation
Activity Theory (Nardi)
Subject, Object[ive], Operations, Artifacts, Env
Activity defines context but it changes as list above changes
Context depends on sensing and inferring, but difficult to know people or goals
Context def 1 - "the interrelated conditions in which something exists or occurs" (
Context def 2 - "that which surrounds and gives meaning to something else" (Free On-line Dictionary of Computing)
Context-aware computing is making computers more aware of the physical and social worlds we live in (loose and vague definition on purpose)
Apr

63. What is Context?
Provides fair criticisms on existing context-aware apps
Response: Focus on the more routine and predictable actions
Response: Focus on simple first-order approx
Examples: Smoke alarms, Calendar reminders
Response: Is occasionally wrong, but won't get better unless we do more research here
Response: Infrastructure enables more research in this area
Apr

64. What is Context? Merging of virtual with physical and social
Mostly implicit input, often secondary to task at hand
People / Places / Things / Virtual / Time
People: Location, Identity, Task, Affect
Places: Activity, People, Temperature, Audio
Things: Location, Identity
Virtual: Services, Bandwidth
Time: All of these Past, Present, and Future
Apr

65. What is Context? But it's a rathole to try to define it…
Workflows? (Greenberg 2001)
Intent and Desire?
Natural Language Processing?
What isn't context?
Instead, operationalize it
Distributed, multi-device, sensor-based
Fairly well-defined and computable concepts (e.g. location and identity, but not intent or workflow)
Apr

66. Why Context-Awareness?
Skeptics toward the ubiquitous computing movement quickly point out that those involved in promoting Mark Weiser's initial vision are mainly interested in pushing technology…
As a reaction against this fair criticism, many researchers spend countless hours trying to dream up the "killer application" that will cause everyone to adopt ubiquitous computing technology.
I think this is the wrong goal to have in mind as a researcher for two reasons. First, we are not schooled in the practices to judge a market…
Second, my interpretation of Weiser's vision is not that there would be a single, most compelling use of ubicomp technology, but rather that the totality of the experience would cause a significant paradigm shift in the way we as humans perceive our relationship to interactive computing.
In short, Weiser argued for what I would like to call a "killer existence," in which many disparate devices would provide an array of services in ways that seamlessly complement our lifestyles.
-- Gregory Abowd, 2001
Apr

67. Why Infrastructure Approach?
Sharing of data
Collaborative filtering, large data sets
Sharing of processing power
Heavy-duty processing on dedicated machines
Sharing of sensors
Sensors in the environment
In-place evolution of services
New algorithms, new features
Supports heterogeneity
Sensors, Devices, OS, Programming Langs
Apr

68. Why Infrastructure Approach?
Building and evaluating now can
Provide more design space for interaction
Let us rapidly prototype and evaluate these apps
Find problems now
Develop the right mechanisms
Evolve the right policies
Apr

69. Some Sensors Temperature Pressure Humidity Soil makeup Acceleration
Noise
Material stress
Biometric
Motion
Distance
Touch
Location
Orientation
Light
Apr

70. Context-Aware Mechanisms
Events
Something interesting has just happened
e.g. "Larry has entered room 306"
Queries
Introspect current context state (discrete)
e.g. "Where is Larry now?"
Filters
Modify events and queries based on preferences and privacy
e.g. "Only show others if in office or not"
Events and Queries are two mechanisms for context-aware applications, but really two sides of same coin
Trigger is computer-initiated use of context information
Query is person-initiated of context information
Focus on discrete queries for now. Continuous queries difficult problem in and of itself.
Triggers / queries typically used for alerts, reconfiguration, or information
Filters for personalization and security
Apr

71. Are Queries and Events Enough?
Query
Event
Triggers
On Event do Action
Metadata Tagging
Query for relevant
context state
Reconfiguration
Query current context state
before and while running
On Event do Reconfigure
Input Specification
Query current context state
to fill in the blanks
"Who is that person sitting next to Uncle Ralph?"
"Where on earth was that picture taken?"
"Was this a photo from my second wedding? Or my third?"
Presentation
Query current state
for more information
On Event do
Update Presentation
Apr