1. Attentive User Interfaces
by Falk Fleischer

2. Overview Introduction Principles & Methodologies Applications Summary
Attention - Sensitive Messaging
Mixed - Initiative Interactions
Summary
11/29/2018
Introduction

3. STRESS Handy Online Calendar Emails Instant Messages File Scout ...
11/29/2018
Introduction

4. Attentive User Interfaces
Human attention and working memory are limited resources!
11/29/2018
Introduction

5. Attentive User Interfaces
Goal of Attentive User Interfaces :
To negotiate the volume and timing of human - computer communication in a natural way
11/29/2018
Introduction

6. Attentive User Interfaces
enhance robustness and fluidity of human - computer collaboration
importance of attentive cues
goals
intentions
topics of interest
signal offering and accepting a contribution in dialogues
11/29/2018
Introduction

7. Principles & Methodologies
Overview
Introduction
Principles & Methodologies
Applications
Attention - Sensitive Messaging
Mixed - Initiative Interactions
Summary
11/29/2018
Principles & Methodologies

8. Principles & Methodologies
Key Features of AUIs
sensing attention
use perceptual sensors
microphones
cameras
GPS
accelerometers
interaction with software devices
online calendar
history of user’s interests and activities
11/29/2018
Principles & Methodologies

9. Principles & Methodologies
Key Features of AUIs
reasoning about attention
model user’s attentive state
reason about future attention and ideal attention-sensitive actions under uncertainty
uses
HMMs
Bayesian Networks
Influence Diagrams
11/29/2018
Principles & Methodologies

10. Principles & Methodologies
Bayesian Networks
abstract probabilisitic knowledge base reflecting causal knowledge
graph with following qualities
1. random variables as nodes
2. directed links between the nodes
3. each node with conditional probability table
4. no directed cycles
11/29/2018
Principles & Methodologies

11. Principles & Methodologies
Bayesian Networks
Body Position
Seminar- Room
Head Pose
Location
Focus of Attention
11/29/2018
Principles & Methodologies

12. Principles & Methodologies
Bayesian Networks
Task: compute probability distribution for a set of query variables given value of some evidence variables
P(Query | Evidence)
 decisions about actions possible
11/29/2018
Principles & Methodologies

13. Principles & Methodologies
Bayesian Networks
kinds of inferences:
Diagnostic - from effect to causes
Causal - from cause to effects
Intercausal - between causes of an effects
Mixed - combinations of the ones above
inference also possible over time using Dynamic Bayesian Networks
11/29/2018
Principles & Methodologies

14. Principles & Methodologies
Influence Diagrams
combines Bayesian Networks with nodes for action and utility
3 types of nodes :
chance nodes : random variables
action nodes : choice of action influencing following nodes
utility nodes : computes the utility of actions given a state of the network
11/29/2018
Principles & Methodologies

15. Principles & Methodologies
Influence Diagrams
Body Position
Value of Action
Location
Focus of Attention
Expected Utility
Actions
Seminar- Room
Head Pose
Cost of Action
11/29/2018
Principles & Methodologies

16. Principles & Methodologies
Influence Diagrams
Algorithm
1. set evidence variable for current state
2. for all possible values of the action node do
a) set action node to specific value
b) calculate utility using standard probabilistic inference algorithm
3. return action with highest utility
11/29/2018
Principles & Methodologies

17. Principles & Methodologies
Key Features of AUIs
graceful negotiation of turns and sense of acknowledgement of the request
determine utility of interruption given the priority of the information
signal information via nonintrusive channel
sense user acknowledgement of the information
11/29/2018
Principles & Methodologies

18. Principles & Methodologies
Key Features of AUIs
Communicating attention
AUIs show their attention to user
communicate focus of the user to other AUIs or remote people requesting user’s attention
11/29/2018
Principles & Methodologies

19. Principles & Methodologies
Key Features of AUIs
Augmenting attentive resources
presents information in estimated focus of user
attenuating peripheral detail
11/29/2018
Principles & Methodologies

20. Principles & Methodologies
Key Features of AUIs
Sensing attention
Reasoning about attention
Negotiation of turns and sense of acknowledgement
Communicating attention
Augmenting attentive resources
11/29/2018
Principles & Methodologies

21. Overview Introduction Principles & Methodologies Applications Summary
Attention - Sensitive Messaging
Mixed - Initiative Interactions
Summary
11/29/2018
Applications

22. The Notification Platform Project
Microsoft Research, Redmond
centered on economic based principles:
value of information
vs.
attentive-sensitive cost of disruption
11/29/2018
Applications

23. The Notification Platform Project
subscription of different sources and output devices possible
considers user’s attention and location under uncertainty
aware of fidelity and relevance of different communication channels
11/29/2018
Applications

24. The Notification Platform Projects
Handy
Online Calendar
Notification Platform
Instant Messages
File Scout
...
11/29/2018
Applications

25. The Notification Platform Project
Universal Inbox
Bayesian attentive model
Context Server & Context Whiteboard
Decision model
Profile of user’s preferences
11/29/2018
Applications

26. The Notification Platform Project
11/29/2018
Applications

27. The Notification Platform Project
Example:
A new message coming into the Universal Inbox
11/29/2018
Applications

28. The Notification Platform Project
1. How important is this message?
- question of informing user now or later
- computes cost of delayed review
 Information Value
11/29/2018
Applications

29. The Notification Platform Project
2. Which device and modality should I use?
a) What is the user doing now?
Focus of Attention
b) Will the user receive my notification on this device?
c) Will I lose fidelity of information when using this device?
 Transmitted Value
Value of Action
Device & Modality
11/29/2018
Applications

30. The Notification Platform Project
3. How expensive is it to alert the user now?
- uses preferences stored in profile
 Attentional cost
Focus of Attention
Cost of Action
Device & Modality
11/29/2018
Applications

31. The Notification Platform Project
for every device and modality compute the Expected Utility =
Transmitted Value - Attentional cost
 choose action with highest utility!
Device & Modality
Value of Action
Cost of Action
Expected Utility
11/29/2018
Introduction

32. The Notification Platform Project
11/29/2018
Applications

33. Overview Introduction Principles & Methodologies Applications Summary
Attention - Sensitive Messaging
Mixed - Initiative Interactions
Summary
11/29/2018
Applications

34. DeepListener Project spoken language system
address the “speech-target problem”
use probabilistic models to compute the likelihood of being the target of speech, of what was said and of different intentions
clarification dialogues
use cues for conversational turn taking
attentional lens or animated agent
11/29/2018
Applications

35. DeepListener Project
11/29/2018
Applications

36. Quartet Project continuous speech recognition system
advanced model of attention:
keyboard events
visual pose
disruption of attention between device and human
natural language processing
realises misrecognition and intention for someone else
11/29/2018
Applications

37. Quartet Project
11/29/2018
Applications

38. Overview Introduction Principles & Methodologies Applications Summary
Attention - Sensitive Messaging
Mixed - Initiative Interactions
Summary
11/29/2018
Summary

39. Summary if we extend devices with attention sensors
advanced representation & inferential methods
communication capabilities
11/29/2018
Summary

40. Summary it is possible to build
attentive
sensible
unintrusive
sociable devices!
but there is still much research to do :
recognition, modelling, reasoning, psychology
11/29/2018
Summary

41. Literature
E. Horvitz, C. M. Kadie, T. Paek, D. Hovel. Models of Attention in Computing and Communications: From Principles to Applications, Communications of the ACM 46(3):52-59, March 2003
Jeffrey S. Shell, Ted Selker, Roel Vertegaal: Interacting with groups of computers. Communications of the ACM 46(3): 40-46, March 2003
Paul P. Maglio, Christopher S. Campbell: Attentive agents. Communications of the ACM 46(3): , March 2003
Roel Vertegaal: Introduction. Communications of the ACM 46(3): 30-33, March 2003
Stuart Russell, Peter Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall, 1995
11/29/2018
Summary

42. Thank you for your kind attention!
11/29/2018