1. 1 Daniel Avrahami - Thesis Proposal Who, What, and When: Supporting Interpersonal Communication over Instant Messaging Daniel Avrahami Committee: Scott Hudson (Chair) Eric Horvitz Robert Kraut Alon Lavie

2. 2 Daniel Avrahami - Thesis Proposal Illustration John is making final changes to a presentation for a client visit. His team member Anne, working at a different site, sends him an instant message asking for some urgent information.

3. 3 Daniel Avrahami - Thesis Proposal Illustration John is making final changes to a presentation for a client visit. His team member Anne, working at a different site, sends him an instant message asking for some urgent information. Since John is pressed for time, he decides to ignore all incoming messages until after he’s done, leaving Anne unable to finish her task.

4. 4 Daniel Avrahami - Thesis Proposal Illustration (cont) Consider now if we were able to: Accurately predict, based on his activity, that John was not likely to respond to Anne’s message for some time Predict, based on past communication patterns, that Anne and John are co-workers Such models could be used, for example, to increase the salience of the alert, indicating to John that Anne’s message may deserve his immediate attention

5. 5 Daniel Avrahami - Thesis Proposal Research goals The two main goals of my proposed thesis work are to provide a better understanding of factors affecting IM interaction in its context, and to use this understanding for the creation of predictive statistical models and tools that support IM communication. In order to achieve these goals my research will use three complementary steps:

6. 6 Daniel Avrahami - Thesis Proposal Research goals Create accurate models that predict responsiveness to incoming IM, and investigate the factors affecting responsiveness (when) Investigate the effect of interpersonal relationships on IM interaction, and create statistical models that use this knowledge to predict relationships (who) Use basic properties of human dialogue to provide support for balancing of responsiveness and performance (what)

7. 7 Daniel Avrahami - Thesis Proposal Background Instant Messaging, or IM, is one of the most popular communication mediums today No longer a medium only for social communication – 12 billion instant messages are sent each day. Nearly 1 billion messages are exchanged by 28 million business users [IDC Market Analysis’05] Useful in many ways: from quick questions and clarifications, coordination and scheduling, to discussions of complex work [Bradner’99; Nardi’00; Handel’02; Herbsleb’02; Isaacs’02]

8. 8 Daniel Avrahami - Thesis Proposal Background Some characteristics of IM: – Sending messages is “cheap” – People can choose when/whether to respond – Asynchrony means that people can (and do) multitask [Nardi’00, Isaacs’02] – Can tell whether a receiver is present But…

9. 9 Daniel Avrahami - Thesis Proposal Background Especially in the workplace, means that messages may often arrive at inconvenient times Presence is not enough Unsuccessful communication can have a negative effect on both sender and receiver – Can disrupt the receiver’s work – Can leave the sender waiting for information – True not only for IM

10. 10 Daniel Avrahami - Thesis Proposal When: Predicting responsiveness to IM

11. 11 Daniel Avrahami - Thesis Proposal Background Want to answer the following question: If an instant message were to arrive right now, would the user respond to it? In how long?

12. 12 Daniel Avrahami - Thesis Proposal How can such models help? senderreceiver  intercept  alert  mask  enhance awareness message

13. 13 Daniel Avrahami - Thesis Proposal sender How can such models help? message receiver  intercept  alert  mask  enhance 

14. 14 Daniel Avrahami - Thesis Proposal sender How can such models help? message receiver  intercept  alert  mask  enhance 

15. 15 Daniel Avrahami - Thesis Proposal sender How can such models help? message receiver  intercept  alert  mask  enhance  

16. 16 Daniel Avrahami - Thesis Proposal sender How can such models help? awareness receiver   intercept  alert  mask  enhance shhhh  

17. 17 Daniel Avrahami - Thesis Proposal sender How can such models help? awareness receiver   intercept  alert  mask  enhance (carefully) not now   

18. 18 Daniel Avrahami - Thesis Proposal Related work Interruptions and disruptions – [Gillie’89, Cutrell’01, Hudson’02, Dabbish’04] Interruptibility and cost of interruption – [Horvitz’99, Horvitz’03, Hudson’03, Begole’04, Horvitz’04, Fogarty’05, Iqbal’06] Models of presence – [Horvitz’02, Begole’03] Responsiveness to Email – [Horvitz’02, Tyler’03]

19. 19 Daniel Avrahami - Thesis Proposal Data collection

20. 20 Daniel Avrahami - Thesis Proposal Data collection Created a plugin for Trillian Pro (written in C) – Non-intrusive collection of IM and desktop events – Why this setup?

21. 21 Daniel Avrahami - Thesis Proposal Data collection (cont) What gets collected: – IM Events Messages, status changes, etc. – Desktop Events Applications, events, etc. Each participants records for at least 4 weeks

22. 22 Daniel Avrahami - Thesis Proposal Data collection (cont.) Privacy of data – Masking messages for example, the message: “This is my secret number: 1234 :-)” was recorded as “AAAA AA AA AAAAAA AAAAAA: DDDD :-)”. Temporary masking – Alerting buddies – Hashing buddy-names

23. 23 Daniel Avrahami - Thesis Proposal Participants 16 participants – Researchers: 6 full-time employees at an industrial research lab (mean age=40.33) – Interns: 2 summer interns at the industrial research lab (mean age=34.5) – Students: 8 Masters students (mean age=24.5)

24. 24 Daniel Avrahami - Thesis Proposal Participants Nearly 5200 hours recorded Over 90,000 messages Over 400 buddies 4 participants provided full text On average, participants exchanged a message every: 8.1, 2.2, 3.1 minutes

25. 25 Daniel Avrahami - Thesis Proposal Responsiveness

26. 26 Daniel Avrahami - Thesis Proposal Responsiveness 50%

27. 27 Daniel Avrahami - Thesis Proposal Responsiveness 92% 50%

28. 28 Daniel Avrahami - Thesis Proposal Defining “IM Sessions” session 92%

29. 29 Daniel Avrahami - Thesis Proposal Defining “Session Initiation Attempts” session used two subsets: 5 minutes (similar to Isaacs’02) and 10 minutes

30. 30 Daniel Avrahami - Thesis Proposal Features For every message: – Features describing IM state. including: Day of week Hour Is the Message-Window open Buddy status (e.g., “Away”) Buddy status duration Time since msg to buddy Time since msg from another buddy Any msg from other in the last 5 minutes log(time since msg with any buddy) Is an SIA-5

31. 31 Daniel Avrahami - Thesis Proposal Features (cont.) For every message: – Features describing desktop state (following Horvitz et al. Fogarty et al. and others). including: Application in focus Application in focus duration Previous application in focus Previous application in focus duration Most used application in past m minutes Duration for most used application in past m minutes Number of application switches in past m minutes Amount of keyboard activity in past m minutes Amount of mouse activity in past m minutes Mouse movement distance in past m minutes

32. 32 Daniel Avrahami - Thesis Proposal What are we predicting? “Seconds Until Response” – computed, for every incoming message from a buddy, by noting the time it took until a message was sent to the same buddy Examined five responsiveness thresholds – 30 seconds, 1, 2, 5, and 10 minutes

33. 33 Daniel Avrahami - Thesis Proposal Modeling method Weka ML toolkit Features selected using a wrapper-based selection technique AdaBoosting on Decision-Tree models 10-fold cross-validation – 10 trials: train on 90%, test on 10% – Next we report combined accuracy

34. 34 Daniel Avrahami - Thesis Proposal Results (full feature-set models) All significantly better than the prior probability (p<.001)

35. 35 Daniel Avrahami - Thesis Proposal Results (buddy-independent models) Previous models used information about the buddy (e.g., time since messaging that buddy) Can predict different responsiveness for different buddies – But what if you wanted just one level of responsiveness? Built models that did not use any buddy- related features

36. 36 Daniel Avrahami - Thesis Proposal Results (buddy-independent models) all significantly better than the prior probability (p<.001) BUT not sig. diff. from previous set

37. 37 Daniel Avrahami - Thesis Proposal Not using content is good and bad Pros: – Privacy preserving – Easy Cons: – Misinterpreting quick responses – Misinterpreting messages that do not need responses – > Can lead to some (machine) learning errors

38. 38 Daniel Avrahami - Thesis Proposal Planned work: Content Need to be able to answer the following question (when a incoming message arrives): Is this message the beginning of a new session or part of the previous session? I plan to develop a method for answering this question automatically using content analysis

39. 39 Daniel Avrahami - Thesis Proposal Planned work: Content

40. 40 Daniel Avrahami - Thesis Proposal Planned work: Content Evaluation plans: – Obtain manual coding of a random subset of segment boundaries identified as correct or incorrect – Use inter-coder agreement to establish a performance benchmark – Compare to accuracy of automated approach

41. 41 Daniel Avrahami - Thesis Proposal Planned work: Probabilities Create a probabilistic model predicting the likelihood of the arrival of the next message in a session – Given evidence about the session so far – What is the probability of Another message within time T ? Another message within time >= T ? Useful in identifying ends of sessions – Good for load balancing (help-desk scenarios)

42. 42 Daniel Avrahami - Thesis Proposal Planned work: Understanding Showed the successful creation of statistical models that predict responsiveness A better understanding is needed of the connection between responsiveness and a person’s communication- and work-context

43. 43 Daniel Avrahami - Thesis Proposal Planned work: Understanding I plan to investigate in detail the contribution of specific features and the interactions between those features to responsiveness – Preliminary results suggest, for example, that indications of work fragmentation significantly affect responsiveness

44. 44 Daniel Avrahami - Thesis Proposal Who: Relationships and communication patterns

45. 45 Daniel Avrahami - Thesis Proposal Relationships and IM communication People use IM for both work and social communication Availability might depend on relationship Wanted to investigate the effect of relationship on basic communication patterns

46. 46 Daniel Avrahami - Thesis Proposal Background Relationship type has significant effects on communication, including the quality, purpose and perceived value [Duck’91] Cues, such as tempo, pauses, speech rates and the frequency of turns, affect the way in which conversation partners perceive each other [Feldstein’94] Frequency affects communication [FTF:Whittaker’94, IM:Isaacs’02]

47. 47 Daniel Avrahami - Thesis Proposal Buddy Coder Co-worker (Senior) Co-worker (Peer) Co-worker (Junior) Co-worker (Other) Friend Family Spouse Significant Other Acquaintance Friend & Co-worker Self Bot [Unknown/Unused]

48. 48 Daniel Avrahami - Thesis Proposal Co-worker (Senior) Co-worker (Peer) Co-worker (Junior) Co-worker (Other) Friend Family Spouse Significant Other Acquaintance Friend & Co-worker Self Bot [Unknown/Unused] Buddy Coder

49. 49 Daniel Avrahami - Thesis Proposal Co-worker (Senior) Co-worker (Peer) Co-worker (Junior) Co-worker (Other) Friend Family Spouse Significant Other Acquaintance Friend & Co-worker Self Bot [Unknown/Unused] Buddy Coder

50. 50 Daniel Avrahami - Thesis Proposal Co-worker (Senior) Co-worker (Peer) Co-worker (Junior) Co-worker (Other) Friend Family Spouse Significant Other Acquaintance Friend & Co-worker Self Bot [Unknown/Unused] Buddy Coder

51. 51 Daniel Avrahami - Thesis Proposal Co-worker (Senior) Co-worker (Peer) Co-worker (Junior) Co-worker (Other) Friend Family Spouse Significant Other Acquaintance Friend & Co-worker Self Bot [Unknown/Unused] Buddy Coder

52. 52 Daniel Avrahami - Thesis Proposal Relationships distribution

53. 53 Daniel Avrahami - Thesis Proposal Session-level measures

54. 54 Daniel Avrahami - Thesis Proposal Session-level measures #TimeMessage Text 117:42:45B:Hey [Participant’s name] 217:42:56B:what time does your group get in the AM? 317:42:57P:hey 417:43:01P:usually around 10 517:43:25B:ok 617:43:38B:i want to start circulating the card in the AM 717:43:58P:ok, good idea 817:44:02P:that's for coordinating this 917:44:13B:no problem 1017:44:27P:thanks :-) 1117:44:35P:sorry bout the typo 1217:44:38B:is ok

55. 55 Daniel Avrahami - Thesis Proposal Session-level measures #TimeMessage Text 117:42:45B:Hey [Participant’s name] 217:42:56B:what time does your group get in the AM? 317:42:57P:hey 417:43:01P:usually around 10 517:43:25B:ok 617:43:38B:i want to start circulating the card in the AM 717:43:58P:ok, good idea 817:44:02P:that's for coordinating this 917:44:13B:no problem 1017:44:27P:thanks :-) 1117:44:35P:sorry bout the typo 1217:44:38B:is ok VariableValue GroupStudent RelationshipWork Duration1.88minutes Message Count12 Turn Count7 Character Count232 Messages per Minute6.4 Messages per Turn1.71 Characters per Message19.3 Seconds Until First Reply1seconds Minimum Gap (between turns)1seconds Maximum Gap (between turns)24seconds Average Gap (between turns)12.2seconds Time of Day5:44pm

56. 56 Daniel Avrahami - Thesis Proposal The effect of relationships Used a repeated-measures ANOVA – Relationship Category (Work, Mix, Social) and Group (Researchers, Interns, Students) were repeated – Participants and BuddyID modeled as random effects – Participants nested in Group – BuddyID nested first in Participants, then in Group – N = 3297

57. 57 Daniel Avrahami - Thesis Proposal Results

58. 58 Daniel Avrahami - Thesis Proposal Summary of Results Sessions with Social contacts were longer and with more messages BUT at a significantly slower pace – Maybe giving less attention to these sessions? Sessions with Work contacts were at a faster pace with longer messages – Grounding? Complex concepts?

59. 59 Daniel Avrahami - Thesis Proposal Results: Session length Significant effect on Session Duration (p<.001) – Social significantly longer sessions than both Mix and Work (Work and Mix n.s.) Similar effects for – Number of Turns Number of Messages Number of Characters Duration correlated at >.85

60. 60 Daniel Avrahami - Thesis Proposal Results: Messaging rate Significant effect on Messaging Rate (p<.01) – Social significantly slower than Mix (p=.003) – Social marginally slower than Work (p=.078) Maximum-Gap (p<.05) Social longer than Work (p=.013)

61. 61 Daniel Avrahami - Thesis Proposal Results: Length of messages Significant effect on Message Length (Characters-per-Message) (p<.001) – Work significantly longer than both Social (p<.001) and Mix (p=.002)

62. 62 Daniel Avrahami - Thesis Proposal Predicting relationships

63. 63 Daniel Avrahami - Thesis Proposal Predicting relationships How can it be used? – Augmenting IM systems Indicators of unavailability Differential alerts – Shared with other mediums E.g. Email – Provide organizational overview

64. 64 Daniel Avrahami - Thesis Proposal Predicting relationships Cross-validation with 16 models (omitting one participant each time) Nominal Logistic Regression

65. 65 Daniel Avrahami - Thesis Proposal Models performance Results from pairs with 2 sessions or more (78% of the data) Classified as WorkSocial Work 40.9% (83) 5.9% (12) Social 14.8% (30) 38.4% (78) Accuracy: 79.3%

66. 66 Daniel Avrahami - Thesis Proposal Models performance Results from pairs with 2 sessions or more (78% of the data) Both significantly better than the prior probability Classified as WorkSocial Work 40.9% (83) 5.9% (12) Social 14.8% (30) 38.4% (78) Accuracy: 79.3% Classified as WorkMixSocial Work 25.3% (74) 5.1% (15) 2.0% (6) Mix 8.2% (24) 14.7% (43) 7.8% (23) Social 9.6% (28) 17.1% (50) 10.2% (30) Overall Accuracy: 50.2% Work vs. Rest: 75.1% Social vs. Rest: 63.5%

67. 67 Daniel Avrahami - Thesis Proposal Additional work Find a way to distinguish Mix from Social Examine the effects of additional aspects of relationships on communication – The effects of physical distance – The effects of buddy-familiarity – The effects of task criticality

68. 68 Daniel Avrahami - Thesis Proposal What: Using content to balance responsiveness and performance

69. 69 Daniel Avrahami - Thesis Proposal Responsiveness / performance tradeoff Users often multitask when using instant messaging [Nardi’00, Isaacs’02, Voida’02] Users often have to choose between – Staying on task and being responsive to IM buddies Current solutions typically force users to choose one or the other: – Update ‘away’ messages – Turn off IM client

70. 70 Daniel Avrahami - Thesis Proposal Quick response - “do you have the figures I need?” Leisurely response - “check out www.cnn.com” Politely deferred - “ru busy?” No response - “going to meeting. ttyl” Expectations for responsiveness

71. 71 Daniel Avrahami - Thesis Proposal The approach: QnA Users ignore, to the best of their ability, the alerts of incoming messages – Transitioning (internally) to being unavailable By observing the content of messages, QnA automatically highlights incoming messages that may deserve their attention – In particular, potential questions and answers

72. 72 Daniel Avrahami - Thesis Proposal Demo (Good luck with the demo)

73. 73 Daniel Avrahami - Thesis Proposal Why questions and answers? A question and an answer form an ‘Adjacency pair’ (Schegloff & Sacks’73) From “Arenas of Language Use” by Clark (1992) “Given a first pair part, a second pair part is conditionally relevant, that is, relevant and expectable, as the next utterance. Once A has asked the question, it is relevant and expectable for B to answer in the next turn.” (p. 157)

74. 74 Daniel Avrahami - Thesis Proposal QnA listens to incoming and outgoing messages – when an outgoing messages is sent if it is a question – remember that expecting a response – when an incoming messages arrives if it is a question and/or we are expecting an answer – wait x seconds to see if user attends to the message – if did not attend then show QnA notification How does it work?

75. 75 Daniel Avrahami - Thesis Proposal is_a_question? Match to list of questions that can be ‘politely deferred’ –(are|r) (you|u) there –busy? Go through list of rules and look for match –(?|/) at end of sentence –what (is|are|r|were|does|do|did|should|can) –did(|n’t|nt) (i|u|you|he|she|they|we) –(are|r) (you|u) –huh

76. 76 Daniel Avrahami - Thesis Proposal Delaying notifications First determine whether the user is engaged in the conversation by listening to these events: – Opens the message window – Clicks on the message window – Types in the message window – Message window was in focus when message arrived Multiple questions from same buddy within same delay period will only get one alert

77. 77 Daniel Avrahami - Thesis Proposal Issues Determining that a message contains a question or an answer can be difficult – interleaved conversations – many short messages that comprise a single turn – loose grammar and spelling Gives buddies a way to increase the salience of their messages. what if they abuse it?

78. 78 Daniel Avrahami - Thesis Proposal QnA summary QnA: A tool that allows users to stay on task, but still seem responsive to buddies who expect it Allows users to transition between work modes – Sits quietly in the background when the user attends to messages – Only notifies when the user ignores messages

79. 79 Daniel Avrahami - Thesis Proposal Future work Collect feedback from users – A few users who have used QnA for over 2 years now – But would like more users Please download QnA from my homepage Improve question identification Implement ‘ignore list’

80. 80 Daniel Avrahami - Thesis Proposal Conclusions

81. 81 Daniel Avrahami - Thesis Proposal Conclusions I have presented planned and completed work on analysis and generation of predictive modeling in support of interpersonal communication over IM: – Completed work along with additional planned work on predictions of responsiveness to IM – Completed work on analysis and predictions of interpersonal relationships and their effect on communication – Completed work on the use of basic properties of human dialogue to allow users to balance responsiveness and performance

82. 82 Daniel Avrahami - Thesis Proposal Contributions This work’s contribution to the HCI field will span both theoretical and applied aspects. – From a theoretical point of view, this work will provide insights into the factors that influence interpersonal communication patterns and responsiveness. – At the applied level, this work will provide predictive statistical models that can be used in many applications. – Finally, this work promotes the creation of tools that use knowledge and predictive models generated from naturally occurring interaction.

83. 83 Daniel Avrahami - Thesis Proposal Acknowledgements No acknowledgements until my defense! Thank you Thi! Good luck to Darren, James, Jeff and Laura

84. 84 Daniel Avrahami - Thesis Proposal Thank you. This work was funded in part by the National Science Foundation under grants IIS 0121560 and IIS 0325351 and this material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. NBCHD030010