1. experience sampling method: preliminary findings sunny consolvo & miriam walker intel research seattle sunny@intel-research.net user-centered evaluations of ubicomp workshop 4 th int’l conference on ubiquitous computing september 29, 2002

2. experience sampling method -- ubicomp 20022 before we get started… this work is preliminary wrapped up data collection this month expect future publication

3. experience sampling method -- ubicomp 20023 outline of talk intro to experience sampling method (esm) case study: the personal server issues wrap-up

4. experience sampling method -- ubicomp 20024 esm: a brief introduction also called: beeper studies time sampling field study technique from psychology used to understand areas such as… mood time use social interactions we’re exploring its uses for ubicomp

5. experience sampling method -- ubicomp 20025 esm: a brief introduction fill out questionnaire when alerted researcher is not present alerted several times per day business as usual until alerted duration is often 1-3 weeks

6. experience sampling method -- ubicomp 20026 running esm: tools need to choose tools for: alerting participant random alerts timed alerts event-based/user-driven delivering questionnaire capturing participant responses

7. experience sampling method -- ubicomp 20027 choosing tools: alerting tools for alerting the participant include: cell phones pagers watches PDAs custom devices some desirable abilities: specify time period during which alerts may go off specify # of alerts per time period & overall

8. experience sampling method -- ubicomp 20028 choosing tools: delivering tools for delivering questionnaires include: paper booklets PDAs phones (traditional or mobile) live researcher recorded message audio recordings (e.g., cassette player)

9. experience sampling method -- ubicomp 20029 choosing tools: capturing tools for capturing responses include: paper booklets PDAs phones (traditional or mobile) live researcher recording service (e.g., Tellme) audio recordings (e.g., cassette player) cameras

10. experience sampling method -- ubicomp 200210 esm: some advantages reduces effects of intrusive observers larger sample size & time interval reduces memory biases by asking about current activity, feelings, etc. it’s all about now

11. experience sampling method -- ubicomp 200211 esm: more advantages less time consuming than observations, yet… study activities as they occur in the field study activities and emotions often not accessible to observers not as disruptive to others

12. experience sampling method -- ubicomp 200212 esm: some disadvantages questionnaires must be brief participant must carry tools at all times power consumption and/or service availability certain events may only show up in very large samples sizes/durations more intrusive than interviews & surveys

13. experience sampling method -- ubicomp 200213 outline of talk intro to experience sampling method (esm) case study: personal server issues wrap-up

14. experience sampling method -- ubicomp 200214 case study: personal server personal server team was interested in: how people use information what they want when they want it where they want it used esm to help collect this data

15. experience sampling method -- ubicomp 200215 pilot study #1 4 participants lab researchers & administrative staff palm m100s for alerting delivering capturing original boston college ESP software for running ESM on palms duration: 1½ - 4 days

16. experience sampling method -- ubicomp 200216 pilot study #1: problems palm dropped  batteries fell out  data lost only method of alert was beep (m100) participants wouldn’t carry it in several situations once volume of beep was set, it couldn’t be changed only response option: multi-choice, single answer difficult for participants to answer some questions e.g., “where are you?” some participants wanted a bag to carry the palm participants wanted to know about missed questionnaires

17. experience sampling method -- ubicomp 200217 pilot study #2 4 participants lab researchers, administrative staff, family modified iESP software palm m500s for alerting and delivering for capturing: 2 participants carried fold-out keyboards (for palm) 2 participants carried paper booklets all 4 carried digital cameras  for describing where & what they were doing duration: 1½ - 4 days

18. experience sampling method -- ubicomp 200218 pilot study #2: justifications upgrading to m500s: ability to alert via beep and/or vibrate almost impossible for batteries to fall out iESP software added functionality, including: freeform text entry multi-choice, multi-answer new formats for multi-choice, single answer slider (e.g., “on a scale of 1 to 5…”) ability to change volume, vibrate, etc. during study ability to check how many questionnaires have been completed vs. asked during study

19. experience sampling method -- ubicomp 200219 our implementation: the study 31 participants recruited from local community pre/post-study interviews + 1 week with equipment 10 questionnaires/day 70 questionnaires total incentive (in USD) $50 for participating and returning equipment $1 per completed questionnaire $120 total possible

20. experience sampling method -- ubicomp 200220 the study (continued) palm m500 to alert, deliver, & capture modified iESP software more about capturing: also used camera free-form data entry: paper booklet for all 31 participants graffiti / soft keyboard for all 31 participants full-sized fold-out keyboard for 9 of 31 participants

21. experience sampling method -- ubicomp 200221 what the participants carried

22. experience sampling method -- ubicomp 200222 type of data collected responses to following types of questions: yes / no multiple choice, single answer multiple choice, multiple answers freeform text entry looks a lot like survey data except multiple responses from same participant in variety of circumstances over time

23. experience sampling method -- ubicomp 200223 other data collected qualitative data from interviews participants’ photos some photos were of interest to the participant other photos documented where s/he was or what s/he was doing

24. experience sampling method -- ubicomp 200224 questions on the palm

25. experience sampling method -- ubicomp 200225 more questions

26. experience sampling method -- ubicomp 200226 participant photos: in transit

27. experience sampling method -- ubicomp 200227 participant photos: at home

28. experience sampling method -- ubicomp 200228 participant photos: at work

29. experience sampling method -- ubicomp 200229 participant photos: retail

30. experience sampling method -- ubicomp 200230 participant photos: pets

31. experience sampling method -- ubicomp 200231 outline of talk intro to experience sampling method (esm) case study: personal server issues wrap-up

32. experience sampling method -- ubicomp 200232 our tool decisions… chose palm because: no service problems—cost or coverage power consumption not an issue for duration know when participant fills out questionnaire probabilities and contingencies can be used if a, then b if b, then not c d 50% of time don’t have to transcribe data

33. experience sampling method -- ubicomp 200233 tool decisions used camera to see if it would help participants describe… where they were what they were doing photos used to jog memories during post-study interviews

34. experience sampling method -- ubicomp 200234 according to the participants… 1/31 would not participate again 2/31 would only do it for more money 28/31 would participate again “it was cool” “I felt like a spy” “friends were jealous of the [fold-out] keyboard” a lot to carry mixed feelings about carry bag several claimed they would be purchasing palms only one participant wanted voice recording option when asked, most did not respond favorably

35. experience sampling method -- ubicomp 200235 from our perspective… woo-hoo! we got all of the equipment back only 1/31 palms was broken of the 31 participants and 70 questionnaires: 28 participants completed 50+ questionnaires 13 of whom completed 60+ questionnaires 2 participants completed 25- questionnaires 1 participant performed a hard reset and returned the palm with no data or software

36. experience sampling method -- ubicomp 200236 outline of talk intro to experience sampling method (esm) case study: personal server issues wrap-up

37. experience sampling method -- ubicomp 200237 wrap up: we’ll use it again pleased with results so far intend to use esm again

38. experience sampling method -- ubicomp 200238 wrap-up: pda software original ESP software available at: http://www2.bc.edu/~barretli/esp/ iESP software (includes upgrades): launching on intel research seattle web site in Oct or Nov 2002 if you want to be notified… business card with “iESP notification” written on it legibly fill out notification sign-up sheet

39. experience sampling method -- ubicomp 200239 acknowledgements drs. lisa & daniel barrett justin silves gaetano borriello ken anderson rahul shah trevor paring roy want john light our pilot testers and our colleagues at intel research seattle

40. experience sampling method -- ubicomp 200240 questions?

41. experience sampling method -- ubicomp 200241 examples of study questions where is your cell phone? in my… hand pocket/belt clip bag car home office somewhere else and with me somewhere else and not with me

42. experience sampling method -- ubicomp 200242 examples of study questions in the last 30 minutes, which of the following would you have liked to have access to but didn’t have available? web pages movie times restaurant reviews personal calendar telephone number(s) computer files music files shopping list “to do” list