experience sampling method: preliminary findings sunny consolvo & miriam walker intel research seattle user-centered evaluations of ubicomp workshop 4 th int’l conference on ubiquitous computing september 29, 2002

experience sampling method -- ubicomp before we get started… this work is preliminary wrapped up data collection this month expect future publication

experience sampling method -- ubicomp outline of talk intro to experience sampling method (esm) case study: the personal server issues wrap-up

experience sampling method -- ubicomp esm: a brief introduction also called: beeper studies time sampling field study technique from psychology used to understand areas such as… mood time use social interactions we’re exploring its uses for ubicomp

experience sampling method -- ubicomp esm: a brief introduction fill out questionnaire when alerted researcher is not present alerted several times per day business as usual until alerted duration is often 1-3 weeks

experience sampling method -- ubicomp running esm: tools need to choose tools for: alerting participant random alerts timed alerts event-based/user-driven delivering questionnaire capturing participant responses

experience sampling method -- ubicomp choosing tools: alerting tools for alerting the participant include: cell phones pagers watches PDAs custom devices some desirable abilities: specify time period during which alerts may go off specify # of alerts per time period & overall

experience sampling method -- ubicomp choosing tools: delivering tools for delivering questionnaires include: paper booklets PDAs phones (traditional or mobile) live researcher recorded message audio recordings (e.g., cassette player)

experience sampling method -- ubicomp choosing tools: capturing tools for capturing responses include: paper booklets PDAs phones (traditional or mobile) live researcher recording service (e.g., Tellme) audio recordings (e.g., cassette player) cameras

experience sampling method -- ubicomp esm: some advantages reduces effects of intrusive observers larger sample size & time interval reduces memory biases by asking about current activity, feelings, etc. it’s all about now

experience sampling method -- ubicomp esm: more advantages less time consuming than observations, yet… study activities as they occur in the field study activities and emotions often not accessible to observers not as disruptive to others

experience sampling method -- ubicomp esm: some disadvantages questionnaires must be brief participant must carry tools at all times power consumption and/or service availability certain events may only show up in very large samples sizes/durations more intrusive than interviews & surveys

experience sampling method -- ubicomp outline of talk intro to experience sampling method (esm) case study: personal server issues wrap-up

experience sampling method -- ubicomp case study: personal server personal server team was interested in: how people use information what they want when they want it where they want it used esm to help collect this data

experience sampling method -- ubicomp pilot study #1 4 participants lab researchers & administrative staff palm m100s for alerting delivering capturing original boston college ESP software for running ESM on palms duration: 1½ - 4 days

experience sampling method -- ubicomp pilot study #1: problems palm dropped  batteries fell out  data lost only method of alert was beep (m100) participants wouldn’t carry it in several situations once volume of beep was set, it couldn’t be changed only response option: multi-choice, single answer difficult for participants to answer some questions e.g., “where are you?” some participants wanted a bag to carry the palm participants wanted to know about missed questionnaires

experience sampling method -- ubicomp pilot study #2 4 participants lab researchers, administrative staff, family modified iESP software palm m500s for alerting and delivering for capturing: 2 participants carried fold-out keyboards (for palm) 2 participants carried paper booklets all 4 carried digital cameras  for describing where & what they were doing duration: 1½ - 4 days

experience sampling method -- ubicomp pilot study #2: justifications upgrading to m500s: ability to alert via beep and/or vibrate almost impossible for batteries to fall out iESP software added functionality, including: freeform text entry multi-choice, multi-answer new formats for multi-choice, single answer slider (e.g., “on a scale of 1 to 5…”) ability to change volume, vibrate, etc. during study ability to check how many questionnaires have been completed vs. asked during study

experience sampling method -- ubicomp our implementation: the study 31 participants recruited from local community pre/post-study interviews + 1 week with equipment 10 questionnaires/day 70 questionnaires total incentive (in USD) $50 for participating and returning equipment $1 per completed questionnaire $120 total possible

experience sampling method -- ubicomp the study (continued) palm m500 to alert, deliver, & capture modified iESP software more about capturing: also used camera free-form data entry: paper booklet for all 31 participants graffiti / soft keyboard for all 31 participants full-sized fold-out keyboard for 9 of 31 participants

experience sampling method -- ubicomp what the participants carried

experience sampling method -- ubicomp type of data collected responses to following types of questions: yes / no multiple choice, single answer multiple choice, multiple answers freeform text entry looks a lot like survey data except multiple responses from same participant in variety of circumstances over time

experience sampling method -- ubicomp other data collected qualitative data from interviews participants’ photos some photos were of interest to the participant other photos documented where s/he was or what s/he was doing

experience sampling method -- ubicomp questions on the palm

experience sampling method -- ubicomp more questions

experience sampling method -- ubicomp participant photos: in transit

experience sampling method -- ubicomp participant photos: at home

experience sampling method -- ubicomp participant photos: at work

experience sampling method -- ubicomp participant photos: retail

experience sampling method -- ubicomp participant photos: pets

experience sampling method -- ubicomp outline of talk intro to experience sampling method (esm) case study: personal server issues wrap-up

experience sampling method -- ubicomp our tool decisions… chose palm because: no service problems—cost or coverage power consumption not an issue for duration know when participant fills out questionnaire probabilities and contingencies can be used if a, then b if b, then not c d 50% of time don’t have to transcribe data

experience sampling method -- ubicomp tool decisions used camera to see if it would help participants describe… where they were what they were doing photos used to jog memories during post-study interviews

experience sampling method -- ubicomp according to the participants… 1/31 would not participate again 2/31 would only do it for more money 28/31 would participate again “it was cool” “I felt like a spy” “friends were jealous of the [fold-out] keyboard” a lot to carry mixed feelings about carry bag several claimed they would be purchasing palms only one participant wanted voice recording option when asked, most did not respond favorably

experience sampling method -- ubicomp from our perspective… woo-hoo! we got all of the equipment back only 1/31 palms was broken of the 31 participants and 70 questionnaires: 28 participants completed 50+ questionnaires 13 of whom completed 60+ questionnaires 2 participants completed 25- questionnaires 1 participant performed a hard reset and returned the palm with no data or software

experience sampling method -- ubicomp outline of talk intro to experience sampling method (esm) case study: personal server issues wrap-up

experience sampling method -- ubicomp wrap up: we’ll use it again pleased with results so far intend to use esm again

experience sampling method -- ubicomp wrap-up: pda software original ESP software available at: iESP software (includes upgrades): launching on intel research seattle web site in Oct or Nov 2002 if you want to be notified… business card with “iESP notification” written on it legibly fill out notification sign-up sheet

experience sampling method -- ubicomp acknowledgements drs. lisa & daniel barrett justin silves gaetano borriello ken anderson rahul shah trevor paring roy want john light our pilot testers and our colleagues at intel research seattle

experience sampling method -- ubicomp questions?

experience sampling method -- ubicomp examples of study questions where is your cell phone? in my… hand pocket/belt clip bag car home office somewhere else and with me somewhere else and not with me

experience sampling method -- ubicomp examples of study questions in the last 30 minutes, which of the following would you have liked to have access to but didn’t have available? web pages movie times restaurant reviews personal calendar telephone number(s) computer files music files shopping list “to do” list