Slide 1 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. The Story of Our Lives From Sensors to Stories in Self-monitoring Systems Dana Pavel, Vic Callaghan, Anind K. Dey (CMU), Francisco Sepulveda, Michael Gardner School of Computer Science and Electronic Engineering University of Essex 12 September 2012

Slide 2 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Outline Motivation and (current) goals MyRoR system Summary of experiments

Slide 3 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. MyRoR Background Lifestyle management is an important area for individuals and societies Most current solutions focus on showing what and less why it happened End users are often left out of the loop

Slide 4 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Main goals Provide a more comprehensive picture of user lives Better understand what information people find interesting Create better correlations and visualisations capable to capture such diverse information Involve the user in all aspects: from information gathering to processing to usage

Slide 5 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Towards a comprehensive user context Environmental noise level Battery level (phone) Signal strength (phone) Devices around (BT, WLAN- based) Location (GPS, cell ID, country code, WLAN based) URLs visited ECG/Heart rate 3-axis accelerom eter Event button (user pressed) data Calendar event Keystrokes Application context Time Mental context (interest, focus, etc.) Availability context (people or resource) Physical context (position, direction, distance, speed, proximity) Temporal context (absolute, relative, duration) Activity context Emotional context Social context (communication, identity)

Slide 6 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission.

Slide 7 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. System view

Slide 8 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Main challenges Synchronizing input devices Different formats for input data Correlating and interpreting data Evolving platform in time (over different situations, over life changes) Presenting information to end users –Detailed vs. abstract –Interactive –Evolving

Slide 9 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Desktop interface in MyRoR Main interface Abstract, story- based visualisations Detailed visualisations

Slide 10 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. MyRoR stories Why: Stories are most natural ways of “representing” information Stories can bring together different types of recorded information in a fun, concise and engaging way Stories require a focus on what is important (“collection of meaningful events” – Kevin M. Brooks ) How: MyRoR stories combine various media created based on interpreting available user information Take into account user annotations One main challenge: What is ‘meaningful’?

Slide 11 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Story creation process MyRoR DB Rule Engine/KBs

Slide 12 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Time Text version Icon version Location- based background Example: An event within a story

Slide 13 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Experiments and studies Online survey – 38 participants –Main focus: Explore attitudes towards self-reflection Test if such system is perceived as useful Explore issues around story creation and customisation – Hands-on experiments – 6 participants –Main focus: Importance of information Story creation and sharing Interactions with the system

Slide 14 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Results from survey (1/2) Q1: Do you often think back about what happened during the day? Q2: Do you think about what triggered a certain emotion or behaviour? Q3: Do you usually propose any change based on self-reflection?

Slide 15 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Results from survey (2/2) Q5: Do you use (or have used) any self- monitoring systems (e.g., fitness-related)? Q6: Would you find useful having a system as presented in the scenario? Q7: If you were to be using such a system would you like to be able to see a story generated based on your activity data?

Slide 16 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Quick Learnings (1/2) It’s good to have more contextual information as it helps remember, understand and reflect on what happened The usefulness of information cannot be detached from the context it was recorded in –We cannot create lifestyle management systems that only focus on a certain type of information Need to create systems that can dynamically change the importance of information based on user, situations and evolve over time Giving people the means to create annotations yields interesting results! –Difference between implicit vs. explicit meaning –Annotations as means for self-reflection –Need better ways for annotating (e.g., watch, pendant?)

Slide 17 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Quick Learnings (2/2) The story-based multi-media concept was preferred over graphs as a quick view into recorded data –Combine narrative with graphic Customization is desired but should not take too long –Background picture is important Stories should be able to change and evolve –Change focus, add new information Fun way of sharing recorded information –Needs means to adapt stories for sharing

Slide 18 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Future… Further develop the stories –Dynamic creation based on changing information, interest and audience Apply system to various areas –E.g., health area (look into embedding various psychological/behavioural/persuasive models)

Slide 19 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Contact info: Online survey: Work is performed under PAL project: Thank you!

Slide 20 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Background material

Slide 21 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Examples of information processing

Slide 22 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Remote access to system (phone-blog) Protected access to collected information Updating system with data (photos, blog entries, documents) Story playing

Slide 23 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Mapping emotional states to colours

Slide 24 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Using colours to represent emotional states

Slide 25 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Visualisations in self-monitoring systems From Affective Diary work at SICS (Stahl, Hook, etc.) From Garmin software Using Google Visualizations

Slide 26 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Daily story Visualisations for information collected and derived stored in the personal database Information collected on demand from remote servers Calendar-based interface Own notes

Slide 27 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Based on user-based actions/annotations Determining “meaningful” events Based on more interesting context data E.g., calls/messages, changes in location, heart rate Allowing end user to select interesting information ! Need to be keep the story short!

Slide 28 © The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. PAL project This work is part of the PAL project ( funded by EPSRC and TSBhttp://palproject.org.uk PAL looks at self-monitoring solutions as part of future healthcare scenarios PAL also includes aspects of security and privacy in the context of various usages of recorded personal data