1. Slide 1 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. The Story of Our Lives From Sensors to Stories in Self-monitoring Systems Dana Pavel, Vic Callaghan, Anind K. Dey (CMU), Francisco Sepulveda, Michael Gardner School of Computer Science and Electronic Engineering University of Essex 12 September 2012

2. Slide 2 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Outline Motivation and (current) goals MyRoR system Summary of experiments

3. Slide 3 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. MyRoR Background Lifestyle management is an important area for individuals and societies Most current solutions focus on showing what and less why it happened End users are often left out of the loop

4. Slide 4 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Main goals Provide a more comprehensive picture of user lives Better understand what information people find interesting Create better correlations and visualisations capable to capture such diverse information Involve the user in all aspects: from information gathering to processing to usage

5. Slide 5 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Towards a comprehensive user context Environmental noise level Battery level (phone) Signal strength (phone) Devices around (BT, WLAN- based) Location (GPS, cell ID, country code, WLAN based) URLs visited ECG/Heart rate 3-axis accelerom eter Event button (user pressed) Email data Calendar event Keystrokes Application context Time Mental context (interest, focus, etc.) Availability context (people or resource) Physical context (position, direction, distance, speed, proximity) Temporal context (absolute, relative, duration) Activity context Emotional context Social context (communication, identity)

6. Slide 6 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission.

7. Slide 7 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. System view

8. Slide 8 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Main challenges Synchronizing input devices Different formats for input data Correlating and interpreting data Evolving platform in time (over different situations, over life changes) Presenting information to end users –Detailed vs. abstract –Interactive –Evolving

9. Slide 9 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Desktop interface in MyRoR Main interface Abstract, story- based visualisations Detailed visualisations

10. Slide 10 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. MyRoR stories Why: Stories are most natural ways of “representing” information Stories can bring together different types of recorded information in a fun, concise and engaging way Stories require a focus on what is important (“collection of meaningful events” – Kevin M. Brooks ) How: MyRoR stories combine various media created based on interpreting available user information Take into account user annotations One main challenge: What is ‘meaningful’?

11. Slide 11 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Story creation process MyRoR DB Rule Engine/KBs

12. Slide 12 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Time Text version Icon version Location- based background Example: An event within a story

13. Slide 13 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Experiments and studies Online survey – 38 participants –Main focus: Explore attitudes towards self-reflection Test if such system is perceived as useful Explore issues around story creation and customisation –http://ieg.essex.ac.uk/myror/survey/intro.phphttp://ieg.essex.ac.uk/myror/survey/intro.php Hands-on experiments – 6 participants –Main focus: Importance of information Story creation and sharing Interactions with the system

14. Slide 14 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Results from survey (1/2) Q1: Do you often think back about what happened during the day? Q2: Do you think about what triggered a certain emotion or behaviour? Q3: Do you usually propose any change based on self-reflection?

15. Slide 15 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Results from survey (2/2) Q5: Do you use (or have used) any self- monitoring systems (e.g., fitness-related)? Q6: Would you find useful having a system as presented in the scenario? Q7: If you were to be using such a system would you like to be able to see a story generated based on your activity data?

16. Slide 16 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Quick Learnings (1/2) It’s good to have more contextual information as it helps remember, understand and reflect on what happened The usefulness of information cannot be detached from the context it was recorded in –We cannot create lifestyle management systems that only focus on a certain type of information Need to create systems that can dynamically change the importance of information based on user, situations and evolve over time Giving people the means to create annotations yields interesting results! –Difference between implicit vs. explicit meaning –Annotations as means for self-reflection –Need better ways for annotating (e.g., watch, pendant?)

17. Slide 17 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Quick Learnings (2/2) The story-based multi-media concept was preferred over graphs as a quick view into recorded data –Combine narrative with graphic Customization is desired but should not take too long –Background picture is important Stories should be able to change and evolve –Change focus, add new information Fun way of sharing recorded information –Needs means to adapt stories for sharing

18. Slide 18 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Future… Further develop the stories –Dynamic creation based on changing information, interest and audience Apply system to various areas –E.g., health area (look into embedding various psychological/behavioural/persuasive models)

19. Slide 19 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Contact info: dmpave@essex.ac.uk Online survey: http://ieg.essex.ac.uk/myror/survey/intro.phphttp://ieg.essex.ac.uk/myror/survey/intro.php Work is performed under PAL project: http://palproject.org.ukhttp://palproject.org.uk Thank you!

20. Slide 20 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Background material

21. Slide 21 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Examples of information processing

22. Slide 22 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Remote access to system (phone-blog) Protected access to collected information Updating system with data (photos, blog entries, documents) Story playing

23. Slide 23 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Mapping emotional states to colours

24. Slide 24 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Using colours to represent emotional states

25. Slide 25 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Visualisations in self-monitoring systems From Affective Diary work at SICS (Stahl, Hook, etc.) From Garmin software Using Google Visualizations

26. Slide 26 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Daily story Visualisations for information collected and derived stored in the personal database Information collected on demand from remote servers Calendar-based interface Own notes

27. Slide 27 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. Based on user-based actions/annotations Determining “meaningful” events Based on more interesting context data E.g., calls/messages, changes in location, heart rate Allowing end user to select interesting information ! Need to be keep the story short!

28. Slide 28 © 2009-2012 The Copyright in this document belongs to the PAL Consortium and no part of this document should be used or copied without their prior written permission. PAL project This work is part of the PAL project (http://palproject.org.uk), funded by EPSRC and TSBhttp://palproject.org.uk PAL looks at self-monitoring solutions as part of future healthcare scenarios PAL also includes aspects of security and privacy in the context of various usages of recorded personal data