1. Context Aware: TNBT, … T hing ? The Big Bet of (CaC) is that The Next Big Thing is: Dramatic Increase of Human Productivity… using computational skills provided by computers as well as …sensing details system that will augment people experience in interaction with reality Gordon Bell www.MyLifeBits.comwww.MyLifeBits.com www.TotalRecallBook.comwww.TotalRecallBook.com 8 March 2010

2. CA: Context Aware Computing… Assume it is TNBT; then is it? New capabilities based on these platforms or technologies? a.All the bits in your e-Memory (Bell-Gemmell PoV) b.“Personal_state(t)”: Loc/pt. x,y,z,ρ,θ,φ; HR,BP, stress; T, Δ’s c.Social Networking d.Cloud enabling new client platforms e.All the above New product(s)? New Service(s)? Features? Featurettes? – Revenue? Business model? CA-thing is very specific as the intersection of ≥2 “dimensions” CA-ness can enable a function or a “point” product/service What problem(s) does having a/more context solve? – How? – Why?... More context, more opportunities for action? – When in doubt, create a taxonomy!

3. CA and CAO: GB thoughts re. definition MS makes $s as an opportunity enabler Context Awareness … the intersection of two or more “state dimensions” providing an opportunity for some kind of action e.g. meeting, insight, path to take/avoid, something to buy… – Time and place – Place and place (e.g.meet friends, shopping) CA is the detection & creation of CAO’s… SEARCH! More state => more opportunities for action More public state => interpersonal opportunities More mobility (physical or cyberspace) => opportunity Availability of entire life => opportunity

4. What is Context? (ISO) 9/10/2008 from Context Vision Users, their goals and tasks, infrastructure (hardware, software, materials), physical and social environment in which the product is used.* * As defined by ISO 9421-11

5. Scope: User 9/2008 9/10/2008 from Context Vision Microsoft Confidential5 Goals & Tasks Activities Health Health History Current Health Traditions Person Family Social Groups Nation Social Relations Psychologic al Type & Current Emotions Life Experience & Personal Interests Contact Identity Data Name, addresses, emails, phones, etc. Biometric “Context combines users, their goals and tasks, infrastructure (hardware, software, materials), physical and social environment in which the product is used.”

6. My context = My e-Memory + my current state

7. eMemory =ground truth bioMemory = metadata... Memory centric view: Context = eMemory + bioMemory (aka State of mind) + environment Web Mail, dcs, books, web, journals, podcasts… Vibe Social: iM, facebook, … calls, conversations… Financial: $ transaction Health: PHR, activity, diet iLife {photo/movie TV music}, trip trajectories Logs … for all “lives” …for all activities Memex aka MyLife Bits aka eMemory “Lives...”

9. Realizing Memex… Digital Capture, Storage, and Utilization of All Personal Information www.MyLifeBits.com www.MyLifeBits.com

10. MyLifeBits Platform c2005 MyLifeBits store database Voice annotation tool Telephone capture tool TV capture tool TV EPG download tool Radio capture & EPG PocketPC transfer tool PocketRadio player Import files MyLifeBits Shell Browser tool Internet IM capture GPS import & Map display SenseCam Screen saver Text annotation tool MAPI interface Legacy email client Outlookinterface files Legacy applications VIBElogging RoomCapture

11. Using an e-Memory to augment b-memory (how did we get the bits for context?) 1998-2001 > Capturing the high order past & present bits – Instigated by Raj Reddy’s Million Book Project – Stimulated by telepresence, Billg “someday you can recall everything…”, idea of tidiness, design for CyberMuseum (Computer History Museum) – “ MyLifeBits is about converting everything to bits” 2001-2005 > Getting bits from e-Memory… as many bits as possible – Stimulated by Bush, Gray (database) … – MyLifeBits Project moves to SQL. Acquire “context” from everything we can – “MyLifeBits is a transaction processing system for a life” 2005: Memex Project. Redmond := Gemmell and Lueder; Project := 0 2009: Total Recall := Book_write_&_publish (Bell_and_Gemmell) – “Mylifebits is an e-memory… to aid bio-memory for life &afterlife” Right Period to create “Memex” based on technologies – We can encode it: Scanning and recognition problem (worked on e-Paper) – We can store it: Terabyte (we thought that would store a digital life) – We can find it: Search came into its prime

12. NOT LIFE-BLOGGING… We think life-Blogging is nuts! LIFE-BLOGGING => GREATER CAO

14. Capturing every step. If I know where I am, what good does it do me or anyone?

15. HEALTH RECORDS BY THE INDIVIDUAL, FOR THE INDIVIDUAL Hospital, GP, specialist, lab, dentist, chiropractor, pharmacist… You need to have control of your records Health: Context matters! When you are about to …

16. Cell Phone as a Platform for Healthcare TechFest 2009 SCS Where was I, what was I doing, who was I with, (what meeting was I in) when I had a high HR or high stress indicator ?

17. The killer app. Health “those spots look a bit better, don’t they?”

18. Navigenics Report

19. 1942 Mayo Clinic visit

20. BIO-METRIC LIFE-LOGGING

21. EKG just before pacemaker 040629 A flaky connection between chambers!

22. Capturing every heartbeat 72.6 beats/min; 38.16 Million beats/year 3.13 billion beats year The important number is 4-4.5 years, or ETS Battery life: the expected trip to surgery!

23. Bypass Feb 83

24. Tracking Test Results……. It’s all context… effecting mental state

25. Philips in-home devices

26. Bodybugg

27. Smart materials

28. In-body health sensing Nanobot in the bloodstream pillcam EndoSure Wireless Pressure Sensor in an aneurysm sac

29. Health Monitoring: “Your husband just died, … here’s his black box”

30. Context awareness actions are based on f(which lives you are in right now, what activity are you engaged in, your accessible digital life)

31. Contacts Office Documents Communication email Profession specific Finance & legal Health records On body recording Books Instructional Mat. Music Photos Videos Location Everything I see Everything I hear Family Parents, Siblings, Children Friends Clubs and Organizations Educational Institutions Work organizations … Projects Office Health Learning Ambience Person Home Job Who: Lives… Lifetime periods Which lives What: Degree “being digital”. Your “digital life” Where: “Location” of your digital life PC … Home … Cellphone … Cloud… Public Cloud Degree of privacy “Lives” aka “Lifebrowsing” “ Personal Role Management” “Autobiographical Memory” for structuring life into organizational groups; general events; and event specific knowledge time periods Who: which Life What: activity Where: you are

33. Personal LifeLog Applications Conservator Baby Book Companion Caretaker Babysitter Advisor Mentor Tutor Autobiography Photo Album Personal Assistant Diary/Journal Biography Medical Manager Executor Obituary OthersSelf Assistant for Elderly Application controlled by: Others Self Application used by: Personal Proxy Parole Officer Personal Flight Recorder Meeting Prep Captain’s Log Trustee Financial Manager Doug Gage, ARPA LifeLog

34. My Bits… then (c2000) and now (c2010) Single system… Where are they? How owns them? Who can access them? How do I keep them private? … Bush’s Memex c1945

35. The vision…

36. Memex Store From Susan Dumais c2005 (I think) My Bits 2010 Dist’d state & Context

37. Where are all of your bits and meta-data? Islands of content & pointers 1.Folder/file hierarchy 2.Outlook (msg, contact, calendar, task); Journal & VIBE 3.OneNote; EverNote 4.Social/professional: iM, Skype, Connection Manager, LinkedIn, Facebook, Twitter, etc. 5.Money… financial institutions (bank, broker, insurance) 6.HealthVault … BodyBugg, … medical providers 7.Music: iTunes, Media Player, Media Center, …Zune 8.Photos: Live photo gallery, Picassa, Flicker 9.Media Center 10.iLife={iPhoto, iMovie, iTunes, iWeb, iBook}

41. Opinions: A Challenge Xerox PARC was unique: they shared and worked to a vision – TNBT: unlikely to come from MSR as a collection of individuals – I still think something like Memex is required… CA, like UI… is techniques or capabilities to enhance/create functions neither is a product, but may be the basis of a compelling product as Apple has repeatedly shown Challenge: What can CaC be used for to create value for MSFT? Google is en route to dominate the C-C (cloud-client) platform – Google development/programming eco-system is one to understand and emulate in light of a complex PC environment – MSR & CS requires a cloud-thin client focus for new relevancy e-Memory i.e. current and past state of an individual is likely or must be at the core or base of (context = state = memory) involving individual action

42. MyLifeBits Project aka Memex Influencing products and research Products & Research Efforts – DARPA LifeLog; DARPA Assist; DARPA CALO and SRI spinoff SIRI – EverNote as a OneNote competitor but focused on e-Memory, – reQall a voice reminder and memory (knows about time and space) – http://www.lifebio.com/ Capture your story for immortality; www.famento.com - your family history http://www.lifebio.com/ www.famento.com – Stanford SALT project to Archive the faculty – IBM Pensieve (IBM Research Personal Memory Organizer) – Korean and Japanese Research Insitutes both Dedicated to MyLifeBits – UK Memories for Life Project… – Caprica TV prequel to Battleship Galactica – Aimee Baldridge, Organize Your Digital Life: How to Store Your Photographs, Music, Videos, and Personal Documents in a Digital World – MSR Global Heritage Project Conferences: – ACM Workshop on Continuous Archival and Retrieval of Personal Experiences (CARPE)Continuous Archival and Retrieval of Personal Experiences – Society for Imaging Science &Technology Conference on Archiving, April 2004 – First Digital Lives Conference, British Libraries, Feb 2009 – National Conference on Archive, West Midlands, UK, November 2009 – Saving Our Present for the Future: Personal Archiving 2010, archive.org, Feb 2010 – Cultural Heritage Track for ACM Multimedia

43. A couple of recent Google slides… Comment by Marc Benioff “Enterprise software had traditionally been built by engineers who worked in corporate office parks with little or no interaction with the customers until a pre-determined “five-year plan” was complete. This isolationist existence has seriously stymied innovation in the technology industry. Companies were not moving fast enough for customers.” -Marc Benioff, Behind The Cloud

44. Cloud-client computing: research role? The PC is a less relevant environment in the face of the cloud and especially IF/When office apps go there. Many users would be happy to have apps run in the cloud and have their data there permanently… CS Research needs to be working and training students for the cloud environments Bing Product cycle: 4 months; Win/Office: >36 mos. – Idea to use is 9X faster; a published idea ships immediately R > AD …. > Product groups > Product (3-5 years at best) Transfer is fundamentally flawed. Product groups can just look at the competitors

45. Memex: A Supported and Active Research Platform for MSR’s Research* Ken Wood, Roger Lueder, Jim Gemmell, Mary Czerwinski, Gordon Bell Supported Platform created and based on: MyLifeBits, VIBE, SenseCam, Digital Geographics, and MLB University Research Program The narcissistic effect: “By having a common platform in use and built on, we can expect exponential increases in functionality through cross-app use and common schema, plus incremental improvements to performance and reliability. This is the nature of all such systems.” The Memex program has four components or goals: 1.continued research on the platform that includes validating WinFS efficacy --especially for the PC for personal data, real time, and a person’s transactions; 2.research coming from Redmond, Cambridge, product efforts and universities; 3.a pre-WinFS platform for building and testing new database-based apps in a real use environment that allows us to”eat our dogfood”; and 4.live use that creates stability, understanding, and new research ideas, problems, and attraction for more apps. *Proposal 6/15/2005

46. End

47. The Agenda for the Lifetime e-Memory 1.Guarantee that data will live forever! “dear appy” problem 2.Data Aggregation from the myriad of data places: cloud, work, home, social sites 3.More content: Cheap, easy, and data-rich (e.g. time, place) capture of: GPS and time everywhere Paper capture has to be as easy as discarding (scanner/shredder) Personal meeting capture... Telephony and audio capture with indexing– speech to text E-book…e-magazines & journals need to have critical mass! Media Center compatible for entertainment (photos, video, TV, radio) Connection to other sensor nets in home, auto, and elsewhere as they slowly come into existence 4.Annotations/meta-information add every-increasing value Easy annotation for aiding search and it becomes the content 5.Automatic classification and organization into taxonomies, facets, etc. 6.Content analysis (critical for photo & video!) 7.Information control: privacy, security, expunge/deniability,… 8.Having to be schizophrenic or have a lobotomy when leaving a “life” 9.One Is dbase intuitive? 9.One dbase for everything (articles, books, conversations,... financial transactions) …vs. long-term use of hierarchical files. Is dbase intuitive? 10.The “killer apps”: Alzheimer, immortality, surrogate memory? 11.Clean living, cloaking, versioning, structure to contacts (family, org charts) 12.GUI’s to improve use (e.g. time to learn, use, retention)

48. HMI 2009: 1.3 Thr;3.6 Zby; 10.8 Pwords Report on US Consumers, Bohn and Short, UCSD Per User / Per Day HoursMegabytesKWords Traditonal TV29.282033261 Gaming incl. computer3.918989911 TV, Games+Movies+DVD33.3275987275 Books, News, Magazines1.16.616 Radio & MP35.338224 Internet, offline programs2.613136 Telephony: Pots, mobile1.122.88

49. From HMI Report on Info Sources in the US