The Second Life of a Sensor: Integrating Real-World Experience in Virtual Worlds using Mobile Phones Mirco Musolesi, Emiliano Miluzzo, Nicholas D. Lane, Shane B. Eisenmany, Tanzeem Choudhury and Andrew T. Campbell Dartmouth College, USA yColumbia University, USA

Outline Introduction to Second Life Introduction to Second Life Motivation Motivation System Design Challenges System Design Challenges Prototype implementation Prototype implementation Evaluation Evaluation Conclusion Conclusion

What ’ s Second Life? An internet based virtual world video game An internet based virtual world video game Enables users, “ Residents ”, to interact with each other through motional avatars Enables users, “ Residents ”, to interact with each other through motional avatars Residents can explore, meet other Residents, socialize, participate in individual and group activities, and create and trade items with one another Residents can explore, meet other Residents, socialize, participate in individual and group activities, and create and trade items with one another

Second Life

Motivation Virtual world still ≠Real world Virtual world still ≠Real world Changes in real world can’t be automatically translated to avatar in virtual world. Changes in real world can’t be automatically translated to avatar in virtual world. Provide a virtual representation, their surroundings as sensed by humans themselves, and interactions with other members. Provide a virtual representation, their surroundings as sensed by humans themselves, and interactions with other members.

Contributions Discuss the systems challenges and design issues in integrating real world people- centric sensor systems and virtual worlds Discuss the systems challenges and design issues in integrating real world people- centric sensor systems and virtual worlds Be able to reproduce human activities and not only inanimate objects Be able to reproduce human activities and not only inanimate objects Discuss a prototype based on implementation of the system on mobile phone Discuss a prototype based on implementation of the system on mobile phone

Activity Recognition and Visualization Activity inference Activity inference –Extraction of high level activities (sitting, running, walking or talking) –Activity mapping Visualization Visualization –User-specified visualization policies –Information accuracy

Intermittent Connectivity Cell phones may experience intermittent connectivity. Cell phones may experience intermittent connectivity. Avatars are set to a meaningful state. Avatars are set to a meaningful state. May involve the use of prediction to determine the current state and location. May involve the use of prediction to determine the current state and location. Employ delay tolerant mechanism to store user status on the phone then later be inputted to virtual world Employ delay tolerant mechanism to store user status on the phone then later be inputted to virtual world

Scalability High status/activities updating rate from users at server end High status/activities updating rate from users at server end Update occurs only when the inferred human status/activity of the human changes Update occurs only when the inferred human status/activity of the human changes Run full/a portion of activity recognition on cell phone Run full/a portion of activity recognition on cell phone

Data Sharing and Group Activities Existing work focuses on the analysis and processing of information collected by the local sensors. Existing work focuses on the analysis and processing of information collected by the local sensors. Sharing of raw or processed data among different people in physical proximity. Sharing of raw or processed data among different people in physical proximity. This allows for a more accurate inference of group activities This allows for a more accurate inference of group activities

External Sensing Devices Activity recognition can be processed by external devices though Bluetooth Activity recognition can be processed by external devices though Bluetooth Bluecel: Bluecel: –Badge size –Accelerometer, light, temperature –GSR sensor to infer stress level

Context Reflection Collected data can be displayed in virtual world by changing the appearance of an environment. Collected data can be displayed in virtual world by changing the appearance of an environment. EX: High humidity (real)  Rain or misty condition (virtual) EX: High humidity (real)  Rain or misty condition (virtual) User may participate in the data collection process such as audio/video media, photos … etc User may participate in the data collection process such as audio/video media, photos … etc

Privacy and Social Implications Be able to interrupt sampling at anytime in virtual world. Be able to interrupt sampling at anytime in virtual world. Actual position and actions of users may be disclosed according to pre-configured policies Actual position and actions of users may be disclosed according to pre-configured policies Personal information exposed to large set of people is an issue Personal information exposed to large set of people is an issue

CenseMe A platform acts as the core service for the retrieval and analysis of the sensed data A platform acts as the core service for the retrieval and analysis of the sensed data Provides information about personal status of its users and their surroundings Provides information about personal status of its users and their surroundings

Architecture of CenseMe

Prototype Design (1/2) CenseMe object behavior is written in LSL and communicates with Second Life server CenseMe object behavior is written in LSL and communicates with Second Life server Linden Script Language is an event oriented language and suited to implement state machine style solution Linden Script Language is an event oriented language and suited to implement state machine style solution A library of potential Second Life activities is maintained A library of potential Second Life activities is maintained

Prototype Design (1/2) 1.When the object is attached to avatar, a periodic timer is initiated. 1.When the object is attached to avatar, a periodic timer is initiated. 2.An HTTP request is made from Second Life server to CenseMe client via CenseMe 2.An HTTP request is made from Second Life server to CenseMe client via CenseMe CenseMe processes the incoming response and returning a list of invoked avatar actions CenseMe processes the incoming response and returning a list of invoked avatar actions Second Life server determines which action should be displayed Second Life server determines which action should be displayed

Screenshots of Avatars

Delay measurement

Future Work Investigate controlling movement in the geographical space Investigate controlling movement in the geographical space Define appropriate privacy policies Define appropriate privacy policies Develop mechanism to address disconnected period Develop mechanism to address disconnected period

Conclusion A mechanism to bridge real and virtual world using commercial mobile phone equipped with standard sensors. A mechanism to bridge real and virtual world using commercial mobile phone equipped with standard sensors. Discuss a number of key issues that need to be tackled. Discuss a number of key issues that need to be tackled. It ’ s the first time that sensors used on everyday mobile phones reflecting a person ’ s activity in Second Life It ’ s the first time that sensors used on everyday mobile phones reflecting a person ’ s activity in Second Life