1. Semantics enhanced Data, Social and and Sensor Webs Talk at Dagstuhl Seminar on Semantic Challenges in Sensor Networking Amit Sheth Kno.e.sis Center, Wright State University

2. Machine Sensing Inert, fixed sensors Carried on moving objects –Vehicles, pedestrians (asthma research) –anonymous data from GPS-enabled vehicles, toll tags, and cellular signaling to mark how fast objects are moving – and overlaying that information with location data and maps (traffic.com, Nokia experiment, …) 2 Text from http://www.geog.ucsb.edu/~good/presentations/icsc.pdf Images credit – flickr.com, cnet.com

3. Semantic Sensor Web

4. Semantically Annotated O&M 2008-03-08T05:00:00,29.1

5. Semantic Sensor ML – Adding Ontological Metadata 5 Person Company Coordinates Coordinate System Time Units Timezone Spatial Ontology Domain Ontology Temporal Ontology Mike Botts, "SensorML and Sensor Web Enablement," Earth System Science Center, UAB Huntsville

6. 6 Semantic Query Semantic Temporal Query Model-references from SML to OWL-Time ontology concepts provides the ability to perform semantic temporal queries Supported semantic query operators include: – contains: user-specified interval falls wholly within a sensor reading interval (also called inside) – within: sensor reading interval falls wholly within the user-specified interval (inverse of contains or inside) – overlaps: user-specified interval overlaps the sensor reading interval Example SPARQL query defining the temporal operator ‘within’

7. Kno.e.sis’ Semantic Sensor Web 7

8. Semantic Sensor Web demo Sensor Discovery on Linked Data demo (Google “Semantic Sensor Web”, demos at the bottom of the Kno.e.sis project page)

9. Citizen Sensing Where humans act as sensors or observers Around them is a network of sensors, computing and communicating with each other –Processing and delivering multi-modal information –Collective Intelligence Information-centric to Experience-centric era –Modeling, processing, retrieving event level information Use of domain knowledge –…. Understanding of casual text 9

10. Citizen Sensors Human beings –6 billion intelligent sensors, 4 million mobile devices –informed observers –rich local knowledge –uplink technology broadband Internet mobile phone 10 Christmas Bird Count

11. mumbai, india

12. november 26, 2008

13. another chapter in the war against civilization

14. and

17. the world saw it Through the eyes of the people

18. the world read it Through the words of the people

19. PEOPLE told their stories to PEOPLE

20. A powerful new era in Information dissemination had taken firm ground

21. Making it possible for us to create a global network of citizens Citizen Sensors – Citizens observing, processing, transmitting, reporting

22. Image Metadata latitude: 18° 54′ 59.46″ N, longitude: 72° 49′ 39.65″ E Image Metadata latitude: 18° 54′ 59.46″ N, longitude: 72° 49′ 39.65″ E Geocoder (Reverse Geo-coding) Geocoder (Reverse Geo-coding) Address to location database 18 Hormusji Street, Colaba Nariman House Identify and extract information from tweets Spatio-Temporal Analysis Structured Meta Extraction Income Tax Office Vasant Vihar

23. Research Challenge #1 Spatio Temporal and Thematic analysis – What else happened “near” this event location? – What events occurred “before” and “after” this event? – Any message about “causes” for this event?

24. Spatial Analysis…. Which tweets originated from an address near 18.916517°N 72.827682°E?

25. Which tweets originated during Nov 27th 2008,from 11PM to 12 PM

26. Giving us Tweets originated from an address near 18.916517°N, 72.827682°E during time interval 27 th Nov 2008 between 11PM to 12PM?

27. Research Challenge #2: Understanding and Analyzing Casual Text Casual text – Microblogs are often written in SMS style language – Slangs, abbreviations

28. Understanding Casual Text Not the same as news articles or scientific literature – Grammatical errors Implications on NL parser results – Inconsistent writing style Implications on learning algorithms that generalize from corpus

29. Nature of Microblogs Additional constraint of limited context – Max. of x chars in a microblog – Context often provided by the discourse Entity identification and disambiguation Pre-requisite to other sophisticated information analytics

30. NL understanding is hard to begin with.. Not so hard – “commando raid appears to be nigh at Oberoi now” Oberoi = Oberoi Hotel, Nigh = high Challenging – new wing, live fire @ taj 2nd floor on iDesi TV stream Fire on the second floor of the Taj hotel, not on iDesi TV

31. Research Opportunities NER, disambiguation in casual, informal text is a budding area of research Another important area of focus: Combining information of varied quality from a – corpus (statistical NLP), – domain knowledge (tags, folksonomies, taxonomies, ontologies), – social context (explicit and implicit communities)

32. Social Context surrounding content Social context in which a message appears is also an added valuable resource Post 1: – “Hareemane House hostages said by eyewitnesses to be Jews. 7 Gunshots heard by reporters at Taj” Follow up post – that is Nariman House, not (Hareemane)

33. Understanding content … informal text I say: “Your music is wicked” What I really mean: “Your music is good” 33

34. Structured text (biomedical literature) Multimedia Content and Web data Web Services Semantic Metadata: Smile is a Track Lil transliterates to Lilly Allen Lilly Allen is an Artist Informal Text (Social Network chatter) Your smile rocks Lil Urban Dictionary MusicBrainz Taxonomy Artist: Lilly Allen Track: Smile Sentiment expression: Rocks Transliterates to: cool, good

35. Example: Pulse of a Community Imagine millions of such informal opinions –Individual expressions to mass opinions “Popular artists” lists from MySpace comments Lilly Allen Lady Sovereign Amy Winehouse Gorillaz Coldplay Placebo Sting Kean Joss Stone

36. Domain Knowledge: A key driver Places that are nearby ‘Nariman house’ –Spatial query Messages originated around this place –Temporal analysis Messages about related events / places –Thematic analysis

37. Research Challenge #3 But Where does the Domain Knowledge come from? Expert and committee based ontology creation … works in some domains (e.g., biomedicine, health care,…) Community driven knowledge extraction –How to create models that are “socially scalable”? –How to organically grow and maintain this model?

38. Building models…seed word to hierarchy creation using WIKIPEDIA Query: “cognition”

39. Today’s Network of Sensors Are sensing, computing, transmitting Are acting in concert –Sharing data –Processing them into meaningful digital representations of the world Researchers using 'sensor webs' to ask new questions or test hypotheses 2009: 1.1 billion PCs, 4 billion mobile devices, 40+ billion mobile sensors (Nokia: Sensing the World with Mobile Devices)Nokia: Sensing the World with Mobile Devices 39

40. Consider that all objects, events and activities in the physical world have a counterpart in the Cyberworld (IoT) multi-facted context of real world is captured in the cyberworld (multilevel & citizen sensors/participatory sensing) each object, event and activity is represented –with semantic annotations (semantic sensor web) for a chosen context, with an ability to explicate and associate variety of relationships and events (Relationship Web, EventWeb) appropriate reasoning and human/social interaction are available and applied, insights extracted (semantic web, social semantic web, experiential computing) Activity anticipated/answers obtained/ decisions reached/communicated/applied 40

41. 41 Psyleron’s Mind-Lamp (Princeton U), connections between the mind and the physical world. Neuro Sky's mind-controlled headset to play a video game. MIT’s Fluid Interface Group: wearable device with a projector for deep interactions with the environment Physical-Cyber divide is narrowing Sensing emotion is increasingly possible and sensors are being developed to capture Emotions.

42. imagine

43. imagine when

44. meets Farm Helper

45. with this Latitude: 38° 57’36” N Longitude: 95° 15’12” W Date: 10-9-2007 Time: 1345h

46. that is sent to Geocoder Farm Helper Services Resource Sensor Data Resource Structured Data Resource Agri DB Soil Survey Agri DB Soil Survey Lat-Long Lawrence, KS Weather data Soil Information Pest information … Weather Resource Location Date /Time Weather Data

47. and

48. Six billion brains

49. imagination today

50. impacts our experience tomorrow

51. What Drives the Spatio-Temporal-Thematic Analysis and Casual Text Understanding Semantics with the help of 1.Domain Models 2.Domain Models 3.Domain Models (ontologies, folksonomies)

52. And who creates these models? YOU, ME, We DO!

53. How do you get comprehensive situational awareness by merging “human sensing” and “machine sensing”? 53

54. Synthetic but realistic scenario an image taken from a raw satellite feed 54

55. an image taken by a camera phone with an associated label, “explosion.” Synthetic but realistic scenario 55

56. Textual messages (such as tweets) using STT analysis Synthetic but realistic scenario 56

57. Correlating to get Synthetic but realistic scenario

58. Create better views (smart mashups)

59. Twitris demo (Search “Twitris” on YouTube)

60. A few more things Use of background knowledge Event extraction from text – time and location extraction Such information may not be present Someone from Washington DC can tweet about Mumbai Scalable semantic analytics – Subgraph and pattern discovery Meaningful subgraphs like relevant and interesting paths Ranking paths

61. The Sum of the Parts Spatio-Temporal analysis – Find out where and when + Thematic – What and how + Semantic Extraction from text, multimedia and sensor data - tags, time, location, concepts, events + Semantic models & background knowledge – Making better sense of STT – Integration + Semantic Sensor Web – The platform = Situational Awareness

62. Structured text (biomedical literature) Informal Text (Social Network chatter) Sensors, Multimedia Content and Web data Web Services Metadata Extraction Patterns / Inference / Reasoning Domain Models Meta data / Semantic Annotations Relationship Web Search Integration Analysis Discovery Question Answering

63. Observation (senses) Observation (sensors) Perception (analysis) Perception (cognition) Communication (language) Communication (services) People Web (human-centric) Sensor Web (machine-centric)

64. Observation Perception Communication Enhanced Experience (humans & machines working in harmony) Semantics for shared conceptualization and interoperability between machine and human Ability to share common communication

65. PARADIGM SHIFT Sensing, Observation, Perception, Semantic, Social Experiential 65

66. From the Semantic Web Community Several key contributing research areas –Operating Systems, networks, sensors, content management and processing, multimodal data integration, event modelling, high-dimensional data visualization …. Semantics and Semantic technologies can play vital role –In the area of processing sensor observations, the Semantic Web is already making strides –Use of core SW capabilities: knowledge representation, use of knowledge bases (ontologies, folkonomies, taxonomy, nomenclature), semantic metadata extraction/annotation, exploiting relationships, reasoning 66

67. THERE IS MORE HAPPENING AT KNO.E.SIS http://knoesis.org 67 Thanks: NSF (SemDis, Spatio-temporal-thematic), NIH, AFRL,SemDisSpatio-temporal-thematicNIH and also Microsoft Research, HP Research, IBM Research. See http://knoesis.wright.edu/projects/funding/

68. KNO.E.SIS MEMBERS – A SUBSET

69. Kno.e.sis Center Labs (3 rd Floor, Joshi) Amit Sheth Semantic Science Lab Semantic Web Lab Service Research Lab TK Prasad Metadata and Languages Lab Shaojun Wang Statistical Machine Learning Pascal Hitzler Formal Semantics & Reasoning lab Michael Raymer Bioinformatics Lab Guozhu Dong Data Mining Lab Keke Chen Data Intensive Analysis and Computing Lab

70. Interested in more background? References/More Details Amit Sheth, Cory Henson, and Satya Sahoo, Semantic Sensor Web, IEEE Internet Computing, July/August 2008, p. 78-83. Also see: http://wiki.knoesis.org/index.php/SSWSemantic Sensor Webhttp://wiki.knoesis.org/index.php/SSW Amit Sheth and Meenakshi Nagarajan, Semantics-Empowered Social Computing. IEEE Internet Computing Jan/Feb 2009, pages 76-80Semantics-Empowered Social Computing Amit Sheth, Semantic Integration of Citizen Sensor Data and Multilevel Sensing: A comprehensive path towards event monitoring and situational awareness, Keynote, February 17, 2009.Semantic Integration of Citizen Sensor Data and Multilevel Sensing: A comprehensive path towards event monitoring and situational awareness Amit Sheth, Citizen Sensing, Social Signals, and Enriching Human Experience, IEEE Internet Computing, July/August 2009, pp. 80-85.Citizen Sensing, Social Signals, and Enriching Human Experience Meena Nagarajan, User-Generated Content on Social Media, Keynote talk at Social Data on the Web (SDOW09) workshop collocated with ISWC, October 2009. Also http://knoesis.wright.edu/research/semweb/projects/socialmedia/User-Generated Content on Social Media http://knoesis.wright.edu/research/semweb/projects/socialmedia/ M. Nagarajan et al., Spatio-Temporal-Thematic Analysis of Citizen-Sensor Data - Challenges and Experiences, Tenth International Conference on Web Information Systems Engineering, Oct 5-7, 2009, Poland.Spatio-Temporal-Thematic Analysis of Citizen-Sensor Data - Challenges and Experiences Amit Sheth, 'Computing for Human Experience: Semantics-Empowered Sensors, Services, and Social Computing on the Ubiquitous Web,' IEEE Internet Computing (Sp. Issue on Internet Predictions: V. Cerf and M. Singh, Eds.), vol. 14, no. 1, pp. 88-91, Jan./Feb. 2010. Also: http://wiki.knoesis.org/index.php/Computing_For_Human_Experience'Computing for Human Experience: Semantics-Empowered Sensors, Services, and Social Computing on the Ubiquitous Web http://wiki.knoesis.org/index.php/Computing_For_Human_Experience Prateek Jain, Pascal Hitzler, Peter Z. Yeh, Kunal Verma and Amit P. Sheth, Linked Data is Merely More Data, AAAI Spring Symposium Linked Data Meets Artificial Intelligence, Stanford, CA, USA, March 22-24, 2010.Linked Data is Merely More Data

71. On-line Demos: [SSW] Semantic Sensor Web – Spatio-teporal-thematic and role-based queries over Mesowest DataSemantic Sensor Web [SDLoD] Sensor Discovery on Linked Data: 1+ billion triple of US storms data on LOD cloud, with Sensor Discovery MashupSensor Discovery on Linked Data [Twitris] Twitris: Twitter through Theme, Space, and Time. ISWC Semantic Web Challenge, October 2009.Twitris: Twitter through Theme, Space, and Time Contact/more details: amit @ knoesis.orgamitknoesis.org Special thanks: Karthik Gomadam, Cory Henson, Meena Nagarajan, Christopher Thomas Partial Funding: NSF (Semantic Discovery: IIS: 071441, Spatio Temporal Thematic: IIS- 0842129), AFRL and DAGSI (Semantic Sensor Web), Microsoft Research and IBM Research (Analysis of Social Media Content),and HP Research (Knowledge Extraction from Community-Generated Content).Semantic DiscoverySpatio Temporal ThematicSemantic Sensor WebAnalysis of Social Media ContentKnowledge Extraction from Community-Generated Content