1. SAMI: Situational Awareness from Multi-modal Input Naveen Ashish

2. Talk Organization  Why are we at RESCUE interested ?  Situational Awareness (SA) –Introduction  System architecture  Research challenges  Expected outcomes and artifacts  Extraction system demonstration

3. Team Naveen Ashish Sharad Mehrotra Nalini Venkatasubramanian Utz Westermann Dmitry Kalashnikov Stella Chen Vibhav Gogate Priya Govindarajan Ram Hariharan John Hutchinson Yiming Ma Dawit Seid Jay Lickfett Chris Davision Quent Cassen Bhaskar Rao Mohan Trivedi Rajesh Hegde Sangho Park Shankar Shivappa Ron Eguchi Mike Mio Jacob Green

4. Information from Various Sources People/Victims at disaster Emergency responders News, video, audio footage GIS, satellite imagery, maps Pushing “Human-as-sensor”

5. More Data ≠ More Information SA Where are the fire personnel ? Have all medical supplies reached ? What areas should we start evacuating first ?

6. Situational Awareness  Wide variety of fields –Beginning in mid-80s, accelerating thru 90s –Fighter aircraft, ATM, Power plants, Manufacturing  Definitions –"the perception of elements in the environment along with a comprehension of their meaning and along with a projection of their status in the near future" –"the combining of new information with existing knowledge in working memory and the development of a composite picture of the situation along with projections of future status and subsequent decisions as to appropriate courses of action to take"  Situational awareness and decision making  Areas –Cognitive science –Information processing –Human factors Knowing what is going on

7. Abstraction of Information Multimodal Input: Text, Audio, Video Events Awareness

8. First-cut Architecture EVENT BASE Querying and Analysis Graph View VISUALIZATION and USER INTERFACES Spatial Indexing PDF Histogram KNOWLEDGE: ONTOLOGIES Text Audio Video Internet RAW DATA EVENT EXTRACTION REFINEMENT Disambiguation Location Centered around EVENTS as fundamental abstractions

9. Research Areas Event Modeling Event Extraction Disambiguation Location Uncertainty Graph Analysis GIS Querying

10. Event Modeling  What is an event ?  Event Representation RELIABILITY PEOPLE EVACUATION LOCATION TIME REPORT TYPE NAME LOCATION AGENCY FROM TO OPERATION NUMBER

11. Domain Knowledge  Captured as Ontologies ROAD EVACUATION AIR EVACUATION IS-A THAILANDSOUTHERN REGION …….PHUKETPHUKET, CHANGWAT

12. Event Extraction  Long history of information extraction –IR (MUC efforts) –Web data extraction  DARPA ACE –Entities, Relations, Events –Events in 2004  Event extraction accuracy is still low  SA Domain –Stream of information –Duplicated, ambiguous –Reliability –Conversations  Modalities –Text

13. Semantics Driven Approach  Semantics Driven  Challenges –Framework –Ontologies  What semantics required for event extraction ?  Application  With NLP, ML techniques  Performance –SA specific  Duplicates, reconciliation, temporal, conversations …..

14. Disambiguation

16. –point-location  in terms of landmarks  uncertain, not (x,y) –reasoning on such data  support all types of queries Uncertainty is a Challenge Report 1: “... a massive accident involving a hazmat truck on I5-N between Sand Canyon and Alton Pkwy...” Report 2: “... a strange chemical smell on Rt133 between I405 and Irvine Blvd...” Report 1 Report 2

17. Implications of Uncertainty in Text How to model uncertainty? –probabilistic model –P(location | report)  e.g. report says “near building A” Queries –cannot be answered exactly...  use probabilistic queries  all events: P(location  R | report) > 0 –SA requirements  triaging capabilities  fast response –top- k –threshold: P(location  R | report) >  –  -RQ, k-RQ, k  -RQ How to map text to probabilities? –use spatial ontologies A B R

18. Graph Analysis  GAAL  Inherent spatio-temporal properties  Graphs are powerful for querying and analysis

19. Current FGDC Search GIS Search

20. Progressive Refinement of Data GIS Search

21. Deliverables, Outcomes, Artifacts  “Vertical” thrusts –Event extraction system (TEXT) –Disambiguation system –GIS search system  Overall system demonstration ?  “By-products” –Ontologies  Computer science research areas Databases Semantic-Web Information Retrieval Intelligent Agents (AI)

22. Thank you ! http://sami.ics.uci.edu