Sensemaking and Ground Truth Ontology Development Chinua Umoja William M. Pottenger Jason Perry Christopher Janneck

Key Terms  Semantic Web - set of technologies and data formats that allow web applications to work and interoperate deeply, using structured knowledge.  Ontology - a representation of a set of concepts within a domain and the relationships between those concepts. OWL – Web Ontology Language  Blackbook - framework for analysis of data, implements Semantic Web technologies and provides a “network” functionality which reveals the link relationships among the objects.  Link Analysis - the active pursuit of identifying relationships and connections between values, entities, and objects.  Ground Truth - a set of data used as a “gold standard” to measure software performance.

Sensemaking  The human task of sorting through a large amount of information to aid situational awareness and decision-making  “A motivated, continuous effort to understand connections (which can be among people, places, and events) in order to anticipate their trajectories and act effectively” (Klein et al, 2006).  e.g. A police detective sorting through large amounts of data i.e. case documents in order to reach a conclusion on a certain case.  An active area of research in Cognitive Science

Pirolli and Card’s Sensemaking Model  Centered on intelligence analysis  Based on Empirical studies (Cognitive Task Analysis)  Capturing the schematic knowledge structures of expertise  Studies have produced evidence that the model is useful for identifying trade-off points that can increase analyst efficiency

Pirolli and Card’s Sensemaking Loop

The Sensemaking Support System  We are developing Sensemaking software for the Semantic Web, using Blackbook as the foundation, that enables police detectives to work with the computer in efforts to solve actual crimes.  Implements the Priolli and Card model of Sensemaking with machine learning to determine which route the detective is taking through the model and provide intelligent feedback.

The Initial Experiment  ~30 detectives will use the system to solve a real-world case using the ground truth data  The software will manage data access to only provide information that the detectives ask for, so the detectives’ skills and instincts are utilized as in a real case.  Goal: determine whether feedback based on the Sensemaking model helps arrive at the solution more efficiently.

Where it all Started…  In August 2006, Dr. Pottenger and others published an article called the “Link Analysis Survey” that set the ground work for my project today.  The process they developed allowed the users to conduct a comprehensive survey of the field’s solutions and metrics, analyze and develop a ground truth dataset on which to evaluate solutions, perform analysis and experimentation within the field of anonymization, propose several standards for use in the field, and identify critical trends and areas of future work.  Laid the foundation for investigation of link-based, knowledge-based systems

My Part…  Bethlehem Police Department, provided information on a case that has previously been solved.  Manually searched through all the documentation used to solve this case and found structure in the information that makes a fully-linked, semantic representation of the case data.  From the analysis of document structure and information content, we develop an ontology with properties and classes that will allow Blackbook to intelligently utilize the data.

My Part (Continued)  Write software to process the raw data and store in an RDF (Resource Description Framework) database corresponding to the Ontology’s types, for integration with Blackbook.  For the files that cannot converted cleanly (i.e. image files) we must create a link that can be used to show the total document that couldn’t be converted.  Ontology and RDF-structured data will be uploaded into Blackbook. Then all of Blackbook’s functionality will be tested with the case data.

What the Future Holds  The experiment will provide understanding of which structures actual detectives find important, so the Ontology can be refined and made more useful.  With analysis of more case documents, we can expand to a ‘deeper’ ontology with more connections. System becomes able to reason more intelligently over more types of real-world data.  Link-based, knowledge-based systems provide genuinely intelligent assistance for all kinds of analysis.

Thank you!