Libraries and Intelligence NSF/NIJ Symposium on Intelligence and Security Informatics. Tucson, AR. Paul Kantor June 2, 2003 Research supported in part by the National Science Foundation under Grant EIA and by the Advanced Research Development Activity under Contract 2002-H The views expressed in this presentation are those of the author, and do not necessarily represent the views of the sponsoring agency.

Relation to General Intelligence and Security Informatics Signal information Map and image information Sound/voice information Geographic information Structured (Database) information Free form textual information in machine readable form

Relation to Librarianship Much of the needed “technology” for managing information related to homeland security is of the same type that librarians have provided “by hand”. But.. –Millions of documents –dozens of languages –many media

Librarianship Cataloging– organizing information according to what it is about –Classification – Machine Learning –Use training examples –Adapt as more data is received –Filter huge streams of potentially relevant data Monitoring Message Streams

Librarianship Reference –Understand what the user wants –Understand both relevance and quality/genre –Learn from a dialog with the user Intelligent Question Answering

Two Projects Filtering/Monitoring Message Streams National Science Foundation (NSF) -- acting for the National Security Agency HITIQA - High quality interactive Question AnsweringMonitoring Message Streams HITIQA Advanced Research Development Activity (ARDA) of the Intelligence Community

Motivation:  monitoring of global satellite communications (though this may produce voice rather than text)  sniffing and monitoring traffic OBJECTIVE: Monitor streams of textualized communication to detect pattern changes and "significant" events

MMS Team Statisticians, computer scientists, experts in info. Retrieval & library science, etc Prof. Fred Roberts – decision rules Prof. David Madigan – statistics Dr. David Lewis –text classification Prof. Paul Kantor – info science Prof. Ilya Muchnik – statistics Prof. Muthu Muthukrishnan – algorithms Dr. Martin Strauss, AT&T Labs – algorithms Dr. Rafail Ostrovsky, Telcordia Technologies, -algorithms Prof. Endre Boros, --Boolean optimization. Dr. Vladimir Menkov programming; Dr. Alex Genkin programming; Mr. Andrei Anghelescu; graduate asisstant Mr. Dmitiry Fradkin; graduate assistant

Given stream of text in any language. Decide whether "new events" are present in the flow of messages. Event: new topic or topic with unusual level of activity. Retrospective or “Supervised” Event Identification: Classification into pre-existing classes. TECHNICAL PROBLEM:

More Complex Problem: Prospective Detection or “Unsupervised” Learning  Classes change - new classes or change meaning  A difficult problem in statistics  Recent new CS approaches 1)Algorithm detects a new class 2)Human analyst labels it; determines its significance

COMPONENTS OF AUTOMATIC MESSAGE PROCESSING (1). Compression of Text -- to meet storage and processing limitations; (2). Representation of Text -- put in form amenable to computation and statistical analysis; (3). Matching Scheme -- computing similarity between documents; (4). Learning Method -- build on judged examples to determine characteristics of document cluster (“event”) (5). Fusion Scheme -- combine methods (scores) to yield improved detection/clustering.

Random Projections Boolean Random Projections Robust Feature Selection Compression Representation Bag of Words Bag of Bits Matching Learning Fusion tf-idf kNN Boolean r-NN Rocchio separator Combinatorial Clustering Naïve Bayes Sparse Bayes Discriminant Analysis Support Vector Machines Non-linear Classifiers Project Components: Rutgers DIMACS MMS

Existing methods use some or all 5 automatic processing components, but don’t exploit the full power of the components and/or an understanding of how to apply them to text data. Lewis' methods used an off-the-shelf support vector machine supervised learner, but tuned it for frequency properties of the data.Very good TREC 2002 results on batch learning. Chinese Academy of Sciences used most basic linear classifier (Roccho model) and achieved the best adaptive learning) Proposed Advances

We can trace a path (called a homotopy) in method space, from a poor Rocchio model to the CAS one - - find some better results along the way. Next steps are:  more sophisticated statistical methods  sophisticated data compression in a pre- processing stage Proposed Advances II

Representations: Boolean representations; weighting schemes Matching Schemes: Boolean matching; nonlinear transforms of individual feature values Learning Methods: new kernel-based methods (nonlinear classification); more complex Bayes classifiers to assign objects to highest probability class Fusion Methods: combining scores based on ranks, linear functions, or nonparametric schemes MORE SOPHISTICATED STATISTICAL APPROACHES :.

Identify best combination of newer methods through careful exploration of variety of tools. Address issues of effectiveness (how well task is done) and efficiency (in computational time and space) Use combination of new or modified algorithms and improved statistical methods built on the algorithmic primitives. Systematic Experimentation on components and on fusion schemes THE APPROACH.

Mercer Kernels Mercer’s Theorem gives necessary and sufficient conditions for a continuous symmetric function K to admit this representation: “Mercer Kernels” This kernel defines a set of functions H K, elements of which have an expansion as: This set of functions is a “reproducing kernel hilbert space” K “pos. semi-definite” Prepared by David L. Madigan

Support Vector Machine Two-class classifier with the form: parameters chosen to minimize: Many of the fitted  ’s are usually zero; x’s corresponding the the non-zero  ’s are the “support vectors.” complexity penalty Gram matrix tuning constant Prepared by David L. Madigan

Regularized Linear Feature Space Model Choose a model of the form: to minimize: Solution is finite dimensional: just need to know K, not  ! prediction is sign(f(x)) A kernel is a function K, such that for all x,z  X where  is a mapping from X to an inner product feature space F Prepared by David L. Madigan

Mixture Models Pr(d|Rel)=af(d)+(1-a)g(d) f, g may be centered at different points in document space. So distinct conceptual representations are accommodated easily. Examples: multinomial distributions.

Example Results on Fusion ,000 documents.

Feature space Random Subspace Score space Learning takes place in two spaces: For matching and filtering, we learn rules in the primary space of document features. For fusion processes we learn rules in a secondary space of “pseudo-features” which are assigned by entire systems, to incoming documents. Relevant

REFERENCE ASPECT Effective Communication with the Analyst User

HITIQA: High-Quality Interactive Question Answering University at Albany, SUNY Rutgers University

HITIQA Team SUNY Albany: –Prof. Tomek Strzalkowski, PI/PM –Prof. Rong Tang –Prof. Boris Yamrom, consultant –Ms. Sharon Small, Research Scientist –Mr. Ting Liu, Graduate Student –Mr. Nobuyuki Shimizu, Graduate Student –Mr. Tom Palen, summer intern –Mr. Peter LaMonica, summer intern/AFRL Rutgers: –Prof. Paul Kantor, co-PI –Prof. K.B. Ng –Prof. Nina Wacholder –Mr. Robert Rittman, Graduate Student –Ms. Ying Sun, Graduate Student –Mr. Peng Song, Graduate student

HITIQA Concept Question: What recent disasters occurred in tunnels used for transportation? Possible Category Axes Seen Vehicle type Losses/Cost location other auto train USER PROFILE; TASK CONTEXT QUESTION NL PROCESSING Clarification Dialogue: S: Are you interested in train accidents, automobile accidents or others? U: Any that involved lost life or a major disruption in communication. Must identify loses. Semantics: What the question “means”: to the system to the user SEMANTIC PROC FUSE & SUMMARIZE Answer & Justification ANSWER GENER. SEARCH & CATEGORIZE KB TEMPLATE SELECTION Focused Information Need QUALITY ASSESSMENT

Key Research Issues Question Semantics –how the system “understands” user requests Human-Computer Dialogue –how the user and the system negotiate this understanding Information Quality Metrics –how some information is better than other Information Fusion –how to assemble the answer that fits user needs.

Document Retrieval Document Retrieval Build Frames Build Frames Process Frames Process Frames Dialogue Manager Dialogue Manager Question Processor Question Processor Wordnet Completed Work question Segment/ Filter Segment/ Filter Cluster Segments Cluster Segments Query Refinement Query Refinement Current Focus DB Gate Answer Generator Answer Generator answer Visualization

Data-Driven NL Semantics  What does the question mean to the user? –The speech act –The focus –User’s task, intention, goal –User’s background knowledge  What does the question mean to the system? –Available information –Information that can be retrieved –The dimensions of the retrieved information

Answer Space Topology KERNEL QUESTION MATCH KERNEL QUESTION MATCH NEAR MISSES, ALTERNATIVE INTERPRETATIONS ALL RETRIEVED FRAMES

Quality Judgments Focus Group: –Sessions conducted: March-April, 2002 –Results: Nine quality aspects generated Expert Sessions: –Sessions Conducted: May-June, 2002 –Results: 100 documents scored twice along 9 quality aspects Student Sessions: –Training and Testing Sessions: June-July, documents judged by experts used for training/testing –Actual Judgment Sessions: June-August, 2002 Qualified students evaluated 10 documents per session –Results: 900 documents scored twice along 9 quality aspects

Factor Analysis of 9 Quality Features Appearance Content

Modeling Quality of Text Kitchen sink approach –160 “independent” variables –Part-of-speech, vocabulary –stylistics, named entities, … Statistical pruning –Statistically significant variables –May be nonsensical to human Human pruning –Only “sensible” variables retained for each quality Pruning improves performance –Kitchen sink overfits –Statistics and Human close in performance –More work needed to understand the relationship

Quality Prediction by Linear Combination of Textual Features (from 5 to 17 variables). Split Half for Training and Testing. Quality FactorsPrediction Rate Depth67% Author Credential55% Accuracy69% Source57% Objectivity64% Grammar79% One Side vs Multi View70% Verbosity63% Readability76% Performance of models

In Summary The two conceptual foundations of librarianship: cataloging and reference, translate to two important problems in managing streams of textual messages: Both involve pattern recognition or machine learning.

Two Roles for Learning Cataloging: learning which features of a message mean that it is significant to the problem at hand Reference: learning which features of a message mean that it is “salient” to a specific user of the system.

Appendix: The following slides were not presented at the conference.

Communicating Credibility A system that is correct 75% or 80% of the time will be wrong one time in every four or 5. Unless it can “shade” its judgments or recommendations, the analyst will lose confidence in it. Credibility must be high enough to avoid extensive rework.

Data Fusion Use multiple methods to assess the relevance of documents or passages, –For a given question, dialogue, or cluster –Each method assigns a “score” Candidates → points in a “score space” Seek patterns to localize the most relevant documents or passages in this “score space” Developed interactive data analysis tool

Background on Fusion Problem There are systems S, T, U, … There are problems to be solved P,Q,R… This defines several fusion problems Local fusion: for a given problem P, and a pair of systems S,T, what is the best fusion rule: Let s(d),t(d) be the scores assigned to document d by systems S and T. Fusion tries to find the “best” combining function f(s,t)

Non-linear “iso-relevance”

Local Fusion Rule A local fusion rule f P (s,t) depends on the specific problem P. –This is relevant if P represents a static problem or profile, which will be considered on many occasions A global fusion rule f(s,t) does not depend on a specific problem P, –and can be safely used on a variety of problems.

Completely rigorous For each topic: 1) Randomly split the documents into two parts: training and testing 2) Do the logistic regression on training part and get the fusion scores for both training and testing documents 3) Calculate p_100 on testing documents. 4) Excellent results (one random sample for each) 5) Test SMART and InQuery on the same random testing set Local Fusion Results are Good

Summary of Local Fusion PROBLEM CASE We ran 5 split half runs on the odd case (318) and the results persist.

Is Local Sensible? Local fusion depends on getting information about a particular topic, and doing the best possible fusion. Not available in an AdHoc (e.g. Google) setting Potentially available in an intelligence applications - -filtering; standing profile

Our Approach to Retrieval Fusion SMART InQuery FUSION PROCESS Request DOCUMENTS SETS Result Set Delivered SET Result Set ADOPT: Fusion System Monitor Fusion Set and Receive Feedback USE: Better System Adaptive “Local” Fusion