A Potpurri of topics Paul Kantor Project overview and cartoon How we did at TREC this year Generalized Performance Plots Remarks on the formal model of decision © Paul Kantor 2002

1. Accumulated documents Rutgers DIMACS: Automatic Event Finding in Streams of Messages Retrospective/Supervised/Tracking 4. Guided Retrieval 7. Track New documents 1. Accumulated documents 2. Unexpected event 3. Initial Profile 6. Revision and Iteration 5.Clustering Analysts Prospective/Unsupervised/Detection 5.. Guided Retrieval 1. Accumulated documents Automatic Event Finding in Streams of Messages has two phases. The first (Year 1) is retrospective -- think of it as 20-20 hindsight. Documents are accumulated (1) until an unexpected event, represented here by a train rushing at us from a tunnel (2) occurs. Analysts build an initial profile (3) , which is used to retrieve likely documents (4) which are clustered (5) to provide a rich document set. Analysts work with this set, to revise and iterate (6), supporting efforts to track down all participants and supporters of the event. In the second and third years of the project attention will focus on prospective detection of significant events. As documents are accumulated (1) continuous clustering and matching (2) identifies documents that do not fit into established patterns. These are grouped and automatically profiled (3) and the results are submitted to analysts. The result, in some cases, will be an anticipated event (4) which, if the analysis is timely, will be prevented. Guided retrieval can then be used, as before, to track down all participants and supporters of the intended event. Technically, the work will be a thorough and systematic exploration of various representations of documents; matching methods; learning methods, and fusion methods, to establish the limits of these technologies. Theoretical work will establish rates of convergence and probabilities of success. Experimental work will test methods using the established TREC collections, and other materials, as appropriate. The work will be done by 7 faculty, and 9 students, post-docs and programmers. 4. Anticipated event 3. Initial Profile 2.Clustering © Paul Kantor 2002 © Paul Kantor 2001

Rutgers-DIMACS KD-D MMS Project Matrix © Paul Kantor 2002

Communication The process converges…. Central limit theorem … What??? Pretty good fit Confidence levels And so on © Paul Kantor 2002

Measures of performance Effectiveness 1. Batch Post-hoc learning. Here there is a large set of already discovered documents, and the system must learn to recognize future instances form the same family 2. Adaptive learning of defined profiles. Here there is a small group of "seed documents" and thereafter the system must learn while it works. Realistic measures must penalize the system for sending document that are of no interest to any analyst, to the human experts. 3. Discovery of new regions of interest. Here the focus is on unexpected patterns of related documents, which are far enough from established patterns to warrant sending them for human evaluation. © Paul Kantor 2002

Measures of performance Effectiveness Efficiency is measured in both time and space resources required to accomplish a given level of effectiveness. Results are best visualized in a set o two or three dimensional plots, as suggested on the following page. © Paul Kantor 2002

Typical measures used Adaptive Filtering Basis: F-measures: precision p=g/n g=number relevant and n = number that the analyst must examine recall R=g/G. G=total number that should be sent to the analyst. F-measures: 1/F = a/p+ (1-a)/R =(1/g)(an+(1-a)G) so F=g/[an+(1-a)G] there is no persuasive argument for using this in TREC2002 a=0.8. A 4:1 weighting © Paul Kantor 2002

Typical measures used Utility-Based measures Pure Measure: U=vg -c(n-g) =-c*n+g*(v-c) Note that sending irrelevant documents drives the score negative. v=2; c=1 “Training Wheels”: To protect groups from having to report astronomically negative results: U →T11SU = [max{U/2G, -0.5} - 0.5]/[1.5] © Paul Kantor 2002

How we have done: TREC2002 Disclaimers and caveats. We report here only on those results that were achieved and validated at the TREC2002 conference. These were done primarily to convince ourselves that we can manage the entire data pipeline, and were not selected to represent the best conceptual approaches that we can think of. © Paul Kantor 2002

Disclaimers and caveats (cont). The TREC Adaptive rules are quite confusing to newcomers. It appears, in conference and post-conference discussions that the two top-ranked systems may not have followed the same set of rules as the other competitors. If this is the case, our results will actually be better than those reported here. © Paul Kantor 2002

Using measure T11SU Adaptive -- Assessor topics - 9th among all teams - 7th among those known to follow rules. Intersection topics - 7th among all teams -- 5th among known to follow the rules. Batch. 6th among all groups on Assessor topics; 3rd among all groups on Intersection topics. Scored above median on almost all. Tops on 24 of 50 © Paul Kantor 2002

Efficiency- Effectiveness Plots Strong and slow Strong and fast 100% Measure of Effectiveness Not good enough for government work Weak but fast 100% Measure of Time Required (Best Baseline method/Method_plotted) © Paul Kantor 2002

BinWorlds --Very simple models Documents live in some number (L) of bins. Some bins have only (b) irrelevant (bad) documents, a few have relevant (good) documents. Documents are delivered randomly from the world, labeled only by their bin numbers. The work has a horizon H, with a payoff v for good documents sent to be judged, and a cost c for bad documents sent to be judged. We consider a hierarchy of models. For example, if only one bin contains good documents, the optimum strategy is either QUIT or continue until seeing one good document, and thereafter only submit documents from this bin to be judged. The expected value of the game is given by: EV=-CostToLearnRightBin+GainThereafter. Since the expected time to learn the right bin is 1+Lb/g EV=-c(1+Lb/g)+(H-(1+Lb/g))(vg-cb)/(b+g). Increasing Horizon H increases EV, while increasing the number of candidate bins, L, makes the game harder. However, if we have failed once on a bin, perhaps it is not wise to test it again. In other models, g,L become parameters to determine, several adjacent bins contain good documents, and the number of good documents varies smoothly. © Paul Kantor 2002

The essential math At any step on the way to the horizon H the decision maker can know only these things: The judgments on submitted documents, and the stage at which they were submitted. Let j(b,i) be the judgment received when a document from bin b was submitted at time step i. As a result of these judgments, the decision maker has a present Bayesian estimate of the chance that each bin is the right bin © Paul Kantor 2002

The challenge Can we find a simple and effective heuristic based on the available history j …….j and the time remaining:H-i . Such an heuristic must exist because the decision rule must be of the form: if the current estimate that a bin is the right one is below some critical value, don’t submit it. Note: this is “obvious but not yet proved.” © Paul Kantor 2002