1. December 2, 2004TDT-2004 Adaptive Topic Tracking at Maryland Tamer Elsayed, Douglas W. Oard, David Doermann University of Maryland, College Park Gary Kuhn National Security Agency

2. Outline Results System design Interpreting the results Next steps

3. Non-Adaptive Topic Tracking No score normalization Cost=0.6507 Bottom left is better

4. Adaptive Topic Tracking No score normalization, unjudged treated as firmly off-topic Cost=0.2438

5. Adaptive Topic Tracking Cost=0.3789 One-pass score normalization, unjudged treated as firmly off-topic

6. Non-Adaptive System Design TDT-5 Evaluation EpochTraining Epoch Compute log-odds ngram weights Compute story scores

7. Log-Odds Term Weights

8. Computing Story Scores

9. Non-Adaptive System Design TDT-5 Evaluation EpochTraining Epoch Compute log-odds ngram weights Compute story scores

10. Adaptive System Design TDT-4TDT-5 Evaluation EpochTraining Epoch Compute log-odds ngram weights Compute story scores Extended Training Epoch Normalize Story scores Compute Normalization factor

11. Interpreting Non-Adaptive Results Lack of normalization probably hurt! What can we say about the effect of incomplete judgments?

12. Interpreting Adaptive Results Normalization hurt! –One-pass design is the problem DET has limitations –Changing the threshold changes our topic model! –Threshold selection is now a critical path item How does judgment density affect the results? Not normalized Normalized

13. Next Steps Further explore normalization –Implement continuous renormalization –Tune parameters on devtest data Decide between TDT-5 and TDT-4 –Is incomplete judging harmful? Define richer training sets –Explicit queries –Many known on-topic/off-topic training stories –Models of (imperfect) behavioral feedback

14. Our Favorite Quote of the Day “It takes time to get the implementation correct” [Yiming] We had 30 days from project initiation to non-adaptive submission