Presentation is loading. Please wait.

Presentation is loading. Please wait.

Capturing User Intent for Analytical Process Eugene Santos Jr. John Wilkinson Fei Yu Deqing Li Keum Kim Dartmouth College Hien Nguyen Jacob Russell Adam.

Similar presentations


Presentation on theme: "Capturing User Intent for Analytical Process Eugene Santos Jr. John Wilkinson Fei Yu Deqing Li Keum Kim Dartmouth College Hien Nguyen Jacob Russell Adam."— Presentation transcript:

1 Capturing User Intent for Analytical Process Eugene Santos Jr. John Wilkinson Fei Yu Deqing Li Keum Kim Dartmouth College Hien Nguyen Jacob Russell Adam Olson University of Wisconsin - Whitewater Presentation at AITP meeting, February 2 nd 2010.

2 Outline Problem Challenges Our User Intent Model Preliminary evaluation Future work Presentation at AITP meeting, February 2 nd 2010.

3 Outline Problem Challenges Our User Intent Model Preliminary evaluation Future work Presentation at AITP meeting, February 2 nd 2010..

4 Problem Presentation at AITP meeting, February 2 nd 2010. APEXB, Senior Analyst Iran’s desk Expertise: Iran, Iraq, Middle East Nuclear weapon programs APEXF, Junior Analyst Iran’s desk Expertise: assessment of Iranian clerical support. APEXL, Advisory Analyst Iran’s desk Expertise: Iraq, iranian political policies

5 Problem Presentation at AITP meeting, February 2 nd 2010. APEXB, Senior Analyst Does not like to talk to anyone APEXF, Junior Analyst Face book, Second life APEXL, Advisory Analyst Rushes through everything

6 Problem Presentation at AITP meeting, February 2 nd 2010. Modeling an analyst’s intent to improve the effectiveness of collaboration among intelligence analysts Capture the process instead of topical interests May help improve the diversity of collaboration

7 Research Challenges Partiality and incompleteness of captured knowledge Dynamic information spaces, user foci, and preferences Uncertainty in analyst’s actions Presentation at AITP meeting, February 2 nd 2010.

8 Research Questions 1.What is user intent and how do we capture it 2.How do we evaluate the effectiveness of the intent model Presentation at AITP meeting, February 2 nd 2010.

9 Outline Problem Challenges Our User Intent Model Preliminary evaluation Future work Presentation at AITP meeting, February 2 nd 2010.

10 Our Intent Model +Analyst intent = Goals + Actions + Commitment Our user model consists of 3 components that are designed to capture intent: –Foci: What is the working space of the analyst and what he is concentrating on? (Goal & Commitment) –Rationale: Why does the analyst have these foci? (Goals) –Action: How are the analyst’s goals accomplished? (Actions) Presentation at AITP meeting, February 2 nd 2010.

11 Our Intent Model Presentation at AITP meeting, February 2 nd 2010. APEXF Search Retain Event database Foci Rationale Action

12 Example - Rationale Presentation at AITP meeting, February 2 nd 2010. Search: Where does the Iranian clerical community stand on Ayatollah Khamenei and President Ahmadinejad's policies with regards to Iran's civilian and military nuclear program? APEXF 12/11/2007 at 11:00 …

13 Example - Foci Presentation at AITP meeting, February 2 nd 2010. Search: Where does the Iranian clerical community stand on Ayatollah Khamenei and President Ahmadinejad's policies with regards to Iran's civilian and military nuclear program? APEXF 12/11/2007 at 11:00 G0G0 iranian_cleric_community nuclear_program ayatollah_khamenei … Commitment level: 1.0

14 Example - Action Presentation at AITP meeting, February 2 nd 2010. Search: Where does the Iranian clerical community stand on Ayatollah Khamenei and President Ahmadinejad's policies with regards to Iran's civilian and military nuclear program? APEXF 12/11/2007 at 11:00 Bayesian network HMM 1 1 2 2 3 3.88.91.82.06.01.06.03.17 Examining Evidence Examining Evidence Searching for Evidence Searching for Evidence Going through Documents Going through Documents

15 Example Presentation at AITP meeting, February 2 nd 2010. Search: Is iranian clerical community supporting Iran's nuclear program? 12/11/2007 at 11:10 Search: Are there clerics who oppose Iran's nuclear program? APEXF 12/11/2007 at 11:05

16 Example - Rationale Presentation at AITP meeting, February 2 nd 2010. APEXF ….

17 Example - Foci Presentation at AITP meeting, February 2 nd 2010. APEXF G0G0 iranian_cleric_community nuclear_program ayatollah_khamenei … Commitment level: 0.33 G1G1 cleric Iran_nuclear_program Iran … Commitment level: 0.5 G2G2 cleric Iran_nuclear_program Iran … Commitment level: 0.66 G3G3 …

18 Example - Action Presentation at AITP meeting, February 2 nd 2010. APEXF 1 1 2 2 3 3.91.82.06.01.06.03.17 Examining Evidence Examining Evidence Searching for Evidence Searching for Evidence Going through Documents Going through Documents HMM Bayesian networks.88

19 Example - Intent Goal: –Iranian clerics and nuclear weapons program Actions: –Searches Commitment: –F(0.33, 0.5, 0.66) Presentation at AITP meeting, February 2 nd 2010. APEXF

20 Outline Problem Motivation Our User Intent Model Preliminary evaluation Future work Presentation at AITP meeting, February 2 nd 2010..

21 Evaluation Do we capture user intent more precisely compared to a simple interest list approach? Do we capture a user’s intent earlier in the analytic process compared to the interest- based approach? Presentation at AITP meeting, February 2 nd 2010.

22 Evaluation Testbed: APEX 07, 8 intelligence analysts Tasks: Where does the Iranian clerical community stand on Ayatollah Khamenei ? President Ahmadinejad's policies with regards to Iran’s civilian and military nuclear program? Database: contain 5613 events, 8 types of actions. Presentation at AITP meeting, February 2 nd 2010.

23 Evaluation Presentation at AITP meeting, February 2 nd 2010.

24 Do we capture user intent more precisely? Analysts: APEXL- APEXC, APEXE-APEXH APEXL- APEXK, APEXF-APEXB. Procedure: Create a user model for each user using 25 consecutive events on December 11, 2007. Precision = Cgoals: a goal node that is found in both intent models. Presentation at AITP meeting, February 2 nd 2010.

25 Do we capture user intent more precisely? Presentation at AITP meeting, February 2 nd 2010. P-value=0.0396<0.05

26 Whether we capture a user’s intent earlier? Presentation at AITP meeting, February 2 nd 2010.. Analysts: APEXF – APEXB –APEXB: nuclear weapon program and Iran. –APEXF: grand Ayatollah Intent (t=5) Interest (t=8)

27 Future work Presentation at AITP meeting, February 2 nd 2010. Research and implementation: –Generation of goal descriptions –Integrating between content-based and action-based information –Link between common intent and collaboration effectiveness Evaluation: –Prediction of action sequences –Define measures of collaboration effectiveness

28 Acknowledgement Presentation at AITP meeting, February 2 nd 2010. This research is funded by CASE program (IARPA) grant #FA8750-07-2-0219 DOD- AFRL (2007-2008). Thanks to the CASE technical advisory team for making the data available, giving constructive feedback and supporting this research.

29 Related work Presentation at AITP meeting, February 2 nd 2010. Eugene Santos Jr. and Hien Nguyen. “Modeling Users for Adaptive Information Retrieval by Capturing User Intent”. Book Chapter. In M. Chevalier, C. Julien, C. Souledupuy (Eds.). Collaborative and Social Information Retrieval and Access: Techniques for Improved User Modeling. 2009. Hien Nguyen and Eugene Santos Jr. (a) “An Evaluation of the Accuracy of Capturing User Intent for Information Retrieval”. In Proceedings of the 2007 International Conference on Artificial Intelligence. Las Vegas, NV. June 2007. Hien Nguyen and Eugene Santos Jr. (b) “Effects of prior knowledge on the effectiveness of a hybrid user model for information retrieval”. In Proceedings of the Homeland Security and Homeland Defense VI conference. Orlando, FL. March 2007. Vol. 6538. Hien Nguyen, Eugene Santos Jr., Aaron Schuet, and Nathan Smith. “Hybrid User Model for Information Retrieval”. In Technical Report of Modeling Others from Observations workshop at AAAI-2006 conference. Hien Nguyen, Eugene Santos Jr., Qunhua Zhao and Hua Wang. “Capturing User Intent for Information Retrieval”. In Proceedings of the 48th Annual Meeting for the Human Factors and Ergonomics Society (HFES-04), New Orleans, LA. 2004. Pages 371-375

30 Past undergraduate research project Presentation at AITP meeting, February 2 nd 2010. - Hybrid Approach to Word Sense Disambiguation. Jacob Russell, Hien Nguyen. Poster presented at National Undergraduate Research Conference. Lacrosse, WI. (nominated by UWW Undergraduate Research Committee to be presented at CUR on the Hill). Implementation of intent-based user model for business domain. Adam Olson, Nathan Lutterman, LQ McDonald III, Hien Nguyen. Poster presented at National Undergraduate Research Conference. Lacrosse, WI. Web-based user model services for analytic process. 2008. Nathan Lutterman, Adam Olson, Hien Nguyen. Poster presented at National Undergraduate Research Conference. Maryland, MD. Simulations of network defects. 2007. Russell Jacob, Nathan Smith, Nicholas Orcust, Hien Nguyen, Jim Harris, Benette Harris. Poster presented at Undergraduate research day, April 4, 2007 and National Conference for Undergraduate research (NCUR 2007), April 12-14. CA. Evaluation of Hybrid User Model for Information Retrieval. 2006. Aaron Schuett, Nathan Smith, and Hien Nguyen. Poster presented at National Conference of Undergraduate Research 2006, Raleigh, NC.

31 Current undergraduate research project Presentation at AITP meeting, February 2 nd 2010. Roulette game – Application of Modeling under Uncertainty. Richard Lilja, Hien Nguyen Intent model for supporting medical trauma treatment. LQ McDonald, Adam Olson, Hien Nguyen


Download ppt "Capturing User Intent for Analytical Process Eugene Santos Jr. John Wilkinson Fei Yu Deqing Li Keum Kim Dartmouth College Hien Nguyen Jacob Russell Adam."

Similar presentations


Ads by Google