1. InfoMagnets : Making Sense of Corpus Data Jaime Arguello Language Technologies Institute

2. Outline InfoMagnets Applications Topic Segmentation Conclusions Q/A

3. Outline InfoMagnets Applications Topic Segmentation Conclusions Q/A

4. Defining Exploratory Corpus Analysis Getting a “sense” of your data How does it relate to: –Information retrieval Need to understand the whole corpus –Data mining Need rich interface to support serendipitous search –Text classification Need to find the “interesting” classes

5. InfoMagnets

6. InfoMagnets Applications Behavioral Research –2 Publishable results (submitted to CHI) CycleTalk Project, LTI –New findings on mechanisms at work in guided exploratory learning Robert Kraut’s Netscan Group, HCII Conversational Interfaces Corpus organization makes authoring conversational agents less intimidating. Rose, Pai, & Arguello (2005); Gweon et al., (2005)

7. InfoMagnets Applications Behavioral Research –2 Publishable results (submitted to CHI) CycleTalk Project, LTI –New findings on mechanisms at work in guided exploratory learning Robert Kraut’s Netscan Group, HCII Conversational Interfaces Corpus organization makes authoring conversational agents less intimidating. Rose, Pai, & Arguello (2005); Gweon et al., (2005)

8. Authoring Conversational Interfaces Goal: Make Authoring CI’s easier Solution: –Guide development with pre-processed sample human-human conversations Addresses different issues –Accessible to non-computational linguists –Developers ≠ domain experts –Consistent with user-centered design: “The user is not like me!”

9. Authoring Conversational Interfaces Topic Segmentation Transcribed human-human conversations A C B CA B Constructing a Master Template

10. Topic Segmentation Preprocess for InfoMagnets But, an important computational linguistics problem in its own right! Previous Work –Marti Hearst’s TextTiling (1994) –Beeferman, Berger, and Lafferty (1997) –Barzilay and Lee (2004) NAACL best paper award! – ….. But, should it all fall under “topic segmentation”?

11. Topic Segmentation of Dialogue Dialogue is Different: –Very little training data –Linguistic Phenomena Ellipsis Telegraphic Content –Coherence is organized around a shared task, not primarily around a single flow of information

12. Lots of places where there is no overlap in “meaningful” content Coherence Defined Over Shared Task

13. Multiple topic shifts in regions w/ zero lexical cohesion

14. Experimental Condition 22 student-tutor pairs Conversation captured through mainstream chat client Thermodynamics domain Training and test data coded by one coder Results shown in terms of p_k (Lafferty & Beeferman, 1999) Significant tests: 2-tailed, t-tests

15. 1 st Attempt: TextTiling TextTiling (Hearst, 1997) –Slide two adjacent “windows” down the text –At each state calculate cosine correlation –Use correlation values to calculate “depth” –“Depth” values higher than a threshold correspond to topic shifts w1 w2

16. TextTiling Results Algorithm(avg) P k NONE0.489703303 ALL0.518040038 EVEN0.518040038 TextTiling0.535304902 TT T-test p-value TT (NONE)0.160036665 TT (ALL)0.089668491 TT (EVEN)0.221969444 Trend for TextTiling to perform worse than degenerate baselines Difference not statistically significant Why doesn’t it work?

17. Lots of gaps where the correlation = 0 Must select boundary heuristically And, still a heuristical improvement on original TextTiling Results

18. But, topic shifts tend NOT to occur where corr > 0. TextTiling Results

19. Cluster utterances Treat each cluster as a “state” Construct HMM –Emission probabilities: state-specific language models –Transition probabilities: based on location and cluster-membership of the utterances Viterbi re-estimation until convergence 2 nd Attempt: Barzilay and Lee (2005)

20. B&L statistically better than TT, but not better than degenerate algorithms B&L Results Algorithm(avg) P k NONE0.489703303 ALL0.518040038 EVEN0.518040038 TextTiling0.535304902 B&L 0.5086039 B&L T-test p-value B&L (NONE)0.586165431 B&L (ALL)0.471531964 B&L (EVEN)0.862335954 B&L (TextTiling)0.038565997

21. B&L Results Too fine grained topic boundaries Most clusters based on “fixed expressions” (e.g. “ok”, “yeah”, “sure” ) Remember: cohesion based on shared task Are state-based language models sufficiently different?

22. Incorporating Dialogue Dynamics Dialogue Act coding scheme Not originally developed for segmentation, but for discourse analysis of human-tutor dialogues 4 main dimensions: –Action: open question, closed question, negation, etc. –Depth: (yes/no) is utterance accompanied with explanation or elaboration –Focus: (binary) is focus on speaker or other agent –Control: Initiation, Response, Feedback Dialogue Exchange (Sinclair and Coulthart, 1975)

23. (Donmez, 2004) Use estimated labels on some dimensions to learn other dimensions 3 types of Features: –Text (discourse cues) –Lexical coherence (binary) –Dialogue Acts labels 10-fold cross-validation Topic Boundaries learned on estimated labels, not hand coded ones! 3 rd Attempt: Cross-Dimensional Learning

24. X-Dimensional Learning Results X-DIM statistically better than TT and degenerate algorithms! Algorithm(avg) P k NONE0.489703303 ALL0.518040038 EVEN0.511727103 TextTiling0.535304902 B&L0.5086039 X-DIM0.45199219 X-DIM T-test p-value X-DIM (NONE)0.024460654 X-DIM (ALL)0.023027116 X-DIM (EVEN)0.022848002 X-DIM (TextTiling)0.011833934

25. Statistically Significant Improvement TTB&LX-DIM NONE NON-SIG SIG ALL NON-SIG SIG EVEN NON-SIG SIG TT SIG B&L SIG

26. Future Directions Merge cross-dimensional learning (w/ dialogue act features) with B&L content modeling HMM approach. Explore other work in topic segmentation of dialogue

27. Recap InfoMagnets and applications Corpus exploration and authoring of CI’s Challenges of topic segmentation of dialogue Description of TextTiling, Barzilay & Lee, X-DIM vs. degenerate methods and each other

28. Q/A Thank you!