1. ADGEN USC/ISI ADGEN: Advanced Generation for Question Answering Kevin Knight and Daniel Marcu (co-PIs) USC/Information Sciences Institute December 4-6

2. ADGEN USC/ISI Of the myriad variations of a text that the machine might produce for an analyst, only a fraction are coherent. What makes a text coherent? New Approach: –We have millions of examples of coherent texts –We can validate ideas empirically, develop models –We can train models automatically ADGEN Research Focus

3. ADGEN USC/ISI ADGEN in AQUAINT 1. Answer generation –Input: collection of text fragments (including phrases and paragraphs) –Fuse phrases into sentences, order sentences to form millions of possible texts –Rank and select most coherent presentation 2. Text improvement –Input: existing text –Apply probabilistic rewriting operations –Select rewrite that most improves coherence without sacrificing any of the basic material

4. ADGEN USC/ISI Text-Level Language Models Given an unordered bag of answers/clauses/sentences, assign an order that yields a coherent text. Typical discourse study: “if we scramble sentences in an English document, the result is not coherent, so text has structure…” Let’s do something about it!

5. ADGEN USC/ISI Models have multiple applications Word-level ordering Text-level ordering Machine Translation Meaning-to-Text Generation Multi-document Summarization Essay Grading ?

6. ADGEN USC/ISI Article Ordering Experiments Test set of 51 scrambled Wall Street articles - re-ordered by human 50% perfectly match original 90% only one off Initial modeling ideas - a good article usually contains reasonable adjacent sentence pairs - adjacent sentences A and B are often related in interesting ways - if A includes sell, B often includes sale - asymmetric -- reverse is not true nearly as often

7. ADGEN USC/ISI Article Ordering Experiments Initial generative story for P(B | A) - Given sentence A = a1 … an: decide on length m for B (in # of words) for i = 1 to m pick a “source word” position j from A pick a “target word” bi for B, according to P(bi | aj) print out sentence B = b1 … bm The sale includes the rights to Germaine Monteil in North and South America and in the Far East, as well as the worldwide rights to the Diane Von Furstenberg cosmetics and fragrance lines and U.S. distribution rights to Lancaster beauty products. A B

8. ADGEN USC/ISI Article Ordering Experiments Initial generative story for P(B | A) - Given sentence A = a1 … an: decide on length m for B (in # of words) for i = 1 to m pick a “source word” position j from A pick a “target word” bi for B, according to P(bi | aj) print out sentence B = b1 … bm The sale includes the rights to Germaine Monteil in North and South America and in the Far East, as well as the worldwide rights to the Diane Von Furstenberg cosmetics and fragrance lines and U.S. distribution rights to Lancaster beauty products. A B

9. ADGEN USC/ISI Article Ordering Experiments Initial generative story for P(B | A) - Given sentence A = a1 … an: decide on length m for B (in # of words) for i = 1 to m pick a “source word” position j from A pick a “target word” bi for B, according to P(bi | aj) print out sentence B = b1 … bm The sale includes the rights to Germaine Monteil in North and South America and in the Far East, as well as the worldwide rights to the Diane Von Furstenberg cosmetics and fragrance lines and U.S. distribution rights to Lancaster beauty products. Terms A B

10. ADGEN USC/ISI Article Ordering Experiments Initial generative story for P(B | A) - Given sentence A = a1 … an: decide on length m for B (in # of words) for i = 1 to m pick a “source word” position j from A pick a “target word” bi for B, according to P(bi | aj) print out sentence B = b1 … bm The sale includes the rights to Germaine Monteil in North and South America and in the Far East, as well as the worldwide rights to the Diane Von Furstenberg cosmetics and fragrance lines and U.S. distribution rights to Lancaster beauty products. Terms weren't disclosed, but industry sources said the price was about $2.5 million. A B

11. ADGEN USC/ISI Article Ordering Experiments Train P(bi | aj) on large, “adjacent-sentence” database from 1987 WSJ - using standard machine-translation package GIZA developed at JHU WS’99 Portion of learned words predicted by judge: Deployment on 51 test articles - for each article, compute P(B | A) costs for every sentence pair in an article - select sentence order that minimizes total cost, using standard TSP algorithm 0.0310724 his 0.0240846 to 0.0204204 " 0.0198243 case 0.0148957 law 0.2191190 judge 0.0674230 court 0.0601497 he 0.0385838 the 0.0329773 that 0.0138114 a 0.0115714 of 0.0112826 ruling 0.0109133 in 0.0108175 against 0.0091600 lawyers 0.0087317 by 0.0083675 appeals 0.0082462 said S1 S2 S3 text boundary S4 Best tour: S4 S3 S1 S2

12. ADGEN USC/ISI Article Ordering Results accuracy (exact match with original article order) Human ordering 47% Random baseline ordering 16% Adjacent sentence model 31% + first sentence detector 41% + squashing of P(B | A)’s 47%

13. ADGEN USC/ISI Disassembling Bags of Sentences Another evaluation: - Given mixed bag of sentences drawn from two articles X1 X4 Y3 X2 Y1 Y2 X3 - Assign an order to them Y2 Y3 Y1 X1 X2 X3 X4 - Does the assigned order separate the two stories? Answer: 86% correct on 49 article pairs.

14. ADGEN USC/ISI Article Ordering Results 51 article test, 4.7 sents/article) 1031 article test, 7.6 sents/article) Human ordering 47% Random baseline ordering 16% 2.8% Adjacent sentence model 31% 7.6% + first sentence detector 41% 9.4% + squashing of P(B | A)’s 47% 9.7%

15. ADGEN USC/ISI Building from the Baseline: What’s Wrong With Machine Articles? Plenty. Is it possible to distinguish original human articles from machine-ordered articles? (If not, we are done!) Classification task: Believed to be human articles Believed to be machine articles 80+%

16. ADGEN USC/ISI What’s Wrong With Machine Articles? Plenty. Is it possible to distinguish original human articles from machine-ordered articles? (If not, we are done!) Classification task: Believed to be human articles Believed to be machine articles 61%

17. ADGEN USC/ISI Exploiting What’s in the Classifier Classifier can’t actually order new articles for us, because it only looks for certain things. But, its knowledge can be put back into the ordering algorithm - large-scale feature-based training - interesting technical challenges Idea: next iteration, the classifier is back to 50% and we start looking for new features. People are good at writing classifiers.

18. ADGEN USC/ISI Why is this important for QA? Outputs in the QA-definitions task: Today: Who is Gunter Blobel? A1: molecular biologist A2: cellular and molecular biologist A3: won the 1999 Nobel Prize in medicine What we want: Dr. Gunter Blobel is a cellular and molecular biologist who won the 1999 Nobel Prize in medicine.

19. ADGEN USC/ISI Why is this important for QA? Outputs of causal questions: Today: sets of sentences. What we want: coherent text. In order to move beyond lists of sentences and sentence fragments, we need general purpose NLG algorithms.

20. ADGEN USC/ISI Next Steps Deal with redundancy Terms weren't disclosed, but industry sources said the price was about $2.5 million. The sale includes the rights to Germaine Monteil in North and South America. Terms were not disclosed by either party. Deal with contradiction Terms weren't disclosed, but industry sources said the price was about $2.5 million. Revlon said it paid $2.2 million for Germaine Monteil. Close the loop on generating/exploiting more discourse features Create useful generation capability

21. ADGEN USC/ISI Thank You

22. ADGEN USC/ISI Sample Problem 1. Terms weren't disclosed, but industry sources said the price was about $2.5 million. 2. Revlon is a cosmetics concern, and Beecham is a pharmaceutical concern. 3. Revlon Group Inc. said it completed the acquisition of the U.S. cosmetics business of Germaine Monteil Cosmetiques Corp., a unit of London-based Beecham Group PLC. 4. The sale includes the rights to Germaine Monteil in North and South America and in the Far East, as well as the worldwide rights to the Diane Von Furstenberg cosmetics and fragrance lines and U.S. distribution rights to Lancaster beauty products.

23. ADGEN USC/ISI 1. Terms weren't disclosed, but industry sources said the price was about $2.5 million. 2. Revlon is a cosmetics concern, and Beecham is a pharmaceutical concern. 3. Revlon Group Inc. said it completed the acquisition of the U.S. cosmetics business of Germaine Monteil Cosmetiques Corp., a unit of London-based Beecham Group PLC. 4. The sale includes the rights to Germaine Monteil in North and South America and in the Far East, as well as the worldwide rights to the Diane Von Furstenberg cosmetics and fragrance lines and U.S. distribution rights to Lancaster beauty products. Sample Problem Correct order: 3, 1, 4, 2