Presentation is loading. Please wait.

Presentation is loading. Please wait.

Extracting Simplified Statements for Factual Question Generation Michael Heilman and Noah A. Smith 1.

Similar presentations


Presentation on theme: "Extracting Simplified Statements for Factual Question Generation Michael Heilman and Noah A. Smith 1."— Presentation transcript:

1 Extracting Simplified Statements for Factual Question Generation Michael Heilman and Noah A. Smith 1

2 Automatic Factual Question Generation (QG) Input: text Output: questions for reading assessment (e.g., for a closed-book quiz) 2 We focus on sentence-level factual questions.

3 …Prime Minister Vladimir V. Putin, the country's paramount leader, cut short a trip to Siberia, returning to Moscow to oversee the federal response. Mr. Putin built his reputation in part on his success at suppressing terrorism, so the attacks could be considered a challenge to his stature…. The Problem In complex sentences, facts can be presented with varied and complex linguistic constructions. 3

4 …Prime Minister Vladimir V. Putin, the country's paramount leader, cut short a trip to Siberia, returning to Moscow to oversee the federal response. Mr. Putin built his reputation in part on his success at suppressing terrorism, so the attacks could be considered a challenge to his stature…. The Problem 4 main clause

5 In complex sentences, facts can be presented with varied and complex linguistic constructions. …Prime Minister Vladimir V. Putin, the country's paramount leader, cut short a trip to Siberia, returning to Moscow to oversee the federal response. Mr. Putin built his reputation in part on his success at suppressing terrorism, so the attacks could be considered a challenge to his stature…. The Problem 5 main clause appositive

6 In complex sentences, facts can be presented with varied and complex linguistic constructions. …Prime Minister Vladimir V. Putin, the country's paramount leader, cut short a trip to Siberia, returning to Moscow to oversee the federal response. Mr. Putin built his reputation in part on his success at suppressing terrorism, so the attacks could be considered a challenge to his stature…. The Problem 6 main clause appositive participial phrase

7 In complex sentences, facts can be presented with varied and complex linguistic constructions. …Prime Minister Vladimir V. Putin, the country's paramount leader, cut short a trip to Siberia, returning to Moscow to oversee the federal response. Mr. Putin built his reputation in part on his success at suppressing terrorism, so the attacks could be considered a challenge to his stature…. The Problem 7 main clause appositive participial phraseconjunction of clauses

8 In complex sentences, facts can be presented with varied and complex linguistic constructions. Prime Minister Vladimir V. Putin cut short a trip to Siberia. Prime Minister Vladimir V. Putin was the country's paramount leader. Prime Minister Vladimir V. Putin returned to Moscow to oversee the federal response. Mr. Putin built his reputation in part on his success at suppressing terrorism. The attacks could be considered a challenge to his stature. The Problem 8 Output:

9 The Rest of the Talk Input: complex sentence Output: set of simple declarative sentences Our method: Uses rules to extract and simplify sentences Is motivated by linguistic knowledge Outperformed a sentence compression baseline 9 Easier to convert into questions

10 Outline Introduction and motivation Our Approach Simplification and extraction operations Evaluation Conclusions 10

11 Alternative: Sentence Compression Input: Complex sentence Output: Simpler sentence that conveys the main point. Suitable for QG? Only one output per input Most methods only delete words 11 Knight & Marcu 2000; Dorr et al. 2003; McDonald 2006; Clarke 2008; Martins & Smith 2009; inter alia Knight & Marcu 2000; Dorr et al. 2003; McDonald 2006; Clarke 2008; Martins & Smith 2009; inter alia

12 Our Approach We extract and simplify multiple statements from complex sentences. We include operations for various syntactic constructions. – encoded with pattern matching rules for trees 12 Similar work: Klebanov et al. 2004

13 Example: Extracting from Appositives 13 Input: Putin, the Russian Prime Minister, visited Moscow. Desired Output: Putin was the Russian Prime Minister.

14 Example: Extracting from Appositives 14 NP Putin visited VBD NP ROOT S, VP,,, NP Siberia NP the Russian Prime Minister (mainverb)(appositive)(noun)

15 Example: Extracting from Appositives 15 NP < (NP=noun !$-- NP $+ (/,/ $++ NP|PP=appositive !$CC|CONJP)) >> (ROOT << /^VB.*/=mainverb) NP Putin visited VBD NP ROOT S, VP,,, NP Siberia NP the Russian Prime Minister (mainverb)(appositive)(noun)

16 Example: Extracting from Appositives 16 NP Putin visited VBDNP the Russian Prime Minister

17 Example: Extracting from Appositives 17 NP Putin was VBDNP the Russian Prime Minister Singular past tense form of be

18 Example: Extracting from Appositives 18 was VBDNP Putin NP the Russian Prime Minister S ROOT VP

19 Implementation Representation: phrase structure trees from the Stanford Parser Syntactic rules are written in the Tregex tree searching language – Tregex operators encode tree relations such as dominance, sisterhood, etc. 19 Klein & Manning 2003 Levy & Andrew 2006

20 Outline Introduction and motivation Our Approach Simplification and extraction operations Evaluation Conclusions 20

21 Encoding Linguistic Knowledge Given an input sentence A that is assumed true, we aim to extract sentences B that are also true. Our operations are informed by two phenomena: semantic entailment presupposition 21

22 Semantic Entailment A entails B: B is true whenever A is true. 22 Levinson 1983

23 A: However, Jefferson did not believe the Embargo Act, which restricted trade with Europe, would hurt the American economy. Simplification by Removing Modifiers 23 Entailment holds when removing certain types of modifiers.

24 A: However, Jefferson did not believe the Embargo Act, which restricted trade with Europe, would hurt the American economy. Simplification by Removing Modifiers 24 Entailment holds when removing certain types of modifiers. discourse marker non-restrictive relative clause

25 A: However, Jefferson did not believe the Embargo Act, which restricted trade with Europe, would hurt the American economy. Simplification by Removing Modifiers 25 B: Jefferson did not believe the Embargo Act would hurt the American economy. Entailment holds when removing certain types of modifiers. discourse marker non-restrictive relative clause

26 Extracting from Conjunctions 26 In most clausal and verbal conjunctions, the individual conjuncts are entailed. A: Mr. Putin built his reputation in part on his success at suppressing terrorism, so the attacks could be considered a challenge to his stature. B 2 : The attacks could be considered a challenge to his stature. B 1 : Mr. Putin built his reputation in part on his success at suppressing terrorism.

27 Extracting from Presuppositions In some constructions, B is true regardless of whether the main clause of sentence A is true. i.e., B is presupposed to be true. In some constructions, B is true regardless of whether the main clause of sentence A is true. i.e., B is presupposed to be true. 27 Levinson 1983 A: Hamilton did not like Jefferson, the third U.S. President. B: Jefferson was the third U.S. President. negation of main clause

28 Presupposition Triggers Many presuppositions have clear syntactic or lexical associations. 28 TriggerExample non-restrictive appositivesJefferson, the third U.S. President, … non-restrictive relative clauses Jefferson, who was the third U.S. President… participial modifiersJefferson, being the third U.S. President, … temporal subordinate clauses Before Jefferson was the third U.S. President, … Jefferson was the third U.S. President.

29 (Over)simplified Pseudocode Take as input a tree t. Extract a set of declarative sentence trees T extracted from constructions in t. For each t’ in T extracted : Simplify t’ by removing modifiers. Extract trees T conjuncts from conjunctions in t’. For each t conjunct in T conjuncts : T result = T result {t conjunct } Return T result 29 by entailment primarily by presupposition

30 Outline Introduction and motivation Our Approach Simplification and extraction operations Evaluation Conclusions 30

31 Baselines HedgeTrimmer – A rule-based sentence compression algorithm – Iteratively performs simplifying operations until the input is less than a specified length (15 here). “Main clause only” – Only the simplified main clause extracted by the full system. 31 Dorr et al Both baselines produce one output per input.

32 Research Questions 1.How long are the simplified outputs and how many are there? – Extracted statements from 25 previously unseen Encyclopedia Britannica articles about cities. 2.How well do the extracted statements cover the information in the input texts? – % of input words in at least one output. 32 Barzilay & Elhadad 2003

33 Results: Length & Coverage 33 Input Texts Hedge- Trimmer Main clause only Full Sentences per input Sentence length Word coverage (%)

34 Research Questions 3.How well does our system preserve fluency and correctness? – Two raters judged simplified outputs for fluency and correctness using 1-5 scales. – We averaged the raters’ scores. 34 Inter-rater agreement: r =.92 for fluency r =.82 for correctness Inter-rater agreement: r =.92 for fluency r =.82 for correctness

35 Results: Fluency & Correctness Input Texts Hedge- Trimmer Main clause only Full Fluency Correctness Rated 5 for both (%) Differences between HedgeTrimmer and Full are statistically significant (p <.05).

36 Outline Introduction and motivation Our Approach Simplification and extraction operations Evaluation Conclusions 36

37 Conclusions Method for extracting simplified declarative statements from complex sentences. Outperformed a text compression baseline. – More outputs and better coverage – Higher % of fluent and correct outputs Future work: evaluation of this as a component in a QG system. 37 Heilman & Smith 2010

38 Questions? 38 Demo & code release available on my website. Demo & code release available on my website.

39 A Whale of a Sentence “As they narrated to each other their unholy adventures, their tales of terror told in words of mirth; as their uncivilized laughter forked upwards out of them, like the flames from the furnace; as to and fro, in their front, the harpooneers wildly gesticulated with their huge pronged forks and dippers; as the wind howled on, and the sea leaped, and the ship groaned and dived, and yet steadfastly shot her red hell further and further into the blackness of the sea and the night, and scornfully champed the white bone in her mouth, and viciously spat round her on all sides; then the rushing Pequod, freighted with savages, and laden with fire, and burning a corpse, and plunging into that blackness of darkness, seemed the material counterpart of her monomaniac commander's soul.” 39 Melville 1851 Gold standard parse: 133 word sentence from Moby Dick:

40 A Whale of a Sentence 40 1.The rushing Pequod seemed the material counterpart of her monomaniac commander's soul. 2.They narrated to each other their unholy adventures. 3.Their uncivilized laughter forked upwards out of them. 4.The harpooneers wildly gesticulated with their huge pronged forks and dippers in their front. 5.The wind howled on. 6.The sea leaped. 7.The ship groaned. 8.The ship dived. 9.The ship steadfastly shot her red hell further and further into the blackness of the sea and the night. 10.The ship scornfully champed the white bone in her mouth. 11.The ship viciously spat round her on all sides. 12.The rushing Pequod was freighted with savages. 13.The rushing Pequod was laden with fire. 14.The rushing Pequod was burning a corpse. 15.The rushing Pequod was plunging into that blackness of darkness. 16.Their unholy adventures were their tales of terror told in words of mirth. System output:


Download ppt "Extracting Simplified Statements for Factual Question Generation Michael Heilman and Noah A. Smith 1."

Similar presentations


Ads by Google