1. Natural Language Generation with Tree Conditional Random Fields Wei Lu, Hwee Tou Ng, Wee Sun Lee Singapore-MIT Alliance National University of Singapore

2. Generating Natural Language (NL) paraphrases for Meaning Representations (MR) Natural Language Generation How many states do not have rivers ? 2 Natural Language Sentence Meaning Representation … … ………… … Lu, Ng & Lee 6 August 2009, EMNLP09

3. Meaning Representation (MR) 3 do nothavestatesrivershow many? QUERY:answer(NUM) NUM:count(STATE) STATE:exclude(STATE STATE) STATE:state(all)STATE:loc_1(RIVER) RIVER:river(all) Lu, Ng & Lee 6 August 2009, EMNLP09

4. Previous Work Chart Generation for Surface Realization –Head-Driven Phrase Structure Grammar (HPSG) (Carroll et al., 1999; Carroll and Oepen, 2005; Nakanishi et al., 2005) –Combinatory Categorial Grammar (CCG) (White and Baldridge, 2003; White, 2004). W ASP -1 ++ by Wong and Mooney (2007) –View the problem as a statistical machine translation task –Inversion of a semantic parser called W ASP, with incorporation of models borrowed from P HARAOH 4 Lu, Ng & Lee 6 August 2009, EMNLP09

5. Hybrid Tree Framework Aims to bridge natural language sentences and their underlying meaning representations On top of the framework, we built a generative model that jointly generates both natural language and MR tree Details presented in our EMNLP 2008 paper for semantic parsing –Wei Lu, Hwee Tou Ng, Wee Sun Lee, and Luke S. Zettlemoyer. 2008. A Generative Model for Parsing Natural Language to Meaning Representations. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP 2008), pages 783– 792. 5 Lu, Ng & Lee 6 August 2009, EMNLP09

6. Hybrid Tree do nothavestatesrivershow many? QUERY:answer(NUM) NUM:count(STATE) STATE:exclude(STATE STATE) STATE:state(all)STATE:loc_1(RIVER) RIVER:river(all) 6 NL-MR Pair Hybrid sequences Lu, Ng & Lee 6 August 2009, EMNLP09

7. The Joint Generative Model Assume the MR tree and NL sentence are jointly generated with a top-down recursive Markov process Able to handle re-ordering of nodes (MR productions) in MR tree during generation process Generation process results in a hybrid tree Shown effective in semantic parsing 7 Lu, Ng & Lee 6 August 2009, EMNLP09

8. NLG with Hybrid Trees The most probable NL w for a given MR m is: 1. Find the most probable hybrid tree T: T* = argmax T p(T|m) 2. The most probable NL sentence w is the yield of the hybrid tree T*: w* = yield(T*) Different assumptions can be made for finding the most probable hybrid tree T Two models: –Baseline: Direct Inversion Model –Tree Conditional Random Field Model 8 Lu, Ng & Lee 6 August 2009, EMNLP09

9. QUERY:answer(NUM) NUM:count(STATE) STATE:exclude(STATE STATE) STATE:state(all)STATE:loc_1(RIVER) RIVER:river(all) 9 do not states have rivers how many? Lu, Ng & Lee 6 August 2009, EMNLP09 Direct Inversion Model NUM STATE STATE 1 STATE 2 RIVER

10. Direct Inversion Model Direct inversion of the semantic parser –The distance d β (w 1,w 2 ) = -log θ(w 2 | β,w 1 ), which is non-negative –Find the most probable sequence from BEGIN to END –Problem equivalent to shortest path problem 10 manySTATE 1 β: NUM:count(STATE) howBEGINEND d β (BEGIN,how)d β (how,many)d β (many,STATE 1 )d β (STATE 1,END) Lu, Ng & Lee 6 August 2009, EMNLP09 thenumberof STATE 1 d β (BEGIN,the) d β (the,number) d β (number,of) d β (of,STATE 1 ) d β (STATE 1,END) ……

11. Direct Inversion Model Problems with the Direct Inversion Model –Strong independence assumptions –Always generates the same phrase below the same MR production, regardless of context MR productions –Modeling dependencies at word level only Need to model dependencies between adjacent hybrid sequences 11 Lu, Ng & Lee 6 August 2009, EMNLP09

12. Tree Conditional Random Fields (CRF) Model –Generate complete phrases instead of words –Explicitly model dependencies between adjacent phrases NLG with Hybrid Trees 12 NUM:count(STATE) BEGIN how many STATE 1 END STATE:exclude(STATE STATE) BEGIN STATE 1 do not STATE 2 END STATE:loc_1(RIVER) BEGIN have RIVER 1 END STATE:state(all) BEGIN states END Lu, Ng & Lee 6 August 2009, EMNLP09

13. Tree CRF Model 13 QUERY:answer(RIVER) RIVER:longest(RIVER) RIVER:exclude(RIVER 1,RIVER 2 ) RIVER:river(all)RIVER:traverse(STATE) STATE:stateid(STATENAME) STATENAME:texas what is RIVER 1 the longest RIVER 1 RIVER 1 that does not RIVER 2 riverrun through STATE 1 STATENAME 1 texas Four sets of features: 1.Hybrid Sequence Features 2.Two-level Hybrid Sequence Features 3.Three-level Hybrid Sequence Features 4.Adjacent Hybrid Sequence Features Lu, Ng & Lee 6 August 2009, EMNLP09

14. Features for Tree CRF Model 14 QUERY:answer(RIVER) RIVER:longest(RIVER) RIVER:exclude(RIVER 1,RIVER 2 ) RIVER:river(all)RIVER:traverse(STATE) STATE:stateid(STATENAME) STATENAME:texas what is RIVER 1 the longest RIVER 1 RIVER 1 that does not RIVER 2 riverrun through STATE 1 STATENAME 1 texas Hybrid Sequence Features Lu, Ng & Lee 6 August 2009, EMNLP09

15. Features for Tree CRF Model 15 QUERY:answer(RIVER) RIVER:longest(RIVER) RIVER:exclude(RIVER 1,RIVER 2 ) RIVER:river(all)RIVER:traverse(STATE) STATE:stateid(STATENAME) STATENAME:texas what is RIVER 1 the longest RIVER 1 RIVER 1 that does not RIVER 2 riverrun through STATE 1 STATENAME 1 texas Two-level Hybrid Sequence Features Lu, Ng & Lee 6 August 2009, EMNLP09

16. QUERY:answer(RIVER) Features for Tree CRF Model 16 RIVER:longest(RIVER) RIVER:exclude(RIVER 1,RIVER 2 ) RIVER:river(all)RIVER:traverse(STATE) STATE:stateid(STATENAME) STATENAME:texas what is RIVER 1 the longest RIVER 1 RIVER 1 that does not RIVER 2 riverrun through STATE 1 STATENAME 1 texas Three-level Hybrid Sequence Features Lu, Ng & Lee 6 August 2009, EMNLP09

17. Features for Tree CRF Model 17 QUERY:answer(RIVER) RIVER:longest(RIVER) RIVER:exclude(RIVER 1,RIVER 2 ) RIVER:river(all)RIVER:traverse(STATE) STATE:stateid(STATENAME) STATENAME:texas what is RIVER 1 the longest RIVER 1 RIVER 1 that does not RIVER 2 river run through STATE 1 STATENAME 1 texas Adjacent Hybrid Sequence Features Lu, Ng & Lee 6 August 2009, EMNLP09

18. Strengths of Tree CRF Model Allows features that specifically model the dependencies between neighboring hybrid sequences in the tree to be used Can efficiently capture long range dependencies between MR productions and hybrid sequences since each hybrid sequence is allowed to depend on the entire MR tree. 18 Lu, Ng & Lee 6 August 2009, EMNLP09

19. Candidate hybrid sequences –Each MR production is associated with a set of candidate hybrid sequences –Tree CRF: the correct hybrid sequence for each MR production is hidden Tree CRF Model 19 NUM:count(STATE) BEGIN how many STATE 1 END BEGIN how many STATE 1 are there END BEGIN what is the number of STATE 1 END BEGIN count the number of STATE 1 END BEGIN give me the number of STATE 1 END …… Lu, Ng & Lee 6 August 2009, EMNLP09

20. Candidate hybrid sequences –Training set consists of hybrid trees which are determined with Viterbi algorithm –Candidate hybrid sequences for each MR production are extracted from these training hybrid trees Tree CRF Model 20 NUM:count(STATE) BEGIN how many STATE 1 END BEGIN how many STATE 1 are there END BEGIN what is the number of STATE 1 END BEGIN count the number of STATE 1 END BEGIN give me the number of STATE 1 END …… Lu, Ng & Lee 6 August 2009, EMNLP09

21. Comparison over two models Two benchmark corpora: Geoquery and Robocup The tree CRF model performs better than the direct inversion model Validates the belief that some long range dependencies are important for NLG task While the direct inversion model performs well on R OBOCUP, it performs substantially worse on G EOQUERY Evaluations (I) G EOQUERY (880)R OBOCUP (300) B LEU N IST B LEU N IST Direct inversion model0.39735.54660.54686.6738 Tree CRF model0.57336.74590.62206.9845 21 Lu, Ng & Lee 6 August 2009, EMNLP09

22. Sample Outputs 22 G EOQUERY Reference what is the largest state bordering texas Direct Inversion model what the largest states border texas Tree CRF model what is the largest state that borders texas R OBOCUP Reference if DR2C7 is true then players 2, 3, 7 and 8 should pass to player 4 Direct Inversion model if DR2C7, then players 2, 3 7 and 8 should ball to player 4 Tree CRF model if the condition DR2C7 is true then players 2, 3, 7 and 8 should pass to player 4 Lu, Ng & Lee 6 August 2009, EMNLP09

23. Comparison with the previous state-of-the-art model Previous model optimizes evaluation metrics directly However, on both corpora, the tree CRF model performs better than the previous model Confirms that longer range dependencies and phrase-level dependencies are more important Evaluations (II) 23 G EOQUERY (880)R OBOCUP (300) B LEU N IST B LEU N IST W ASP -1 ++0.53706.48080.60226.8976 Tree CRF Model0.57336.74590.62206.9845 Lu, Ng & Lee 6 August 2009, EMNLP09

24. Evaluations (III) Comparison on other languages (G EOQUERY -250) Achieves better performance than the previous state-of-the-art system on all languages 24 EnglishJapanese B LEU N IST B LEU N IST W ASP -1 ++0.60355.71330.65854.6648 Tree CRF Model0.62655.89070.67884.8486 SpanishTurkish B LEU N IST B LEU N IST W ASP -1 ++0.61755.72930.48244.3283 Tree CRF Model0.63825.84880.50964.5033 Lu, Ng & Lee 6 August 2009, EMNLP09

25. Conclusions Built two novel models for NLG, on top of the hybrid tree framework –Direct Inversion Model –Tree CRF Model Evaluation shows the tree CRF model performs better than the direct inversion model Further evaluation shows the proposed tree CRF model performs better than a previous state-of-the-art system reported in the literature 25 Lu, Ng & Lee 6 August 2009, EMNLP09

26. Questions? 26 Lu, Ng & Lee 6 August 2009, EMNLP09