1. Text Summarisation based on Human Language Technologies and its Applications Elena Lloret Pastor Supervisor: Dr. Manuel Palomar Seminar - June 2011

2. Outline Introduction State of the Art COMPENDIUM Text Summarisation Tool Evaluation and Experiments COMPENDIUM in HLT Applications Conclusion 2

3. Introduction MOTIVATION Human Language Technologies (HLT) ▫Allow people to communicate with machines by using natural language (Cole, 1997) Intelligent applications based on HLT ▫Information retrieval ▫Question Answering ▫Text Classification ▫Opinion Mining ▫Text Summarisation 3

4. Introduction MOTIVATION Why is Text Summarization (TS) needed? ▫To condense information, keeping at the same time, the most relevant one ▫Help users to manage and process large amounts of information 4 The 2008 Summer Olympics took place in Beijing, China, from August 8 to August 24, 2008. A total of 11,028 athletes from 204 National Olympic Committees (NOCs) competed in 28 sports and 302 events. It was the third time that the Summer Olympic Games were held in Asia, after Tokyo, Japan in 1964 and Seoul, South Korea in 1988. The program for the Beijing Games was quite similar to that of the 2004 Summer Olympics held in Athens. There were 28 sports and 302 events. Moreover, there were 43 new world records and 132 new Olympic records set at the 2008 Summer Olympics. Chinese athletes won the most gold medals, with 51, and 100 medals altogether, while the United States had the most medals total with 110. There were many memorable champions but it was Michael Phelps and Usain Bolt who stole the headlines. Source documents: http://en.wikipedia.org/wiki/2008_Summer_Olympics http://en.beijing2008.cn/# http://www.olympic.org/beijing-2008-summer-olympics 17.500.000 results

5. State of the Art TYPES OF SUMMARIES 5

6. State of the Art TEXT SUMMARISATION PROCESS Topic identification ▫What the document is about Interpretation or topic fusion ▫Important topics are expressed using new formulation Summary generation ▫Natural Language Generation is applied to build the final summary 6

7. State of the Art GENERATION OF SUMMARIES Approaches ▫Statistical-based  tf, tf*idf (e.g. Or ă san, 2009) ▫Topic-based  event words (e.g. Kuo & Chen, 2008) ▫Graph-based  LexRank (e.g. Erkan & Radev, 2004) ▫Discourse-based  lexical chains (e.g. Barzilay & Elhadad, 1999) ▫Machine learning-based  neuronal nets (e.g. Svore et al., 2007) 7

8. State of the Art GENERATION OF SUMMARIES New types of summaries ▫Personalised summaries  user profiles (e.g. Díaz & Gervás, 2007) ▫Update summaries  “history” (e.g. Li et al., 2008) ▫Sentiment-based summaries  multi-aspect rating model (e.g. Titov, & McDonald, 2008) ▫Surveys summaries  Wikipedia articles (e.g. Sauper & Barzilay, 2009) ▫Abstractive summaries  sentence compression (e.g. Filippova, 2010) 8

9. State of the Art GENERATION OF SUMMARIES New scenarios ▫Literary text  books (e.g. Ceylan & Mihalcea, 2009) ▫Patent claims  (e.g. Trappey et al., 2009) ▫Image captioning  (e.g. Aker & Gaizauskas, 2010) ▫Web 2.0 textual genres  blogs (e.g. Balahur et al., 2009) 9

10. State of the Art EVALUATION OF SUMMARIES 10 Types of evaluation ▫Intrisic  evaluate the summary on its own  Informativeness assessment  Quality assessment ▫Extrinsic  evaluate how good the summaries are to perform other tasks Pyramid QARLA ROUGE Basic Elements Indicativeness Grammaticality Coherence Non-redundancy

11. COMPENDIUM TS tool TYPES OF SUMMARIES 11

12. 12 Legend: Additional Stages Types of Summaries (output) Core Stages Input for the additional stages Input COMPENDIUM TS tool ARCHITECTURE

13. Suface Linguistic Analysis ▫Pre-process the input text by employing state-of-the-art tools  Sentence segmentation  Tokenisation  Part-of-Speech tagging  Stemming  Stop word identification 13 COMPENDIUM TS tool CORE STAGES SURFACE LINGUISTIC ANALYSIS

14. Redundancy Detection ▫Identify and remove repeated information  Textual Entailment (Ferrández, 2009) ▫The main idea behind the use of TE for detecting redundancy is that those sentences whose meaning is already contained in other sentences can be discarded, as the information has been previously mentioned 14 COMPENDIUM TS tool CORE STAGES T: The man was killed last week H: The man is dead T: The man was shot in his shoulder H: The man is dead TRUEFALSE REDUNDANCY DETECTION

15. Topic Identification ▫Identify the most relevant topics  Term frequency (Luhn, 1958) ▫Most frequent words (without considering stop words) can be considered the main topics of a document 15 COMPENDIUM TS tool CORE STAGES TOPIC IDENTIFICATION

16. Relevance Detection ▫Compute a weight for each sentence, depending on its importance  The Code Quantity Principle (Givón, 1990)  Coding element  noun phrase ▫Sentences containing a noun phrase including high frequent words will be considered more important ▫Score for each sentence  16 COMPENDIUM TS tool CORE STAGES RELEVANCE DETECTION

17. Summary Generation ▫Summary size  number of words  compression rate ▫The highest scored sentences up to a desired length are selected and extracted ▫Sentences are ordered as they appear in the document Type of summaries (output) ▫Generic extracts  COMPENDIUM E 17 COMPENDIUM TS tool CORE STAGES SUMMARY GENERATION

18. Query Similarity ▫Cosine similarity  qSim Type of summaries (output) ▫Query-focused extract  COMPENDIUM QE ▫Score for each sentence 18 COMPENDIUM TS tool ADDITIONAL STAGES

19. Subjective Information Detection ▫Opinion mining techniques (Balahur-Dobrescu et al., 2009 ) Type of summaries (output) ▫Sentiment-based extract  COMPENDIUM SE ▫Select the highest relevant sentences among the subjective ones 19 COMPENDIUM TS tool ADDITIONAL STAGES

20. Information compression and fusion ▫Word graphs Type of summaries (output) ▫Abstractive-oriented summary  COMPENDIUM E-A ▫Combine extractive and new information 20 COMPENDIUM TS tool ADDITIONAL STAGES

21. 21 EVALUATION AND EXPERIMENTS EVALUATION METHODOLOGY Type of evaluation ▫intrinsic What are we going to assess? ▫ COMPENDIUM in different domains and contexts Which criteria are we going to use for the evaluation? ▫Content (automatically)  ROUGE (Lin, 2004) ▫Quality (manually)  readability & user satisfaction

22. Newswire ▫Single-document generic extracts: ~ 45% (F-measure, ROUGE-1) ▫Multi-document: ~ 30% (F-measure, ROUGE-1) Blogs ▫Multi-document sentiment-based summaries: ~ 64% (F-measure, Pyramid) Image captions ▫Multi-document query-focused summaries: ~36% (F-measure, ROUGE-1) Medical research papers ▫Single-document abstractive-oriented summaries: ~ 42% (F- measure, ROUGE-1) 22 EVALUATION AND EXPERIMENTS RESULTS

23. 23 Question answering ▫ Allows users to formulate questions in natural language and provide them with the exact information required Objective ▫Integrate COMPENDIUM with a Web-based question answering approach  COMPENDIUM QE COMPENDIUM in HLT APPLICATIONS QUESTION ANSWERING

24. 24 COMPENDIUM in HLT QUESTION ANSWERING Proposed approach Question analysis ▫Question type, focus and keywords Information retrieval ▫Retrieve the first 20 documents in Google Summarisation ▫ COMPENDIUM QE ▫Summary size  length of snippets Answer extraction ▫Named Entities ▫Semantic roles

25. 25 Data ▫100 factual questions  Person  Location  Temporal  Organization Evaluation ▫Correct ▫Incorrect ▫Non-answered Question (temporal) When was the first Barbie produced? Answer1959 Question (location) Where is the pancreas located? Answerabdomen COMPENDIUM in HLT QUESTION ANSWERING F-measure (%)

26. Results ▫Named entity-based QA ▫Semantic role-based QA Question type PersonOrganizationTemporalLocation Snippets53.352.248.961.2 COMPENDIUM QE 56.553.366.765.3 26 COMPENDIUM in HLT QUESTION ANSWERING Question type PersonOrganizationTemporalLocation Snippets43.925.024.248.9 COMPENDIUM QE 53.741.260.062.5 NE-based QA 12% SR-based QA 48%

27. The proposed techniques are appropriate for TS ▫Textual entailment  appropriate to tackle redundancy ▫Code Quantity Principle  detecting relevant information ▫Word graph-based algorithms  compress and merge information Summaries, although imperfect in their nature, can improve the performance of other HLT tasks ▫Question Answering 27 CONCLUSION

28. Text Summarisation based on Human Language Technologies and its Applications Elena Lloret Pastor Supervisor: Dr. Manuel Palomar Seminar - June 2011