Presentation is loading. Please wait.

Presentation is loading. Please wait.

AUTOMATIC TEXT SUMMARIZATION By Chetana Gavankar Subhabrata Mukherjee Kedharnath Narahari Sarbartha Sengupta under guidance of: Prof Pushpak Bhattacharya.

Similar presentations


Presentation on theme: "AUTOMATIC TEXT SUMMARIZATION By Chetana Gavankar Subhabrata Mukherjee Kedharnath Narahari Sarbartha Sengupta under guidance of: Prof Pushpak Bhattacharya."— Presentation transcript:

1 AUTOMATIC TEXT SUMMARIZATION By Chetana Gavankar Subhabrata Mukherjee Kedharnath Narahari Sarbartha Sengupta under guidance of: Prof Pushpak Bhattacharya

2 PRESENTATION CONTENT Motivation Types of summaries Challenges Single-Document Summarization o Early work o Machine Learning Methods  Supervised Methods  Unsupervised Method o Deep Natural Language Analysis methods Multi-Document Summarization Evaluation Conclusion

3 MOTIVATION Download papers and get the summary.. You have list of s about sports event get the summary of those s in one para… You have to study loads of books for the exam and the summarizer gives the key concepts of the books as few pages notes… Value for researchers Get me everything Papers say about “Automatic Text Summarization”

4 MOTIVATION

5 DEFINITION Automatic Summaries Should be less than half of original text Should convey important information May be produced from single or multiple documents (Radev et al) Dipanjan Das, Andre F.T. Martins (2007). A Survey on Automatic Text Summarization. Literature Survey for the Language and Statistics II course at CMU, Pittsburg

6 APPLICATIONS - NEWS AGGREGATOR

7 APPLICATIONS – MOVIE REVIEWS

8 TYPES OF SUMMARIES With respect to content: Indicative: provide an idea what the text is about, but do not render the content Informative: shortened versions of the text With respect to the way of creating: Extracts: identify important sections of the text Abstracts: produce important material in a new way Dipanjan Das, Andre F.T. Martins (2007). A Survey on Automatic Text Summarization. Literature Survey for the Language and Statistics II course at CMU, Pittsburg

9 TYPES OF SUMMARIES With respect to Input Restricted vs. Unrestricted domain Single-document vs. Multiple-document With respect to Purpose Generic vs. Query based Background vs. just-the-news nd vs. just-the-news Eduard Hovy and Lin C. Y. "Automated Text Summarization in summarist", MIT Press

10 TYPES OF SUMMARIES With respect to Input Restricted vs. Unrestricted domain Single-document vs. Multiple-document With respect to Purpose Generic vs. Query based Background vs. just-the-news Eduard Hovy and Lin C. Y. (1998 )"Automated Text Summarization and the summarist system", TIPSTER III Final Report (SUMMAC)

11 TOOLS – WORD SUMMARIZER Microsoft Word AutoSummarize

12 TOOLS – GNOME SUMMARIZER

13 TOOLS – SWESUM SUMMARIZER

14 CHALLENGES Selecting pieces from the input and concatenating them to yield a summary High reduction rates like headline Methods for evaluating summaries Multiple languages Multiple Hybrid sources Hahn U. and Mani I. (2000) “The Challenges of Automatic Summarization”, Computer, IEEE Computer Society 14

15 EARLY WORK ATS has its roots in the last 50’s and has been developed continuously… A word frequency based ATS [Luhn, 1958]. An ATS based on multiple features [Edmundson, 1969]. …….. ……. Still unsolved !

16 WORD FREQUENCY Content words indicate topic of a text Frequency of a content word – measure of its significance Retrieve the top n frequent occurring content words Rank a sentence according to the frequency of those words present in it WORDS FREQUENCY E Resolving power of significant words Luhn, H. P. (1958). The automatic creation of literature abstracts. IBM Journal of Research Development, 2(2):

17 Position of a Sentence Sentences occurring under certain headings are positively relevant Topic sentences tend to occur very early or very late in a document and its paragraphs Optimum Position Policy a ranked list that indicates in what ordinal positions in the text the high-topic-bearing sentences tend to occur. ( Lin and Hovy, 97). Ex: [T1, P1S1, P1S2,...] for a News Article Edmundson, H. P. (1969). New methods in automatic extracting. Journal of the ACM, 16(2):

18 Cue words in a Text Probable relevance of a sentence affected by Cue words: Bonus words : positively affecting the relevance of a sentence ( e.g. “Significant”, “Greatest” ) Stigma words : negatively affecting the relevance of a sentence ( e.g. “Impossible”, “Hardly” ) Null words : irrelevant Edmundson, H. P. (1969). New methods in automatic extracting. Journal of the ACM, 16(2):

19 NAÏVE BAYES Let s be a particular sentence, S the set of sentences making up the summary and F 1, … F k be the set of features Assume feature independence – Additional Features Sentence Length Presence of Uppercase Words – Position, Cue features with Sentence Length performed best Kupiec, J., Pedersen, J., and Chen, F. (1995). A trainable document summarizer, In Proceedings SIGIR '95, pages 68-73

20 Naïve Bayes Contd… Richer features – Tf-idf to derive signature words – Named Entity Tagger to retrieve tokens – Shallow Discourse Analysis to maintain cohesion – Synonym and Morphological variants of lexical terms merged using WordNet. Aone, C., Okurowski, M. E., Gorlinsky, J., and Larsen, B. (1999), A trainable summarizer with knowledge acquired from robust nlp techniques, In Mani, I. and Maybury, M. T., editors, Advances in Automatic Text Summarization, pages 71-80

21 Decision Tree Feature independence assumption not valid in real world situation Creation of feature vector – Baseline – Title – tf & tf-idf scores – Position score – Query Signature – IR Signature – Sentence Length – Average Lexical Connectivity – Numerical Data – Proper Name – Pronoun & Adjective – Weekday & Month – Quotation – First Sentence Lin, C.-Y. (1999). Training a selection function for extraction, In Proceedings of CIKM '99, pages 55-62

22 Decision Tree Contd… Scores of all the features combined by automated learning process using decision tree and normalized Remarks – Decision Tree performs best over all dataset – Naïve combination beats Decision Tree in 3 topics Possible Reason ???? Features were Independent

23 Hidden Markov Model Drawbacks of earlier approches – Feature based bag-of-words model – Non-sequential Use sequential model to account for local dependencies between sentences Features – Position of sentence in the document – Number of terms in the sentence – Likeliness of the sentence terms given the document terms Conroy, J. M. and O'leary, D. P. (2001). Text summarization via hidden markov models, In Proceedings of SIGIR '01, pages

24 Hidden Markov Model Contd… 2s+1 states alternating between s summary states & s+1 non-summary states Odd state summary state, Even state non- summary state Transition Matrix M whose element (i,j) is the probability of transition from state i to j Output function b i (o)= Pr(O|state i) where O is an observed vector of features Assumption : features are multivariate normal M & O learnt from training data

25 Log-Linear Models Let c be a label, d the item we are interested in labeling, f i the i th feature and λ i the corresponding feature weight Z(d) = Ʃ c exp( Ʃ i λ i f i (c,d)) is the normalization constant f w,c’ (d,c)= 0 c≠c’ 1 c=c’ Larger value of λ i means f i is a strong indicator of class c GIS, IIS used to iteratively tune model parameters. Outperformed Naïve Bayes DRAWBACK : Overfitting Osborne, M. (2002). Using maximum entropy for sentence extraction. In Proceedings of the ACL'02 Workshop on Automatic Summarization

26 Drawbacks of Earlier Methods 1.Performance might degrade, if the text consists of multiple topics. 2.Anaphors in the extracted sentences might not have any antecedent in the summary. 3.The summary might be incoherent, since the sentences are just extracted from various parts of the text.

27 Drawbacks Contd … Consider the following two sequences: 1. “ Dr.Kenny has invented an anesthetic machine. This device controls the rate at which an anesthetic is pumped into the blood. ” 2. “ Dr.Kenny has invented an anesthetic machine. The doctor spent two years on this research. ” “Dr.Kenny” appears once in both sequences and so does “machine”. But sequence 1 is about the machine, and sequence 2 is about the “doctor”.

28 Cohesion Cohesion ( Halliday & Hasan 1976) – “stitching together” different parts of the text – Use of semantically related terms, co-reference, ellipsis, conjunctions Lexical Cohesion – Semantically related words – Reiteration category Repetations, synonyms, hyponyms – Collocation category Words occurring in same lexical context – Ex: She works as a teacher in the school.

29 Lexical Chain General Approach 1. Select a set of candidate words; 2. For each candidate word, find an appropriate chain relying on a relatedness criterion among members of the chains; 3. If it is found, insert the word in the chain and update it accordingly. Relations – Extra-strong (between word & its repetation) No restriction – Strong (between 2 words connected by WordNet reln) Window of 7 sentences – Medium-strong (path length > 1 hop ) Window of 3 sentences Preference : Extra-strong > Strong > Medium-strong

30 Drawback Mr. Kenny is the person that invented an anesthetic machine which uses micro-computers to control the rate at which an anesthetic is pumped into the blood. Such machines are nothing new. But his device uses two micro-computers to achieve much closer monitoring of the pump feeding the anesthetic into the patient. (Morris & Hirst 1991) [lex "Mr.", sense {mister, Mr.}] [lex "person", sense {person, individual,someone, man, mortal, human, soul}]. First sense of machine in WordNet – “ an efficient person” – a holonym of “person” and thus wrongly disambiguated

31 Component and Graph Connectivity Barzilay, R. and Elhadad, M. (1997). Using lexical chains for text summarization. In Proceedings ISTS'97

32 Component & Graph Connectivity Contd…

33 Multi Document Summarization Multiple sources of information – Similarity between topics – Supplement each other – Occasionally contradictory Key Tasks – Identifying Key concepts across documents – Coping with redundancy – Ensuring final Summary is coherent and complete Applications: news clustering systems – Google News, Columbia NewsBlaster, News in Essence etc

34 TOPIC-DRIVEN SUMMARIZATION Carbonell, J. and Goldstein, J. (1998). The use of MMR, diversity- based re-ranking for reordering documents and producing summaries.

35 TOPIC DRIVEN SUMMARIZATION(contd.) Carbonell, J. and Goldstein, J. (1998). The use of MMR, diversity-based re-ranking for reordering documents and producing summaries. In proceedings of SIGIR '98.

36 GRAPH SPREADING Mani, I. and Bloedorn, E. (1997). Multi-document summarization by graph search and matching. In AAAI/IAAI, pages

37 Example for nodes and links in the graph(mani and bloerdon 97’) Mani, I. and Bloedorn, E. (1997). Multi-document summarization by graph search and matching. In AAAI/IAAI, pages

38 GRAPH SPREADING(Contd..) Words and phrases are intialized according to their TF-IDF scores. For each sentence in both documents, two scores are computed. One score that reflects the presence of common nodes, which is computed as the average weight of these nodes; Other score that computes instead the average weights of difference nodes. the sentences that have higher common and different scores are highlighted and accordingly the ouput is generated.

39 CENTROID-BASED SUMMARY Does not make use of language generation model. Documents are modeled as bag-of-words. Topic Detection :  Clustering algorithm that uses TF-IDF vector repesentations of documents  Successively add documents and recomputes centroids.   Centroids are pseudo documents that include words with TF-IDF score above some threshold. d’-truncated document C j – j th cluster Radev, D. R., Jing, H., and Budzikowska, M. (2000). Centroid-based summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies.

40 CENTROID BASED SUMMARIZATION Sentence Identification: 2metrics o C luster B ased R elative U tility-how relevant a sentence is to particular topic of the cluster. o C ross S entence I nformational S ubsumption- measure of redundancy among sentences. Radev, D. R., Jing, H., and Budzikowska, M. (2000). Centroid-based summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies. In NAACL-ANLP 2000 Workshop on Automatic summarization, pages 21-30, Morristown, NJ, USA.

41 MULTILINGUAL MULTI-DOCUMENT SUMMARY Target language(English) in which summary is written. Source documents present in both preferred language and foreign language(Arabic). Use IBM’s translational model to translate documents in source language to target language. Check for similarity between translated sentences in two documents. If similarities found, retain documents in source language, since they can be more grammatically correct. Evans, D. K. (2005). Similarity-based multilingual multi-document summarization. Technical Report CUCS , Columbia University.

42 SHORT SUMMARIES Witbrock, M. J. and Mittal, V. O. (1999). Ultra-summarization (poster abstract): a statistical approach to generating highly condensed non- extractive summaries.

43 Witbrock, M. J. and Mittal, V. O. (1999). Ultra-summarization (poster abstract): a statistical approach to generating highly condensed non- extractive summaries. In Proceedings of SIGIR '99, pages 315{316, New York, NY, USA.

44 SENTENCE COMPRESSION Compression of sentence used for summarization. Uses noisy-channel model which considers that one starts with a short summary s, according to source model P (s). Subjected to noisy-channel to make full sentence t, in a process guided by channel model, P(t/s ). Now observing t, recover the summary as: s’=argmax s P(s/t)= argmax s P(s).P(t/s) Advantage of decoupling the goals of grammatical correctness and preserving important information. Knight, K. and Marcu, D. (2000). Statistics-based summarization - step one: Sentence compression. In AAAI/IAAI, pages

45 Knight, K. and Marcu, D. (2000). Statistics-based summarization - step one: Sentence compression. In AAAI/IAAI, pages

46 Evaluation Difficult task. (There does not exist an ideal summary for a given document or set of document.) Agreement between human summarizers is quite low. Difficult to evaluate the summary content. Absence of a standard human or automatic evaluation metric.

47 Evaluation Lin and Hovy (2002).  describe and compare various human and automatic metrics to evaluate summaries.  Focus on the evaluation procedure used in the Document Understanding Conference 2001(DUC-2001).  compared manually written ideal summaries with summaries generated automatically by summarization systems and baseline summaries. Lin, C.-Y. and Hovy, E. (2002). Manual and automatic evaluation of summaries. In Proceedings of the ACL-02 Workshop on Automatic Summarization, pages 45-51

48 Evaluation Lin and Hovy (2002).  Each text was decomposed into a list of units (sentences).  stepped through each model unit (MU) from the ideal summaries.  marked all system units (SU) sharing content with the current model unit. All (4) Most (3) Some (2) or Hardly any (1)  Grammaticality, cohesion, and coherence were also rated

49 Evaluation Lin and Hovy (2002).  The weighted recall at threshold ‘t’ (t=1 to 4).  outline an accumulative n-gram matching score (which they call NAMS), Taken from: Dipanjan Das, Andre F.T. Martins, A Survey on Automatic Text Summarization2007

50 Evaluation Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004).  Let be a set of reference summary, and let be a summary generated automatically by a system. Let be a binary vector representing n-grams contained in a document d.  The metric ROUGE-N is an n-gram recall based statistic where denotes the usual inner product of vectors Lin, C.-Y. (2004). Rouge: A package for automatic evaluation of summaries. In Marie-Francine Moens, S. S., editor, Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74-81

51 Evaluation Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004).  Example: d := “Ram helped Shyam to learn German.” r1:= “Shyam learnt German by Ram.” r2:= “Ram taught German to Shyam.” s := “Shyam learn German from Ram.” Φ n (r1) := [10101]Φ n (r2) := [10111] Φ n (s) := [11101] = 6/7

52 Evaluation Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004).  ROUGE-N can be used for multiple reference summaries.  An alternative is taking the most similar summary in the reference set.

53 Evaluation Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004).  Example: d := “Ram helped Shyam to learn German.” r1:= “Shyam learnt German by Ram.” r2:= “Ram taught German to Shyam.” s := “Shyam learn German from Ram.” Φ n (r1) := [10101]Φ n (r2) := [10111] Φ n (s) := [11101] = 3/3

54 Evaluation Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004).  Another metric in (Lin, 2004) applies the concept of longest common subsequence (LCS).  Let r 1,…,r u be the reference sentences of the documents in R, and s a candidate summary

55 Evaluation Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004).  Example: s := “Ram pushed Shyam into the Pool.” r1:= “Ram push Shyam into the Pool.” r2:= “Shyam push Ram into the Pool.” ROUGE-N: r1=r2 (“Ram”, “Shyam into the Pool”) ROUGE-L: r1: = 5/6 (“Ram”, “Shyam into the Pool”) r2: = 4/6 (“Ram”, “into the Pool”) r1>r2

56 Evaluation Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004).  Yet another measure ROUGE-S, which can be seen as a modified version of ROUGE-N for n = 2

57 Evaluation Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004).  Example: s := “Ram pushed Shyam into the Pool.” r1:= “Ram push Shyam into the Pool.” r2:= “Shyam push Ram into the Pool.” r3:= “Shyam into the Pool Ram pushed.” ROUGE-N: r3>r1=r2 ROUGE-L: r2>r3=r4 ROUGE-S: r1: = 10/15 r2: = 9/15 r3: = 7/15 r1>r2>r3

58 Evaluation Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin 2004).  The various versions of ROUGE were evaluated by computing the correlation coefficient between ROUGE scores and human judgment scores

59 Conclusion a need to develop efficient and accurate summarization systems. attention has drifted from summarizing scientic articles to news articles, electronic mail messages, advertisements, and blogs. Both abstractive and extractive approaches have been attempted. simple extraction of sentences have produced satisfactory results in large-scale applications. This survey emphasizes extractive approaches to summarization using statistical methods.

60 References Primary Reference o Dipanjan Das, Andre F.T. Martins (2007). A Survey on Automatic Text Summarization. Literature Survey for the Language and Statistics II course at CMU, Pittsburg Secondary Reference o Edmundson, H. P. (1969). New methods in automatic extracting. Journal of the ACM, 16(2): o Luhn, H. P. (1958). The automatic creation of literature abstracts. IBM Journal of Research Development, 2(2): o Lin, C.-Y. (1999). Training a selection function for extraction. In Proceedings of CIKM '99, pages o Ono, K., Sumita, K., and Miike, S. (1994). Abstract generation based on rhetorical structure extraction. In Proceedings of Coling '94, pages o Barzilay, R. and Elhadad, M. (1997). Using lexical chains for text summarization. In Proceedings ISTS'97 o Mani, I. and Bloedorn, E. (1997). Multi-document summarization by graph search and matching. In AAAI/IAAI, pages

61 References Contd… Evans, D. K. (2005). Similarity-based multilingual multi-document summarization Technical Report CUCS , Columbia University Radev, D. R., Jing, H., and Budzikowska, M. (2000). Centroid-based summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies. In NAACL-ANLP 2000 Workshop on Automatic summarization, pages Witbrock, M. J. and Mittal, V. O. (1999). Ultra-summarization (poster abstract): a statistical approach to generating highly condensed non- extractive summaries. In Proceedings of SIGIR '99, pages Knight, K. and Marcu, D. (2000). Statistics-based summarization - step one: Sentence compression. In AAAI/IAAI, pages Lin, C.-Y. and Hovy, E. (2002). Manual and automatic evaluation of summaries. In Proceedings of the ACL-02 Workshop on Automatic Summarization, pages Lin, C.-Y. (2004). Rouge: A package for automatic evaluation of summaries. In Marie-Francine Moens, S. S., editor, Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages Lin, C.-Y., Cao, G., Gao, J., and Nie, J.-Y. (2006). An information-theoretic approach to automatic evaluation of summaries. In Proceedings of HLT- NAACL '06, pages


Download ppt "AUTOMATIC TEXT SUMMARIZATION By Chetana Gavankar Subhabrata Mukherjee Kedharnath Narahari Sarbartha Sengupta under guidance of: Prof Pushpak Bhattacharya."

Similar presentations


Ads by Google