2. Natural Language Processing: Overview & Current Applications Noriko Tomuro April 7, 2006

3. What is NLP? Natural Language Processing (NLP) is a field in Computer Science devoted to creating computers that use natural language as input and/or output. Natural Language Processing (NLP) is a field in Computer Science devoted to creating computers that use natural language as input and/or output.

4. NLP involves other disciplines.. Linguistics Linguistics –NLP is also called ”Computational Linguistics” Psychology Psychology Mathematics and Statistics Mathematics and Statistics Information Theory Information Theory

5. Machines that Can Speak (1) HAL 9000 in “2001: A Space Odyssey” HAL 9000 in “2001: A Space Odyssey”

6. Machines that Can Speak (2) C3PO in Star Wars C3PO in Star Wars KITT in Knight Rider KITT in Knight Rider

7. But Still a Sci-Fi… “However, as 2001 approached it became clear that 2001's predictions in computer technology were far fetched. Natural language, lip reading, planning and plain common sense in computers were still the stuff of science fiction.” HAL 9000 - Wikipedia “However, as 2001 approached it became clear that 2001's predictions in computer technology were far fetched. Natural language, lip reading, planning and plain common sense in computers were still the stuff of science fiction.” HAL 9000 - Wikipedia

8. NLP is AI-complete “The most difficult problems in AI manifest themselves in human language phenomena.” “The most difficult problems in AI manifest themselves in human language phenomena.” Use of language is the touchstone of intelligent behavior. Use of language is the touchstone of intelligent behavior.

9. Test for Intelligence – Turing Test Alan Turing (1950) proposed a test of a machine's capability to perform human-like conversation. Alan Turing (1950) proposed a test of a machine's capability to perform human-like conversation. A human judge engages in a natural language conversation with two other parties, one a human and the other a machine; if the judge cannot reliably tell which is which, then the machine is said to pass the test. A human judge engages in a natural language conversation with two other parties, one a human and the other a machine; if the judge cannot reliably tell which is which, then the machine is said to pass the test.

10. Early Conversational Programs ELIZA (by Joseph Weizenbaum), 1966 ELIZA (by Joseph Weizenbaum), 1966 –A psychotherapist –No real understanding; simple pattern- matching to respond to user input (my ?x depresses me) (why does your ?x depress you) 5 (life ?x) (why do you say it ?x) 3 (I could ?x) (you could ?x) 2 (because ?x) (that is a good reason) 3 (?x) (tell me more) 0

11. Loebner Prize & Chatterbots/Chatbots (1) In 1990, Hugh Loebner started an annual Turing Test competition. In 1990, Hugh Loebner started an annual Turing Test competition. Conversational Programs are nowadays called Chatterbots (or Chatbots). Conversational Programs are nowadays called Chatterbots (or Chatbots). $100,000 to the first bot that judges cannot distinguish from a real human in a Turing test that includes text, visual, and auditory input. $100,000 to the first bot that judges cannot distinguish from a real human in a Turing test that includes text, visual, and auditory input. The Prize dissolves once the $100,000 prize is won. The Prize dissolves once the $100,000 prize is won.

12. Loebner Prize & Chatterbots/Chatbots (2) Nobody has won the prize yet. Nobody has won the prize yet.

13. NLP Applications NLP can be stand-along applications or components embedded in other systems. NLP can be stand-along applications or components embedded in other systems. NLP components/tasks include: NLP components/tasks include: –Part-of-speech tagging –Named Entity identification –Chunking (Partial parsing) I discuss some current NLP stand-alone applications. I discuss some current NLP stand-alone applications.

14. 1. Machine Translation (MT) One of the very earliest pursuits in computer science (after WWII). One of the very earliest pursuits in computer science (after WWII). Broad application domain – military to literary to search engines Broad application domain – military to literary to search engines Basic approaches: Basic approaches: –Inter-lingual (rule-based) –Direct translation (corpus-based)  more popular these days

15. Let’s translate! Google includes a MT engine (based on SYSTRAN system developed in EC). Google includes a MT engine (based on SYSTRAN system developed in EC).MT engineMT engine Let’s translate “Saddam Hussein has dismissed evidence suggesting he approved the execution of people under 18.” (BBC world news, April 5, 2006) Let’s translate “Saddam Hussein has dismissed evidence suggesting he approved the execution of people under 18.” (BBC world news, April 5, 2006)

16. 2. Text Summarization Create a summary of a text or texts. Create a summary of a text or texts. Many difficult problems, including: Many difficult problems, including: –Paraphrases –Anaphora (e.g.“it”, “they”)

17. 3. Question Answering Finds an answer (not a document) to a question typed in as a natural language sentence (not keywords). Finds an answer (not a document) to a question typed in as a natural language sentence (not keywords). Most systems can only answer simple, trivial pursuit type questions. Most systems can only answer simple, trivial pursuit type questions. Ask.com Ask.com Ask.com FAQFinder FAQFinder Some search engines perform limited, phrase-based Q&A, e.g. Google Some search engines perform limited, phrase-based Q&A, e.g. GoogleGoogle

18. Let’s ask questions! “Who wrote “The Da Vinci Code”? ” “Who wrote “The Da Vinci Code”? ” “What is the longest river in Australia?” “What is the longest river in Australia?” “What is the name of the main character in James Joyce's “Ulysses"? “ “What is the name of the main character in James Joyce's “Ulysses"? “

22. 4. Analyzing Web Documents Recently there have been many NLP applications which analyze (not just retrieve) web documents Recently there have been many NLP applications which analyze (not just retrieve) web documents –Blogs – for semantic analysis, sentiment (polarity/opinion) identification Blogs –Email Spam Filtering – but most often systems utilize simple word probability –A general approach “Web as a corpus” – web as the vast collection of documents

23. 5. Tutoring Systems We’ll hear from Peter shortly We’ll hear from Peter shortly

24. Why is NLP so hard..? Understanding natural languages is hard … because of inherent ambiguity Understanding natural languages is hard … because of inherent ambiguity Engineering NLP systems is also hard … because of Engineering NLP systems is also hard … because of –Huge amount of data resource needed (e.g. grammar, dictionary, documents to extract statistics from) –Computational complexity (intractable) of analyzing a sentence

25. Ambiguity (1) “At last, a computer that understands you like your mother.” “At last, a computer that understands you like your mother.” Three possible (syntactic) interpretations: Three possible (syntactic) interpretations: 1.It understands you as well as your mother understands you. 2.It understands that you like your mother. 3.It understands you as well as it understands your mother.

26. Ambiguity (2) At the acoustic level, At the acoustic level, –“.. computer that understands you like your mother” –“.. computer that understands you lie cured mother” At the semantic level, a “mother” is: At the semantic level, a “mother” is: –a female parent ? –a cask used in vinegar-making ?

27. Ambiguity (3) Word segmentation Word segmentation e.g. 社長兼業務部長 Possibilities include: - 社長 兼 業務 部長 president both business general-manager - 社長 兼業 務 部長 president multi-business Tsutomu general-manager

28. Ambiguity -- summary Ambiguity occurs at different levels. Ambiguity occurs at different levels. To resolve ambiguities, deep linguistic (as well as common-sense) knowledge is required. To resolve ambiguities, deep linguistic (as well as common-sense) knowledge is required. Two immediate ramifications: Two immediate ramifications: 1.Knowledge bottleneck – How do we acquire and encode ALL such information? 2.Computational Complexity – O(c n ), exponential w.r.t. the length of the sentence

29. The Bottom Line Complete NL Understanding (thus general intelligence) is impossible. Complete NL Understanding (thus general intelligence) is impossible. But we can make incremental progress. But we can make incremental progress. Also we have made successes in limited domains. Also we have made successes in limited domains. [But NLP is costly – Lots of work and resources are needed, but the return is sometimes not worth it.] [But NLP is costly – Lots of work and resources are needed, but the return is sometimes not worth it.]

30. Sentence Analysis (ACTION ingest (ACTOR John-1) (PATIENT food)) Syntactic structureSemantic structure “John ate the cake” S NPV “John”“ate” “the cake”

31. Syntactic Parsing Grammar R0: R1: R2: R3: R4: R5: R6: R7: cake"" N the"" Det ate"" V John"" NP V VG NPVG VP N Det NP VP NP S         S VP V NP “John”“ate” “the” DetN “cake” The process of deriving the phrase structure of a sentence is called “parsing”. The process of deriving the phrase structure of a sentence is called “parsing”. The structure (often represented by a Context Free parse tree) is based on the grammar. The structure (often represented by a Context Free parse tree) is based on the grammar.

32. Parsing Algorithms Top-down Parsing -- (top-down) derivation Top-down Parsing -- (top-down) derivation Bottom-up Parsing Bottom-up Parsing Chart Parsing Chart Parsing Earley’s Algorithm – most efficient, O(n 3 ) Earley’s Algorithm – most efficient, O(n 3 ) Left-corner Parsing – optimization of Earley’s Left-corner Parsing – optimization of Earley’s and lots more… and lots more…

33. (Bottom-up) Chart Parsing “John ate the cake” 0 1 2 3 4 (2) shift 2 (1) shift 1 “John” (7) shift 2 (9) reduce (10) reduce (5) shift 2 (3) shift 1 “ate”(6) shift 1 “the”(8) shift 1 “cake” (4) shift 2 (11) reduce --- 01234 Grammar

34. Earley’s Algorithm “John ate the cake” 0 1 2 3 4 Grammar (2) scanner “John” (4) predictor (5) scanner “ate” (7) predictor (8) scanner “the” (9) scanner “cake” (10) completor (11) completor (1) predictor (3) predictor (6) completor (12) completor

35. Demo using my CF parser

36. Probabilistic Parsing For ambiguous sentences, we’d like to know which parse tree is more likely than others. For ambiguous sentences, we’d like to know which parse tree is more likely than others. So we must assign probability to each parse tree … but how? So we must assign probability to each parse tree … but how? A probability of a parse tree t is where r is a rule used in t. A probability of a parse tree t is where r is a rule used in t. and p(r) is obtained from a (annotated) corpus.

37. Partial Parsing Parsing fails when the coverage of the grammar is not complete – but it’s almost impossible to write out all legal syntax (without accepting ungrammatical sentences). Parsing fails when the coverage of the grammar is not complete – but it’s almost impossible to write out all legal syntax (without accepting ungrammatical sentences). We’d like to at least get pieces even when full parsing fails. We’d like to at least get pieces even when full parsing fails. Why not abandon full parsing and aim for partial parsing from the start… Why not abandon full parsing and aim for partial parsing from the start…

38. Semantic Analysis (1) Derive the meaning of a sentence. Derive the meaning of a sentence. Often applied on the result of syntactic analysis. Often applied on the result of syntactic analysis. “John ate the cake.” NP V NP NP V NP ((actionINGEST); syntactic verb (actorJOHN-01); syntactic subj (objectFOOD)); syntactic obj ((actionINGEST); syntactic verb (actorJOHN-01); syntactic subj (objectFOOD)); syntactic obj To do semantic analysis, we need a (semantic) dictionary (e.g. WordNet, http://www.cogsci.princeton.edu/~wn/ ). To do semantic analysis, we need a (semantic) dictionary (e.g. WordNet, http://www.cogsci.princeton.edu/~wn/ ). http://www.cogsci.princeton.edu/~wn/

39. Semantic Analysis (2) Semantics is a double-edged sword… Semantics is a double-edged sword… –Can resolve syntactic ambiguity “I saw a man on the hill with a telescope” “I saw a man on the hill with a telescope” “I saw a man on the hill with a hat” “I saw a man on the hill with a hat” –But introduces semantic ambiguity “She walked towards the bank” “She walked towards the bank” But in human sentence processing, we seem to resolve both types of ambiguities simultaneously (and in linear time)… But in human sentence processing, we seem to resolve both types of ambiguities simultaneously (and in linear time)…

40. Demo using my Unification parser