1. CS 4100 Artificial Intelligence Prof. C. Hafner Class Notes April 3and5, 2012

2. Why Natural Language Processing ? Huge amounts of data –Internet = at least 20 billion pages –Intranet Applications for processing large amounts of texts require NLP expertise Classify text into categories Index and search large texts Automatic translation Speech understanding –Understand phone conversations Information extraction –Extract useful information from resumes Automatic summarization –Condense 1 book into 1 page Question answering Knowledge acquisition Text generation / dialogues

3. Natural? Natural Language? –Refers to the language spoken by people, e.g. English, Japanese, Swahili, as opposed to artificial languages, like C++, Java, etc. Natural Language Processing –Applications that deal with natural language in a useful way (beyond token/string matching) Computational Linguistics –Doing linguistics on computers –More on the linguistic side than NLP, but closely related ]

4. Why Natural Language Processing? kJfmmfj mmmvvv nnnffn333 Uj iheale eleee mnster vensi credur Baboi oi cestnitze Coovoel2^ ekk; ldsllk lkdf vnnjfj? Fgmflmllk mlfm kfre xnnn!

5. Computers Lack Knowledge! Computers “see” text in English the same you have seen the previous text! People naturally have –“Common sense” –Reasoning capacity –Years of life experience Computers naturally have –No common sense –No reasoning capacity –No life experience

6. Where does it fit in the CS taxonomy? Computers Artificial Intelligence AlgorithmsDatabasesNetworking Robotics Search Natural Language Processing Information Retrieval Machine Translation Language Analysis SemanticsParsing

7. Linguistics Levels of Analysis Speech and text (and sign language) Levels –Phonology: sounds / letters / pronunciation –Morphology: the structure of words –Syntax: how these sequences are structured –Semantics: meaning of the strings –Pragmatics: what we use language to accomplish Interaction between levels

8. Issues in Syntax Shallow parsing: “the dog chased the bear” “the dog” “chased the bear” subject - predicate Identify basic structures NP-[the dog] VP-[chased the bear] Deeper analysis: “the dog ate my homework” –Who did what? (literal meaning) - semantics –The meaning in context - pragmatics

9. Issues in Syntax Full parsing: John loves Mary Help figuring out (automatically) questions like: Who did what and when?

10. More Issues Anaphora Resolution: discourse “The dog entered my room. It scared me” Preposition Attachment (syntax & semantics) “I saw the man in the park with a telescope”

11. Issues in Semantics Understand language! How? “plant” = industrial plant “plant” = living organism Words are ambiguous Importance of semantics? –Machine Translation: wrong translations –Information Retrieval: wrong information –Anaphora Resolution: wrong referents

12. The sea is home to million of plants and animals English  French [commercial MT system] Le mer est a la maison de billion des usines et des animaux French  English The sea is at the home for billions of factories and animals Why Semantics?

13. Issues in Semantics How to learn the meaning of words? From dictionaries: word senses plant, works, industrial plant -- (buildings for carrying on industrial labor; "they built a large plant to manufacture automobiles") plant, flora, plant life -- (a living organism lacking the power of locomotion) They are producing about 1,000 automobiles in the new plant The sea flora consists in 1,000 different plant species The plant was close to the farm.

14. Issues in Semantics Learn from annotated examples: –Assume 100 examples containing “plant” previously tagged by a human –Train a learning algorithm –How to choose the learning algorithm? –How to obtain the 100 tagged examples?

15. Issues in Pragmatics Why? To modify the beliefs of other agents Why? To change the actions of other agents

16. Issues in Learning Semantics Learning? –Assume a (large) amount of annotated data = training –Assume a new text not annotated = test Learn from previous experience (training) to classify new data (test) Bayes nets, decision trees, memory based learning (e.g. nearest neighbor), neural networks

17. Issues in Information Extraction “There was a group of about 8-9 people close to the entrance on Highway 75” Who? “8-9 people” Where? “highway 75” Extract information Detect new patterns: –Detect hacking / hidden information / etc. Gov./mil. puts lots of money put into IE research

18. Issues in Information Retrieval General model: –A huge collection of texts –A query Task: find documents that are relevant to the given query How? Create an index, like the index in a book More … –Vector-space models –Boolean models Examples: Google, Yahoo, etc.

19. Issues in Information Retrieval Retrieve specific information Question Answering “What is the height of mount Everest?” 11,000 feet

20. Issues in Information Retrieval Find information across languages! Cross Language Information Retrieval “What is the minimum age requirement for car rental in Italy?” Search also Italian texts for “eta minima per noleggio macchine” Integrate large number of languages Integrate into performant IR engines

21. Issues in Machine Translation Text to Text Machine Translations Speech to Speech Machine Translations Most of the work has addressed pairs of widely spread languages like English-French, English- Chinese

22. Issues in Machine Translations How to translate text? –Learn from previously translated data  Need parallel corpora French-English, Chinese-English have the Hansards Reasonable translations Chinese-Hindi – no such tools available today!

23. Speech Act Theory “I pronounce you husband & wife” “I sentence you to 5 years”

26. Natural languages are NOT context free – but almost!

27. About 40% of words in NY Times are not in a (large) dictionary – Natural language is “productive”

29. Example: “I SAW A MAN IN THE PARK WITH A TELESCOPE”

30. Parsing Parsing with CFGs refers to the task of assigning correct trees to input strings Correct here means a tree that covers all and only the elements of the input and has an S at the top It doesn’t actually mean that the system can select the correct tree from among the possible trees As with everything of interest, parsing involves a search that involves the making of choices Example: “I SAW A MAN IN THE PARK WITH A TELESCOPE”

31. The problem of “scaling up” – the same as in knowledge representation, planning, etc but even more difficult

32. Sentence-Types Declaratives: A plane left –S -> NP VP Imperatives: Leave! –S -> VP Yes-No Questions: Did the plane leave? –S -> Aux NP VP WH Questions: When did the plane leave? –S -> WH Aux NP VP

33. Potential Problems in CFG Agreement Subcategorization Movement

34. Agreement This dog Those dogs This dog eats Those dogs eat *This dogs *Those dog *This dog eat *Those dogs eats

35. Subcategorization Sneeze: John sneezed Find: Please find [a flight to NY] NP Give: Give [me] NP [a cheaper fare] NP Help: Can you help [me] NP [with a flight] PP Prefer: I prefer [to leave earlier] TO-VP Told: I was told [United has a flight] S *John sneezed the book *I prefer United has a flight *Give with a flight Subcat expresses the constraints that a predicate (verb for now) places on the number and type of the argument it wants to take

36. So? So the various rules for VPs overgenerate. –They permit the presence of strings containing verbs and arguments that don’t go together –For example –VP -> V NP therefore –Sneezed the book is a VP since “sneeze” is a verb and “the book” is a valid NP –Subcategorization frames can help with this problem (“slow down” overgeneration)

37. Movement Core example – [[My travel agent] NP [booked [the flight] NP ] VP ] S I.e. “book” is a straightforward transitive verb. It expects a single NP arg within the VP as an argument, and a single NP arg as the subject.

38. Movement What about? Which flight did the travel agent book ? (“Which flight” is the object of the verb “book”. It was “moved” to the front of the sentence!!) – Which flight do you want me to have the travel agent book? The direct object argument to “book” can be a long way from where its supposed to appear. Here it is separated from its verb by 2 other verbs. Therefore NL cannot be a finite state language

39. Semantics: Fillmore’s Case Grammar “Cases” are semantically based not grammatical ones like in Latin or German Charles Fillmore, “The Case for Case”, 1968 Produced more than one version

40. Case Grammar Semantics Case grammar semantics: Treats the verb as a predicate and the subject, objects, and other subordinate clauses as “arguments”. Labels the arguments with their relationship to the verb- predicate (called “cases”) [uses subcategorization info] Ex: John sold his car – agent and object cases Ex: John sold his car to Mary – agent, object and recipient cases

41. Fillmore’s list of cases Agentive (A): the case of the typically animate perceived instigator of the action identified by the verb. Instrumental (I) the case of the inanimate force or object causally involved in the action of state identified by the verb.

42. Fillmore’s list of cases Dative (D) - later Experiencer (E): the case of the animate being affected by the state or action identified by the verb. Factitive (F) - later Goal (G): the case of the object or being resulting from the action or state identified by the verb, or understood as a part of the meaning of the verb.

43. Fillmore’s list of cases Locative (L): the case which identified the location or spatial orientation of the state or action identified by the verb. Objective (O): the semantically most neutral case, the case of anything representable by a noun whose role in the action or state identified by the verb is identified by the semantic interpretation of the verb itself.

44. Case grammar semantics Semantic (case) roles don’t depend simply on syntactic roles Ex: John sold his car to Mary – agent, object and recipient cases Ex: John sold Mary his car Ex: John broke the window Ex: John broke the window with a hammer – agent, object and instrument cases Ex: A hammer broke the window

45. Informal quiz: Consider these sentences: 1.The burglar opened the door. 2.The door was opened by the burglar. 3.The burglar opened the door with a crowbar. 4.The door was opened by a crowbar. 5.The crowbar opened the door. 6.The door opened.

46. Case analysis : 1.The burglar opened the door. SOSO 2.The door was opened by the burglar. SASA 3.The burglar opened the door with a crowbar. SOASOA 4.The door was opened by a crowbar. SASA 5.The crowbar opened the door. SOSO 6.The door opened. S

47. Strengths Only one Noun Phrase occupies each case role in relation to a particular verb Therefore one could classify verbs in terms of which case roles they took. e.g.: o“open” - O, {A} {I} o“shout” - A, O, {E} {} denotes optional elements This model has been used in Artificial Intelligence, along with the sub-categorization of verbs (described earlier)

48. Weaknesses Researchers could not agree on a standard set of cases. Not always easy in practice to allocate particular Noun Phrases to cases. When it gets difficult there is a temptation to use the Objective (O) as a kind of “dustbin case” for all the NPs that don’t seem to fit anywhere else.