1. CIS 8590 – Fall 2008 NLP 1 Introduction to Natural Language Processing (aka, Computational Linguistics) Slides by me, Martha Palmer, Eleni Miltsakaki, Dan Jurafsky, Tarkan Kacmaz, and others

2. Overview General Methods in NLP (3 weeks) –Low-level NLP problems and techniques –Graphical Models for NLP –Text Mining basics Information Retrieval Overview (2 weeks) Information Extraction Overview (2 weeks) Selected topics in Information Extraction (4 weeks) CIS 8590 – Fall 2008 NLP 2

3. Practical Matters Prereqs: General understanding of probability and statistics Grading: 20% In-class participation, paper presentations 30% Course projects 50% Midterm (and possibly quizzes) Course projects –Please start finding project partners soon! –I will supply some ideas for projects later CIS 8590 – Fall 2008 NLP 3

4. WHAT IS LANGUAGE?

5. When we study human language, we are approaching what some might call the “human essence”, the distinctive qualities of mind that are, so far as we know, unique to man. Noam Chomsky

6. WHAT IS LANGUAGE? Definition with respect to form: Language is a system of speech symbols. It is realized acoustically (sound waves), visually-spatially (sign language) and in written form. Definition with respect to function: Language is the most important means of human communication. It is used to convey and exchange information (informative function) Multiplicity of languages: We know of about 7000 languages, which is about 1% of all the languages that ever existed.

7. LANGUAGE AND THE BRAIN

9. THEORIES OF LANGUAGE Noam Chomsky claims that language is innate. B. F. Skinner claims that language is learned; it is basically a stimulus-response mechanism.

10. WHAT IS GRAMMAR ? When we learn a language we also learn the rules that govern how language elements, such as words, are combined to produce meaningful language. These elements and rules constitute the Grammar of a language. The Grammar is “what we know” Grammar represents our linguistic competence.

11. DESCRIPTIVE vs PRESCRIPTIVE GRAMMAR Prescriptive (should be) Descriptive (is)

12. Areas of Linguistics phonetics - the study of speech sounds phonology - the study of sound systems morphology- the rules of word formation syntax - the rules of sentence formation semantics - the study of word meanings pragmatics – the study of discourse meanings sociolinguistics - the study of language in society applied linguistics –the application of the methods and results of linguistics to such areas as language teaching, national language policies, lexicography, translation, language in politics etc.

13. What is phonetics? Phonetics is the science of speech. We all speak. But how many of us know how we speak? Or what speech is like? Phonetics seeks to answer those questions.

14. Orthography and Sounds The English language is not phonetic. Words are not spelled as they are pronounced There is no one to one correspondence between the letters and the sounds or phonemes.

15. Orthography and Sounds Did he believe that Caesar could see the people seize the seas. The silly amoeba stole the key to the machine

16. Articulatory Phonetics The production of any speech sound involves the movement of an air stream. Most speech sounds are produced by pushing the air out of the lungs through the mouth (oral) and sometimes through the nose (nasal).

17. SPEECH ORGANS

21. Phonology Phonology deals with the system and pattern of speech sounds in a language. Phonology of a language is the system and pattern of speech sounds.

22. Phonology Phonological knowledge permits us to: produce sounds which form meaningful utterances, to recognize a “foreign” accent, to make up new words, To know what is or is not a sound in one’s language to know what different sound strings may represent

23. Phonetics vs Phonology Phonetics The study of speech sounds. Phonology The study of the way speech sounds form patterns.

24. Sequences of Phonemes b l ı kI b k ı k l ı bı l b k b ı l kb k ı l k ı l bı b l k k k b b l l ı ı possibleimpossible “I just bought a beautiful new blick” What is a blick? “I just bought a beautiful new bkli” WHAT!!

25. Sequences of Phonemes Your knowledge of English “tells” you that certain strings of phonemes are permissible and others are not. That’s why /bkli/ does not sound like an English word. It violates the restrictions on the sequencing of phonemes; i.e. it violates the phonological rules of English.

26. Delete a word-final /b/ when it occurs after a /m/ as in:But not! Rules of Phonology bomb crumb lamb tomb bombard crumble limber tumble

27. Morphology & Syntax Morphology deals with the combination of morphemes into words. Syntax deals with the combination of words into sentences.

28. What is the meaning of ‘meaning’? Learning a language includes learning the “agreed upon” meanings of certain strings of sounds and, Learning how to combine these meaningful units into larger units which also convey meaning.

29. Morphemes Morpheme is the smallest linguistic unit that has meaning. Morpheme is a grammatical unit in which there is an arbitrary union of sound and a meaning and, which cannot be further analysed.

30. Morphemes A morpheme may be represented by a single sound: e.g. the plural morpheme [s] in cat+s A morpheme may be represented by a syllable (monosyllabic): e.g. child+ish

31. Morphemes A morpheme may be represented by more than one syllable (polysyllabic): e.g. lady, water or three syllables: e.g. crocodile or four syllables: e.g. salamander

32. 32 Words Two basic ways to form words –Inflectional (e.g. English verbs) Open + ed = opened Open + ing = opening –Derivational (e.g. adverbs from adjectives, nouns from adjectives) Happy  happily Happy  happiness (nouns from adjectives)

33. 33 Syntax The study of classes of words and the rules that govern how the words can combine to make phrases and sentences.

34. 34 Basic classes of words Classes of words aka parts of speech (POS) –Nouns –Verbs –Adjectives –Adverbs The above classes of word belong to the type open class words We also have closed class words –Articles, pronouns, prepositions, particles, quantifiers, conjunctions

35. 35 Basic phrases A word from an open class can be used to form the basis of a phrase The basis of a phrase is called the head

36. 36 Examples of phrases Noun phrases –The manager of the institute –Her worry to pass the exams –Several students from the English Department Adjective phrases –easy to understand –mad as a dog –glad that he passed the exam

37. 37 Examples of phrases Adverb phrases –fast like the wind –outside the building Verb phrases –ate her sandwich –went to the doctor –believed what I told him

38. 38 “Complements” Notice that to be meaningful the verb “go”, for example requires a phrase for “location” –*John went –John went home Such phrases “complete” the meaning of the verb (or other type of head) and are called complements

39. 39 Inside the noun phrase NPs are used to refer to things: objects, places, concepts, events, qualities, etc NPs may consist of: –A single pronoun (he, she, etc) –A name or proper noun (John, Athens, etc) –A specifier and a noun –A qualifier and a noun –A specifier and a qualifier and a noun (e.g., the first three winners)

40. 40 Specifiers Specifiers indicate how many objects are described and also how these objects relate to the speaker Basis types of specifiers –Ordinals (e.g., first, second) –Cardinals (e.g., one, two) –Determiners (see next slide)

41. 41 Determiners Basic types of determiners –Articles (the, a, an) –Demonstratives (this, that, these, those) –Possessives (‘s, her, my, whose, etc) –Wh-determiners (which, what –in questions) –Quantifying determiners (some, every, most, no, any, etc.)

42. 42 Qualifiers Basic types of qualifiers –Adjectives Happy cat Angry feelings –Noun modifiers Cook book University hospitals

43. 43 Inside the verb phrase A simple VP –Adverbial modifier + head verb + complements Types of verbs –Auxiliary (be, do, have) –Modal (will, can, could) –Main (eat, work, think)

44. 44 Types of verb complements Intransitive verbs do not require complements Transitive verbs require an object as a complement (e.g. find a key) Transitive verbs allow passive forms (e.g. a key was found) Ditransitive verbs require one direct and on indirect object (e.g. give Mary a book)

45. 45 Other verb complements Clausal complements –Some verbs require clausal complements Mary knows that John left Prepositional phrase complements –Some verbs requires specific PP complements Mary gave the book to John –Others require any PP complement John put the book on the shelf/in the room/under the table

46. 46 Adjective phrases Simple –Angry, easy, etc Complex –Pleased with the prize –Angry at the committee –Willing to read the book Complex AdjP normally do not precede nouns, they are used as complements of verbs such as be or seem

47. 47 Adverbial phrases Indicators of –Degree –Location –Manner –The time of something (now, yesterday, etc) –Frequency –Duration Location in the sentence –Initial –Medial –Final

48. 48 Grammars and parsing What is syntactic parsing –Determining the syntactic structure of a sentence Basic steps –Identify sentence boundaries –Identify what part of speech is each word –Identify syntactic relations

49. Context Free Grammar S -> NP VP NP -> det (adj) N NP -> Proper N NP -> N VP -> V, VP -> V PP VP -> V NP VP -> V NP PP, PP -> Prep NP VP -> V NP NP LING 2000 - 2006 NLP 49

50. LING 2000 - 2006 NLP 50 Parses V PP VP S NP the mat satcat on NP Prep The cat sat on the mat Det N N

51. LING 2000 - 2006 NLP 51 Parses V PP VP S NP time an arrow flies like NP Prep Time flies like an arrow. N DetN

52. LING 2000 - 2006 NLP 52 Parses VNP VP S NP flies like an N Det Time flies like an arrow. N time arrow N

53. LING 2000 - 2006 NLP 53 Features C for Case, Subjective/Objective –She visited her. P for Person agreement, (1 st, 2 nd, 3 rd ) –I like him, You like him, He likes him, N for Number agreement, Subject/Verb –He likes him, They like him. G for Gender agreement, Subject/Verb –English, reflexive pronouns He washed himself. –Romance languages, det/noun T for Tense, –auxiliaries, sentential complements, etc. –* will finished is bad

54. LING 2000 - 2006 NLP 54 Probabilistic Context Free Grammars Adding probabilities Often, lexicalizing the probabilities

55. A PCFG S -> NP VP (0.5) S -> ADVP NP VP (0.5) NP -> det (adj) N (0.7) NP -> Proper N (0.15) NP -> N (0.15) VP -> V, (0.1); VP -> V PP (0.1) VP -> V NP (0.4); VP -> V NP PP (0.4) PP -> Prep NP (1) CIS 8590 – Fall 2008 NLP 55

56. A Lexicalized PCFG Sample rules: S_give -> NP VP_give (1.0) NP_friend -> det (adj) N_friend (1.0) NP_Sally -> ProperN_Sally (1.0) VP_give -> V_give NP NP (0.3) VP_give -> V_give NP PP_to (0.7) CIS 8590 – Fall 2008 NLP 56

57. Parsing Computational task: Given a set of grammar rules and a sentence, find a valid parse of the sentence (efficiently) Naively, you could try all possible combinations of rules until you get to a parse tree that has “S” at the root, and the right words at the leaves. But that takes exponential time in the number of words. CIS 8590 – Fall 2008 NLP 57

58. CKY Parsing (aka, CYK) CKY parsing is a dynamic programming solution I bring it up now because dynamic programming shows up all the time in NLP Dynamic programming: simplifying a complicated problem by breaking it down into simpler subproblems in a recursive manner CIS 8590 – Fall 2008 NLP 58

59. CKY – Basic Idea Let the input be a string S consisting of n characters: a 1... a n. Let the grammar contain r nonterminal symbols R 1... R r. This grammar contains the subset R s which is the set of start symbols. Let P[n,n,r] be an array of booleans. Initialize all elements of P to false. At each step, the algorithm sets P[i,j,k] to be true if the subsequence of words (span) starting from i of length j can be generated from R k We will start with spans of length 1 (individual words), and then proceed to increasingly larger spans, and determining which ones are valid given the smaller spans that have already been processed. CIS 8590 – Fall 2008 NLP 59

60. CKY Algorithm For each i = 1 to n For each unit production R j -> a i, set P[i,1,j] = true. For each i = 2 to n -- Length of span For each j = 1 to n-i+1 -- Start of span For each k = 1 to i-1 -- Partition of span For each production R A -> R B R C If P[j,k,B] and P[j+k,i-k,C] then set P[j,i,A] = true If any of P[1,n,x] is true (x is iterated over the set s, where s are all the indices for R s ) Then S is member of language Else S is not member of language CIS 8590 – Fall 2008 NLP 60

61. CKY In Action http://homepages.uni- tuebingen.de/student/martin.lazarov/demo s/cky.html CIS 8590 – Fall 2008 NLP 61

62. Finding the Best Parse With a PCFG (or lexicalized PCFG), it’s possible to score the trees to find the best (highest probability) parse Instead of a boolean array P, you would need to store weights (or probabilities) in the array; for the rest, the algorithm is almost identical. CIS 8590 – Fall 2008 NLP 62