Natural Language Processing Slides adapted from Pedro Domingos What’s the problem? Input? Natural Language Sentences Output? Parse Tree Semantic Interpretation (Logical Representation)

Example Applications Enables great user interfaces! Spelling and grammar checkers. Http://www.askjeeves.com/ Document understanding on the WWW. Spoken language control systems: banking, shopping Classification systems for messages, articles. Machine translation tools.

NLP Problem Areas Morphology: structure of words Syntactic interpretation (parsing): create a parse tree of a sentence. Semantic interpretation: translate a sentence into the representation language. Pragmatic interpretation: incorporate current situation into account. Disambiguation: there may be several interpretations. Choose the most probable

Some Difficult Examples From the newspapers: Squad helps dog bite victim. Helicopter powered by human flies. Levy won’t hurt the poor. Once-sagging cloth diaper industry saved by full dumps. Ambiguities: Lexical: meanings of ‘hot’, ‘back’. Syntactic: I heard the music in my room. Referential: The cat ate the mouse. It was ugly.

Parsing Context-free grammars: (2 + X) * (17 + Y) is in the grammar. EXPR -> NUMBER EXPR -> VARIABLE EXPR -> (EXPR + EXPR) EXPR -> (EXPR * EXPR) (2 + X) * (17 + Y) is in the grammar. (2 + (X)) is not. Why do we call them context-free?

Using CFG’s for Parsing Can natural language syntax be captured using a context-free grammar? Yes, no, sort of, for the most part, maybe. Words: nouns, adjectives, verbs, adverbs. Determiners: the, a, this, that Quantifiers: all, some, none Prepositions: in, onto, by, through Connectives: and, or, but, while. Words combine together into phrases: NP, VP

An Example Grammar S -> NP VP VP -> V NP NP -> NAME NP -> ART N ART -> a | the V -> ate | saw N -> cat | mouse NAME -> Sue | Tom

Example Parse The mouse saw Sue.

“Sue bought the cat biscuits Ambiguity “Sue bought the cat biscuits S -> NP VP VP -> V NP VP -> V NP NP NP -> N NP -> N N NP -> Det NP Det -> the V -> ate | saw | bought N -> cat | mouse |biscuits | Sue | Tom

Example: Chart Parsing Three main data structures: a chart, a key list, and a set of edges Chart: Name of terminal or non-terminal 4 length 3 2 1 1 2 3 4 Starting points

Key List and Edges Key list: Push down stack of chart entries “the” “box” “floats” Edges: rules that can be applied to chart entries to build up larger entries 4 length 3 box 2 floats 1 the 1 2 3 4 detthe o

Chart Parsing Algorithm Loop while entries in key list 1. Remove entry from key list 2. If entry already in chart, Add edge list Break 3. Add entry from key list to chart 4. For all rules that begin with entry’s type, add an edge for that rule 5. For all edges that need the entry next, add an extended edge (see algorithm on right) 6. If the edge is finished, add an entry to the key list with type, start point, length, and edge list To extend an edge with chart entry c Create a new edge e’ Set start (e’) to start (e) Set end(e’) to end(e) Set rule(e’) to rule(e) with “o” moved beyond c. Set the righthandside(e’) to the righthandside(e)+c

Try it S -> NP VP VP -> V NP -> Det N Det -> the N -> box V -> floats

Semantic Interpretation Our goal: to translate sentences into a logical form. But: sentences convey more than true/false: It will rain in Seattle tomorrow. Will it rain in Seattle tomorrow? A sentence can be analyzed by: propositional content, and speech act: tell, ask, request, deny, suggest

Propositional Content We develop a logic-like language for representing propositional content: Word-sense ambiguity Scope ambiguity Proper names --> objects (John, Alon) Nouns --> unary predicates (woman, house) Verbs --> transitive: binary predicates (find, go) intransitive: unary predicates (laugh, cry) Quantifiers: most, some Example: Mary: Loves(John, Mary)

Statistical NLP(see book by Charniak, “Statistical Language Learning”, MIT Press, 1993 Consider the problem of tagging part-of-speech: “The box floats” “The”  Det; “Box”  N; “Floats”  V; Given a sentence w(1,n), where w(i) is the i-th word, we want to find tags t(i) assigned to each word w(i)

The Equations Find the t(1,n) that maximizes Assume that P[t(1,n)|w(1,n)]=P[w(1,n)|t(1,n)]/P(w(1,n)) So, only need to maximize P[w(1,n)|t(1,n)] Assume that A word depends only on previous tag A tag depends only on previous tag We have: P[w(j)|w(1,j-1),t(1,j)]=P[w(j)|t(j)], and P[t(j)|w(1,j-1),t(1,j-1)] = P(t(j)|t(j-1)] Thus, want to maximize P[w(n)|t(n-1)]*P[t(n+1)|t(n)]*P[w(n-1)|t(n-2)]*P[t(n)|t(n-1)]…

Example “The box floats”: given a corpus (a training set) Assignment one: T(1)=det, T(2) = V, T(3)=V P(V|det) is rather low, so is P(V|V). Thus is less likely compared to Assignment two: T(t)=det, T(2) = N; t(3) = V P(N|det) is high, and P(V|N) is high, thus is more likely! In general, can use Hidden Markov Models to find probabilities floats the box V det N

Experiments Charniak and Colleagues did some experiments on a collection of documents called the “Brown Corpus”, where tags are assigned by hand. 90% of the corpus are used for training and the other 10% for testing They show they can get 95% correctness with HMM’s. A really simple algorithm: assign t to w by the highest probability tag P(t|w)  91% correctness!

Natural Language Summary Parsing: context free grammars with features. Semantic interpretation: Translate sentences into logic-like language Use additional domain knowledge for word-sense disambiguation. Use context to disambiguate references.