CS460/IT632 Natural Language Processing/Language Technology for the Web Lecture 24 (14/04/06) Prof. Pushpak Bhattacharyya IIT Bombay Word Sense Disambiguation

Word sense disambiguation (WSD) Cardinal problem in NLP also called as “Lexical Disambiguation” Other disambiguation are: –Structure disambiguation –Prepositional phrase attachment

Example I saw the mountain with a telescope. I saw the dog with two tails

WSD and POS tagging POS tagging is also disambiguation (local context needed). WSD is sense finding (long distance sense needed) Example –“The play could not be held due to rain, which flooded the open air theatre.”

WSD usage WSD is crucial for –Information extraction QA Summary generation –IR

Ambiguity Ambiguity arises from / refers to (written) –Homography The concept have no relation but are represented by same word/graph. –Polysemy The sense are related (many semantics).

Example Homography – “Bank” River side of Financial institute Depend (verb) Polysemy –“Fall” The tree chopped at he root, fall. The kingdom completely mismanaged, fall.

Approaches to WSD Knowledge based –Uses human-crafted knowledge base Data driven –Uses training data and machine learning.

Knowledge based WSD Resources needed –Argument-selection preference –Sense repository Wordnet Ontology

Example for Knowledge based –“The restaurant serves many delicious dishes.” –“Washing dishes after meals is a pain.” Hit the argument frame of the verb. Verb have arguments: Agent and Object. Example: –Serve:- Agent: dishObject: restaurant –Wash:- Agent: ?Object: utensil, car

Pitfalls Pitfall in Argument-selection preference: –Metaphor Example: –“I had to eat lots of cabbage in France” –“I had to eat my words” –“When she saw the demon eat the building, she ran in panic.” Here selection preference won’t work.

Lesk algorithm Another knowledge based approach. Uses Wordnet, heavily. Steps: –Create words from context –Watch words in Wordnet –Find intersection to obtain the sense Performance is not so good.

Data Driven approaches Have to use sense marked training data. Most common is “Semcor corpus”. Supervised category: –Naïve Bayes –Information Theoretic Un-supervised category: –Clustering

Naïve Bayes S ^ = argmax P(s|w) s Є S P(s|w) = P(w|s). P(s) P(w) Naïve Bayes uses following formula: