1. 9/13/1999JHU CS 600.465/ Intro to NLP/Jan Hajic1 Textbooks you need Manning, C. D., Sch ü tze, H.: Foundations of Statistical Natural Language Processing. The MIT Press. 1999. ISBN 0-262-13360-1. [required - on order] Allen, J.: Natural Language Understanding. The Benjamins/Cummins Publishing Co. 1994. ISBN 0-8053-0334-0. [required - available] Jurafsky, D. and J. H. Martin: Speech and Language Processing. Prentice-Hall. 2000. ISBN 0-13-095069-6.Speech and Language Processing

2. 9/13/1999JHU CS 600.465/ Intro to NLP/Jan Hajic2 Other reading Charniak, E: –Statistical Language Learning. The MIT Press. 1996. ISBN 0-262-53141-0. Cover, T. M., Thomas, J. A.: –Elements of Information Theory. Wiley. 1991. ISBN 0-471-06259-6. Jelinek, F.: –Statistical Methods for Speech Recognition. The MIT Press. 1998. ISBN 0- 262-10066-5 Proceedings of major conferences: –ACL (Assoc. of Computational Linguistics) –EACL (European Chapter of ACL) –ANLP (Applied NLP) –COLING (Intl. Committee of Computational Linguistics)

3. 9/13/1999JHU CS 600.465/ Intro to NLP/Jan Hajic3 Course segments Intro & Probability & Information Theory (3) –The very basics: definitions, formulas, examples. Language Modeling (3) –n-gram models, parameter estimation –smoothing (EM algorithm) A Bit of Linguistics (3) –phonology, morphology, syntax, semantics, discourse Words and the Lexicon (3) –word classes, mutual information, bit of lexicography.

4. 9/13/1999JHU CS 600.465/ Intro to NLP/Jan Hajic4 Course segments (cont.) Hidden Markov Models (3) –background, algorithms, parameter estimation Tagging: Methods, Algorithms, Evaluation (8) –tagsets, morphology, lemmatization –HMM tagging, Transformation-based, Feature-based NL Grammars and Parsing: Data, Algorithms (9) –Grammars and Automata, Deterministic Parsing –Statistical parsing. Algorithms, parameterization, evaluation Applications (MT, ASR, IR, Q&A,...) (4)

5. 9/13/1999JHU CS 600.465/ Intro to NLP/Jan Hajic5 Goals of the HLT Computers would be a lot more useful if they could handle our email, do our library research, talk to us … But they are fazed by natural human language. How can we make computers have abilities to handle human language? (Or help them learn it as kids do?)

6. 9/13/1999JHU CS 600.465/ Intro to NLP/Jan Hajic6 A few applications of HLT Spelling correction, grammar checking … Better search engines Information extraction, gisting Psychotherapy; Harlequin romances; etc. New interfaces: –Speech recognition (and text-to-speech) –Dialogue systems (USS Enterprise onboard computer) –Machine translation; speech translation (the Babel tower??) Trans-lingual summarization, detection, extraction …

7. 9/13/1999JHU CS 600.465/ Intro to NLP/Jan Hajic7 Levels of Language Phonetics/phonology/morphology: what words (or subwords) are we dealing with? Syntax: What phrases are we dealing with? Which words modify one another? Semantics: What’s the literal meaning? Pragmatics: What should you conclude from the fact that I said something? How should you react?

8. 9/13/1999JHU CS 600.465/ Intro to NLP/Jan Hajic8 What’s hard – ambiguities, ambiguities, all different levels of ambiguities John stopped at the donut store on his way home from work. He thought a coffee was good every few hours. But it turned out to be too expensive there. [from J. Eisner] - donut: To get a donut (doughnut; spare tire) for his car? - Donut store: store where donuts shop? or is run by donuts? or looks like a big donut? or made of donut? - From work: Well, actually, he stopped there from hunger and exhaustion, not just from work. - Every few hours: That’s how often he thought it? Or that’s for coffee? - it: the particular coffee that was good every few hours? the donut store? the situation - Too expensive: too expensive for what? what are we supposed to conclude about what John did?

9. 9/13/1999JHU CS 600.465/ Intro to NLP/Jan Hajic9 NLP: The Main Issues Why is NLP difficult? –many “ words ”, many “ phenomena ” --> many “ rules ” OED: 400k words; Finnish lexicon (of forms): ~2. 10 7 sentences, clauses, phrases, constituents, coordination, negation, imperatives/questions, inflections, parts of speech, pronunciation, topic/focus, and much more! irregularity (exceptions, exceptions to the exceptions,...) potato -> potato es (tomato, hero,...); photo -> photo s, and even: both mango -> mango s or -> mango es Adjective / Noun order: new book, electrical engineering, general regulations, flower garden, garden flower,...: but Governor General

10. 9/13/1999JHU CS 600.465/ Intro to NLP/Jan Hajic10 Difficulties in NLP (cont.) –ambiguity books: NOUN or VERB? –you need many books vs. she books her flights online No left turn weekdays 4-6 pm / except transit vehicles (Charles Street at Cold Spring) –when may transit vehicles turn: Always? Never? Thank you for not smoking, drinking, eating or playing radios without earphones. (MTA bus) –Thank you for not eating without earphones?? –or even: Thank you for not drinking without earphones!? My neighbor ’ s hat was taken by wind. He tried to catch it. –...catch the wind or...catch the hat ?

11. 9/13/1999JHU CS 600.465/ Intro to NLP/Jan Hajic11 (Categorical) Rules or Statistics? Preferences: –clear cases: context clues: she books --> books is a verb –rule: if an ambiguous word (verb/nonverb) is preceded by a matching personal pronoun -> word is a verb – less clear cases: pronoun reference –she/he/it refers to the most recent noun or pronoun (?) (but maybe we can specify exceptions) –selectional: –catching hat >> catching wind (but why not?) –semantic: –never thank for drinking in a bus! (but what about the earphones?)

12. 9/13/1999JHU CS 600.465/ Intro to NLP/Jan Hajic12 Solutions Don ’ t guess if you know: morphology (inflections) lexicons (lists of words) unambiguous names perhaps some (really) fixed phrases syntactic rules? Use statistics (based on real-world data) for preferences (only?) No doubt: but this is the big question!

13. 9/13/1999JHU CS 600.465/ Intro to NLP/Jan Hajic13 Statistical NLP Imagine: –Each sentence W = { w 1, w 2,..., w n } gets a probability P(W|X) in a context X (think of it in the intuitive sense for now) –For every possible context X, sort all the imaginable sentences W according to P(W|X): –Ideal situation: best sentence (most probable in context X) NB: same for interpretation P(W) “ ungrammatical ” sentences

14. 9/13/1999JHU CS 600.465/ Intro to NLP/Jan Hajic14 Real World Situation Unable to specify set of grammatical sentences today using fixed “ categorical ” rules (maybe never, cf. arguments in MS) Use statistical “ model ” based on REAL WORLD DATA and care about the best sentence only (disregarding the “ grammaticality ” issue) best sentence P(W) W best W worst