1. 9/8/20151 Natural Language Processing Lecture Notes 1

2. 9/8/20152 Today Administration and Syllabus –course web page Introduction

3. 9/8/20153 Natural Language Processing What is it? –What goes into getting computers to perform useful and interesting tasks involving human languages. –Secondarily: insights that such computational work give us into human languages and human processing of language.

4. 9/8/20154 Natural Language Processing Foundations are in computer science (AI, theory, algorithms,…); linguistics; mathematics; logic and statistics; and psychology

5. 9/8/20155 Why Should You Care? Two trends 1.An enormous amount of knowledge is now available in machine readable form as natural language text 2.Conversational agents are becoming an important form of human-computer communication

6. 9/8/20156 Knowledge of Language Words (words and their composition) Syntax (structure of sentences) Semantics (explicit meaning of sentence) Discourse and pragmatics (implicit and contextual meaning)

7. 9/8/20157 Small Applications Line breakers Hyphenators Spelling correctors Optical Character Recognition software Grammar and style checkers

8. 9/8/20158 Big Applications Question answering Conversational agents Text summarization Machine translation

9. 9/8/20159 Note NLP, as in many areas of AI: –We’re often dealing with ill-defined problems –We don’t often come up with perfect solutions/algorithms –We can’t let either of those facts get in our way

10. 9/8/201510 Course Material We’ll be intermingling discussions of: –Linguistic topics Syntax and meaning representations –Computational techniques Context-free grammars –Applications Translation and QA systems

11. 9/8/201511 Chapter 1 Knowledge of language Ambiguity Models and algorithms History

12. 9/8/201512 Knowledge of Language Phonetics and phonology: speech sounds, their production, and the rule systems that govern their use Morphology: words and their composition from more basic units –Cat, cats (inflectional morphology) –Child, children –Friend, friendly (derivational morphology)

13. 9/8/201513 Knowledge of Language Syntax: the structuring of words into legal larger phrases and sentences

14. 9/8/201514 Semantics The meaning of words and phrases –Lexical semantics: the study of the meanings of words –Compositional semantics: how to combine word meanings –Word-sense disambiguation River bank vs. financial bank

15. 9/8/201515 Pragmatics Indirect speech acts: –Do you have a stapler? Presupposition: –Have you stopped beating your wife? Deixis and point of view: –Zoe was angry at Joe. Where was he? Implicature: -Yes, there are 3 flights to Boston. In fact, there are 4. * The general was assassinated. In fact, he isn’t dead.

16. 9/8/201516 Discourse Utterance interpretation in the context of the text or dialog –Sue took the trip to New York. She had a great time there. Sue/she; New York/there; took/had (time)

17. 9/8/201517 Ambiguity Almost all of the non-trivial tasks performed by NLP systems are ambiguity resolution tasks There is ambiguity at all levels of language

18. 9/8/201518 Ambiguity I saw the woman with the telescope Syntactically ambiguous: –I saw (NP the woman with the telescope) –I saw (NP the woman) (PP with the telescope)

19. 9/8/201519 “I made her duck” I cooked waterfowl for her I cooked waterfowl belonging to her I create the duck she owns I caused her to lower her head quickly… Part of speech tagging: is “duck” a noun or verb? Parsing syntactic structure: is “her” part of the “duck” NP? Word-sense disambiguation (lexical semantics): does “make” mean create, lower head, or cook?

20. 9/8/201520 Dealing with Ambiguity Two approaches: –Tightly coupled interaction among processing levels; knowledge from other levels can help decide among choices at ambiguous levels. –Pipeline processing Most NLP systems are probabilistic: they make the most likely choices

21. 9/8/201521 Models and Algorithms Models (as we are using the term here): –Formalisms to represent linguistic knowledge Algorithms: –Used to manipulate the representations and produce the desired behavior choosing among possibilities and combining pieces

22. 9/8/201522 Models State Machines: finite state automata, finite state transducers Formal rule systems: context free grammars Logical formalisms: first-order predicate calculus; higher-order logics Models of uncertainty: Bayesian probability theory

23. 9/8/201523 Algorithms Many of the algorithms that we’ll study will turn out to be transducers; algorithms that take one kind of structure as input and output another.

24. 9/8/201524 Algorithms In particular.. –State-space search To manage the problem of making choices during processing when we lack the information needed to make the right choice –Dynamic programming To avoid having to redo work during the course of a state-space search –Machine Learning (classifiers, EM, etc)

25. 9/8/201525 State Space Search States represent pairings of partially processed inputs with partially constructed answers –E.g. sentence + partial parse tree Goal is to arrive at the right/best structure after having processed all the input. –E.g. the best parse tree spanning the sentence As with most interesting AI problems the spaces are too large and the criteria for “bestness” are difficult to encode (so heuristics, probabilities)

26. 9/8/201526 Dynamic Programming Don’t do the same work over and over. Avoid this by building and making use of solutions to sub-problems that must be invariant across all parts of the space.