1. 1 LING 6932 Spring 2007 LING 6932 Topics in Computational Linguistics Hana Filip Lecture 1: Introduction to Field, History, Quick Review of Regular Expressions, Start of Finite Automata

2. 2 LING 6932 Spring 2007 Today 1/11 Week 1 Overview and history of the field Knowledge of language The role of ambiguity Models and Algorithms Eliza, Turing, and conversational agents History of speech and language processing Administration Overview of course topics 1 week on each topic http://plaza.ufl.edu/hfilip/http://plaza.ufl.edu/hfilip/ (later also WebCT) Regular expressions Finite State Automata Deterministic Recognition of Finite State Automata

3. 3 LING 6932 Spring 2007 Computational Linguistics and Natural Language Processing What is it? Getting computers to perform useful tasks involving human languages for: –Enabling human-machine communication –Improving human-human communication –Doing stuff with language objects Examples: –Question Answering http://www.humana-military.com/ –Machine Translation http://www.google.com/language_tools –Spoken Conversational Agents http://www.cs.rochester.edu/research/trains/ http://www.cs.rochester.edu/research/trains/ The Trains Project: James Allen (University of Rochester)

4. 4 LING 6932 Spring 2007 Kinds of knowledge needed? Consider the following interaction with HAL 9000 the computer from 2001: A Space Odyssey 2001: A Space Odyssey (1968) is a science fiction novel by Arthur C. Clarke which was developed concurrently with Stanley Kubrick's film version HAL = Heuristically programmed ALgorithmic computer Dave: Open the pod bay doors, Hal. HAL: I’m sorry Dave, I’m afraid I can’t do that.

5. 5 LING 6932 Spring 2007 Knowledge needed to build HAL? Speech recognition and synthesis Dictionaries (how words are pronounced) Phonetics (how to recognize/produce each sound of English) Natural language understanding Knowledge of the English words involved –What they mean - lexical semantics –What the smallest meaning bearing units of words are - morphology –How they combine (what is a `pod bay door’?)  Knowledge of syntactic structure –I’m I do, Sorry that afraid Dave I’m can’t Knowledge of compositional semantics –How the meaning of a complex expression is determined by its structure and the meanings of its constituents

6. 6 LING 6932 Spring 2007 What’s needed? Dialog and pragmatic knowledge “open the door” is a REQUEST (as opposed to a STATEMENT or information-question) It is polite to respond, even if you’re planning to kill someone. It is polite to pretend to want to be cooperative (I’m afraid, I can’t…) What is `that’ in `I can’t do that’? Even a system to book airline flights needs much of this kind of knowledge

7. 7 LING 6932 Spring 2007 Modern conversational agents Switchboard Receptionists Implications: need to know how a native or non-native speaker of English is likely to pronounce a random American name. Airline and other simple dialogue systems Lots of fabulously fun pragmatic issues –Grounding –Error handling In general Now that there are commercial installations Big incentive to use linguistic or any other kind of knowledge if it actually improves performance

8. 8 LING 6932 Spring 2007 Eliza: Weizenbaum (1966) http://i5.nyu.edu/~mm64/x52.9265/january1966.html User: You are like my father in some ways. ELIZA: What resemblance do you see User: You are not very aggressive but I think you don’t want me to notice that. ELIZA: What makes you think I am not aggressive User: You don’t argue with me ELIZA: Why do you think I don’t argue with you User: You are afraid of me ELIZA: Does it please you to believe I am afraid of you

9. 9 LING 6932 Spring 2007 Question Answering What does “door” mean? What year was Abraham Lincoln born? How many states were in the United States when Lincoln was born? Was there a military draft during the Hoover administration? What do US scientists think about whether human cloning should be legal?

10. 10 LING 6932 Spring 2007 Modern QA systems Still in infancy Simple factoid questions beginning to work OK Annual government-sponsored “bakeoff” called TREC TREC = Text REtrieval Conference … to encourage research in information retrieval from large text collections http://trec.nist.gov/

11. 11 LING 6932 Spring 2007 Machine Translation Chinese gloss: Dai-yu alone on bed top think-of-with-gratitude Bao-chai again listen to window outside bamboo tip plantain leaf of on-top rain sound sigh drop clear cold penetrate curtain not feeling again fall down tears come Hawkes translation: As she lay there alone, Dai-yu’s thoughts turned to Bao-chai… Then she listened to the insistent rustle of the rain on the bamboos and plantains outside her window. The coldness penetrated the curtains of her bed. Almost without noticing it she had begun to cry.

12. 12 LING 6932 Spring 2007 Machine Translation The Story of the Stone or the Dream of the Red Chamber (Cao Xueqin 1792) classic novel from the Qing dynasty, considered the greatest work of Chinese fiction Issues: (“Language Differences”) Sentence segmentation Zero anaphoric pronouns Coding of tense/aspect  Penetrate -> penetrated Stylistic differences across languages –Bamboo tip plantain leaf -> bamboos and plantains Cultural knowledge –Curtain -> curtains of her bed

13. 13 LING 6932 Spring 2007 Ambiguity Computational linguists are obsessed with ambiguity Ambiguity is a fundamental problem of computational linguistics Resolving ambiguity is a crucial goal

14. 14 LING 6932 Spring 2007 Ambiguity Find at least 5 meanings of this sentence: I made her duck

15. 15 LING 6932 Spring 2007 Ambiguity Find at least 5 meanings of this sentence: I made her duck I cooked waterfowl for her benefit (to eat) I cooked waterfowl belonging to her I created the (plaster?) duck she owns I caused her to quickly lower her head or body I waved my magic wand and turned her into undifferentiated waterfowl At least one other meaning that’s inappropriate for gentle company.

16. 16 LING 6932 Spring 2007 Ambiguity is Pervasive I caused her to quickly lower her head or body Lexical category: “duck” can be a N or V I cooked waterfowl belonging to her. Lexical category: “her” can be a possessive (“of her”) or dative (“for her”) pronoun I made the (plaster) duck statue she owns Lexical Semantics: “make” can mean “create” or “cook” Lexical disambiguation part-of-speech tagging word sense disambiguation Syntactic disambiguation: “her duck” two syntactic phrases: NP VP one syntactic phrase: [Det N] NP

17. 17 LING 6932 Spring 2007 Ambiguity is Pervasive Grammar: “Make” can be: Transitive: (verb has a noun direct object) –I cooked [waterfowl belonging to her] Ditransitive: (verb has 2 noun objects) –I made [her] (into) [undifferentiated waterfowl] Action-transitive (verb has a direct object and another verb) I caused [her] [to move her body]

18. 18 LING 6932 Spring 2007 Ambiguity is Pervasive Phonetics! I mate or duck I’m eight or duck Eye maid; her duck Aye mate, her duck I maid her duck I’m aid her duck I mate her duck I’m ate her duck I’m ate or duck I mate or duck

19. 19 LING 6932 Spring 2007 Models and Algorithms Models: formalisms used to capture the various kinds of linguistic structure. State machines (fsa, transducers, markov models) Formal rule systems (context-free grammars, feature systems) Logic (predicate calculus, inference) Probabilistic versions of all of these + others (gaussian mixture models, probabilistic relational models, etc etc) Algorithms used to manipulate representations to create structure.

20. 20 LING 6932 Spring 2007 Hidden Markov Chain A hidden Markov model (HMM) is a statistical model in which the system being modeled is assumed to be a Markov process with unknown parameters, and the challenge is to determine the hidden parameters from the observable parameters.statistical modelMarkov processobservable x: hidden states (weather: sunny, cloudy, rainy) y: observable outputs (your friend: walking in the park, shopping, cleaning the apartment) a: transition probabilities b:output probabilities

21. 21 LING 6932 Spring 2007 Context-free Grammar Chomsky 1957 “Syntactic Structures” In linguistics and computer science, a context-free grammar (CFG) is a formal grammar in which every production rule is of the form linguistics computer science formal grammar V—> w where V is a nonterminal symbol and w is a string consisting of terminals and/or non-terminals. nonterminal symbol The term "context-free" expresses the fact that the non-terminal V can always be replaced by w, regardless of the context in which it occurs. A formal language is context-free if there is a context-free grammar that generates it. formal language context-free

22. 22 LING 6932 Spring 2007 Language, Thought, Understanding A Gedanken Experiment: Turing Test 1950 paper "Computing Machinery and Intelligence" Alan Turing (1912-1954 ) founder of computer science, mathematician, philosopher, cryptographer Question “can a machine think” is not operational. Operational version: 2 people and a computer Interrogator talks to contestant and computer via teletype Task of machine is to convince interrogator it is human Task of contestant is to convince interrogator she and not machine is human.

23. 23 LING 6932 Spring 2007 Eliza: Weizenbaum (1966) User: You are like my father in some ways. ELIZA: What resemblance do you see User: You are not very aggressive but I think you don’t want me to notice that. ELIZA: What makes you think I am not aggressive User: You don’t argue with me ELIZA: Why do you think I don’t argue with you User: You are afraid of me ELIZA: Does it please you to believe I am afraid of you

24. 24 LING 6932 Spring 2007 History: foundational insights 1940s-1950s Automaton: Turing 1936 McCulloch-Pitts neuron (1943) –http://diwww.epfl.ch/mantra/tutorial/english/mcpits/ht ml/http://diwww.epfl.ch/mantra/tutorial/english/mcpits/ht ml/ Kleene (1951/1956) Shannon (1948) link between automata and Markov models Chomsky (1956)/Backus (1959)/Naur(1960): CFG Probabilistic/Information-theoretic models Shannon (1948) Bell Labs speech recognition (1952)

25. 25 LING 6932 Spring 2007 History: the two camps: 1957-1970 Symbolic Zellig Harris 1958 Transformation and Discourse Analysis Project - first parser? –Cascade of finite-state transducers Chomsky AI workshop at Dartmouth (McCarthy, Minsky, Shannon, Rochester) Newell and Simon: Logic Theorist, General Problem Solver Statistical Bledsoe and Browning (1959): Bayesian OCR Mosteller and Wallace (1964): Bayesian authorship attribution Denes (1959): ASR combining grammar and acoustic probability

26. 26 LING 6932 Spring 2007 Four paradigms: 1970-1983 Stochastic Hidden Markov Model 1972 –Independent application of Baker (CMU) and Jelinek/Bahl/Mercer lab (IBM) following work of Baum and colleagues at IDA Logic-based Colmerauer (1970,1975) Q-systems Definite Clause Grammars (Pereira and Warren 1980) Kay (1979) functional grammar, Bresnan and Kaplan (1982) unification Natural language understanding Winograd (1972) Shrdlu Schank and Abelson (1977) scripts, story understanding Influence of case-role work of Fillmore (1968) via Simmons (1973), Schank. Discourse Modeling Grosz and colleagues: discourse structure and focus Perrault and Allen (1980) BDI model

27. 27 LING 6932 Spring 2007 Empiricism and Finite State Redux: 1983-1993 Finite State Models Kaplan and Kay (1981): Phonology/Morphology Church (1980): Syntax Return of Probabilistic Models: Corpora created for language tasks Early statistical versions of NLP applications (parsing, tagging, machine translation) Increased focus on methodological rigor: –Can’t test your hypothesis on the data you used to build it! –Training sets and test sets

28. 28 LING 6932 Spring 2007 The field comes together: 1994- 2006 Statistical models standard ACL conference: –1990: 39 articles 1 statistical –2003 62 articles 48 statistical Machine learning techniques key Information retrieval meets NLP Unified field: NLP, Machine Translation (MT), Automatic Speech Recognition (ASR), Text-to-Speech (TTS), Dialog, Information Retrieval (IR)

29. 29 LING 6932 Spring 2007 Language, Thought, Understanding A Gedanken Experiment: Turing Test 1950 paper "Computing Machinery and Intelligence" Alan Turing (1912-1954 ) founder of computer science, mathematician, philosopher, cryptographer Question “can a machine think” is not operational. Operational version: 2 people and a computer Interrogator talks to contestant and computer via teletype Task of machine is to convince interrogator it is human Task of contestant is to convince interrogator she and not machine is human.

30. 30 LING 6932 Spring 2007 Eliza: Weizenbaum (1966) User: You are like my father in some ways. ELIZA: What resemblance do you see User: You are not very aggressive but I think you don’t want me to notice that. ELIZA: What makes you think I am not aggressive User: You don’t argue with me ELIZA: Why do you think I don’t argue with you User: You are afraid of me ELIZA: Does it please you to believe I am afraid of you

31. 31 LING 6932 Spring 2007 History: foundational insights 1940s-1950s Automaton: Turing 1936 McCulloch-Pitts neuron (1943) –http://diwww.epfl.ch/mantra/tutorial/english/mcpits/ht ml/http://diwww.epfl.ch/mantra/tutorial/english/mcpits/ht ml/ Kleene (1951/1956) Shannon (1948) link between automata and Markov models Chomsky (1956)/Backus (1959)/Naur(1960): CFG Probabilistic/Information-theoretic models Shannon (1948) Bell Labs speech recognition (1952)

32. 32 LING 6932 Spring 2007 History: the two camps: 1957-1970 Symbolic Zellig Harris 1958 Transformation and Discourse Analysis Project - first parser? –Cascade of finite-state transducers Chomsky AI workshop at Dartmouth (McCarthy, Minsky, Shannon, Rochester) Newell and Simon: Logic Theorist, General Problem Solver Statistical Bledsoe and Browning (1959): Bayesian OCR Mosteller and Wallace (1964): Bayesian authorship attribution Denes (1959): ASR combining grammar and acoustic probability

33. 33 LING 6932 Spring 2007 Four paradigms: 1970-1983 Stochastic Hidden Markov Model 1972 –Independent application of Baker (CMU) and Jelinek/Bahl/Mercer lab (IBM) following work of Baum and colleagues at IDA Logic-based Colmerauer (1970,1975) Q-systems Definite Clause Grammars (Pereira and Warren 1980) Kay (1979) functional grammar, Bresnan and Kaplan (1982) unification Natural language understanding Winograd (1972) Shrdlu Schank and Abelson (1977) scripts, story understanding Influence of case-role work of Fillmore (1968) via Simmons (1973), Schank. Discourse Modeling Grosz and colleagues: discourse structure and focus Perrault and Allen (1980) BDI model

34. 34 LING 6932 Spring 2007 Empiricism and Finite State Redux: 1983-1993 Finite State Models Kaplan and Kay (1981): Phonology/Morphology Church (1980): Syntax Return of Probabilistic Models: Corpora created for language tasks Early statistical versions of NLP applications (parsing, tagging, machine translation) Increased focus on methodological rigor: –Can’t test your hypothesis on the data you used to build it! –Training sets and test sets

35. 35 LING 6932 Spring 2007 The field comes together: 1994- 2006 Statistical models standard ACL conference: –1990: 39 articles 1 statistical –2003 62 articles 48 statistical Machine learning techniques key Information retrieval meets NLP Unified field: NLP, Machine Translation (MT), Automatic Speech Recognition (ASR), Text-to-Speech (TTS), Dialog, Information Retrieval (IR)

36. 36 LING 6932 Spring 2007 Some brief demos Machine Translation http://translate.google.com/translate_t Text-To-Speech: http://www- 306.ibm.com/software/pervasive/tech/demos/tts.shtml Question Answering (LCC): http://www.languagecomputer.com/demos/question_an swering/internet_demo/more_examples.html