1. 현재 언어처리 기술 현황과 통 계적 접근, 사용하는 이유 3 주 강의

2. The Dream It’d be great if machines could –Process our email (usefully) –Translate languages accurately –Help us manage, summarize, and aggregate information –Use speech as a UI (when needed) –Talk to us / listen to us But they can’t: –Language is complex, ambiguous, flexible, and subtle –Good solutions need linguistics and machine learning knowledge So:

3. What is NLP? Fundamental goal: deep understand of broad language –Not just string processing or keyword matching! End systems that we want to build: –Ambitious: speech recognition, machine translation, information extraction, dialog interfaces, question answering, trend finding … –Modest: spelling correction, text categorization…

4. Speech Systems Automatic Speech Recognition (ASR) –Audio in, text out –SOTA: 0.3% for digit strings, 5% dictation, 50%+ TV Text to Speech (TTS) –Text in, audio out –SOTA: totally intelligible (if sometimes unnatural) Speech systems currently: –Model the speech signal –Model language

5. Machine Translation Translation systems encode: –Something about fluent language –Something about how two languages correspond (middle of term) SOTA: for easy language pairs, better than nothing, but more an understanding aid than a replacement for human translators

6. Information Extraction Information Extraction (IE) –Unstructured text to database entries –SOTA: perhaps 70% accuracy for multi-sentence temples, 90%+ for single easy fields

7. Question Answering Question Answering: –More than search –Ask general comprehension questions of a document collection –Can be really easy: “What’s the capital of Wyoming?” –Can be harder: “How many US states’ capitals are also their largest cities?” –Can be open ended: “What are the main issues in the global warming debate?” SOTA: Can do factoids, even when text isn’t a perfect match

8. What is nearby NLP? Computational Linguistics –Using computational methods to learn more about how language works –We end up doing this and using it Cognitive Science –Figuring out how the human brain works –Includes the bits that do language –Humans: the only working NLP prototype! Speech? –Mapping audio signals to text –Traditionally separate from NLP, converging? –Two components: acoustic models and language models –Language models in the domain of stat NLP

9. What is this Class? Three aspects to the course: –Linguistic Issues What are the range of language phenomena? What are the knowledge sources that let us disambiguate? What representations are appropriate? –Technical Methods Learning and parameter estimation Increasingly complex model structures Efficient algorithms: dynamic programming, search –Engineering Methods Issues of scale Sometimes, very ugly hacks We’ll focus on what makes the problems hard, and what works in practice…

10. Class Requirements and Goals Class requirements –Uses a variety of skills / knowledge: Basic probability and statistics Basic linguistics background Decent coding skills (Java) –Most people are probably missing one of the above –We’ll address some review concepts with sections, TBD Class goals –Learn the issues and techniques of statistical NLP –Build the real tools used in NLP (language models, taggers, parsers, translation systems) –Be able to read current research papers in the field –See where the gaping holes in the field are!

11. Rational versus Empiricist Approaches to Language (I) Question: What prior knowledge should be built into our models of NLP? Rationalist Answer: A significant part of the knowledge in the human mind is not derived by the senses but is fixed in advance, presumably by genetic inheritance (Chomsky: poverty of the stimulus). Empiricist Answer: The brain is able to perform association, pattern recognition, and generalization and, thus, the structures of Natural Language can be learned.

12. Rational versus Empiricist Approaches to Language (II) Chomskyan/generative linguists seek to describe the language module of the human mind (the Ilanguage) for which data such as text (the Elanguage) provide only indirect evidence, which can be supplemented by native speakers intuitions. Empiricists approaches are interested in describing the E-language as it actually occurs. Chomskyans make a distinction between linguistic competence and linguistic performance. They believe that linguistic competence can be described in isolation while Empiricists reject this notion.

13. Empiricist Seeks methods that can work on raw text as it exists –Knowledge induction (automatic learning), not by disambiguation American structuralism –The work of Shannon –Assign probabilities on linguistic events compared to concentrating on categorical judgments about rare types of sentences

14. 의문 언어학이 대답해야 하는 것 – 사람은 무엇을 말하는가 ? – 세상에 대해 무엇을 말하고, 묻고, 요구하는가 ? 인간이 어떻게 언어를 배우고, 또 실시간에 언어 를 생성하고 이해하는가 ? 문법적으로는 바르나 어색한 문장 –In additions to this, she insisted that women were regarded as a different existence from men unfairly. ( 언어학적 논쟁은 알아서 볼 것 ) –take a while, sort of/kind of, I kind of love you.( 고어 )

15. Some Early NLP History 1950’s: –Foundational work: automata, information theory, etc. –First speech systems –Machine translation (MT) hugely funded by military (imagine that) Toy models: MT using basically word-substitution –Optimism! 1960’s and 1970’s: NLP Winter –Bar-Hillel (FAHQT) and ALPAC reports kills MT –Work shifts to deeper models, syntax –… but toy domains / grammars (SHRDLU, LUNAR) 1980’s: The Empirical Revolution –Expectations get reset –Corpus-based methods become central –Deep analysis often traded for robust and simple approximations –Evaluate everything

16. Today’s Approach to NLP From ~1970-1989, people were concerned with the science of the mind and built small (toy) systems that attempted to behave intelligently. Recently, there has been more interest on engineering practical solutions using automatic learning (knowledge induction). While Chomskyans tend to concentrate on categorical judgements about very rare types of sentences, statistical NLP practitioners concentrate on common types of sentences.

17. Why is NLP Difficult? NLP is difficult because Natural Language is highly ambiguous. Example: “The company is training workers” has 2 or more parse trees (i.e., syntactic analyses). “List the sales of the products produced in 1973 with the products produced in 1972” has 455 parses. Therefore, a practical NLP system must be good at making disambiguation decisions of word sense, word category, syntactic structure, and semantic scope.

18. Methods that don’t work well Maximizing coverage while minimizing ambiguity is inconsistent with symbolic NLP. Furthermore, hand-coded syntactic constraints and preference rules are time consuming to build, do not scale up well and are brittle in the face of the extensive use of metaphor in language. Example: if we code animate being --> swallow --> physical object I swallowed his story, hook, line, and sinker. The supernova swallowed the planet.

19. Classical NLP: Parsing Write symbolic or logical rules: Use deduction systems to prove parses from words –Minimal grammar on “Fed raises” sentence: 36 parses –Simple 10-rule grammar: 592 parses –Real-size grammar: many millions of parses This scaled very badly, didn’t yield broad-coverage tools

20. NLP: Annotation Much of NLP is annotating text with structure which specifies how it’s assembled. –Syntax: grammatical structure –Semantics: “meaning,” either lexical or compositional

21. What Made NLP Hard? The core problems: –Ambiguity –Sparsity –Scale –Unmodeled Variables

22. Problem: Ambiguities Headlines: –Iraqi Head Seeks Arms –Ban on Nude Dancing on Governor’s Desk –Juvenile Court to Try Shooting Defendant –Teacher Strikes Idle Kids –Stolen Painting Found by Tree –Kids Make Nutritious Snacks –Local HS Dropouts Cut in Half –Hospitals Are Sued by 7 Foot Doctors – 일본 차는 인도로 인도로 – 축구장서 날아온 공이 자전거에 맞아 넘어져 숨졌다면 … Why are these funny?

23. Syntactic Ambiguities Maybe we’re sunk on funny headlines, but normal, borin g sentences are unambiguous? Our company is training workers. Fed raises interest rates 0.5 % in a measure against inflation

24. Dark Ambiguities Dark ambiguities: most analyses are shockingly bad (meaning, they don’t have an interpretation you can get your mind around) Unknown words and new usages Solution: We need mechanisms to focus attention on the best ones, probabilistic techniques do this

25. Semantic Ambiguities Even correct tree-structured syntactic analyses don’t always nail down the meaning Every morning someone’s alarm clock wakes me up John’s boss said he was doing better

26. Other Levels of Language Tokenization/morphology: –What are the words, what is the sub-word structure? –Often simple rules work (period after “Mr.” isn’t sentence break) –Relatively easy in English, other languages are harder: Segmentation Morphology Discourse: how do sentences relate to each other? Pragmatics: what intent is expressed by the literal meaning, how to react to an utterance? Phonetics: acoustics and physical production of sounds Phonology: how sounds pattern in a language

27. Disambiguation for Applications Sometimes life is easy –Can do text classification pretty well just knowing the set of words used in the document, same for authorship attribution –Word-sense disambiguation not usually needed for web search because of majority effects or intersection effects (“jaguar habitat” isn’t the car) Sometimes only certain ambiguities are relevant Other times, all levels can be relevant (e.g., translation) he hoped to record a world record

28. Problem: Scale People did know that language was ambiguous! –…but they hoped that all interpretations would be “good” ones (or ruled out pragmatically) –…they didn’t realize how bad it would be

29. Corpora A corpus is a collection of text –Often annotated in some way –Sometimes just lots of text –Balanced vs. uniform corpora Examples –Newswire collections: 500M+ words –Brown corpus: 1M words of tagged “balanced” text –Penn Treebank: 1M words of parsed WSJ –Canadian Hansards: 10M+ words of aligned French / English sentences –The Web: billions of words of who knows what

30. Corpus-Based Methods A corpus like a treebank gives us three important tools: –It gives us broad coverage

31. Corpus-Based Methods –It gives us statistical information This is a very different kind of subject/object asymmetry than what many linguists are interested in.

32. Corpus-Based Methods –It lets us check our answers!

33. Problem: Sparsity However: sparsity is always a problem –New unigram (word), bigram (word pair), and rule rates in newswire

34. The (Effective) NLP Cycle Pick a problem (usually some disambiguation) Get a lot of data (usually a labeled corpus) Build the simplest thing that could possibly work Repeat: –See what the most common errors are –Figure out what information a human would use –Modify the system to exploit that information Feature engineering Representation design Machine learning methods We’re going to do this over and over again

35. Language isn’t Adversarial One nice thing: we know NLP can be done! Language isn’t adversarial: –It’s produced with the intent of being understood –With some understanding of language, you can often tell what knowledge sources are relevant But most variables go unmodeled –Some knowledge sources aren’t easily available (real world knowledge, complex models of other people’s plans) –Some kinds of features are beyond our technical ability to model (especially cross-sentence correlations)

36. 바르게 보인다고 바르지 않다 !! Epistemological accuracy!! 쓰레기통행이다. 한국해이다. 한국해이를, 한국해를 미국해이다. 미국해이를 미국행이다.

37. What Statistical NLP can do for us Disambiguation strategies that rely on hand-coding produce a knowledge acquisition bottleneck and perform poorly on naturally occurring text. A Statistical NLP approach seeks to solve these problems by automatically learning lexical and structural preferences from corpora. In particular, Statistical NLP recognizes that there is a lot of information in the relationships between words. The use of statistics offers a good solution to the ambiguity problem: statistical models are robust, generalize well, and behave gracefully in the presence of errors and new data

38. Corpora Brown Corpus – 1 million words British National Corpus – 100 mil. Words American National Corpus – 10 mil. words -> 100 Penn TreeBank - parsed WSJ text Canadian Hansard – parallel corpus (bilingual) Dictionaries: Longman Dictionary of Contemporary English WordNet (hierarchy of synsets)

39. Things that can be done with Text Corpora (I) Word Counts Word Counts to find out: –What are the most common words in the text. –How many words are in the text (word tokens and word types). –What the average frequency of each word in the text is. Limitation of word counts: Most words appear very infrequently and it is hard to predict much about the behavior of words that do not occur often in a corpus. ==> Zipf’s Law.

40. Things that can be done with Text Corpora (II) Zipf’s Law If we count up how often each word type of a language occurs in a large corpus and then list the words in order of their frequency of occurrence, we can explore the relationship between the frequency of a word, f, and its position in the list, known as its rank, r. Zipf’s Law says that: f ∝ 1/r Significance of Zipf’s Law: For most words, our data about their use will be exceedingly sparse. Only for a few words will we have a lot of examples.

41. Common words in Tom Sawyer

42. Frequencies of frequencies in Tom Sawyer

43. Zipf's law in Tom Sawyer

45. Zipf’s Law

46. Zipf's law for the Brown corpus

47. Mandelbrot's formula for the Brown corpus

48. Things that can be done with Text Corpora (III) Collocations A collocation is any turn of phrase or accepted usage where somehow the whole is perceived as having an existence beyond the sum of its parts (e.g., disk drive, make up, bacon and eggs). Collocations are important for machine translation. Collocations can be extracted from a text (example, the most common bigrams can be extracted). However, since these bigrams are often insignificant (e.g., “at the”, “of a”), they can be filtered.

49. 이용 예 drive  disk drive, make up 사고  교통 사고  사고 방법 전하  전하번호  전하를 걸다 전하를 너무 일찍 걸었다. 자주 거짓말을 하는 순희는 아주 예쁘지만, 영희는 그렇지 않다. 그리 예쁜 여자가 거짓말을 하다니 개발일정완전재조정 ( 띄어 읽기, 띄어 쓰기 )

50. 순서 bigram: of the, in the, to the, on the …. New York, he said, as a Filtering :: adjective+noun, noun+noun –last year, next year ::: 예외적

51. Commonest bigrams in the NYT

52. Filtered common bigrams in the NYT

53. Things that can be done with Text Corpora (IV) Concordances Finding concordances corresponds to finding the different contexts in which a given word occurs. One can use a Key Word In Context (KWIC) concordancing program. Concordances are useful both for building dictionaries for learners of foreign languages and for guiding statistical parsers.

54. KWIC display

55. Syntactic frames for showed in Tom Sawyer

56. Why study NLP Statistically? Up until the late 1980’s, NLP was mainly investigated using a rule-based approach. However, rules appear too strict to characterize people’s use of language. This is because people tend to stretch and bend rules in order to meet their communicative needs. Methods for making the modeling of language more accurate are needed and statistical methods appear to provide the necessary flexibility.

57. Subdivisions of NLP Parts of Speech and Morphology (words, their syntactic function in sentences, and the various forms they can take). Phrase Structure and Syntax (regularities and constraints of word order and phrase structure). Semantics (the study of the meaning of words (lexical semantics) and of how word meanings are combined into the meaning of sentences, etc.) Pragmatics (the study of how knowledge about the world and language conventions interact with literal meaning).

58. Topics Covered in this course

59. Tools and Resources Used Probability/Statistical Theory: Statistical Distributions, Bayesian Decision Theory. Linguistics Knowledge: Morphology, Syntax, Semantics and Pragmatics. Corpora: Bodies of marked or unmarked text to which statistical methods and current linguistic knowledge can be applied in order to discover novel linguistic theories or interesting and useful knowledge organization.

60. Textbook and other useful information Foundations of Statistical Natural Language Processing, by Chris Manning and Hinrich Sch ü tze, MIT Press, 1999. Course Website: borame.cs.pusan.ac.kr