1. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-6801 עיבוד שפות טבעיות - שיעור שבע Tagging, Partial Parsing Context Free Grammar עידו דגן המחלקה למדעי המחשב אוניברסיטת בר אילן

2. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-6802 Up from bigrams The POS tagging model we described used a history of just the previous tag: P(t i |t 1,…,t i-1 ) ≈ P(t i |t i-1 ) i.e. a First Order Markovian Assumption In this case each state in the HMM corresponds to a POS tag One can build an HMM for POS trigrams P(t i |t 1,…,t i-1 ) ≈ P(t i |t i-2,t i-1 )

3. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-6803 POS Trigram HMM Model More accurate (also empirically) than a bigram model –He clearly marked –is clearly marked Sparseness problem – smoothing, back-off In such a model the HMM states do NOT correspond to POS tags. Why not 4-grams? –Too many states, not enough data!

4. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-6804 Supervised/Unsupervised Is the HMM based tagging a supervised algorithm? –Yes, because we need a tagged corpus to estimate the transition and emission probabilities (!) What do we do if we don’t have an annotated corpus but, –Have a dictionary –Have an annotated corpus from a different domain and an un-annotated corpus in desired domain.

5. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-6805 Baum-Welch Algorithm also known as the Forward-Backward Algorithm An EM algorithm for HMMs. Maximization by Iterative hill climbing The algorithm iteratively improves the model parameters based on un- annotated training data.

6. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-6806 Baum-Welch Algorithm… Start of with parameters based on the dictionary: –P(w|t) = 1 if t is possible tag for w –P(w|t) = 0 otherwise –Uniform distribution on state transitions This is enough to bootstrap from. Could also be used to tune a system to a new domain. But best results, and common practice, is using supervised estimation

7. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-6807 Completely unsupervised? What if there is no dictionary and no annotated corpus?  Then POS tagging becomes clustering – doesn’t correspond to linguistic POS

8. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-6808 Syntax The study of grammatical relations between words and other units within the sentence. The Concise Oxford Dictionary of Linguistics the way in which linguistic elements (as words) are put together to form constituents (as phrases or clauses) Merriam-Webster Dictionary

9. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-6809 Brackets “I prefer a morning flight” [ S [ NP [ pro I]][ VP [ V prefer][ NP [ Det a] [ Nom [ N morning] [ N flight]]]]]]

10. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68010 Parse Tree Noun Nom NounDet VerbPronoun Ipreferamorning flight NP S VP

11. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68011 Parsing The problem of mapping from a string of words to to its parse tree is called parsing.

12. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68012 Generative Grammar A set of rules which indicate precisely what can be and cannot be a sentence in a language. A grammar which precisely specifies the membership of the set of all the grammatical sentences in the language in question and therefore excludes all the ungrammatical sentences.

13. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68013 Formal Language Approach The set of all grammatical sentences in a given natural language –In practice – a complete grammar is not available Are natural languages regular?

14. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68014 English is not a regular language! a n b n is not regular Look at the following English sentences: –John and Mary like to eat and sleep, respectively. –John, Mary, and Sue like to eat, sleep, and dance, respectively. –John, Mary, Sue, and Bob like to eat, sleep, dance, and cook, respectively. Anti missile missile

15. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68015 Constituents Certain groupings of words behave as constituents. Constituents are able to occur in various sentence positions: –ראיתי את הילד הרזה –ראיתי אותו מדבר עם הילד הרזה –הילד הרזה גר ממול

16. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68016 The Noun Phrase (NP) Examples: –He –Ariel Sharon –The prime minister –The minister of defense during the war in Lebanon. They can all appear in a similar context: ___ was born in Kfar-Malal

17. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68017 Prepositional Phrases Examples: –the man in the white suit –Come and look at my paintings –Are you fond of animals? –Put that thing on the floor

18. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68018 Verb Phrases Examples: –He went –He was trying to keep his temper. –She quickly showed me the way to hide.

19. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68019 Chunking Text chunking is dividing sentences into non- overlapping phrases. Noun phrase chunking deals with extracting the noun phrases from a sentence. While NP chunking is much simpler than parsing, it is still a challenging task to build an accurate and very efficient NP chunker.

20. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68020 What is it good for Chunking is useful in many applications: –Information Retrieval & Question Answering –Machine Translation –Preprocessing before full syntactic analysis –Text to speech –…

21. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68021 What kind of structures should a partial parser identify? Different structures useful for different tasks: –Partial constituent structure [ NP I] [ VP saw [ NP a tall man in the park]]. –Prosodic segments [I saw] [a tall man] [in the park]. –Content word groups [I] [saw] [a tall man] [in the park].

22. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68022 Chunk Parsing Goal: divide a sentence into a sequence of chunks. Chunks are non-overlapping regions of a text: –[I] saw [a tall man] in [the park]. Chunks are non-recursive –a chunk can not contain other chunks Chunks are non-exhaustive –not all words must be included in chunks

23. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68023 Chunk Parsing Examples Noun-phrase chunking: –[I] saw [a tall man] in [the park]. Verb-phrase chunking: –The man who [was in the park] [saw me]. Prosodic chunking: –[I saw] [a tall man] [in the park].

24. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68024 Chunks and Constituency Constituents: [a tall man in [the park]]. Chunks: [a tall man] in [the park]. Chunks are not constituents –Constituents are recursive Chunks are typically subsequences of Constituents –Chunks do not cross constituent boundaries

25. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68025 Chunk Parsing: Accuracy Chunk parsing achieves higher accuracy –Smaller solution space –Less word-order flexibility within chunks than between chunks –Better locality: Fewer long-range dependencies Less context dependence –No need to resolve attachment ambiguity –Less error propagation

26. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68026 Chunk Parsing: Domain Specificity Chunk parsing is less domain specific: Dependencies on lexical/semantic information tend to occur at levels "higher" than chunks: –Attachment –Argument selection –Movement Fewer stylistic differences within chunks

27. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68027 Chunk Parsing: Efficiency Chunk parsing is more efficient –Smaller solution space –Relevant context is small and local –Chunks are non-recursive –Chunk parsing can be implemented with a finite state machine

28. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68028 Psycholinguistic Motivations Chunk parsing is psycholinguistically motivated: Chunks as processing units –Humans tend to read texts one chunk at a time – Eye-movement tracking studies Chunks are phonologically marked –Pauses, Stress patterns Chunking might be a first step in full parsing

29. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68029 Chunk Parsing Techniques Chunk parsers usually ignore lexical content Only need to look at part-of-speech tags Techniques for implementing chunk parsing: –Regular expression matching / Finite State Machines (see next) –Transformation Based Learning –Memory Based Learning –Other tagging-style methods (see further)

30. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68030 Regular Expression Matching Define a regular expression that matches the sequences of tags in a chunk –A simple noun phrase chunk regexp: ? * * –Chunk all matching subsequences: the/DT little/JJ cat/NN sat/VBD on/IN the/DT mat/NN [the/DT little/JJ cat/NN] sat/VBD on/IN [the/DT mat/NN] –If matching subsequences overlap, the first one or longest one gets priority

31. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68031 Chunking as Tagging Map Part of Speech tag sequences to {I,O,B}*  I – tag is part of an NP chunk  O – tag is not part of  B – the first tag of an NP chunk which immediately follows another NP chunk  Alternative tags: Begin, End, Outside Example: –Input: The teacher gave Sara the book –Output: I I O I B I

32. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68032 Chunking State of the Art When addressed as tagging – methods similar to POS tagging can be used –HMM – combining POS and IOB tags –TBL – rules based on POS and IOB tags Depending on task specification and test set: 90-95% for NP chunks

33. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68033 Homework

34. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68034 Context Free Grammars Putting the constituents together

35. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68035 Problems with FS grammars Reasons why finite-state grammars may be inadequate: Cannot represent constituency adequately Cannot represent structural ambiguity Cannot deal with recursion Recursion occurs when an expansion of a non- terminal includes the non-terminal itself, eg.NP containing a PP, which contains an NP

36. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68036

37. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68037

38. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68038 Context-free grammars A context-free grammar consists of –a lexicon – set of terminals –a set of non-terminals –a set of rules or productions

39. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68039 A baby CF grammar for NPs NP  Det Nominal NP  ProperNoun NP  Pronoun Nominal  Noun Nominal  Noun Nominal Det  a Det  the Pronoun  I Noun  flight

40. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68040 Context-free rules Each context-free rule is of the form: A  γ a single non-terminal symbol  an ordered list of one or more terminals and non-terminals Start symbol S (sentence)

41. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68041 Context-free grammars CFGs can be used for analyzing or for generating „  “ means: rewrite the symbol on the left with the string of symbols on the right Example: NP  Det Nominal  Det Noun  a Noun  a flight

42. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68042 Context-free grammars A sequence of rule expansions is called the derivation of a string of words. Formally, a particular CF language is a set of strings which can be derived from a particular CF grammar A derivation is represented by a parse tree

43. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68043

44. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68044

45. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68045 The combined baby CFG S  NP VP NP  Pronoun |Det Nominal |ProperNoun Nominal  Noun|Noun Nominal VP  Verb NP | Verb NP PP | Verb PP PP  Preposition NP Noun  flight|flights |trip |morning |... Verb  is |prefer |like|need |want |fly |... Adjective  cheapest|non-stop|first|direct|... Pronoun  me|I |you|it|... ProperNoun  Los Angeles |Chicago |Alaska |... Det  a |the|an |this | these|that|... Preposition  from|to |on |near|... Conjunction  and |or|but|...

46. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68046

47. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68047 Formal definition of a CFG A context-free grammar is a 4-tuple: –A set of non-terminal symbols N –A set of terminal symbols Σ (disjoint from N) –A set of productions P, each of the form A  α, where A is a non-terminal and α is a string of symbols from the infinite set of strings (Σ  N)* –A designated start symbol S

48. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68048 CF language If A  β is a production of P and α and γ are any strings in the set (∑  N)*,we say that αAγ directly derives α β γ, or αAγ  α β γ Derivation is a generalization of direct derivation –Let α1, α2,..., αm be strings in (∑  N)*, m  1, such that α1  α2, α2  α3,..., α(m-1)  αm –we say that α1 derives αm, or α1  * αm

49. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68049 CF language The language L(G) generated by a grammar G is the set of strings composed of terminal symbols which can be derived from the designed start symbol S: L(G)= {W| w is in ∑ * and S  * w}

50. Syllabus Text Books Classes Reading Material Assignments Grades Links Forum Text Books 88-68050 Grammar equivalence Two grammars are strongly equivalent if they generate the same set of strings and if they assign the same phrase structure to each sentence Two grammars are weakly equivalent if they generate the same set of strings but do not assign the same phrase structure to each sentence