MPS 2016 ELI SHAMIR, HEBREW UNIVERSITY JERUSALEM PARENTAL VIEW OF CONTEXT – FREE BIRTH AND EVOLUTION

Formal Languages Theory- Mid 50’s Confluence of several directions: -Natural Languages [NLP], Syntax Specifications -Early Prog. Languages, Syntax Specifications -Automata & Machine, Formal Specifications -Combinatorial Math. Sets of strings -Biological: L Systems -…

Formal Languages Generative Hierarchy Chomsky+ Recursive enumerable Context-sensitive (LBA) Mildly context- sensitive Context free* Regular = FA definable Subsequently integrated into space/time complexity hierarchy- the backbone of theoretical computer science. * Several sub-models studied, related to compiler constructions for programming languages.

Context-free [CF] central position due to: equivalence of several distinct models Algebraic equations [MPS] DUAL APPROACH IN ARGGEMENT AND PROOFS Production rules and trees BNF- Backus NF, Syntax of early prog. languages Categorical grammars Dependency structures Lambek algebraic calculus Pushdown Automata… Rich algebraic, combinatorial, algorithmic properties and problems, significant applications.

: Boston- Jerusalem Correspondence Linguists : MIT N. Chomsky Y. Bar Hillel HUJI Mathem: Harvard MPS (MARCO) H. Gaifman, M. Perles, E. Shamir Paris [Math PhD students] Main articles, monographs mainly on CF [listed next: 2-5, 19]. Up to 1969, Many other researches and groups in USA, Europe, Japan joined. See publication lists [next few slides]. Inclusion as a basic topic in CS education.

Central Publications up to J. Hopcroft and J. Ullman, Formal Languages and their relations to Automata, Assidon-Wesley, [Extensive reference list] 2.Y. Bar-Hillel, H. Gaifman and E. Shamir, On categorical and phrase structure grammars. Bulletin research council of Israel, vol. 9f (1960), Y. Bar-Hillel, M. Perles and E. Shamir, On formal properties of simple phrase, structure grammars, Z. Phonetik, Sprachwiss. Kommun., 14 (1961), & 3 reproduced in Y. Bar-Hillel, Language and information, Assidon-Wesley, appeared as a monograph in Russian, N. Chomsky, On certain formal properties of grammars, Inf. and Control, 2:2 (1959), N. Chomsky and M. P. Schutzenberger, The algebraic theory of context-free languages, Computer Programming and Formal Systems, North Holland, [Appeared as a monograph] 6.J. Evey, The theory and application of pushdown store machines, Doctoral Thesis, Harvard University, R. W. Floyd, The syntax of programming languages- a survey, Professional Group Electronic Computers [PGEC], 13: 4 (1964),

9.S. Ginsburg, and H. G. Rice, Two families of languages related to ALGOL, JACM, 9: 3, , S. Ginsburg, The mathematical theory of context-free languages, S. Greibach, A new normal form theorem for context-free grammars, JACM, 12:1, 42-52, D. E. Knuth, a characterization of parenthesis languages, Inf. and Control, 11: 3, , P. S. Landweber, Three theorems on phrase structure grammars of type 1, Inf. and Control, 6:2, , M. Nivat, Transduction des langages de Chomsky, PhD Thesis. Univ. de Paris, [Also in Annales de l’Institut Fourier, 18: , 1968]. 15.R. J. Parikh, On context-free languages, JACM, 13, , D. J. Rosenkrantz, Matrix equations and normal forms for context-free grammars, JACM, 14:3, , J. Rhodes and E. Shamir, Complexity of grammars by group- theoretic methods, Journal of Combinatorial Theory, , E. Shamir, A representation theorem for algebraic and context-free power series in noncommuting variables, Inf. and Control, 11, , M. P. Schutzenberger [Several articles: ] 20.D. H. Younger, Recognition and parsing of context-free languages in time n, Inf. and Control, 10: 2, ,

Chosen Books & Publications After J. Autebert, J. Berstel and L. Boasson, Context-free language and pushdown automata. Chap. 3 In: handbook of formal languages Vol 1. G. Rozenberg and A. Salomaa (eds.), Springer-Verlag [Extensive reference list] 2.M. Droste, W. Kuich, H. Vogler (Eds.), Handbook of Weighted Automata, Springer S. Greibach. The hardest context-free language. SIAM J. on computing 3 (1973), M. Harrison, Introduction to Formal Language Theory, Addison- Wesley, L. Kallmeyer, Parsing Beyond Context Free Grammars, Springer, E. Shamir, Some inherently ambiguous context-free languages. Inf. and Control 18 (1971). 7.J. Berstel, Transductions and context-free languages, Teubner Verlag, A. Salomaa, Formal Languages, Academic Press, J. Sakarovitch, Pushdown automata with terminal languages, 421 in Publication RIMS, Kyoto University, 1981, pp S. Eilenberg, Automata, Languages and Machines, Vol. A & B, Academic Press, G. Rozenberg and A. Salomaa. The mathematical theory of L systems, Springer P. Flojolet, Analytic models and ambiguity of context free languages Theor. Comp. Sci 49,

Hindsight of Central CF Results Chomsky- Schutzenberger Theorems: and their impact Each CFL L= h (DykeᴖR) Dyke= {well bracketed strings}, R= regular language A non-ambiguous L has an algebraic generating function (Sh 1967): Each CFL maps into Non-deter. lifting of 1 sided Dyke hence it is A universal CFL thus a “hardest CFL”. map a  φ(a)= […+…+], φ(a 1 a 2 … a n )= φ(a 1 )… φ(a n )= =[…+…+] […+…+]… […+…+] (multinom product) wϵL(G) iff opening multinom product gives a term in DYKE. (BGS 1960 ): Non-deter. lifting of CAT is also universal (hardest) CFL

DYKE-j: All well-bracketed strings with j pairs. CAT: Well-cancelled categories-strings. a  a / b b, a / b  a / b / c c, a  a / b / c b/ c they are determ. CFLs, their non-det. liftings are “Hardest CFL. Algebraic path: Gauss elim-> Greib.NF->SH. Thm. & Pushdown Automat. Derivation path: triplets (p, A, q) [in BPS 1960] -> Pushdown Autom -> Greib. Normal Form and SH. Thm Algorithm and Complexity : impact of the non-decidability results (BPS 1960). Membership and parsing – tabular dynamic prog. algorithms (CYK, Earley,…). Time complexity reduced to multip. of Boolean matrices (L. Valiant, L. Lee). (Hindsight (continued

Ambiguity- Complex Issues In (Linear)CFG, in Transductions, in Algeb Equations Inherent ambiguity proofs using pumping in D - trees and by generating function method (Ph. Flajolet) Effect of Transformations on ambiguity Effects on Parsing of product ambiguity degree Inherently 1 or infinite? Open question Eilenberg problem: decomposition of bounded degree language to union of 1 degree languages - open

Ambiguity in NLP: Ambiguity in natural languages can be resolved (or created) by cyclic rotation of the sentence: Bible Book of Job chapter 6 verse 14 (six Hebrew words). Translated : "a friend should extend # mercy to the sufferer $, even if he abandons God's fear." Anaphoric ambiguity: the pronoun "he" refers to the sufferer or to the friend? A poetic beautiful answer: to Both. Cyclic rotated sentences, starting at the symbols # and $, resolve the ambiguity towards one way or the other. Political loaded example: the policeman shot # the boy $ with the gun.

SRT: SPREAD - ROTATE TRANSFORMATION Of a grammar G, its trees and derived strings internal nodes labelled by prodacts of grammars: SRT TREE root label = #G, leaves labels = H(i) – linear grammars Thm ( invariance claim) 1-1 onto U {D – trees of H(i)} D - trees of #G mapped Mod. Cyclic rotations (of trees and derives strings) But Works perfect for non – expansive CF grammars (quasi-rational) but also for mild context – sensitive with CF skeleton (E.G.LIG grammars) SRT: enhance parsing alg, property tests, and applications cosmetics of the CFG model to enhance its NLP adequacy: *Avoid expansive pumping B BB BUT ADD GENER. POWER BY LOCAL STACKS (AS IN INDEXED GRAMMARS)

Top Trunk Rotation of MN to (M*N^) M M EXIT N^ x1x1 x2x2 y1y1 y2y2 x1x1 x2x2 y2y2 y1y1 N derived strings: m x 1 x 2 … n^ …y 2 y 1 …y 2 y 1 m x 1 x 2 … n^ for trees: M* 180 Cyclic rotation of

N grammar (top trunk) M* grammar B  B’C B’  CB B  DB’ B’  BD B  B^, B^  α B^= root(M) All productions not involving [B] carry over from N to M*; those of M unchanged. Note: Since M may contain symbols of [B] duplicate symbols [B] needed only for the new top trunk of M* The TTR rotation is invertible, one-one onto for the derivation trees, preserving weights and ambiguity degree in ‘cyclic rotated’ sense. SRT For grammars:

Example (from [Sh., 1971]) (M)(N) = (u $J u ) (v J$ v), u, v ε {0.1}* = J u = reversal of u, It has unbounded "direct (product) ambiguity" which increases time in CYK algorithm to n In one TTR step (see below) MN is rotated to (M*)(N^) = (v u $ J u v ) (J $), which has a linear grammar, with 3 pump classes. All (product ambiguity) trees are rotated to (union ambiguity) trees for M*N^. Each derived terminal string is CYC-rotated as well! R R R R R 3

MILD Context-Sensitive Models & SRT Many models proposed incl. 4 equivalent ones: Linear-Index [LIG], Tree-Adjoint [TAG],…. Should satisfy some formal requirements: Proper extension of CFG, Poly-time parsing algor… We define NE-LIG as follows: Has NE-CFG skeleton aux. symbols A, B,… Each pump-class [B] maintains stack (pushdown) index, stack empty at enter & Exit of several consecutive pump blocks- THUS, it can, with skeleton -symbols as “states”, simulate any PDM, any CFG. The form of production rules is: B[index] C B’[index’], Bˆ[ ]  D[ ] E [ ] Push Pop

Glossary CFG/L- Context Free Grammars/Language LIG- Linear Indexed Grammar TAG- Tree Adjoining Grammar NLP- Natural Language Processing CYK- Cocke, Younger, Kasami CNF- Chomsky Normal Form GNF- Greibach Normal Form SRT- Spread Rotate Tree D-Tree- Derivation Tree EPOS- Epoch Semi-Order TTR- Top Trunk Rotation DP- Dynamic Programming NE- Non Expansive POS- Parts of Speech PDM- Pushdown Machine NT- Non terminals (symbols)