Presentation is loading. Please wait.

Presentation is loading. Please wait.

Parsing Compiler Baojian Hua Front End source code abstract syntax tree lexical analyzer parser tokens IR semantic analyzer.

Published by Modified over 8 years ago

Similar presentations

Presentation on theme: "Parsing Compiler Baojian Hua Front End source code abstract syntax tree lexical analyzer parser tokens IR semantic analyzer."— Presentation transcript:

1 Parsing Compiler Baojian Hua bjhua@ustc.edu.cn

2 Front End source code abstract syntax tree lexical analyzer parser tokens IR semantic analyzer

3 Parsing The parser translates the source program into abstract syntax trees Token sequence: from the lexer abstract syntax trees: check validity of programs cook compiler internal data structures for programs Must take account the program syntax

4 Conceptually token sequence abstract syntax tree parser language syntax

5 Syntax: Context-free Grammar Context-free grammars are (often) given by BNF expressions (Backus-Naur Form) read Dragon sec 2.2 More powerful than RE in theory Good for defining language syntax

6 Context-free Grammar (CFG) A CFG consists of 4 components: a set of terminals (tokens): T a set of nonterminals: N a set of production rules: P s -> t1 t2 … tn with s  N, and t1, …, tn  (T ∪ N) a unique start nonterminal: S

7 Example // Recall the min-ML language in “code3” // (simplified) N = {decs, dec, exp} T = {SEMICOLON, VAL, ID, ASSIGN, NUM} S = decs decs -> dec SEMICOLON decs | dec -> VAL ID ASSIGN exp exp -> ID | NUM

8 Derivation A derivation: Starts with the unique start nonterminal S repeatedly replacing a right-hand nonterminal s by the body of a production rule of the nonterminal s stop when right-hand are all terminals The final string consists of terminals only and is called a sentence (program)

9 Example decs -> dec SEMICOLON decs | dec -> VAL ID ASSIGN exp exp -> ID | NUM val x = 5; val y = x; derive me decs -> … (a choice)

10 Example decs -> dec SEMICOLON decs | dec -> VAL ID ASSIGN exp exp -> ID | NUM val x = 5; val y = x; derive me decs -> dec SEMICOLON decs -> VAL ID ASSIGN exp SEMICOLON decs -> VAL ID ASSIGN NUM SEMICOLON decs -> VAL ID ASSIGN NUM SEMICOLON dec SEMICOLON decs -> … -> VAL ID ASSIGN NUM SEMICOLON VAL ID ASSIGN ID SEMICOLON decs

11 Another Way to Derive the same Program decs -> dec SEMICOLON decs | dec -> VAL ID ASSIGN exp exp -> ID | NUM val x = 5; val y = x; derive me decs -> dec SEMICOLON decs -> dec SEMICOLON dec SEMICOLON decs -> …

12 Derivation For same string, there may exist many derivations left-most derivation right-most derivation Parsing is the problem of taking a string of terminals and figure out whether it could be derived from a CFG error-detection

13 Parse Trees Derivation can also be represented as trees useful to understand AST (discussed later) Idea: each internal node is labeled with a non-terminal each leaf node is labeled with a terminal each use of a rule in a derivation explains how to generate children in the parse tree from the parents

14 Example decs -> dec SEMICOLON decs | dec -> VAL ID ASSIGN exp exp -> ID | NUM val x = 5; val y = x; derive me decs dec SEMI decs VAL ID =exp 5 decSEMIdecs similar case

15 Different Derivations, same Tree decs -> dec SEMICOLON decs -> VAL ID ASSIGN exp SEMICOLON decs -> … decs -> dec SEMICOLON decs -> dec SEMICOLON dec SEMICOLON decs -> … val x = 5; val y = x; derive me decs dec SEMI decs VAL ID =exp 5 decSEMIdecs similar case

16 Parse Tree has Meanings: post-order traversal decs -> dec SEMICOLON decs -> VAL ID ASSIGN exp SEMICOLON decs -> … decs -> dec SEMICOLON decs -> dec SEMICOLON dec SEMICOLON decs -> … val x = 5; val y = x; derive me decs dec SEMI decs VAL ID =exp 5 decSEMIdecs similar case

17 Ambiguous Grammars A grammar is ambiguous if the same sequence of tokens can give rise to two or more different parse trees

18 Example exp -> num -> id -> exp + exp -> exp * exp 3+4*5 derive me exp -> exp + exp -> 3 + exp -> 3 + exp * exp -> 3 + 4 * exp -> 3 + 4 * 5 exp -> exp * exp -> exp + exp * exp -> 3 + exp * exp -> 3 + 4 * exp -> 3 + 4 * 5

19 Example exp -> num -> id -> exp + exp -> exp * exp exp -> exp + exp -> 3 + exp -> 3 + exp * exp -> 3 + 4 * exp -> 3 + 4 * 5 exp -> exp * exp -> exp + exp * exp -> 3 + exp * exp -> 3 + 4 * exp -> 3 + 4 * 5 exp + 3 * 5 4 * 5 + 4 3

20 Ambiguous Grammars Problem: compilers make use of parse trees to interpret the meaning of parsed programs different parse trees have different meanings eg: 4 + 5 * 6 is not (4 + 5) * 6 languages with ambiguous grammars are DISASTROUS; the meaning of programs isn ’ t well-defined! You can ’ t tell what your program might do! Solution: rewrite grammar to equivalent forms

21 Eliminating ambiguity In programming language syntax, ambiguity often arises from missing operator precedence or associativity * is of high precedence than + both + and * are left-associative Why or why not? Rewrite grammar to take account of this

22 Example exp -> num -> id -> exp + exp -> exp * exp exp -> exp + term -> term term -> term * factor -> factor factor -> num -> id Q: is the right grammar ambiguous? Why or why not?

23 Parser A program to check whether a program is derivable from a given grammar expensive in general must be fast to compile a 2000k lines of kernel even for small application code Theorists have developed specialized kind of grammar which may be parsed efficiently LL(k) and LR(k)

24 Predictive parsing A.K.A: Recursive descent parsing, top-down parsing simple to code by hand efficient can parse a large set of grammar Key idea: one (recursive) function for each nonterminal one clause for each right-hand production rule

25 Example decs -> dec SEMICOLON decs | dec -> VAL ID ASSIGN exp exp -> ID | NUM (* step #1: represent tokens *) datatype token = Val | Id of string | Num of int | Assign | Semicolon | Eof (* step #2: connect with lexer *) token current = ref getToken (); fun advance () = current := getToken (); fun eat (token t) = if !current = t then advance () else error (“want “, t, “but got “, !current)

26 decs -> dec SEMICOLON decs | dec -> VAL ID ASSIGN exp exp -> ID | NUM (* step #1: represent tokens *) datatype token = Val | Id of string | Num of int | Assign | Semi | Eof (* step #2: connect with lexer *) token current = ref getToken (); fun advance () = current := getToken (); fun eat (token t) = …; (* step #3: build the parser *) fun parseDecs() = case !current of VAL => parseDec (); eat (Semi); parseDecs (); | EOF => () | _ => error (“want VAL or EOF”) fun parseDec () = … fun parseExp () = …

27 Moral The key point in predicative parsing is to determine the production rule to use (recursive function to call) must know the “ start ” symbols of each rule “ start ” symbol must not overlap ex: exp -> NUM | ID This motivates the idea of first and follow sets

28 Moral S -> w1 -> w2 -> … -> wn Current nonterminal is S, and the current input token is t if wk starts with t, then choose wk, or if wk derives empty string, and the string follow S starts with t First symbol sets of wi (1<=i<=n) don ’ t overlap to avoid backtracking

29 Nullable, First and Follow sets To use predicative parsing, we must compute: Nullable: nonterminals that derive empty string First(ω) : set of terminals that can begin any string derivable from ω Follow(X): set of terminals that can immediately follow any string derivable from nonterminal X Read Dragon sec 4.4.2 and Tiger sec 3.2 Fixpoint algorithms

30 Nullable, First and Follow sets Which symbol X, Y and Z can derive empty string? What terminals may the string derived from X, Y and Z begin with? What terminals may follow X, Y and Z? Z -> d -> X Y Z Y -> c -> X -> Y -> a

31 Nullable If X can derive an empty string, iff: base case: X -> inductive case: X -> Y1 … Yn Y1, …, Yn are n nonterminals and may all derive empty strings

32 Computing Nullable Nullable <- {}; while (Φ still change) for (each production X -> α) switch (α) case  : Nullable ∪ = {X}; break; case Y1 … Yn: if (Y1  Nullable && … && Yn  Nullable) Nullable ∪ = {X}; break;

33 Example: Nullables Z -> d -> X Y Z Y -> c -> X -> Y -> a Round012 Φ{}

34 Example: Nullables Z -> d -> X Y Z Y -> c -> X -> Y -> a Round012 Φ{}{Y, X}

35 Example: Nullables Z -> d -> X Y Z Y -> c -> X -> Y -> a Round012 Φ{}{Y, X}

36 First(X) Set of terminals that X begins with: X => a … Rules base case: X -> a First (X) ∪ = {a} inductive case: X -> Y1 Y2 … Yn First (X) ∪ = First(Y1) if Y1  Nullable, First (X) ∪ = First(Y2) if Y1,Y2  Nullable, First (X) ∪ = First(Y3) …

37 Computing First // Suppose Nullable has been computed First(X) <- {}; // for each X while (First still change) for (each production X -> α) switch (α) case a: First(X) ∪ = {a}; break; case Y1 … Yn: First(X) ∪ = First(Y1); if (Y1\not\in Nullable) break; First(X) ∪ = First(Y1); …; // Similar as above

38 Example: First Z -> d -> X Y Z Y -> c -> X -> Y -> a Round0123 First(Z){} First(Y){} First(X){} Nullable = {X, Y}

39 Example: First Z -> d -> X Y Z Y -> c -> X -> Y -> a Round0123 First(Z){}{d} First(Y){}{c} First(X){}{c, a} Nullable = {X, Y}

40 Example: First Z -> d -> X Y Z Y -> c -> X -> Y -> a Round0123 First(Z){}{d}{d, c, a} First(Y){}{c} First(X){}{c, a} Nullable = {X, Y}

41 Example: First Z -> d -> X Y Z Y -> c -> X -> Y -> a Round0123 First(Z){}{d}{d, c, a} First(Y){}{c} First(X){}{c, a} Nullable = {X, Y}

42 Parsing with First Z -> d {d} -> X Y Z {a, c, d} Y -> c {c} -> {} X -> Y {c} -> a {a} First(Z){d, c, a} First(Y){c} First(X){c, a} Nullable = {X, Y} Now consider this string: d Suppose we choose the production: Z -> X Y Z But we get stuck at: X -> Y -> a neither can accept d! Why?

43 Follow(X) Set of terminals that may follow X: S => … X a … Rules: Base case: Follow (X) = {} inductive case: Y -> ω1 X ω2 Follow(X) ∪ = Fisrt(ω2) if ω2 is Nullable, Follow(X) ∪ = Follow(Y)

44 Computing Follow(X) Follow(X) <- {}; while (Follow still change) { for (each production Y -> ω1 X ω2 ) Follow(X) ∪ = First (ω2); if ( ω2 is Nullable) Follow(X) ∪ = Follow (Y);

45 Example: Follow Z -> d -> X Y Z Y -> c -> X -> Y -> a Round0123 First(Z) Follow(Z) {d, c, a} {} First(Y) Follow(Y) {c} {} First(X) Follow(X) {c, a} {} Nullable = {X, Y}

46 Example: Follow Z -> d -> X Y Z Y -> c -> X -> Y -> a Round0123 First(Z) Follow(Z) {d, c, a} {}{$} First(Y) Follow(Y) {c} {}{d, c, a} First(X) Follow(X) {c, a} {}{d, c, a} Nullable = {X, Y}

47 Example: Follow Z -> d -> X Y Z Y -> c -> X -> Y -> a Round0123 First(Z) Follow(Z) {d, c, a} {}{$} First(Y) Follow(Y) {c} {}{d, c, a} First(X) Follow(X) {c, a} {}{d, c, a} Nullable = {X, Y}

48 Predicative Parsing Table With Nullables, First(), and Follow(), we can make a parsing table P(N,T) each entry contains a set of productions t1 t2 t3 t4 … $(EOF) N1 ri N2 rk N3 rj …

49 Predicative Parsing Table For each rule X -> ω for each a  First(ω), add X -> ω to P(X, a) if X is nullable, add X -> ω to P(X, b) for each b  Follow (X) all other entries are “ error ” t1 t2 t3 t4 … $(EOF) N1 r1 N2 rk N3 ri …

50 Example: Predicative Parsing Table First(X) Follow(X) {c, a} {c, d, a} First(Y) Follow(Y) {c} {c, d, a} First(Z) Follow(Z) {d, c, a} {$} Z -> d -> X Y Z Y -> c -> X -> Y -> a Nullable = {X, Y} acd ZZ->X Y Z Z->d Z->X Y Z YY->Y->c Y-> XX->Y X->a X->Y

51 Example: Predicative Parsing Table First(X) Follow(X) {c, a} {c, d, a} First(Y) Follow(Y) {c} {c, d, a} First(Z) Follow(Z) {d, c, a} {$} Z -> d -> X Y Z Y -> c -> X -> Y -> a Nullable = {X, Y} acd ZZ->X Y Z Z->d Z->X Y Z YY->Y->c Y-> XX->Y X->a X->Y

52 LL(1) A context-free grammar is called LL(1) if it can be parsed this way: Left-to-right parsing Leftmost derivation 1 token lookahead This means that in the predicative parsing table, there is at most one production in every entry

53 Speeding up set Construction All these sets (Nullable, First, Follow) can be computed simultaneously see Tiger algorithm 3.13 Order the computation: What ’ s the optimal order to compute these set?

54 Example: Speeding up set Construction Z -> d -> X Y Z Y -> c -> X -> Y -> a Round0123 First(Z){} First(Y){} First(X){} Nullable = {X, Y} Q1: What ’ s reasonable order here? Q2: How to set this order?

55 Directed Graph Model Z -> d -> X Y Z Y -> c -> X -> Y -> a Nullable = {X, Y} Q1: What ’ s reasonable order here? Q2: How to set this order? Z X Y {c} {c, a} {d, c, a} Order: Y X Z

56 Reverse Topological Sort Quasi-topological sort the directed graph Quasi: topo-sort general directed graph is impossible also known as reverse depth-first ordering Reverse: information (First) flows from successors to predecessors Refer to your favorite algorithm book

57 Problem LL(1) can only be used with grammars in which every production rules for a nonterminal start with different terminals Unfortunately, many grammars don ’ t have this perfect property

58 Example exp -> num -> id -> exp + exp -> exp * exp exp -> exp + term -> term term -> term * factor -> factor factor -> num -> id Q: is the right grammar LL(1)? Why or why not?

59 Solutions Left-recursion elimination Left-factoring Read: dragon sec4.3.2, 4.3.3, 4.3.4 tiger sec3.2

60 Example exp -> term exp’ exp’ -> + term exp’ -> term -> factor term’ term’-> * factor term’ -> factor -> num -> id Q: is the right grammar LL(1)? are those two grammars equivalent? exp -> exp + term -> term term -> term * factor -> factor factor -> num -> id

61 LL(k) LL(1) can be further generalized to LL(k): Left-to-right parsing Leftmost derivation k token lookahead Q: table size? other problems with this approach?

62 Summary Context-free grammar is a math tool for specifying language syntax and others … Writing parsers for general grammar is hard and costly LL(k) and LR(k) LL(1) grammars can be implemented efficiently table-driven algorithms (again!)

Download ppt "Parsing Compiler Baojian Hua Front End source code abstract syntax tree lexical analyzer parser tokens IR semantic analyzer."

Similar presentations

Parsing 4 Dr William Harrison Fall 2008

Parsing 4 Dr William Harrison Fall 2008
Mooly Sagiv and Roman Manevich School of Computer Science

Mooly Sagiv and Roman Manevich School of Computer Science
6/12/2015Prof. Hilfinger CS164 Lecture 111 Bottom-Up Parsing Lecture 11-12 (From slides by G. Necula & R. Bodik)

6/12/2015Prof. Hilfinger CS164 Lecture 111 Bottom-Up Parsing Lecture (From slides by G. Necula & R. Bodik)
Top-Down Parsing.

Top-Down Parsing.
Parsing Discrete Mathematics and Its Applications Baojian Hua

Parsing Discrete Mathematics and Its Applications Baojian Hua
By Neng-Fa Zhou Syntax Analysis lexical analyzer syntax analyzer semantic analyzer source program tokens parse tree parser tree.

By Neng-Fa Zhou Syntax Analysis lexical analyzer syntax analyzer semantic analyzer source program tokens parse tree parser tree.
CS 381 - Summer 2005 Top-down and Bottom-up Parsing - a whirlwind tour June 20, 2005 Slide acknowledgment: Radu Rugina, CS 412.

CS Summer 2005 Top-down and Bottom-up Parsing - a whirlwind tour June 20, 2005 Slide acknowledgment: Radu Rugina, CS 412.
Context-Free Grammars Lecture 7

Context-Free Grammars Lecture 7
COS 320 Compilers David Walker.

COS 320 Compilers David Walker.
Parsing III (Eliminating left recursion, recursive descent parsing)

Parsing III (Eliminating left recursion, recursive descent parsing)
COS 320 Compilers David Walker. last time context free grammars (Appel 3.1) –terminals, non-terminals, rules –derivations & parse trees –ambiguous grammars.

COS 320 Compilers David Walker. last time context free grammars (Appel 3.1) –terminals, non-terminals, rules –derivations & parse trees –ambiguous grammars.
CS 310 – Fall 2006 Pacific University CS310 Parsing with Context Free Grammars Today’s reference: Compilers: Principles, Techniques, and Tools by: Aho,

CS 310 – Fall 2006 Pacific University CS310 Parsing with Context Free Grammars Today’s reference: Compilers: Principles, Techniques, and Tools by: Aho,
1 Foundations of Software Design Lecture 23: Finite Automata and Context-Free Grammars Marti Hearst Fall 2002.

1 Foundations of Software Design Lecture 23: Finite Automata and Context-Free Grammars Marti Hearst Fall 2002.
1 The Parser Its job: –Check and verify syntax based on specified syntax rules –Report errors –Build IR Good news –the process can be automated.

1 The Parser Its job: –Check and verify syntax based on specified syntax rules –Report errors –Build IR Good news –the process can be automated.
Professor Yihjia Tsai Tamkang University

Professor Yihjia Tsai Tamkang University
COS 320 Compilers David Walker. last time context free grammars (Appel 3.1) –terminals, non-terminals, rules –derivations & parse trees –ambiguous grammars.

COS 320 Compilers David Walker. last time context free grammars (Appel 3.1) –terminals, non-terminals, rules –derivations & parse trees –ambiguous grammars.
Top-Down Parsing.

Top-Down Parsing.
COP4020 Programming Languages

COP4020 Programming Languages
Chapter 2 Chang Chi-Chung 2008.03 rev.1. A Simple Syntax-Directed Translator This chapter contains introductory material to Chapters 3 to 8  To create.

Chapter 2 Chang Chi-Chung rev.1. A Simple Syntax-Directed Translator This chapter contains introductory material to Chapters 3 to 8  To create.
Chapter 3 Chang Chi-Chung 2015.05.18. Parse tree intermediate representation The Role of the Parser Lexical Analyzer Parser Source Program Token Symbol.

Chapter 3 Chang Chi-Chung Parse tree intermediate representation The Role of the Parser Lexical Analyzer Parser Source Program Token Symbol.

Similar presentations

Ads by Google