Review 1.Lexical Analysis 2.Syntax Analysis 3.Semantic Analysis 4.Code Generation 5.Code Optimization
Syntax Analysis Often called parsing Groups tokens of source program into grammatical phrases that are used by the compiler to check for correct syntax and to help in generating code Creation of a hierarchical structure called a syntax tree –Tree helps us determine if program is syntactically correct –Also aids in the translation of source program to target language
Grammar Example San FranciscoSeattle Lexical Ogay orthnay eightay undrendhay ilesmay Syntactical Miles go hundred north eight Logical Go eight hundred miles (facing south) Run-Time Go eight hundred miles (facing West)
Grammars – Defining the Language Rules Terminals (tokens) Non-terminals - Syntactic variable. Contains groups of tokens that define some part of the language –Example: expression, if statement, etc Start symbol - A special non-terminal (i.e. a program) Productions –The manner in which terminals and non-terminals are combined to form statements –A non-terminal in LHS and a string of terminals and non- terminals in RHS
Example Variable Declaration –A type followed by one or more comma separated identifiers that end with a semi-colon. -> ->, and are non-terminals The comma is a terminal The two lines are called productions
Grammars We will be defining a grammar for the entire JO99 programming language. We will then have JCUP produce for us a parser that detects if the JO99 program is syntactically correct. In addition, the parser will create for us a tree that represents the program and allows us to be able to do other things like semantic checks and code generation.
Example Simple arithmetic expressions with + and * – – * 45.3 –( ) * (6.035 * -( )) Terminals (or tokens) –num for all the numbers –‘+’, ‘-’, ‘*’, ‘(‘, ‘)’ What is the grammar for all possible expressions?
Example ( ) - num + *
Categories of Parsers ()
–L - parse from left to right –R - parse from right to left ()
Categories of Parsers –L - leftmost derivation –R - rightmost derivation ()
Categories of Parsers –Number of look ahead characters ()
Categories of Parsers –Examples: LL(0) – Parse Left to Right, Derive the tree using a leftmost derivation (top down), no look ahead characters LR(1) – Parse Left to Right, Derive the tree using a rightmost derivation (bottom up), 1 look ahead character. –Each category of parsing handles a different type of language. –We will be learning about LR(k) parsers and will implement an LR(k) parser.
Why LR(k)? Virtually all programming language grammars can be parsed using a LR(k) technique Most general parsing method for programming grammars Can build a very efficient parser engine given just the syntax rules of the language. Can detect a syntactic error as soon as it is possible to do so Because its so general, programs have been written (JCUP) that produce the parser instead of writing it from scratch.
LR(k) Parser implementation Sometimes called a Shift-Reduce Parser Parse from left to right (get the tokens from left to right) Bottom up parsing (same as rightmost derivation)
Actions of a Shift-Reduce Parser Parse Tree
Actions of a Shift-Reduce Parser Parse Tree Parse Tree
Actions of a Shift-Reduce Parser Parse Tree Parse Tree
Actions of a Shift-Reduce Parser Parse Tree Parse Tree
Actions of a Shift-Reduce Parser Parse Tree Parse Tree
Actions of a Shift-Reduce Parser Parse Tree Parse Tree
Actions of a Shift-Reduce Parser Parse Tree Parse Tree
Actions of a Shift-Reduce Parser Parse Tree Parse Tree
Actions of a Shift-Reduce Parser Parse Tree
Actions of a Shift-Reduce Parser Parse Tree
Actions of a Shift-Reduce Parser Parse Tree
Actions of a Shift-Reduce Parser Parse Tree
Actions of a Shift-Reduce Parser How do we build this tree? As productions are recognized, a portion of the tree is created. This portion of the tree will be needed later to build bigger portions of the tree and therefore must be saved for future use. This requires a stack. The stack plus the next token read from the source program determines the action.
Actions of a Shift-Reduce Parser Stack Current Symbol stack Parser Action Parser Engine
Actions of a Shift-Reduce Parser Shift –Shift the current element into top of the stack –Move the current pointer (next token) Reduce –Apply a production (we recognize a part of the program) –Top of the stack should match the RHS of the grammar –Remove those symbols from the stack –Add the LHS non-terminal Accept –End of stream reached and stack only has the start symbol Reject –End of stream reached but stack has more than the start symbol
Shift-Reduce Parser Example *(+num)
Shift-Reduce Parser Example *(+num) ( ) - num + - *
Shift-Reduce Parser Example *(+num) ( ) - num + - *
Shift-Reduce Parser Example *(+num) ( ) - num + - * num SHIFT
Shift-Reduce Parser Example *(+num) ( ) - num + - * num REDUCE
Shift-Reduce Parser Example *(+num) num ( ) - num + - * REDUCE
Shift-Reduce Parser Example (+num) num* ( ) - num + - * * SHIFT
Shift-Reduce Parser Example (+num) num* ( ) - num + - * * REDUCE
Shift-Reduce Parser Example (+num) num* ( ) - num + - * REDUCE
Shift-Reduce Parser Example +num) num*( ( ) - num + - * ( SHIFT
Shift-Reduce Parser Example *+num) num( ( ) - num + - * ( num SHIFT
Shift-Reduce Parser Example *+num) num( ( ) - num + - * ( num REDUCE
Shift-Reduce Parser Example *+num) num( ( ) - num + - * ( REDUCE
Shift-Reduce Parser Example *num) num( + ( ) - num + - * ( + SHIFT
Shift-Reduce Parser Example *num) num( + ( ) - num + - * ( + REDUCE
Shift-Reduce Parser Example *num) num( + ( ) - num + - * ( REDUCE
Shift-Reduce Parser Example *) num( + ( ) - num + - * ( num SHIFT
Shift-Reduce Parser Example *) num( + ( ) - num + - * ( num REDUCE
Shift-Reduce Parser Example *) num( + ( ) - num + - * ( REDUCE
Shift-Reduce Parser Example *) num( + ( ) - num + - * ( REDUCE
Shift-Reduce Parser Example *) num( + ( ) - num + - * ( REDUCE
Shift-Reduce Parser Example *) num( + ( ) - num + - * ( ) SHIFT
Shift-Reduce Parser Example *) num( + ( ) - num + - * ( ) REDUCE
Shift-Reduce Parser Example *) num( + ( ) - num + - * REDUCE
Shift-Reduce Parser Example *) num( + REDUCE ( ) - num + - *
Shift-Reduce Parser Example *) num( + REDUCE ( ) - num + - *
Shift-Reduce Parser Example *) num( + ACCEPT ( ) - num + - *
What does the parser engine do? If the top symbols of the stack match the RHS of a production then do the reduction –Pop the RHS from the top of the stack –Push the LHS symbol onto the stack If no production is found do the shift – Push the current input into the stack If the input is empty –Accept if only the start symbol is on the stack –Reject otherwise Parser Engine
This is not that simple! Many choices of reductions if there are multiple RHS. Which LHS do we put on stack in its place? Choice between shift and reduce –Stack matches a RHS –But that may not be the right match –May need to shift an input onto stack and later find a different reduction
Shift-Reduce Parser Example ( ) - num + - * Change in the Grammar
Shift-Reduce Parser Example ( ) - num + - * Change in the Grammar
Shift-Reduce Parser Example ( ) - num + - * Change in the Grammar
Shift-Reduce Parser Example ( ) - num + - * Change in the Grammar
Shift-Reduce Parser Example -num ( ) - num + - *
Shift-Reduce Parser Example -num ( ) - num + - * num
Shift-Reduce Parser Example -num ( ) - num + - * num SHIFT
num Shift-Reduce Parser Example -num ( ) - num + - * num REDUCE
Shift-Reduce Parser Example -num ( ) - num + - * num REDUCE
- Shift-Reduce Parser Example num ( ) - num + - * num - SHIFT
- Shift-Reduce Parser Example num ( ) - num + - * num - We have a choice!!! REDUCE
- Shift-Reduce Parser Example num ( ) - num + - * num - But not the right thing to do!! REDUCE
Shift-Reduce Parser Example num ( ) - num + - * num - REDUCE But not the right thing to do!!
num Shift-Reduce Parser Example num ( ) - num + - * num - But not the right thing to do!! SHIFT
num Shift-Reduce Parser Example num ( ) - num + - * num - But not the right thing to do!! REDUCE
Shift-Reduce Parser Example num ( ) - num + - * num - But not the right thing to do!! REDUCE
Shift-Reduce Parser Example num ( ) - num + - * num - But not the right thing to do!! No more actions!!! ERROR
Shift-Reduce Parser Example But this is perfectly valid input for the grammar We chose the wrong production and thus the wrong LHS Lets see what happens when we choose the right production and the right LHS
- Shift-Reduce Parser Example num ( ) - num + - * num - We have a choice REDUCE The step before we went wrong
- Shift-Reduce Parser Example num ( ) - num + - * num - Use the other production REDUCE
Shift-Reduce Parser Example num ( ) - num + - * num - REDUCE Use the other production
num Shift-Reduce Parser Example num ( ) - num + - * num - SHIFT
num Shift-Reduce Parser Example num ( ) - num + - * num - REDUCE
Shift-Reduce Parser Example num ( ) - num + - * num - REDUCE
Shift-Reduce Parser Example num ( ) - num + - * num - REDUCE
Shift-Reduce Parser Example num ( ) - num + - * num - ACCEPT
Shift-Reduce Parser Parser Engine is far more complicated that it appears. Requires it to know all possible productions that would match the top of the stack and the given input symbol.
Constructing a LR(k) Parser What is in the parse engine –decide between shift and reduce –decide on the right reduction
Constructing a LR(k) Parser Create a DFA –Encodes all the possible states that the parser can be in –DFA state transition occurs on terminals and non- terminals Create a Parser Table – From the DFA create a transition table that stores what action should be taken for the current state and current input character Maintain a stack of states in parallel with the stack of symbols
LR(k) Parser Engine Current Symbol Parser Action LR(k) Parser Engine Symbol Stack State Stack
Parser Tables Look-up [top of state stack] [ input symbol] in the parser table Carry-out the described action
Parser Tables Shift to sn –Push input token into the symbol stack –Push sn into state stack –Advance to next input symbol
Parser Tables Reduce (n) –Pop both stacks as many times as the number of symbols on the RHS of rule n –Push LHS of rule n into symbol stack –Lookup [top of the state stack][top of symbol stack] –Push that state (in goto k) into state stack
Parser Tables Accept –Stop parsing and report success
Parser Tables Error –Stop parsing and report failure
LR example The grammar $(1) ( )(2) ( )(3)
Question The grammar $(1) ( )(2) ( )(3)
Parser Table in Action The grammar $(1) ( )(2) ( )(3)
Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3)
Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) $
Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $
Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )()($
Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )()($
Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )()($
( s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ ))($
s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ ))(($
s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ ))(($
s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ ))(($
s2 ( ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ ))(($
s2 ( ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s2 ( ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s2 ( ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s5 ) s2 ( ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s5 ) s2 ( ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s5 ) s2 ( ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s5 ) s2 ( ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s5 ) s2 ( ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s5 ) s2 ( ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(() s5 ) s2 ( $
X ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
X s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
X s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s3 X s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s3 X s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s3 X s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s3 X s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s4 ) s3 X s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s4 ) s3 X s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s4 ) s3 X s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s4 ) s3 X s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s4 ) s3 X s2 ( Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$ s4 ) s3 X s2 (
X Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
X Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
X Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s1 X Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s1 X Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s1 X Parser Table in Action Parser Table The grammar $(1) ( )(2) ( )(3) s0 $ )(()$
s1 X Parser Table in Action The grammar $(1) ( )(2) ( )(3) s0 $ )(()$ Accept
Parser Table The table (DFA) and the stacks are called Push Down Automaton (PDA). There is an algorithm for converting a grammar to a PDA. Parser generators – given a grammar, produce a parser table with a stack (PDA) We will not study how this is done because its somewhat complicated. JCUP will convert our grammar into a shift-reduce parser.