9/27/2006Prof. Hilfinger, Lecture 141 Parsing Odds and Ends Lecture 14 (P. N. Hilfinger, plus slides adapted from R. Bodik)

9/27/2006Prof. Hilfinger, Lecture 142 Administrivia Trial run of project autograder on Wednesday night (sometime). No other runs until Friday.

9/27/2006Prof. Hilfinger, Lecture 143 Topics Syntax-directed translation and LR parsing Syntax-directed translation and recursive descent Dealing with errors in LR parsing: quick and dirty approach

9/27/2006Prof. Hilfinger, Lecture 144 Syntax-Directed Translation and LR Parsing Idea: Add semantic stack, parallel to the parsing stack: –each symbol (terminal or non-terminal) on the parsing stack stores its value on the semantic stack –each reduction uses the values at the top of the stack to compute the new value to be associated with the symbol that’s produced –when the parse is finished, the semantic stack will hold just one value: the translation of the root non-terminal, which is the translation of the whole input.

9/27/2006Prof. Hilfinger, Lecture 145 Semantic actions during parsing when shifting –push the value of the terminal on the semantic stack when reducing –pop k values from the semantic stack, where k is the number of symbols on production’s RHS –push the production’s value on the semantic stack

9/27/2006Prof. Hilfinger, Lecture 146 An LR example Grammar + translation rules: E  E + ( E )$$ = $1 + $4 E  int $$ = $1 Input: 2 + ( 3 ) + ( 4 )

9/27/2006Prof. Hilfinger, Lecture 147 Shift-Reduce Example with evaluations parsing stacksemantic stack I int + (int) + (int)$ shift I

9/27/2006Prof. Hilfinger, Lecture 148 Shift-Reduce Example with evaluations I int + (int) + (int)$ shift I int I + (int) + (int)$ red. E  int2 I

9/27/2006Prof. Hilfinger, Lecture 149 Shift-Reduce Example with evaluations I int + (int) + (int)$ shift I int I + (int) + (int)$ red. E  int2 I E I + (int) + (int)$ shift 3 times2 I

9/27/2006Prof. Hilfinger, Lecture 1410 Shift-Reduce Example with evaluations I int + (int) + (int)$ shift I int I + (int) + (int)$ red. E  int2 I E I + (int) + (int)$ shift 3 times2 I E + (int I ) + (int)$ red. E  int2 ‘+’ ‘(‘ 3 I

9/27/2006Prof. Hilfinger, Lecture 1411 Shift-Reduce Example with evaluations I int + (int) + (int)$ shift I int I + (int) + (int)$ red. E  int2 I E I + (int) + (int)$ shift 3 times2 I E + (int I ) + (int)$ red. E  int2 ‘+’ ‘(‘ 3 I E + (E I ) + (int)$ shift 2 ‘+’ ‘(‘ 3 I

9/27/2006Prof. Hilfinger, Lecture 1412 Shift-Reduce Example with evaluations I int + (int) + (int)$ shift I int I + (int) + (int)$ red. E  int2 I E I + (int) + (int)$ shift 3 times2 I E + (int I ) + (int)$ red. E  int2 ‘+’ ‘(‘ 3 I E + (E I ) + (int)$ shift2 ‘+’ ‘(‘ 3 I E + (E) I + (int)$ red. E  E + (E) 2 ‘+’ ‘(‘ 3 ‘)’ I

9/27/2006Prof. Hilfinger, Lecture 1413 Shift-Reduce Example with evaluations I int + (int) + (int)$ shift I int I + (int) + (int)$ red. E  int2 I E I + (int) + (int)$ shift 3 times2 I E + (int I ) + (int)$ red. E  int2 ‘+’ ‘(‘ 3 I E + (E I ) + (int)$ shift2 ‘+’ ‘(‘ 3 I E + (E) I + (int)$ red. E  E + (E) 2 ‘+’ ‘(‘ 3 ‘)’ I E I + (int)$ shift 3 times 5 I

9/27/2006Prof. Hilfinger, Lecture 1414 Shift-Reduce Example with evaluations I int + (int) + (int)$ shift I int I + (int) + (int)$ red. E  int2 I E I + (int) + (int)$ shift 3 times2 I E + (int I ) + (int)$ red. E  int2 ‘+’ ‘(‘ 3 I E + (E I ) + (int)$ shift 2 ‘+’ ‘(‘ 3 I E + (E) I + (int)$ red. E  E + (E) 2 ‘+’ ‘(‘ 3 ‘)’ I E I + (int)$ shift 3 times 5 I E + (int I )$ red. E  int5 ‘+’ ‘(‘ 4 I

9/27/2006Prof. Hilfinger, Lecture 1415 Shift-Reduce Example with evaluations I int + (int) + (int)$ shift I int I + (int) + (int)$ red. E  int2 I E I + (int) + (int)$ shift 3 times2 I E + (int I ) + (int)$ red. E  int2 ‘+’ ‘(‘ 3 I E + (E I ) + (int)$ shift 2 ‘+’ ‘(‘ 3 I E + (E) I + (int)$ red. E  E + (E) 2 ‘+’ ‘(‘ 3 ‘)’ I E I + (int)$ shift 3 times 5 I E + (int I )$ red. E  int5 ‘+’ ‘(‘ 4 I E + (E I )$ shift5 ‘+’ ‘(‘ 4 I

9/27/2006Prof. Hilfinger, Lecture 1416 Shift-Reduce Example with Evaluations I int + (int) + (int)$ shift I int I + (int) + (int)$ red. E  int2 I E I + (int) + (int)$ shift 3 times2 I E + (int I ) + (int)$ red. E  int2 ‘+’ ‘(‘ 3 I E + (E I ) + (int)$ shift 2 ‘+’ ‘(‘ 3 I E + (E) I + (int)$ red. E  E + (E) 2 ‘+’ ‘(‘ 3 ‘)’ I E I + (int)$ shift 3 times 5 I E + (int I )$ red. E  int5 ‘+’ ‘(‘ 4 I E + (E I )$ shift5 ‘+’ ‘(‘ 4 I E + (E) I $ red. E  E + (E) 5 ‘+’ ‘(‘ 4 ‘)’ I

9/27/2006Prof. Hilfinger, Lecture 1417 Shift-Reduce Example with evaluations I int + (int) + (int)$ shift I int I + (int) + (int)$ red. E  int2 I E I + (int) + (int)$ shift 3 times2 I E + (int I ) + (int)$ red. E  int2 ‘+’ ‘(‘ 3 I E + (E I ) + (int)$ shift 2 ‘+’ ‘(‘ 3 I E + (E) I + (int)$ red. E  E + (E) 2 ‘+’ ‘(‘ 3 ‘)’ I E I + (int)$ shift 3 times 5 I E + (int I )$ red. E  int5 ‘+’ ‘(‘ 4 I E + (E I )$ shift5 ‘+’ ‘(‘ 4 I E + (E) I $ red. E  E + (E) 5 ‘+’ ‘(‘ 4 ‘)’ I E I $ accept9 I

9/27/2006Prof. Hilfinger, Lecture 1418 Taking Advantage of Derivation Order So far, rules have been functional; no side effects except to define (once) value of LHS. LR parsing produces reverse rightmost derivation. Can use the ordering to do control semantic actions with side effects.

9/27/2006Prof. Hilfinger, Lecture 1419 Example of Actions with Side Effects E  E + T print “+”, E  T pass T  T * F print “*”, T  F pass F  int print $1, F  ( E ) pass We know that reduction taken after all the reductions that form the nonterminals on right-hand side. So what does this print for 3+4*(7+1)? * +

9/27/2006Prof. Hilfinger, Lecture 1420 Recursive-Descent Translation Translating with recursive descent is also easy. The semantic values (what Bison calls $$, $1, etc.), become return values of the parsing functions We’ll also assume that the lexer has a way to return lexical values (e.g., the scan function introduced in Lecture 9 might do so).

9/27/2006Prof. Hilfinger, Lecture 1421 E  T | E+T T  P | T*P P  int | ‘(‘ E ‘)’ def E(): T () while next() == “+”: scan(“+”); T() def T(): P() while next() == “*”: scan(“*”); P() Example of Recursive-Descent Translation def P(): if next()==int: scan (int) elif next()== “ ( “ : scan( “ ( “ ) E() scan( “ ) ” ) else: ERROR() (we ’ ve cheated and used loops; see Recursive Descent for Real slide from lecture 9)

9/27/2006Prof. Hilfinger, Lecture 1422 E  T | E+T T  P | T*P P  int | ‘(‘ E ‘)’ def E(): v = T () while next() == “+”: scan(“+”); v += T() return v def T(): v = P() while next() == “*”: scan(“*”); v *= P() return v Example contd.: Add Semantic Values def P(): if next()==int: v = scan (int) elif next()== “ ( “ : scan( “ ( “ ) v = E() scan( “ ) ” ) else: ERROR() return v

9/27/2006Prof. Hilfinger, Lecture 1423 Table-Driven LL(1) We can automate all this, and add to the LL(1) parser method from Lecture 9. However, this gets a little involved, and I’m not sure it’s worth it. (That is, let’s leave it to the LL(1) parser generators for now!)

9/27/2006Prof. Hilfinger, Lecture 1424 Dealing with Syntax Errors One purpose of the parser is to filter out errors that show up in parsing Later stages should not have to deal with possibility of malformed constructs Parser must identify error so programmer knows what to correct Parser should recover so that processing can continue (and other errors found) Parser might even correct error (e.g., PL/C compiler could “correct” some Fortran programs into equivalent PL/1 programs!)

9/27/2006Prof. Hilfinger, Lecture 1425 Identifying Errors All of the valid parsers we’ve seen identify syntax errors “as soon as possible.” Valid prefix property: all the input that is shifted or scanned is the beginning of some valid program … But the rest of the input might not be So in principle, deleting the lookahead (and subsequent symbols) and inserting others will give a valid program.

9/27/2006Prof. Hilfinger, Lecture 1426 Automating Recovery Unfortunately, best results require using semantic knowledge and hand tuning. –E.g., a(i].y = 5 might be turned to a[i].y = 5 if a is statically known to be a list, or a(i).y = 5 if a function. Some automatic methods can do an OK job that at least allows parser to catch more than one error.

9/27/2006Prof. Hilfinger, Lecture 1427 Bison’s Technique The special terminal symbol error is never actually returned by the lexer. Gets inserted by parser in place of erroneous tokens. Parsing then proceeds normally.

9/27/2006Prof. Hilfinger, Lecture 1428 Example of Bison’s Error Rules Suppose we want to throw away bad statements and carry on stmt : whileStmt | ifStmt | … | error NEWLINE ;

9/27/2006Prof. Hilfinger, Lecture 1429 Response to Error Consider erroneous text like if x y: … When parser gets to the y, will detect error. Then pops items off parsing stack until it finds a state that allows a shift or reduction on ‘error’ terminal Does reductions, then shifts ‘error’. Finally, throws away input until it finds a symbol it can shift after ‘error’

9/27/2006Prof. Hilfinger, Lecture 1430 Error Response, contd. So with our example: stmt : whileStmt | ifStmt | … | error NEWLINE ; We see ‘y’, throw away the ‘if x’, so as to be back to where a stmt can start. Shift ‘error’ and away more symbols to NEWLINE. Then carry on. Bad input: if x y: … x = 0

9/27/2006Prof. Hilfinger, Lecture 1431 Of Course, It’s Not Perfect “Throw away and punt” is sometimes called “panic-mode error recovery” Results are often annoying. For example, in our example, there’s an INDENT after the NEWLINE, which doesn’t fit the grammar and causes another error. Bison compensates in this case by not reporting errors that are too close together But in general, can get cascade of errors. Doing it right takes a lot of work.