1. Chapter 2 Chang Chi-Chung 2007.3.15

2. Lexical Analyzer The tasks of the lexical analyzer:  Remove white space and comments  Encode constants as tokens  Recognize Keywords and Identifiers  Store identifier names in a symbol table.

3. Lexical Analyzer Lexical analyzer Lexer() if (peek == ‘\n’) line = line +1 Parser or Syntax-Directed Translator Parser() token token Attribute

4. Remove white space and comments for ( ; ; peek = next character ) { if ( peek is a blank or a tab ) do nothing; else if (peek is a newline) line = line + 1; else break; } For white spaces and comments  Eliminated by the lexical analyzer.  Modifying the grammar to incorporate it into the syntax. ( not easy )

5. Reading Ahead A lexical analyzer may need ahead some characters before it can decide on the token to be returned to the parser.  Examples To distinguish between 1 and 10. To distinguish between t and ture. To distinguish between > and >=. Approachs  To maintain an input buffer.  One-character read-ahead.

6. Encode constants as tokens For a sequence of digits, the lexical analyzer must pass to the parser a token.  The token consists of the terminal along with an integer- valued attribute computed from the digits. Example  31 + 28 + 29  if ( peek holds a digit ) { v = 0; do { v = v * 10 + integer value of digit peek; peek = next input character; } while ( peek holds a digit) return token ; }

7. Recognize Keywords and Identifiers Keyword  A fixed character string as punctuation marks or to identify constructs.  Example for 、 while 、 if Identifier  Use to name variables, arrays, functions, and the like.  Parser treat identifiers as terminals.  Example count = count + increment; =

8. Recognize Keywords and Identifiers The lexical analyzer uses a table to hold character strings.  A string table can be implemented by a hash table.  Single Representation  Reserved Words. if ( peek holds a letter ) { collect letters or digits into a buffer b ; s = string formed from the characters in b ; w = token returned by words.get(s) ; if (w is not null) return w; else { Enter the key-value pair (s, ) into words return token ; }

9. Create a Lexical Analyzer Token scan() { skip white space. (A) handle numbers. (B) handle reserved words and identifiers. (C) Token t = new Token ( peek ); peek = blank; (D) return t ; }

10. Complete Lexical Analyzer (1) package lexer; public class Token { public final int tag; public Token(int t) { tag = t; } } public class Tag { public final static int NUM = 256, ID = 257, TRUE = 258, FALSE = 259; } public class Num extends Token { public final int value; public Num(int v) { super(Tag.NUM); value = v; } public class Word extends Token { public final String lexeme; public Word(int t, String s) { super(t); lexeme = new String(s); } class Token +int tag class Num +int value class Word +string lexeme

11. Complete Lexical Analyzer (2) package lexer; import java.io.*; import java.util.*; public class Lexer { public int line = 1; private char peek = ' '; private Hashtable words = new Hashtable(); void reserve(Word t) { words.put(t.lexeme, t); } public Lexer() { reserve( new Word(Tag.TRUE, "true") ); reserve( new Word(Tag.FALSE, "false") ); }

12. Complete Lexical Analyzer (3) public Token scan() throws IOException { for ( ; ; peek = (char) System.in.read() ) { if ( peek == ' ' || peek == '\t' ) continue; else if ( peek == '\n' ) line = line + 1; else break; } if ( Character.isDigit(peek) ) { int v = 0; do { v = v * 10 + Character.digit(peek, 10); peek = (char) System.in.read(); } while ( Character.isDigit(peek) ) return new Num(v); } } C D

13. Complete Lexical Analyzer (4) public Token scan() throws IOException { if ( Character.isLetter(peek) ) { StringBuffer b = new StringBuffer(); do { b.append(peek); peek = (char) System.in.read() } while ( Character.isLetterOrDigit(peek) ); String s = b.toString(); Word w = (Word) words.get(s); if (w != null) return w; w = new Word(Tag.ID, s); words.put(s, w); return w; } Token t = new Token(peek); peek = ' '; return t; } A B

14. Symbol Tables Symbol tables are data structures  Used by compilers to hold information about source-program constructs. Scope of identifier x  The scope of a particular declaration x Scope  A portion of a program that is the scope of one or more declaration.

15. Symbol Tables w xint y w y bool zint B0B0 B1B1 B3B3 { int x 1, int y 1 ; { int w 2 ; bool y 2 ; int z 2 ; w 2 ; x 1 ; y 2 ; z 2 ; } w 0 ; x 1 ; y 1 ; }

16. Symbol Tables package symbols; import java.util.*; public class Env { private Hashtable table; protected Env prev; public Env(Env p) { table = new Hashtable(); prev = p; } public void put(String s, Symbol sym) { table.put(s, sym); } public Symbol get(String s) { for (Env e = this; e != null; e = e.prev) { Symbol found = (Symbol)(e.table.get(s)); if (found != null) return found; } return null; } w xint y w y bool zint B0B0 B1B1 B3B3

17. The Use of Symbol Tables program → block { top = null; } block → ‘{‘ { saved = top; top = new Env(top); print(“{ “); } decls stmts ‘}’ { top = saved; print(“} “); } decls → decls decl | ε decl → type id ; { s = new Symbol; s.type = type.lexeme; top.put(id.lexeme, s); } stmts → stmts stmt | ε stmt → block | factor ; { print(“; “); } factor → id { s = top.get(id.lexeme); print(id.lexeme); print(“:”); print(s.type); }

18. Intermediate Code Generation Two most important intermediate representations.  Trees Parse trees, syntax trees (abstract trees) Examples  while ( expr ) stmt  op: while E 1 : expr E 2 : stmt  Linear representations Three-address code x, y, z: names, constants, compiler-generated temporaries. Examples  ifFalse x goto L  ifTrue x goto L  goto L  x = y  x [ y ] = z  x = y [ z ] op E1E1 E2E2 x = y op z

19. Three-Address Code ifFalse x goto after code to compute expr into x code for stmt 1 after if expr then stmt 1  ifFalse x goto after x = i – j + k t1 = i – j t2 = t1 + k x = t2 temporary x = 2 * a [ i ] t1 = a [ i ] t2 = 2 * t1 x = t2

20. Intermediate Code Generation Parser or Syntax-Directed Translator Parser() If eq peek assign (int) ‘\n’ line 1 + 1: t1 = (int) ‘\n’ 2: ifFalse peek == t1 goto 4 3: line = line + 1 4: or if (peek == ‘\n’) line = line +1

21. Static Checking Static checks are consistency checks that are done during compilation.  Syntactic Checking An identifier being declared at most once in a scope. A break statement must have an enclosing loop or switch statement.  Type Checking l-values and r-values  r-values are as values.  l-values are locations.  Examples i = 5 i = i + 1

22. Syntax Trees Concrete SyntaxAbstract Syntax = || && == != = > + - * / % ! - unary [ ] assign cond rel op not minus access

23. Syntax Trees seq if while null some tree for an expression some tree for an expression some tree for an expression some tree for an expression