Lexical Analysis Dragon Book: chapter 3

Intermediate code generator Compiler structure Source program Lexical analyzer Syntax analyzer Error handling Symbol table Semantic analyzer Intermediate code generator Code optimizer Code generator Target program

Compiler structure Lexical analyzer Syntax analyzer Error handling Source program Lexical analyzer token Get next token Syntax analyzer Error handling Symbol table

Tokens in programming languages Sample instances Description if id keyword rel <, <=, <>, >=, > relation count, length, point2 variable num 3.1415927, 7, 145e-3 Numerical constant str “abc”, “some space” “\7\” is a char” Constant string

Tokens may be difficult to recognize Fortran: DO 5 I=1.25 DO 5 I=1,25 (spaces do not count). PL/I: IF THEN THEN THEN=ELSE; ELSE ELSE=THEN; (no reserved keywords). PL/I: PR1(2, 7, 18, D*3, 175.14)=3 (proc. call or array reference).

Strings, languages. A sequence of characters over some alphabet, e.g., 0100110 over {0, 1}. In computers, usually ASCII or EBCDIC. Length of strings: number of characters. Empty string:  (size 0). Concatenation: putting one string after another. X=dog, Y=house, XY=doghouse (also X.Y). Prefix: ban is prefix of banana. Suffix: ana is prefix of banana.

Language: a set of strings The alphabet is a language: L={A, B, …, Z, a, b, …, z}. Constant languages: X={ab, ba}, Y={a}. Concatenation: X.Y = {aba, baa}. Y.X = {aab, aba}. Union: XY=X+Y=X|Y={ab, ba, a}. Exponentation: X3 = X.X.X Star: X* = zero or more occurrences. L* = all words with letters from L. L+= all words with one or more letters from L.

Regular expressions X|Y = XY= { s | sX or sY }. X.Y = { x.y | xX and yY }. X* = i=0, Xi. X+ = i=1, Xi.

Examples a|b = {a, b}. (a|b).(a|b) = {aa, ab, ba, bb}. a* = { , a, aa, aaa, … }. (a|b)* = { , a, b, ab, ba, aa, aba, … }

Defining tokens digit  [0-9] digits  digit+ fraction  . digits |  exponent  E ( + | - |  ) digits |  const  digits fraction exponent

Not everything is regular! All the words of the form w c w, where w is a word and c a letter. The syntax of a program, e.g., the recursive definition of if-then-else. stmtif expr then stmt else stmt.

If a>8 then goto nextloop else begin while z>8 do Reading the input If a>8 then goto nextloop else begin while z>8 do Token starts here Last character read Need sometimes to “lookahead”. For example: identifying the variable done. May need to “unread” a character.

Returning: token + attributes. if xyz > 11 then if, keyword id, value=xyz op, value=“>”. const, value=11 then, keyword.

Finite Automata Includes: States {s1,s2,…,s5}. Initial states {s1}. Accepting states {s3,s5}. Alphabet {a, b, c}. Transitions: {(s1,a,s2), (s2, a, s3), …}. s1 a b b s2 b a s5 c b a c s4 s3 a Deterministic?

Automaton. What is the language? b s0 a b s1 Formally: An input is a word over the alphabet . A run over a word is an alternating sequence of states and letters, starting from the initial state. Accepting run: ends with an accepting state.

Example s0 s1 b a b Input: aabbb Run: s0 a s0 a s0 b s1 b s1 b s1. Accepts. Input: aba Run: s0 a s0 b s1 a s0. Does not accept.

Automaton. What is the language? b s0 s1 a b a

Automaton. What is the language? b s1 a s0 b a

Identifying tokens F I T H E N E L S E letter letter|digit

Non deterministic automata 0,1 1 s3 Allows more than a single transition from a state with the same label. There does not have to be a transition from every state with every label. Allows multiple initial states. Allows  transitions.

Nondeterministic runs 0,1 1 s3 Input: 0100 Run 1: s0 0 s0 1 s0 0 s0 0 s0. Does not accept. Run 2: s0 0 s0 1 s1 0 s2 0 s3. Accepts. Accepts when there exists an accepting run.

Determinizing Automata s0 s1 s2 0,1 1 s3 Each state of D is a set of the states of N. S—aT when T={t|sS and s—at}. The initial state of D includes all the initial states of N. Accepting states in D include at least one accepting state of N.

Determinization s0 s1 s2 s3 1 0,1 0,1 0,1 s0 s0,s3 s0,s2 s0,s1 1 1 1 1

Determinization 000 100 010 101 110 111 011 001 1 1

Translating regular expressions into automata     L1L2 L2 L1.L2 L1  L2   L   L*

Automatic translation (a|b).(a.b)=(ab)(ab)=(a+b).(a+b)=… a a b  b a  a     b b    

Determinization with  transitions. b b s8 s10 s2 s4     Add to each set states reachable using  transitions. s0,s1,s2 s3,s5,s6,s7,s8 s9,s11 s4,s5,s6,s7,s8 s10,s11 a b

Minimization a b p1 p3 p0 p2 p4  Group all the states together.  Separate states according to available exit transitions.  Separate a set to two if from some of its states one can reach another set and with others one cannot. Repeat until cannot separate.

Minimization a b p1 p3 p0 p2 p4 Group all the states together.

Minimization p0 p1 p3 p2 p4 a b Separate states according to available exit transitions.

Minimization p0 p1 p3 p2 p4 a b  Separate a set to two if from some of its states one can reach another set and with others one cannot. Repeat until cannot separate.

Can minimize now a b   a a b b

Lex Declarations %% Translation rules Auxiliary procedures

Lex behavior Lex Program a.out Lex source program lex.l lex.yy.c C Compiler Lex source program lex.l lex.yy.c a.out a.out Input streem Output tokens

Lex behavior Translates the definitions into an automaton. The automaton looks for the longest matching string. Either return some value to the reading program (parser), or looks for next token. Lookahead operator: x/y  allow the token x only if y follows it (but y is not part of the token).

Lex Project Project collection date: Feb 11th. Work in pairs (singles). Use lex to take a text and check whether the number of open parentheses of any kind is equal to the number of closed parentheses. Exception: Inside quotes. \” is not a closing quote.