1. Compiler Construction Lecture Three: Lexical Analysis - Part Two CSC 2103: Compiler Construction Lecture Three: Lexical Analysis - Part Two Joyce Nakatumba-Nabende 1

2. BUILD FINITE AUTOMATAS FOR THE FOLLOWING REGULAR EXPRESSIONS FOR SOME LEXICAL TOKENS Building Finite Automatas for Lexical Tokens ( Brief review and addition to lecture 2) 2 if {return IF;} [a - z] [a - z0 - 9 ] * {return ID;} [0 - 9] + {return NUM;} ([0 - 9] + “.” [0 - 9] *) | ([0 - 9] * “.” [0 - 9] +) {return REAL;} (“--” [a - z]* “\n”) | (“ ” | “ \n ” | “ \t ”) + {/* do nothing*/}. {error ();}

3. Building Finite Automatas for Lexical Tokens 3 3 The NFA for a symbol i is: i 12 start The NFA for the regular expression if is: f 3 1 start 2 i The NFA for a symbol f is: f 2 start 1 IF if {return IF;}

4. Cont’d 4 a-z 2 1 start ID [a-z] [a-z0-9 ] * {return ID;} 0-9 a-z

5. ([0 - 9] + “.” [0 - 9] *) | ([0 - 9] * “.” [0 - 9] +) {return REAL;} 5 5 1 start REAL 0-9 2 3. 5 4. 6 ε

6. Cont’d Note: Please build the finite automatas for the remaining lexical tokens for your own practice 6

7. Regular Expressions to NFAs 7 Lexical Specification Regular expressions NFA DFA Table-driven Implementation of DFA

8. Regular Expressions to NFAs RE  NFA ◦ Build NFA for each term; connect with  moves NFA  DFA ◦ Simulate the NFA using the subset construction DFA  minimized DFA ◦ Merge equivalent states 8

9. Regular Expressions to NFAs Question: How do we define an NFA that accepts a regular expression? Solution: It is very simple. Remember that a regular expression is formed by the use of: ◦ alternation ◦ concatenation ◦ repetition Thus all we need to do is to know how to build the NFA for a single NFA symbol, and how to compose NFAs. 9

10. Regular Expressions to NFAs For each kind of rexp, define an equivalent NFA. Notation: NFA for rexp M 10 M

11. Regular Expressions to NFAs For ε For input a 11 i ε i a j j

12. Regular Expressions to NFAs For AB 12 ε A B

13. Regular Expressions to NFAs For A|B 13 i ε A B ε ε ε j

14. Regular Expressions to NFAs For A* 14 iC j A ε ε ε ε

15. 15 1 4 a 3 a a b b b b 2 a Exam result

16. Regular Expressions to NFAs Construct an NFA that matches the regular expression r = (a|b)*a that matches {a, aa, ba, aaa, bba, ababa, ……}. Parse the regular expression into its sub expressions involving alphabet symbols a and b and ε : ε, a, b, a|b, ()* 16

17. Regular Expressions to NFAs NFA accepting the RExp (a|b)*a 17 E C D F G H B ε ε ε ε ε ε ε a ε a b I A

18. RE to NFAs Examples Draw an NFAs that accepts the following regular expressions 1.a*|b 2.(a|b)* 3.(a|b)*cd 18

19. Conversion of NFAs to DFAs 19 Lexical Specification Regular expressions NFA DFA Table-driven Implementation of DFA

20. Non-Deterministic Features of NFA Non-Deterministic Features of NFA There are three main cases of non- determinism in NFAs: 1. Transition to a state without consuming any input. 2. Multiple transitions on the same input symbol. 3. No transition on an input symbol. To convert NFAs to DFAs we need to get rid of non- determinism from NFAs.

21. Conversion of NFAs to DFAs Given: ◦ A non-deterministic finite state machine (NFA). Goal: ◦ Convert to an equivalent deterministic finite state machine (DFA). Why? ◦ Faster recognizer! 21

22. Conversion of NFAs to DFAs To convert an NFA to a DFA we use a concept of subset construction. Central to this we use the concept of the closure. A major step of the subset construction is to find the ε -closure for each state. ε -closure of a state s is the set of all states reachable from zero or more ε -transitions. 22

23. Subset Construction Method NFA without  -transitions 1 3 2 4 5 a a a a a b b b a,b

24. Subset Construction Method: Step One NFA without  -transitions 1 3 2 4 5 a a a a a b b b a,b Construct a transition table showing all reachable states for every state for every input symbols.

25. Subset Construction Method: Step One NFA without  -transitions 1 3 2 4 5 a a a a a b b b a,b state δ (q,a) δ (q,b) A{A,B,C,D,E}{D,E} B ∅{C,E} C{B} ∅ D{E}{D} E ∅∅

26. Subset Construction Method: Step One NFA without  -transitions 1 3 2 4 5 a a a a a b b b a,b state (q) δ (q,a) δ (q,b) 1{1,2,3,4,5}{4,5} 2 ∅{3,5} 3{2} ∅ 4{5}{4} 5 ∅∅ Transition from state q with input a Transition from state q with input b

27. Transition table Subset Construction Method: StepTwo The set of states resulting from every transition function constitutes a new state. Calculate all reachable states for every such state for every input symbol. state (q) δ (q,a) δ (q,b) 1{1,2,3,4,5}{4,5} 2 ∅{3,5} 3{2} ∅ 4{5}{4} 5 ∅∅

28. Transition table q δ (q,a) δ (q,b) A{A,B,C,D,E}{D,E} Starts with Initial state state δ (q,a) δ (q,b)1{1,2,3,4,5}{4,5} 2 ∅{3,5} 3{2} ∅ 4{5}{4} 5 ∅∅ Subset Construction table Subset Construction Method: Step Two

29. Transition table q δ (q,a) δ (q,b) A{A,B,C,D,E}{D,E} {A,B,C,D,E} {D,E} Starts with Initial state state δ (q,a) δ (q,b)1{1,2,3,4,5}{4,5} 2 ∅{3,5} 3{2} ∅ 4{5}{4} 5 ∅∅ Subset Construction table Subset Construction Method: Step Two Repeat this process until no more new states are reachable

30. Transition table q δ (q,a) δ (q,b) A{A,B,C,D,E}{D,E} {A,B,C,D,E}{A,B,C,D,E} {3,4,5} {3,4,5} {D,E}ED {C,D,E} {C,D,E}{B,E}D E ∅∅ DED {B,E} ∅{C,E} {C,E}B ∅ B ∅{C,E} Starts with Initial state state δ (q,a) δ (q,b) 1 A {1,2,3,4,5}{4,5} 2B ∅{3,5} 3C{2} ∅ 4D{5}{4} 5E ∅∅ Subset Construction table Subset Construction Method: Step Two

31. Conversion of NFAs to DFAs 31 1 3 2 4 5 a a a a a b b b a,b

32. Conversion of NFAs to DFAs A DFA has at most one edge from each state for a given symbol and is a suitable basis for a transition table. We need to eliminate the  -transitions by subset construction. 32

33. Conversion of NFAs to DFAs : Definitions OperationDescription  -closure(s)Set of NFA states reachable from NFA state s on  -transitions alone.  -closure(T)Set of NFA states that are reachable from the set of states T on  -transitions alone. Move(T,a)Set of states to which there is a transition on input symbol a from some NFA state in T. 33

34. Conversion of NFAs to DFAs NFA accepting the RExp (a|b)*a 34 E C D F G H B ε ε ε ε ε ε ε a ε a b I A

35. Conversion of NFAs to DFAs Start state: ε -Closure(A) = {A,B,C,E,H} = 1 35 E C D F G A H B ε ε ε ε ε ε ε a ε a b J

36. Conversion of NFAs to DFAs Start state: ε -Closure(A) = {A,B,C,E,H} = 1 Move DFA (1,a) = {D,J} = ε -Closure(Move DFA (1,a)) = ε -Closure({D,J}) = {B,C,D,E,G,H,J} = 2 36 E C D F G A H B ε ε ε ε ε ε ε a ε a b J

37. Conversion of NFAs to DFAs Start state: ε -Closure(A) = {A,B,C,E,H} = 1 Move DFA (1,b) = {F} = ε -Closure(Move DFA (1,b)) = ε -Closure({F}) = {B,C,E,F,G,H} = 3 37 E C D F G A H B ε ε ε ε ε ε ε a ε a b J

38. Conversion of NFAs to DFAs Process 2 = {B,C,D,E,G,H,J} Move DFA (2,a) = {D,J} = ε -Closure(Move DFA (2,a)) = ε -Closure({D,J}) = {B,C,D,E,G,H,J} = 2 38 E C D F G A H B ε ε ε ε ε ε ε a ε a b J

39. Conversion of NFAs to DFAs Process 2 = {B,C,D,E,G,H,J} Move DFA (2,b) = {F} = ε -Closure(Move DFA (2,a)) = ε -Closure({F}) = {B,C,E,F,G,H} = 3 39 E C D F G A H B ε ε ε ε ε ε ε a ε a b J

40. Conversion of NFAs to DFAs Process 3 = {B,C,E,F,G,H} Move DFA (3,a) = {D,J} = ε -Closure(Move DFA (3,a)) = ε -Closure({D,J}) = {B,C,D,E,G,H,J} = 2 40 E C D F G A H B ε ε ε ε ε ε ε a ε a b J

41. Conversion of NFAs to DFAs Process 3 = {B,C,E,F,G,H} Move DFA (3,b) = {F} = ε -Closure(Move DFA (3,b)) = ε -Closure({F}) = {B,C,E,F,G,H} = 3 41 E C D F G A H B ε ε ε ε ε ε ε a ε a b J

42. Conversion of NFAs to DFAs 42 E C D F G A H B ε ε ε ε ε ε ε a ε a b J 1 2 3 b b b a a a

43. Conversion of NFAs to DFAs: Exercise One 43 Consider the NFA below accepting the RExp (a|b)*abb. Convert the NFA to a DFA

44. Conversion of NFAs to DFAs: Exercise Two Consider the NFA accepting the RExp aa*|bb*. Convert the NFA to a DFA. 44

45. Conclusion 45 Lexical Specification Regular expressions NFA DFA Table-driven Implementation of DFA