Download presentation

Presentation is loading. Please wait.

Published byTabitha Hood Modified over 4 years ago

1
LEXICAL ANALYSIS Phung Hua Nguyen University of Technology 2006

2
Faculty of IT - HCMUTLexical Analysis2 Outline Introduction to Lexical Analysis Token specification –Language –Regular Expressions (REs) Token recoginition –REs NFA (Thompson’s construction, Algorithm 3.3) –NFA DFA (subset construction, Algorithm 3.2) –DFA minimal DFA (Algorithm 3.6) Programming

3
Faculty of IT - HCMUTLexical Analysis3 Introduction Read the input characters Produce as output a sequence of tokens Eliminate white space and comments lexical analyzer parser symbol table source program token get next token

4
Faculty of IT - HCMUTLexical Analysis4 Why ? Simplify design Improve compiler efficiency Enhance compiler portability

5
Faculty of IT - HCMUTLexical Analysis5 Tokens, Patterns, Lexemes TokenSample Lexeme Informal description of pattern const if relation,>= or >= idpi, count, x2letter followed by letters or digits num3.14, 25, 6.02E3any numeric constant literal“core dumped”any characters between “ and “ except “

6
Faculty of IT - HCMUTLexical Analysis6 Outline Introduction Token specification –Language –Regular Expressions (REs) Token recoginition –REs NFA (Thompson’s construction, Algorithm 3.3) –NFA DFA (subset construction, Algorithm 3.2) –DFA minimal DFA (Algorithm 3.6) Programming

7
Faculty of IT - HCMUTLexical Analysis7 Alphabet, Strings and Languages Alphabet ∑ : any finite set of symbols –The Vietnamese alphabet {a, á, à, ả, ã, ạ, b, c, d, đ,…} –The binary alphabet {0,1} –The ASCII alphabet String: a finite sequence of symbols drawn from ∑ : –Length |s| of a string s: the number of symbols in s –The empty string, denoted , | | = 0 Language: any set of strings over ∑ ; –its two special cases: : the empty set { }

8
Faculty of IT - HCMUTLexical Analysis8 Examples of Languages ∑ ={ a, á, à, ả, ã, ạ, b, c, d, đ,… } –Vietnamese language ∑ = { 0,1 } –A string is an instruction –The set of Pentium instructions ∑ = the ASCII set –A string is a program –The set of C programs

9
Faculty of IT - HCMUTLexical Analysis9 Terms (Fig.3.7) TermDefinition prefix of sa string obtained by removing 0 or more trailing symbols of s; e.g. ban is a prefix of banana suffix of sa string formed by deleting 0 or more the leading symbols of s; e.g. na is a suffix of banana substring of sa string obtained by deleting a prefix and a suffix from s; e.g. nan is a substring of banana proper prefix, suffix or substring of s Any nonempty string x that is, respectively, a prefix, suffix os substring of s such that s x

10
Faculty of IT - HCMUTLexical Analysis10 String operations String concatenation –If x and y are strings, xy is the string formed by appending y to x. E.g.: x = hom, y = nay xy = homnay – is the identity: y = y; x = x String exponentiation –s 0 = –s i = s i-1 s E.g. s = 01, s 0 = , s 2 = 0101, s 3 = 010101

11
Faculty of IT - HCMUTLexical Analysis11 Language Operations (Fig 3.8) TermDefinition union: L ML M = { s | s L or s M } concatenation: LM LM= { st | s L or t M } Kleene closure: L * L * = L 0 L LL LLL … where L 0 = { } 0 or more concatenations of L positive closure: L + L + = L LL LLL … 1 or more concatenations of L

12
Faculty of IT - HCMUTLexical Analysis12 Examples L = {A,B,…,Z,a,b,…,z} D = {0,1,…,9} ExampleLanguage L D LD L 4 L * L(L D) * D + letters and digits strings consists of a letter followed by a digit all four-letter strings all strings of letters, including all strings of letters and digits beginning with a letter all strings of one or more digits

13
Faculty of IT - HCMUTLexical Analysis13 Regular Expressions (Res) over Alphabet ∑ Inductive base: 1. is a RE, denoting the RL { } 2.a ∑ is a RE, denoting the RL {a} Inductive step: Suppose r and s are REs, denoting the language L(r) and L(s). Then 3.(r)|(s) is a RE, denoting the RL L(r) L(s) 4.(r)(s) is a RE, denoting the RL L(r)L(s) 5.(r)* is a RE, denoting the RL (L(r))* 6.(r) is a RE, denoting the RL L(r)

14
Faculty of IT - HCMUTLexical Analysis14 Precedence and Associativity Precedence: –“*” has the highest precedence –“concatenation” has the second highest precedence –“|” has the lowest precedence Associativity: –all are left-associative E.g.: (a)|((b)*(c)) a|b*c Unnecessary parentheses can be removed

15
Faculty of IT - HCMUTLexical Analysis15 Example ∑ = {a, b} 1.a|b denotes {a,b} 2.(a|b)(a|b) denotes {aa,ab,ba,bb} 3.a* denotes { ,a,aa,aaa,aaaa,…} 4.(a|b)* denotes ? 5.a|a*b denotes ?

16
Faculty of IT - HCMUTLexical Analysis16 Notational Shorthands One or more instances +: r+ = rr* –denotes the language (L(r))+ –has the same precedence and associativity as * Zero or one instance ?: r? = r| –denotes the language (L(r) { }) Character classes –[abc] denotes a|b|c –[A-Z] denotes A|B|…|Z –[a-zA-Z_][a-zA-Z0-9_]* denotes ?

17
Faculty of IT - HCMUTLexical Analysis17 Outline Introduction Token specification –Language –Regular Expressions (REs) Token recoginition –REs NFA (Thompson’s construction, Algorithm 3.3) –NFA DFA (subset construction, Algorithm 3.2) –DFA minimal DFA (Algorithm 3.6) Programming

18
Faculty of IT - HCMUTLexical Analysis18 Overview RE NFADFA mDFA 3.5 3.6 3.2 3.3

19
Faculty of IT - HCMUTLexical Analysis19 Nondeterministic finite automata A nondeterministic finite automaton (NFA) is a mathematical model that consists of –a finite set of states S –a set of input symbols ∑ –a transition function move: S ∑ S –a start state s 0 –a finite set of final or accepting states F

20
Faculty of IT - HCMUTLexical Analysis20 Transition graph state transition start state final state AB a A A A

21
Faculty of IT - HCMUTLexical Analysis21 Transition table ab 0{0,1}{0} 1-{2} 2-{3} Input symbol State

22
Faculty of IT - HCMUTLexical Analysis22 Acceptance A NFA accepts an input string x iff there is some path in the transition graph from start state to some accepting state such that the edge labels along this path spell out x. A B 0 1 01010 01011 A B A B A B 010 10 A B A B A ? 0 1 0 11 error 0 1 0

23
Faculty of IT - HCMUTLexical Analysis23 Deterministic finite automata A deterministic finite automaton (DFA) is a special case of NFA in which 1.no state has an -transition, and 2.for each state s and input symbol a, there is at most one edge labeled a leaving s.

24
Faculty of IT - HCMUTLexical Analysis24 Thompson’s construction of NFA from REs guided by the syntactic structure of the RE r For , For a in ∑ if if a

25
Faculty of IT - HCMUTLexical Analysis25 Thompson’s construction (cont’d) Suppose N(s) and N(t) are NFA’s for REs s and t –For s|t, –For st, –For s*, –For (s), use N(s) itself N(s) N(t) i f N(s) i f N(t) i f

26
Faculty of IT - HCMUTLexical Analysis26 Outline Introduction Token specification –Language –Regular Expressions (REs) Token recoginition –REs NFA (Thompson’s construction) –NFA DFA (subset construction) –DFA minimal DFA (Algorithm 3.6) Programming

27
Faculty of IT - HCMUTLexical Analysis27 Subset construction OperationDescription -closure(s) Set of NFA states reachable from state s on -transition alone -closure(T) Set of NFA states reachable from some state s in T on -transition alone move(T,a)Set of NFA states to which there is a transition on input a from some state s in T s : an NFA state T : a set of NFA states

28
Faculty of IT - HCMUTLexical Analysis28 Subset construction (cont’d) Let s 0 be the start state of the NFA; Dstates contains the only unmarked state -closure(s 0 ); while there is an unmarked state T in Dstates do begin mark T for each input symbol a do begin U := -closure(move(T; a)); if U is not in Dstates then Add U as an unmarked state to Dstates; DTran[T; a] := U; end;

29
Faculty of IT - HCMUTLexical Analysis29 DFA Let ( ∑, S, T, F, s 0 ) be the original NFA. The DFA is: The alphabet: ∑ The states: all states in Dstates The transitions: DTran The accepting states: all states in Dstates containing at least one accepting state in F of the NFA The start state: -closure(s0)

30
Faculty of IT - HCMUTLexical Analysis30 Outline Introduction Token specification –Language –Regular Expressions (REs) Token recoginition –REs NFA (Thompson’s construction) –NFA DFA (subset construction) –DFA minimal DFA (Algorithm 3.6) Programming

31
Faculty of IT - HCMUTLexical Analysis31 Minimise a DFA Initially, create two states: 1.one is the set of all final states: F 2.the other is the set of all non-final states: S - F while (more splits are possible) { Let S = {s 1,…, s n } be a state and c be any char in ∑ Let t 1,…, t n be the successor states to s 1,…, s n under c if (t 1,…, t n don't all belong to the same state) { Split S into new states so that s i and s j remain in the same state iff t i and t j are in the same state }

32
Faculty of IT - HCMUTLexical Analysis32 Example ABD E C b b b b b a a a a a Step1: {A,B,C,D}{E} For a, {B,B,B,B} For b, {C,D,C,E} Split {A,B,C} {D}{E} Step 2: For b, {C,D,C} Split {A,C} {B} {D} {E} Step 3: For a, {B,B} For b, {C,C} Terminate ABD E b b b b b a a a a

33
Faculty of IT - HCMUTLexical Analysis33 Outline Introduction Token specification –Language –Regular Expressions (REs) Token recoginition –REs NFA (Thompson’s construction) –NFA DFA (subset construction) –DFA minimal DFA (Algorithm 3.6) Programming

34
Faculty of IT - HCMUTLexical Analysis34 Input Buffering begin…begin… Scanner eof if (forward at end of first half) { reload second half forward++ } else if (forward at end of second half) { reload first half forward = 0 } else forward++

35
Faculty of IT - HCMUTLexical Analysis35 Input Buffering begin…begin… Scanner eof forward = forward + 1 if (forward↑=eof) { if (forward at end of first half) { reload second half forward++ } else if (forward at end of second half) { reload first half forward = 0 } else terminate the analysis }

36
Faculty of IT - HCMUTLexical Analysis36 Transition Diagrams relop 01 2 3 4 < = > other return(relop,LE) return(relop,NE) return(relop,LT) id letter(letter|digit)* 56 7 letter letter or digit other return(id,lexeme) Transition diagram is a DFA in which there is no edge leaving out of a final state

37
Faculty of IT - HCMUTLexical Analysis37 Implementation token nexttoken() { while (1) { switch (state) { case 0: c = nextchar(); if (c == ‘<‘) state = 1; else state = fail(0); break; case 1: c = nextchar(); if (c == ‘=‘) state = 2; else if (c == ‘>’ state = 3; else state = 4; break; case 2: retract(0); return new Token(relop,”<=”); case 4: retract(1); return new Token(relop,”<”); case 5: c = nextchar(); if (Character.isLetter(c)) state = 6; else state = fail(5); break; case 6: c = nextchar(); if (Character.isLetter(c) ||Character.isDigit(c)) continue; else state = 7; break; case 7: retract(1); return new Token(id, getLexeme());

38
Faculty of IT - HCMUTLexical Analysis38 Implemetation (cont’d) int fail(int current_state) { forward = beginning; switch (current_state) { case 0: return 5; case 5: error(); } void retract(int flag) { if (flag ==1) move forward back get lexeme from beginning to forward move forward onward beginning = forward state = 0 } b│e│g│i│n│:│=│ │ │…

39
Faculty of IT - HCMUTLexical Analysis39 Outline Introduction Token specification –Language –Regular Expressions (REs) Token recoginition –REs NFA (Thompson’s construction) –NFA DFA (subset construction) –DFA minimal DFA (Algorithm 3.6) Programming

Similar presentations

© 2019 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google