Download presentation
Presentation is loading. Please wait.
Published byGwenda Fleming Modified over 8 years ago
1
Compiler Principles Fall 2014-2015 Compiler Principles Lecture 1: Lexical Analysis Roman Manevich Ben-Gurion University
2
Agenda 2 Understand role of lexical analysis in a compiler Lexical analysis theory Implementing professional scanner via scanner generator
3
Javascript example var currOption = 0; // Choose content to display in lower pane. function choose ( id ) { var menu = ["about-me", "publications", "teaching", "software", "activities"]; for (i = 0; i < menu.length; i++) { currOption = menu[i]; var elt = document.getElementById(currOption); if (currOption == id && elt.style.display == "none") { elt.style.display = "block"; } else { elt.style.display = "none"; } Can you some identify basic units in this code? 3
4
Javascript example var currOption = 0; // Choose content to display in lower pane. function choose ( id ) { var menu = ["about-me", "publications", "teaching", "software", "activities"]; for (i = 0; i < menu.length; i++) { currOption = menu[i]; var elt = document.getElementById(currOption); if (currOption == id && elt.style.display == "none") { elt.style.display = "block"; } else { elt.style.display = "none"; } Can you some identify basic units in this code? 4 keyword ? ? ? ? ? ? ?
5
Javascript example var currOption = 0; // Choose content to display in lower pane. function choose ( id ) { var menu = ["about-me", "publications", "teaching", "software", "activities"]; for (i = 0; i < menu.length; i++) { currOption = menu[i]; var elt = document.getElementById(currOption); if (currOption == id && elt.style.display == "none") { elt.style.display = "block"; } else { elt.style.display = "none"; } Can you some identify basic units in this code? 5 keyword operator identifier numeric literal punctuation whitespace comment string literal
6
Role of lexical analysis First part of compiler front-end Convert stream of characters into stream of tokens – Split text into most basic meaningful strings Simplify input for syntax analysis High-level Language (scheme) Executable Code Lexical Analysis Syntax Analysis Parsing ASTSymbol Table etc. Inter. Rep. (IR) Code Generation 6
7
From scanning to parsing 59 + (1257 * xPosition) )id*num(+ Lexical Analyzer program text token stream Parser Grammar: E id E num E E + E E E * E E ( E ) + num x * Abstract Syntax Tree valid syntax error 7 Lexical error valid
8
Scanner output var currOption = 0; // Choose content to display in lower pane. function choose ( id ) { var menu = ["about-me", "publications“, "teaching", "software", "activities"]; for (i = 0; i < menu.length; i++) { currOption = menu[i]; var elt = document.getElementById(currOption); if (currOption == id && elt.style.display == "none") { elt.style.display = "block"; } else { elt.style.display = "none"; } 1: VAR 1: ID(currOption) 1: EQ 1: INT_LITERAL(0) 1: SEMI 3: FUNCTION 3: ID(choose) 3: LP 3: ID(id) 3: EP 3: LCB... Stream of Tokens LINE: ID(value) 8
9
Tokens 9
10
What is a token? Lexeme – substring of original text constituting an identifiable unit – Identifiers, Values, reserved words, … Record type storing: – Kind – Value (when applicable) – Start-position/end-position – Any information that is useful for the parser Different for different languages 10
11
Example tokens TypeExamples Identifierx, y, z, foo, bar NUM42 FLOATNUM-3.141592654 STRING“so long, and thanks for all the fish” LPAREN( RPAREN) IFif … 11
12
C++ example 1 Splitting text into tokens can be tricky How should the code below be split? vector > myVector >> operator >, > two tokens or ? 12
13
C++ example 2 Splitting text into tokens can be tricky How should the code below be split? vector > myVector >, > two tokens 13
14
Separating tokens TypeExamples Comments/* ignore code */ // ignore until end of line White spaces\t \n Lexemes are recognized but get consumed rather than transmitted to parser – if i f i/*comment*/f 14
15
Preprocessor directives in C TypeExamples Include directives#include Macros#define THE_ANSWER 42 15
16
First step of designing a scanner Define each type of lexeme – Reserved words: var, if, for, while – Operators: < = ++ – Identifiers: myFunction – Literals: 123 “hello” – Annotations: @SuppressWarnings How can we define lexemes of unbounded length 16 ?
17
First step of designing a scanner Define each type of lexeme – Reserved words: var, if, for, while – Operators: < = ++ – Identifiers: myFunction – Literals: 123 “hello” – Annotations: @SuppressWarnings How can we define lexemes of unbounded length – Regular expressions 17 ?
18
Agenda 18 Understand role of lexical analysis in a compiler – Convert text to stream of tokens Lexical analysis theory Implementing professional scanner via scanner generator
19
Regular expressions 19
20
Regular languages refresher Formal languages – Alphabet = finite set of letters – Word = sequence of letter – Language = set of words Regular languages defined equivalently by – Regular expressions – Finite-state automata 20
21
Regular expressions Empty string: Є Letter: a Concatenation: R 1 R 2 Union: R 1 | R 2 Kleene-star: R* – Shorthand: R + stands for R R* scope: (R) Example: (0* 1*) | (1* 0*) – What is this language? 21
22
Exercise 1 - Question Language of Java identifiers – Identifiers start with either an underscore ‘_’ or a letter – Continue with either underscore, letter, or digit 22
23
Exercise 1 - Answer Language of Java identifiers – Identifiers start with either an underscore ‘_’ or a letter – Continue with either underscore, letter, or digit – (_|a|b|…|z|A|…|Z)(_|a|b|…|z|A|…|Z|0|…|9)* 23
24
Exercise 1 – Better answer Language of Java identifiers – Identifiers start with either an underscore ‘_’ or a letter – Continue with either underscore, letter, or digit – (_|a|b|…|z|A|…|Z)(_|a|b|…|z|A|…|Z|0|…|9)* – Using shorthand macros First= _|a|b|…|z|A|…|Z Next= First|0|…|9 R= First Next* 24
25
Exercise 2 - Question Language of rational numbers in decimal representation (no leading, ending zeros) – Positive examples: 0 123.757.933333 0.7 – Negative examples: 007 0.30 25
26
Exercise 2 - Answer Language of rational numbers in decimal representation (no leading, ending zeros) – Digit= 1|2|…|9 Digit0 = 0|Digit Num= Digit Digit0* Frac= Digit0* Digit Pos= Num |.Frac | 0.Frac| Num.Frac PosOrNeg = (Є|-)Pos R= 0 | PosOrNeg 26
27
Exercise 3 - Question Equal number of opening and closing parenthesis: [ n ] n = [], [[]], [[[]]], … 27
28
Exercise 3 - Answer Equal number of opening and closing parenthesis: [ n ] n = [], [[]], [[[]]], … Not regular Context-free Grammar: S ::= [] | [S] 28
29
Finite automata 29
30
Finite automata start a b b c accepting state start state transition An automaton is defined by states and transitions 30
31
Automaton running example start a b b c Words are read left-to-right cba 31
32
Automaton running example start a b b c Words are read left-to-right cba 32
33
Automaton running example start a b b c Words are read left-to-right cba 33
34
Automaton running example start a b b c Words are read left-to-right word accepted cba 34
35
Word outside of language start a b b c cbb 35
36
Word outside of language Missing transition means non-acceptance start a b b c cbb 36
37
Word outside of language start a b b c bba 37
38
Word outside of language start a b b c bba 38
39
Word outside of language start a b b c bba 39 Final state is not an accepting state
40
Exercise - Question What is the language defined by the automaton below? start a b b c 40
41
Exercise - Answer What is the language defined by the automaton below? – a b* c – Generally: all paths leading to accepting states start a b b c 41
42
A little about me Joined Ben-Gurion University two years ago Research interests – Advanced compilation and synthesis techniques – Language-supported parallelism – Static analysis and verification 42
43
I am here for Teaching you theory and practice of popular compiler algorithms – Hopefully make you think about solving problems by examples from the compilers world – Answering questions about material Contacting me – e-mail: romanm@cs.bgu.ac.il – Office hours: see course web-pageweb-page Announcements Forums (per assignment) 43
44
Tentative syllabus Front End Scanning Top-down Parsing (LL) Bottom-up Parsing (LR) Attribute Grammars Intermediate Representation Lowering Optimizations Local Optimizations Dataflow Analysis Loop Optimizations Code Generation Register Allocation Instruction Selection 44 mid-termexam
45
Nondeterministic Finite automata 45
46
Non-deterministic automata Allow multiple transitions from given state labeled by same letter start a a b c b c 46
47
NFA run example cba start a a b c b c 47
48
NFA run example Maintain set of states cba start a a b c b c 48
49
NFA run example cba start a a b c b c 49
50
NFA run example Accept word if any of the states in the set is accepting cba start a a b c b c 50
51
NFA+Є automata Є transitions can “fire” without reading the input start a b c Є 51
52
NFA+Є run example start a b c cba Є 52
53
NFA+Є run example Now Є transition can non-deterministically take place start a b c cba Є 53
54
NFA+Є run example start a b c cba Є 54
55
NFA+Є run example start a b c cba Є 55
56
NFA+Є run example start a b c cba Є 56
57
NFA+Є run example start a b c cba Є Word accepted 57
58
Reg-exp vs. automata Regular expressions are declarative – Offer compact way to define a regular language by humans – Don’t offer direct way to check whether a given word is in the language Automata are operative – Define an algorithm for deciding whether a given word is in a regular language – Not a natural notation for humans 58
59
From Regular expressions to NFA 59
60
From reg. exp. to automata Theorem: there is an algorithm to build an NFA+Є automaton for any regular expression Proof: by induction on the structure of the regular expression – For each sub-expression R we build an automaton with exactly one start state and one accepting state – Start state has no incoming transitions – Accepting state has no outgoing transitions 60
61
From reg. exp. to automata Theorem: there is an algorithm to build an NFA+Є automaton for any regular expression Proof: by induction on the structure of the regular expression 61 start
62
Base cases R = R = a start a 62
63
Construction for R 1 | R 2 start R1R1 R2R2 63
64
Construction for R 1 R 2 start R1R1 R2R2 64
65
Construction for R* start R 65
66
NFA determinization 66
67
From NFA+Є to DFA Construction requires O(n) states for a reg-exp of length n Running an NFA+Є with n states on string of length m takes O(m·n 2 ) time – Can we reduce the n 2 factor? 67
68
From NFA+Є to DFA Construction requires O(n) states for a reg-exp of length n Running an NFA+Є with n states on string of length m takes O(m·n 2 ) time – Can we reduce the n 2 factor? Theorem: for any NFA+Є automaton there exists an equivalent deterministic automaton Proof: determinization via subset construction – Number of states in the worst-case O(2 n ) – Running time O(m) 68
69
Subset construction For an NFA+Є with states M={s 1,…,s k } Construct a DFA with one state per set of states of the corresponding NFA – M’={ [], [s 1 ], [s 1,s 2 ], [s 2,s 3 ], [s 1,s 2,s 3 ], …} Simulate transitions between individual states for every letter 69 a s1s1 s2s2 a [s 1,s 4 ] [s 2,s 7 ] NFA+Є DFA a s4s4 s7s7
70
Subset construction For an NFA+Є with states M={s 1,…,s k } Construct a DFA with one state per set of states of the corresponding NFA – M’={ [], [s 1 ], [s 1,s 2 ], [s 2,s 3 ], [s 1,s 2,s 3 ], …} Extend macro states by states reachable via Є transitions 70 Є s1s1 s4s4 [s 1,s 2 ] [s 1,s 2,s 4 ] NFA+Є DFA
71
Recap We know how to define any single type of lexeme We know how to convert any regular expression into a recognizing automaton But is this enough for scanning? 71
72
Designing a scanner 72
73
Scanning challenges Regular expressions allow us to recognize whether a given text is a sequence of legal lexemes – Define the language of all sequences of lexemes Automata theory provides an algorithm for checking membership of words – But we are interested in splitting the text not just deciding on membership 1.How do we split the text into lexemes? 2.How do we handle ambiguities – lexemes matching more than one token? 73
74
Separating lexemes ID = (a|b|…|z) (a|b|…|z)* ONE = 1 Input: abb1 How do we return ID(abb), ONE? 74
75
Separating lexemes ID = (a|b|…|z) (a|b|…|z)* ONE = 1 Input: abb1 How do we return ID(abb), ONE? 75 start a-z 1 ID ONE
76
Maximal munch algorithm ID = (a|b|…|z) (a|b|…|z)* ONE = 1 Input: abb1 How do we return ID(abb), ONE? Solution: find longest matching lexeme – Keep reading text until automaton fails – remember text position of last accepting state – Return token corresponding to last accepting state – Reset – go back to start state and continue reading input from remembered text position 76
77
Handling ambiguities ID = (a|b|…|z) (a|b|…|z)* IF = if Input: if Matches both tokens What should the scanner output? 77 start a-z i ID IF f NFA
78
Handling ambiguities ID = (a|b|…|z) (a|b|…|z)* IF = if Input: if Matches both tokens What should the scanner output? Solution: break tie using order of definitions – Output: ID(if) 78 start a-z i ID IF f NFA
79
Handling ambiguities ID = (a|b|…|z) (a|b|…|z)* IF = if Input: if Matches both tokens What should the scanner output? Solution: break tie using order of definitions – Output: ID(if) 79 start a-z\i i a-z ID ID < IF f ID a-z\f DFA a-z
80
Handling ambiguities IF = if ID = (a|b|…|z) (a|b|…|z)* Input: if Matches both tokens What should the scanner output? Solution: break tie using order of definitions – Output: IF 80 Conclusion: list keyword token definitions before identifier definition start a-z\i i a-z ID IF < ID f ID a-z\f a-z DFA
81
Filtering illegal combinations Which tokens should the scanner return for “123foo”? 81
82
Filtering illegal combinations Which tokens should the scanner return for “123foo”? – We sometimes want to rule out certain token concatenations prior to parsing – How can we do that with what we’ve seen so far? 82
83
Filtering illegal combinations Which tokens should the scanner return for “123foo”? – We sometimes want to rule out certain token concatenations prior to parsing – How can we do that with what we’ve seen so far? Define “error” lexemes 83
84
Putting the algorithm pieces together 84
85
Scanner construction recap 85 ……………… List of regular expressions (one per lexeme) NFA+Є DFA Code implementing maximal munch with tie breaking policy minimization
86
Running the scanner Maintain variables for – ltpos = position from which we scan for next token – lposa = Last position of accepting state – state = Current state While not reaching end of text do Read until getting stuck If not end-of-file return token = input[ ltpos, lposa ] ltpos = lposa + 1 state = q1 (initial) lposa = N/A 86
87
Scanning exercise You are given the following lexeme: – RAT: (1-9)(0-9)*. (0-9)*(1-9) | 0. (0-9)*(1-9) Construct the corresponding scanner automaton Run it on the inputs 1.23.2 1.230.2 87
88
Scanner run on 1.23.2 88 tokencurr. statecurr. lettercurr. positionLast accept pos.Scan from q110N/A0 q2.1N/A0 q322N/A0 q53320 1.23q5.430 ERRORq1.4N/A4 q1 start q2 1-9 0-9. q5 q3 q4 0 1-9 0 0 q6 0.
89
Scanner run on 1.230.2 89 tokencurr. statecurr. lettercurr. positionLast accept pos.Scan from q110N/A0 q2.1N/A0 q322N/A0 q53320 0430 1.23q4.530 q104N/A4 q6.5N/A4 q326N/A4 0.2q5EOF 64 q1 start q2 1-9 0-9. q5 q3 q4 0 1-9 0 0 q6 0.
90
Agenda 90 Understand role of lexical analysis in a compiler – Convert text to stream of tokens Lexical analysis theory – Theory of regular languages + maximal munch + precedence Implementing professional scanner via scanner generator
91
Implementing a scanner 91
92
Implementing modern scanners Manual construction of automata + determinization + maximal munch + tie breaking – Very tedious – Error-prone – Non-incremental Fortunately there are tools that automatically generate robust code from a specification for most languages – C: Lex, Flex Java: JLex, JFlex 92
93
Using JFlex Define tokens (and states) Run JFlex to generate Java implementation Usually MyScanner.nextToken() will be called in a loop by parser Lexical Specification JFlexMyScanner.java Stream of characters Tokens MyScanner.lex 93
94
Common format for reg-exps 94
95
Escape characters What is the expression for one or more + symbols? – (+)+ won’t work – (\+)+ will backslash \ before an operator turns it to standard character \*, \?, \+, … Newline: \n or \r\n depending on OS Tab: \t 95
96
Shorthands Use names for expressions – letter = a | b | … | z | A | B | … | Z – letter_ = letter | _ – digit = 0 | 1 | 2 | … | 9 – id = letter_ (letter_ | digit)* Use hyphen to denote a range – letter = a-z | A-Z – digit = 0-9 96
97
Catching errors What if input doesn’t match any token definition? – Want to gracefully signal an error Trick: add a “catch-all” rule that matches any character and reports an error – Add after all other rules 97
98
Next lecture: parsing
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.