1. Sung-Dong Kim, Dept. of Computer Engineering, Hansung University
LEX & Yacc
Sung-Dong Kim,
Dept. of Computer Engineering,
Hansung University

2. LEX Input: tiny.l Output: lex.yy.c or lexyy.c Procedure yylex
Table-driven implementation of a DFA
Similar to “getToken” RE +
action
Scanner
(C code)
Lex
(2011-1) Compiler

3. LEX Convention (1) Metacharacters Quotes: actual characters Backslash
For not metacharacters: “if”, if
For metacharacters: “(”
Backslash
\(\* = “\*”
\n, \t
(aa|bb)(a|b)*c? = (“aa”|“bb”)(“a”|“b”)* “c”?
(2011-1) Compiler

4. LEX Convention (2) [...] : any one of them Hyphen
[abxz]: any one of the characters a, b, x, z
(aa|bb)(ab)*c?
Hyphen
Ranges of characters
[0-9]
(2011-1) Compiler

5. LEX Convention (3) . ^ Represents a set of characters
Any character except a newline ^
Complementary sets
[^0-9abc]: any character that is not a digit and is not one of the letter a, b, c
(2011-1) Compiler

6. LEX Convention (4) Square bracket
Most of the metacharacters lose their special status
[-+] == (“+”|“-”)
[+-]: from “+”, all characters
[.”?]: any of the three characters ., ”, ?
[\^\\]: ^ or \
(2011-1) Compiler

7. LEX Convention (5) Curly bracket Names of regular expressions
nat = [0-9]+
signedNat = (“+”|“-”)? nat
nat [0-9]+
signedNat (“+”|“-”)? {nat}
(2011-1) Compiler

8. Format of LEX Input (1) Input file = regular expression + C code
Definitions
Any C code that must be inserted to any function - %{…}%
Names of regular expressions
Rules
Regular expressions + C code (action)
Auxiliary routines (optional)
C code + main program (if needed)
(2011-1) Compiler

9. Format of LEX Input (2) Layout {definitions} %% {rules}
{auxiliary routines}
(2011-1) Compiler

10. Example 1: scanner that adds line numbers to text%{
/* a Lex program that adds line numbers to lines of text,
printing the new text to the standard output */
#include <stdio.h>
int lineno = 1; %}
line .*\n %%
{line} {printf(“%5d %s”,lineno++,yytext); }
main()
{ yylex(); return 0; }
(2011-1) Compiler

11. Example 2: prints the count of # of replacements%{
/* a Lex program that changes all numbers from decimal to hexadecimal notation, printing a summary statistic stderr */
#include <stdlib.h>
#include <stdio.h>
int count = 0; %}
digit [0-9]
number {digit}+ %%
{number} { int n = atoi(yytext);
printf(“%x”, n);
if (n > 9) count++; }
(2011-1) Compiler

12. fprintf(stderr, “number of replacements = %d”, count); return 0; }
main()
{ yylex();
fprintf(stderr, “number of replacements = %d”, count);
return 0; }
(2011-1) Compiler

13. Example 3: prints all input lines that begin or end with the ‘a’%{
/* Selects only lines that end or begin with the letter ‘a’.
Deletes everything else. */
#include <stdio.h> %}
ends_with_a .*a\n
begins_with_a a.*\n %%
{ends_with_a} ECHO;
{begins_with_a} ECHO;
.*\n ;
main()
{ yylex(); return 0; }
(2011-1) Compiler

14. Summary (1) Ambiguity resolution The principles of longest substring
Substring with equal length: first-match first-serve
No match: copy the next character and continue
(2011-1) Compiler

15. Summary (2) Insertion of C Code %{ … %}: exact copy
Auxiliary procedure section: exact copy at the end
Any code following a RE (action): at the appropriate place in yylex
(2011-1) Compiler

16. Lex Internal Names lex.yy.c: Lex output file name or lexyy.c
yylex: Lex scanning routine
yytext: String matched on current action
yyin: Lex input file (default: stdin)
yyout: Lex output file (default: stdout)
input: Lex buffered input routine
ECHO: Lex default action (print yytext to yyout)
(2011-1) Compiler

17. LEX for TINY %{ #include “globals.h” #include “util.h”
#include “scan.h”
/* lexeme of identifier or reserved word */
char tokenString[MAXTOKENLEN+1]; */
digit [0-9]
number {digit}+
letter [a-zA-Z]
identifier {letter}+
newline \n
whitespace [ \t] %%
(2011-1) Compiler

18. “repeat” { return REPEAT; } “until” { return UNTIL; }
“if” { return IF; }
“then” { return THEN; }
“else” { return ELSE; }
“end” { return END; }
“repeat” { return REPEAT; }
“until” { return UNTIL; }
“read” { return READ; }
“write” { return WRITE; }
“:=” { return ASSIGN; }
“=” { return EQ; }
“<” { return LT; }
“+” { return PLUS; }
“-” { return MINUS; }
“*” { return TIMES; }
“/” { return OVER; }
“(” { return LPAREN; }
“)” { return RPAREN; }
“;” { return SEMI; }
(2011-1) Compiler

19. {number} { return NUM; } {identifier} { return ID; }
{newline} { lineno++; }
{whitespace} { /* skip whitespace */ }
“{” { char c; do
{ c = input();
if (c == ‘\n’) lineno++;
} while (c != ‘}’); }
. { return ERROR; } %%
(2011-1) Compiler

20. TokenType getToken(void) { static int firstTime = TRUE;
TokenType currentToken;
if (firstTime)
{ firstTime = FALSE;
lineno++;
yyin = source;
yyout = listing; }
currentToken = yylex();
strncpy(tokenString, yytext, MAXTOKENLEN);
if (TraceScan) {
fprintf(listing, “\t%d: “, lineno);
printToken(currentToken, tokenString);
return currentToken;
(2011-1) Compiler

21. YACC LALR(1) parser generator Yet another compiler compiler syntax
spec.
parser
(2011-1) Compiler

22. YACC Basics (1) Input/output Specification file format Yacc filename.y
y.tab.c
ytab.c
filename.tab.c
{definitions} %%
{rules}
{auxiliary routines}
(2011-1) Compiler

23. YACC Basics (2) Definitions Rules Auxiliary routines
Information about tokens, data types, grammar rules
C code  output file
Rules
Modified BNF format
C code
Auxiliary routines
Procedure and function declarations
main()  yyparse()  yylex()
(2011-1) Compiler

24. #include <stdio.h> #include <ctype.h> %} %token NUMBER %% %{
#include <stdio.h>
#include <ctype.h> %}
%token NUMBER %%
command : exp {printf(“%d\n”,$1);}
exp : exp ‘+’ term {$$ = $1 + $3;}
| exp ‘-’ term {$$ = $1 - $3;}
| term {$$ = $1;} ;
term : term ‘*’ factor {$$ = $1 * $3;}
| factor {$$ = $1;}
factor : NUMBER {$$ = $1;}
| ‘(’ exp ‘)’ {$$ = $2;}
(2011-1) Compiler

25. while((c = getchar()) == ‘ ‘); /* blank 제거 */ if (isdigit(c)) {
main() {
return yyparse(); }
int yylex(void) {
int c;
while((c = getchar()) == ‘ ‘); /* blank 제거 */
if (isdigit(c)) {
ungetc(c,stdin);
scanf(“%d”,&yylval);
return(NUMBER);
if (c == ‘\n’) return 0; /* 파싱 정지 */
return(c);
void yyerror(char *s) {
fprintf(stderr,”%s\n”,s); /* 에러메시지 출력*/
return 0;
(2011-1) Compiler

26. YACC Options (1) -d Header file generation yacc –d filename.y
y.tab.h, ytab.h, filename.tab.h
Other file
#include y.tab.h
Call yylex()
(2011-1) Compiler

27. YACC Options (2) -v option Verbose option yacc –d filename.y y.output
(2011-1) Compiler

28. $accept : command_$end $end accept . error state 2 command : exp_ (1)
NUMBER shift 5
( shift 6
error
command goto 1
exp goto 2
term goto 3
factor goto 4
state 1
$accept : command_$end
$end accept
. error
state 2
command : exp_ (1)
exp : exp_+ term
exp : exp_- term
+ shift 7
- shift 8
. reduce 1
state 3
exp : term_ (4)
term : term_* factor
* shift 9
. reduce 4
state 4
term : factor_ (6)
. reduce 6
(2011-1) Compiler

29. state 5 factor : NUMBER_ (7) . reduce 7 state 6 factor : (_exp )
NUMBER shift 5
( shift 6
. error
exp goto 10
term goto 3
factor goto 4
state 7
exp : exp +_term
NUMBER shift 5
( shift 6
. error
term goto 11
factor goto 4
state 8
exp : exp -_term
term goto 12
(2011-1) Compiler

30. state 9 term : term *_factor NUMBER shift 5 ( shift 6 . error
factor goto 13
state 10
exp : exp_+ term
exp : exp_- term
factor : ( exp_)
+ shift 7
- shift 8
) shift 14
state 11
exp : exp + term_ (2)
term : term_* factor
* shift 9
. reduce 2
state 12
exp : exp – term_ (3)
. reduce 3
state 13
term : term * factor_ (5)
. reduce 5
(2011-1) Compiler

31. 8/127 terminals, 4/600 nonterminals
state 14
factor : ( exp )_ (8)
. reduce 8
8/127 terminals, 4/600 nonterminals
9/300 grammar rules, 15/1000 states
0 shift/reduce, 0 reduce/reduce conflicts reported
9/601 working sets used
memory: states, etc. 36/2000, parser 11/4000
9/601 distinct lookahead sets
6 extra closures
18 shift entries, 1 exceptions
8 goto entries
4 entries saved by goto default
Optimizer space used: input 50/2000, output 218/4000
218 table entries, 202 zero
maximum spread: 257, maximum offset: 43
(2011-1) Compiler