1. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 1/27 Automatic Generation of Language-based Tools: The LISA Approach Marjan Mernik UNIVERSITY OF MARIBOR FACULTY OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE

2. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 2/27 Outline of the Presentation How to specify a programming language? Formal methods for programming language definition LISA compiler/interpreter generator Language-based tools generated by LISA

3. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 3/27 How to specify a programming language? Using the natural language –Advantages: descriptions are understandable, accessible to a wide variety of users. –Disadvantages: lack of clarity, ambiguities, various interpretations.

4. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 4/27 How to specify a programming language? Using a formal method –Advantages: syntax and the semantics are defined in a precise and unambiguous manner, possibility for automatic generation of compilers or interpreters, tool for programming language design –Disadvantages: required detail knowledge

5. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 5/27 Formal methods for programming language definition Lexicon (regular definitions, FSA) Syntax (BNF) Semantics (axiomatic, attribute grammars, denotational, algebraic, structural operational/natural, action, abstract state machines,....)

6. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 6/27 Formal methods for programming language definition Possibility for automatic generation of compilers or interpreters –Attribute Grammars: Synthesizer Generator –Denotational: PSG –Algebraic: ASF+SDF –Structural operational/Natural: Centaur –Action: ASD –Abstract-state machines: Gem-Mex

7. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 7/27 Formal methods for programming language definition From formal language definitions many other language-based tools can be automatically generated, such as: –syntax-directed editors, –type checkers, –dataflow analyzers, –partial evaluators, –debuggers, –test case generators, –animators, etc.

8. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 8/27 Formal methods for programming language definition The core language definitions have to be augmented or Just a part of formal language definitions is enough for automatic tool generation or Implicit information must be extracted from formal language definition

9. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 9/27 Formal methods for programming language definition Automatic generation is possible whenever a tool can be built from a fixed part and a variable part; and also the variable part, language dependent, has to be systematically derivable from the language specifications (Table 1).

10. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 10/27 Table 1 Generated ToolFormal Specification Fixed Part Variable part Lexerregular definitionsalgorithm which interpret action table action table: State Parser (LR)BNFalgorithm which interpret action table and goto table action table: State T Action goto table: State (T N) State EvaluatorAttribute Grammartree walk algorithm semantic functions Language knowledgeable editor regular definitions (extracted from AG) matching algorithm same as lexer

11. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 11/27 Table 1 (cont.) Generated ToolFormal Specification Fixed Part Variable part FSA visualizationregular definitions (extracted from AG) FSA layout algorithmsame as lexer Syntax tree visualization BNF (extracted from AG) Syntax tree layout algorithm syntax tree Dependency graph visualization extracted from AGDG layout algorithmdependency graph Semantic evaluator animation extracted from AGSemantic tree layout algorithm decorated syntax tree & semantic functions

12. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 12/27 Regular definitions An example – arithmetic expressions: e.g. (23+2)*3 integer [0-9] + operator + | * separator ( | )

13. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 13/27 Regular definitions (variable part) action table: State State 0..9+,*(,)\t, \n, 01234 11 2 3 4 4 0 1 2 3 4 0..9 +,* (,) \t,\n 0..9 \t,\n tInteger tOperator tSeparator tIgnore

14. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 14/27 Regular definitions (variable part) void initAutomata() { for (int i = 0; i<=maxState; i++) { for (int j = 0; j<256; j++) automata[i][j] = noEdge; } for (int i = '0'; i<='9'; i++) automata[0][i] = automata[1][i] = 1; automata[0]['+'] = automata[0]['*'] = 2; automata[0]['('] = automata[0][')'] = 3; automata[0]['\n']=automata[0][' ']=automata[0]['\t']=4; automata[4]['\n']=automata[4][' ']=automata[4]['\t']=4; finite[0] = tLexError; finite[1] = tInteger; finite[2] = tOperator; finite[3] = tSeparator; finite[4] = tIgnore; }

15. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 15/27 Regular definitions (fixed part) Token nextToken() { int currentState = startState; string lexem; int startColumn = column; int startRow = row; do { int tempState = getNextState(currentState, peek()); if (tempState!=noEdge) { currentState = tempState; lexem += (char)read(); } else { if (isFiniteState(currentState)) { Token token(lexem, startColumn, startRow, getFiniteState(currentState), eof()); if (token.getToken()==tIgnore) return nextToken(); else return token; } else { return Token("", startColumn, startRow, tLexError, eof());} } }while (true); }

16. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 16/27 BNF An example – arithmetic expressions: e.g. (23+2)*3 E ::= T EE EE ::= + T EE | T ::= F TT TT ::= * F TT | F ::= (E) | #integer

17. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 17/27 LL(1) Parser (fixed part) a T a if token = 'a' then nextToken else error 1 2 n... if token IN FIRST( 1 ) then T( 1 ) else if token IN FIRST( 2 ) then T( 2 )... else if token IN FIRST( n ) then T( n ) else error A N A call A; n 2 1 T( 1 );T( 2 );...;T( n )...

18. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 18/27 LL(1) Parser (variable part) bool EE() { if (scanner->currentToken().getLexem()=="+") { scanner->nextToken(); return T() && EE(); } return true; } bool F(){ if (scanner->currentToken().getToken()==Scanner::tInteger) { scanner->nextToken(); return true; } else if (scanner->currentToken().getLexem()=="(") { scanner->nextToken(); bool zac = E(); if (zac && scanner->currentToken().getLexem()==")") { scanner->nextToken(); return true; } else return false; }

19. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 19/27 SLR(1) Parser (variable part) i()+-# SETi()+-# 0ss 1234 1 sss 675 2r2 3r5 4ss 10234 5 a 6ss 834 7ss 934 8r3 9r4 10 sss 1167 r6 F: State T Action G: State (T N) State

20. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 20/27 SLR(1) Parser (fixed part) PUSH(stack,0) current := nextToken while true do if F(TOPV(stack), current) = s then {shift} T := G(TOPV(stack), current) PUSH(stack,T) current := nextToken elseif F(TOPV(stack), current) = r k then {reduce k-th production} for j = 1 to SIZE(k) POP(stack) T := G(TOPV(stack),LHS(k)) PUSH(stack,T) elseif F(TOPV(stack),current) = a then accept else error endif enddo

21. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 21/27 Attribute Grammars Productions (p P):Semantic functions (f p,a, a S(X 0 )): E T EEE.val = EE.val EE.inval = T.val EE 0 + T EE 1 EE 0.val = EE 1.val EE 1.inval = EE 0.inval + T.val EE 0 EE 0.val = EE 0.inval T F TTT.val = TT.val TT.inval = F.val TT 0 * F TT 1 TT 0.val = TT 1.val TT 1.inval = TT 0.inval * F.val TT 0 TT 0.val = TT 0.inval F ( E )F.val = E.val F IntegerF.val = Str2Int(Integer.lexVal)

22. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 22/27 Attribute Grammars bool EE(int inVal, int &val) { if (scanner->currentToken().getLexem()=="+") { scanner->nextToken(); int tempVal; bool ok = T(tempVal); return ok && EE(inVal+tempVal,val); } else { val = inVal; return true; } bool T(int &val) { int tempVal; bool ok = F(tempVal); return ok && TT(tempVal, val); }

23. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 23/27 LISA ver. 1 Developed 1994 Mernik, Korbar, Žumer. LISA: A Tool for Automatic Language Implementation, ACM Sigplan Notices, Vol. 30, No. 4, pp. 71 – 79, 1995.

24. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 24/27 LISA ver. 2.0 LISA ver. 2.0 (joint work with M. Lenič, E. Avdičaušević, V. Žumer) –started in summer 1997 –finished in summer 2000 Incremental language development Educational tool

25. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 25/27 LISA ver. 2.0

26. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 26/27 LISA ver. 2.0 LISA generates also other language- based tools –Editors, –Inspectors, –visualizers/animators (Slovene-Portugal Project)

27. Socrates/Erasmus Programme, University of Minho, Braga, May 29, 2003 27/27 LISA ver. 2.0 LISA tool demonstration More info: http://marcel.uni-mb.si/lisa