1 CSE305 Programming Languages Syntax What is it? How is it specified? Who uses it? Why is it needed?

2 Note These notes are based on the Sebesta text. The tree diagrams in these slides are from the lecture slides provided in the instructor resources for the text, and were made by David Garrett.

3 Introduction: syntax and semantics Syntax: a formal description of the structure of programs in a given language. Semantics: a formal description of the meaning programs in a given language. Together the syntax and semantics define a language.

4 Who uses a language definition? Those who design a language Those who implement a language (e.g. write compilers for it) The who use the language (i.e. software developers) Those who make tools for developers (e.g. JDT in Eclipse)

5 Language & grammar A given language can have more than one grammar which describes it. The grammar presented to a user is not necessarily the same as the grammar used in an implementation. –implementation requires a very detailed grammar –user needs a human-readable grammar

6 Syntax and semantics of programming languages I have cautioned against getting too hung up on the syntax of a programming language. But, you still need to learn the syntax of any language you work with so that you can read and write programs in the language. To understand the meaning of programs expressed in a language you also have know the semantics of the language.

7 General background Chomsky hierarchy Context-free grammars Backus-Naur form

8 Chomsky hierarchy Noam Chomsky defined a hierarchy of grammars and languages known as the Chomsky hierarchy: –regular languages (most restrictive) –context-free languages –context-sensitive languages –unrestricted languages (least restrictive)

9 Chomsky hierarchy regular languages context-free languages context-sensitive languages unrestricted languages

10 Context-free (CF) grammar A CF grammar is formally presented as a 4- tuple G=(T,NT,P,S), where: –T is a set of terminal symbols (the alphabet) –NT is a set of non-terminal symbols –P is a set of productions (or rules), where P  NT  (T  NT)* –S  NT

11 Example 1 A small formal language L 1 = { 0, 00, 1, 11 } G 1 = ( {0,1}, {S}, { S  0, S  00, S  1, S  11 }, S )

12 Example 2 A small fragment of English L 2 = { the dog chased the dog, the dog chased a dog, a dog chased the dog, a dog chased a dog, the dog chased the cat, … } G 2 = ({a, the, dog, cat, chased}, {S, NP, VP, Det, N, V}, {S  NP VP, NP  Det N, Det  a | the, N  dog | cat, VP  V | V NP, V  chased}, S ) Notes: S = Sentence, NP = Noun Phrase, N = Noun VP = Verb Phrase, V = Verb, Det = Determiner

13 Language terminology (from Sebesta, p. 125) A language is a set of strings of symbols, drawn from some finite set of symbols (called the alphabet of the language). “The strings of a language are called sentences” “Formal descriptions of the syntax […] do not include descriptions of the lowest-level syntactic units […] called lexemes.” “A token of a language is a category of its lexemes.” Syntax of a programming language is often presented in two parts: –regular grammar for token structure (e.g. structure of identifiers) –context-free grammar for sentence structure

14 Examples of lexemes and tokens LexemesTokens fooidentifier i sumidentifier -3integer_literal 10integer_literal 1 ;statement_separator =assignment_operator

15 Backus-Naur Form (BNF) Backus-Naur Form (1959) –Invented by John Backus to describe ALGOL 58, modified by Peter Naur for ALGOL 60 –BNF is equivalent to context-free grammar –BNF is a metalanguage used to describe another language, the object language –Extended BNF: adds syntactic sugar to produce more readable descriptions

16 BNF Fundamentals Sample rules [p. 128] → = → if then → if then else non-terminals/tokens surrounded by lexemes are not surrounded by keywords in language are in bold → separates LHS from RHS | expresses alternative expansions for LHS → if then | if then else = is in this example a lexeme

17 BNF Rules A rule has a left-hand side (LHS) and a right-hand side (RHS), and consists of terminal and nonterminal symbols A grammar is often given simply as a set of rules (terminal and non-terminal sets are implicit in rules, as is start symbol)

18 Describing Lists There are many situations in which a programming language allows a list of items (e.g. parameter list, argument list). Such a list can typically be as short as empty or consisting of one item. Such lists are typically not bounded. How is their structure described?

19 Describing lists The are described using recursive rules. Here is a pair of rules describing a list of identifiers, whose minimum length is one: -> ident | ident, Notice that ‘, ’ is part of the object language (the language being described by the grammar).

20 Derivation of sentences from a grammar A derivation is a repeated application of rules, starting with the start symbol and ending with a sentence (all terminal symbols)

21 Recall example 2 G 2 = ({a, the, dog, cat, chased}, {S, NP, VP, Det, N, V}, {S  NP VP, NP  Det N, Det  a | the, N  dog | cat, VP  V | VP NP, V  chased}, S)

22 Example: derivation from G 2 Example: derivation of the dog chased a cat S  NP VP  Det N VP  the N VP  the dog VP  the dog V NP  the dog chased NP  the dog chased Det N  the dog chased a N  the dog chased a cat

23 Example 3 L 3 = { 0, 1, 00, 11, 000, 111, 0000, 1111, … } G 3 = ( {0, 1}, {S, ZeroList, OneList}, {S  ZeroList | OneList, ZeroList  0 | 0 ZeroList, OneList  1 | 1 OneList }, S )

24 Example: derivations from G 3 Example: derivation of S  ZeroList  0 ZeroList  0 0 ZeroList  ZeroList  Example: derivation of S  OneList  1 OneList  1 1 OneList  1 1 1

25 Observations about derivations Every string of symbols in the derivation is a sentential form. A sentence is a sentential form that has only terminal symbols. A leftmost derivation is one in which the leftmost nonterminal in each sentential form is the one that is expanded. A derivation can be leftmost, rightmost, or neither.

26 An example programming language grammar fragment -> | ; -> = -> a | b | c | d -> + | - -> | const

27 A leftmost derivation of a = b + const => => = => a = => a = + => a = b + => a = b + const

28 Parse tree A parse tree is an hierarchical representation of a derivation: const a = b +

29 Parse trees and compilation A compiler builds a parse tree for a program (or for different parts of a program). If the compiler cannot build a well-formed parse tree from a given input, it reports a compilation error. The parse tree serves as the basis for semantic interpretation/translation of the program.

30 Extended BNF Optional parts are placed in brackets [ ] -> ident [( )] Alternative parts of RHSs are placed inside parentheses and separated via vertical bars -> (+|-) const Repetitions (0 or more) are placed inside braces { } -> letter {letter|digit}

31 Comparison of BNF and EBNF sample grammar fragment expressed in BNF -> + | - | -> * | / | same grammar fragment expressed in EBNF -> {(+ | -) } -> {(* | /) }

32 Ambiguity in grammars A grammar is ambiguous if and only if it generates a sentential form that has two or more distinct parse trees Operator precedence and operator associativity are two examples of ways in which a grammar can provide an unambiguous interpretation.

33 Operator precedence ambiguity The following grammar is ambiguous: -> | const -> / | - The grammar treats the '/' and '-' operators equivalently.

34 An ambiguous grammar for arithmetic expressions -> | const -> / | - const --//

35 Disambiguating the grammar If we use the parse tree to indicate precedence levels of the operators, we can remove the ambiguity. The following rules give / a higher precedence than - -> - | -> / const | const const / -

36 Links to BNF-style grammars for actual programming languages Below are some links to grammars for real programming languages. Look at how the grammars are expressed. – – In the ones listed below, find the parts of the grammar that deal with operator precedence. – – – Jacques.Levy/poly/mainB/node23.htmlhttp:// Jacques.Levy/poly/mainB/node23.html –

37 Derivation of 2+5*3 using C grammarC grammar *

38 Recursion and parentheses To generate 2+3*4 or 3*4+2, the parse tree is built so that + is higher in the tree than *. To force an addition to be done prior to a multiplication we must use parentheses, as in (2+3)*4. Grammar captures this in the recursive case of an expression, as in the following grammar fragment:  + |  * |  | | “(” “)”

39 Associativity of operators When multiple operators appear in an expression, we need to know how to interpret the expression. Some operators (e.g. +) are associative, meaning that the meaning of an expression with multiple instances of the operator is the same no matter how it is interpreted: (a+b)+c = a+(b+c) Some operators (e.g. -) are not associative: (a-b)-c  a-(b-c)e.g. try a=10, b=8, c=6 (10-8)-6 = -4 but 10-(8-6)=8 - and / are both left-associative, meaning a-b-c is interpreted as (a-b)-c. Exponentiation (**) is right-associative. This means that 2**3**2 is interpreted as 2**(3**2) (i.e. 2**9) rather than (2**3)**2 (i.e. 8**2 or 2**6).

40 Associativity of Operators Operator associativity can be encoded by a grammar. The following grammar fragment does not do this: the left and right operands of '-' are treated symmetrically. -> - | -> | | “(” “)”

41 Associativity of Operators However, the following rules ensure that '-' is left- associative, because they prevent direct recursion with '-' in the right-hand operand. -> - | -> | | “(” “)” - -

42 Decision timing: Design time vs. Implementation time (to come) Java and precedence/associativity/left-to- right evaluation vs. C++ (?)

43 Theory vs. Reality (to come) Java/C# vs. C/C++ (size of representation – but this is not the slide to address this on: see next point). Also, effect of fixed size of representations on associativity: –mathematically, (x+y)+z = x+(y+z) –in practice (+ is not always associative): (large+small)+small = large large+(small+small) > large Dealing with fixed-size numeric representations