1. 1 Simplifications of Context-Free Grammars

2. 2 A Substitution Rule Substitute Equivalent grammar

3. 3 A Substitution Rule Equivalent grammar Substitute

4. 4 In general: Substitute equivalent grammar

5. 5 Nullable Variables Nullable Variable:

6. 6 Removing Nullable Variables Example Grammar: Nullable variable

7. 7 Substitute Final Grammar

8. 8 Unit-Productions Unit Production: (a single variable in both sides)

9. 9 Removing Unit Productions Observation: Is removed immediately

10. 10 Example Grammar:

11. 11 Substitute

12. 12 Remove

13. 13 Substitute

14. 14 Remove repeated productions Final grammar

15. 15 Useless Productions Some derivations never terminate... Useless Production

16. 16 Another grammar: Not reachable from S Useless Production

17. 17 In general: if then variable is useful otherwise, variable is useless contains only terminals

18. 18 A production is useless if any of its variables is useless Productions useless Variables useless

19. 19 Removing Useless Productions Example Grammar:

20. 20 First: find all variables that can produce strings with only terminals Round 1: Round 2:

21. 21 Keep only the variables that produce terminal symbols: (the rest variables are useless) Remove useless productions

22. 22 Second: Find all variables reachable from Use a Dependency Graph not reachable

23. 23 Keep only the variables reachable from S Final Grammar (the rest variables are useless) Remove useless productions

24. 24 Removing All Step 1: Remove Nullable Variables Step 2: Remove Unit-Productions Step 3: Remove Useless Variables

25. 25 Normal Forms for Context-free Grammars

26. 26 Chomsky Normal Form Each productions has form: variable or terminal

27. 27 Examples: Not Chomsky Normal Form Chomsky Normal Form

28. 28 Convertion to Chomsky Normal Form Example: Not Chomsky Normal Form

29. 29 Introduce variables for terminals:

30. 30 Introduce intermediate variable:

31. 31 Introduce intermediate variable:

32. 32 Final grammar in Chomsky Normal Form: Initial grammar

33. 33 From any context-free grammar (which doesn’t produce ) not in Chomsky Normal Form we can obtain: An equivalent grammar in Chomsky Normal Form In general:

34. 34 The Procedure First remove: Nullable variables Unit productions

35. 35 Then, for every symbol : In productions: replace with Add production New variable:

36. 36 Replace any production with New intermediate variables:

37. 37 Theorem: For any context-free grammar (which doesn’t produce ) there is an equivalent grammar in Chomsky Normal Form

38. 38 Observations Chomsky normal forms are good for parsing and proving theorems It is very easy to find the Chomsky normal form for any context-free grammar

39. 39 Greinbach Normal Form All productions have form: symbolvariables

40. 40 Examples: Greinbach Normal Form Not Greinbach Normal Form

41. 41 Conversion to Greinbach Normal Form: Greinbach Normal Form

42. 42 Theorem: For any context-free grammar (which doesn’t produce ) there is an equivalent grammar in Greinbach Normal Form

43. 43 Observations Greinbach normal forms are very good for parsing It is hard to find the Greinbach normal form of any context-free grammar

44. 44 Compilers

45. 45 Compiler Program v = 5; if (v>5) x = 12 + v; while (x !=3) { x = x - 3; v = 10; }...... Add v,v,0 cmp v,5 jmplt ELSE THEN: add x, 12,v ELSE: WHILE: cmp x,3... Machine Code

46. 46 Lexical analyzer parser Compiler program machine code input output

47. 47 A parser knows the grammar of the programming language

48. 48 Parser PROGRAM STMT_LIST STMT_LIST STMT; STMT_LIST | STMT; STMT EXPR | IF_STMT | WHILE_STMT | { STMT_LIST } EXPR EXPR + EXPR | EXPR - EXPR | ID IF_STMT if (EXPR) then STMT | if (EXPR) then STMT else STMT WHILE_STMT while (EXPR) do STMT

49. 49 The parser finds the derivation of a particular input 10 + 2 * 5 Parser E -> E + E | E * E | INT E => E + E => E + E * E => 10 + E*E => 10 + 2 * E => 10 + 2 * 5 input derivation

50. 50 10 E 25 E => E + E => E + E * E => 10 + E*E => 10 + 2 * E => 10 + 2 * 5 derivation derivation tree EE EE + *

51. 51 10 E 25 derivation tree EE EE + * mult a, 2, 5 add b, 10, a machine code

52. 52 Parsing

53. 53 grammar Parser input string derivation

54. 54 Example: Parser derivation input ?

55. 55 Exhaustive Search Phase 1: All possible derivations of length 1 Find derivation of

56. 56

57. 57 Phase 2 Phase 1

58. 58 Phase 2 Phase 3

59. 59 Final result of exhaustive search Parser derivation input (top-down parsing)

60. 60 Time complexity of exhaustive search Suppose there are no productions of the form Number of phases for string :

61. 61 Time for phase 1: possible derivations For grammar with rules

62. 62 Time for phase 2: possible derivations

63. 63 Time for phase : possible derivations

64. 64 Total time needed for string : Extremely bad!!! phase 1 phase 2 phase 2|w|

65. 65 There exist faster algorithms for specialized grammars S-grammar: symbolstring of variables appears once Pair

66. 66 S-grammar example: Each string has a unique derivation

67. 67 In the exhaustive search parsing there is only one choice in each phase For S-grammars: Total time for parsing string : Time for a phase:

68. 68 For general context-free grammars: There exists a parsing algorithm that parses a string in time (we will show it in the next class)

69. 69 The CYK Parser

70. 70 The CYK Membership Algorithm Input: Grammar in Chomsky Normal Form String Output: find if

71. 71 The Algorithm Grammar : String : Input example:

72. 72

73. 73

74. 74

75. 75

76. 76 Therefore: Time Complexity: The CYK algorithm can be easily converted to a parser (bottom up parser) Observation: