1. CS 208: Computing Theory Assoc. Prof. Dr. Brahim Hnich Faculty of Computer Sciences Izmir University of Economics

2. Contact Info Room 415 Email: brahim.hnich@ieu.edu.trbrahim.hnich@ieu.edu.tr Office hours Tuesdays: 14—16

3. Textbooks The course lecture are based on the following books Harry R. Lewis and C.H. Papadimitriou. "Elements of the Theory of Computation", ISBN 0-13-272741-2, Prentice Hall. (Available in library) Michael Sipser. "Introduction to the Theory of Computation", ISBN 053494728X, PWS Publishing Company.

4. Textbooks The course lecture are based on the following books Michael R. Garey and David S. Johnson. “Computer and Intractability", ISBN 0-7167-1045-5, W.H. Freeman and Company. C.H. Papadimitriou. “Computational Complexity", ISBN 0-201-53082-1, Addison-Wesley Publishing Company, Inc.

5. Grading and exams 6 Homeworks: 30% Midterm Exam 30% Final Exam : 40%

6. Programme Part A: Preliminaries Introduction Mathematical Notations for Computations

7. Programme Part B: Automata Theory Finite State Automata Context Free Grammars

8. Programme Part C: Computability Theory Turing Machines Decidability and Undecidability

9. Programme Part D: Complexity Theory Time Complexity Intractability

10. Programme Part E: Summary and Concluding Remarks

11. Good Luck!

12. Outline Introduction to the course Goal Organization Outline Mathematical Notations for Computations

13. What is computation? Definition: Processing of information based on a finite set of operations or rules

14. What is computation? Paper and pencil Arithmetic 99 - 8 = 91 Calculators Bar code scanner Digital camera Computer programs in C and Java …

15. What do we want in a “theory” Generality Technology-independent Abstraction: ignores inessential details Precision Mathematical, formal Can prove theorems about computation Positive: what can be computed Negative: what cannot be computed

16. So you want to be a good computer scientist/ software engineer? Learn the rules Algorithms, data structures, … Learn basic principles Data abstraction, … Understand which problems you can solve with computers and which you simply cannot … and why!

17. Learn the rules Algorithms, data structures, … Learn basic principles Data abstraction, … Understand which problems you can solve efficiently with computers and which you cannot! So you want to be a good computer scientist/ software engineer?

18. Course Overall Objective Understand the fundamental capabilities and limitations of computers Make a theory out of the idea of computation

19. Why study theory? Pragmatic Reasons Apply efficient algorithms to tractable problems Avoid intractable or impossible problems Learn to tell the difference

20. Why study theory? Mathematical education Automata Theory Computability Theory Complexity Theory

21. Why study theory? Philosophy What is computation? What is provable?

22. Course of two parts

23. The Good News You only have to take this course once Except those of you who will not pay attention to the lectures At the end, you can say that you understand what computers are really about Good for postgraduate and professional life

24. 2nd half of the course

25. The Bad News Theory course Formal Requires mathematics Understanding Proofs Analytical skills

26. The Bad News Theory course Formal Requires mathematics Understanding Proofs Analytical skills But, we will try to make it fun because it is really interesting!

27. End of Introduction Next: Topics to be covered

28. Three major topics What is a computer? Automata Theory What can (cannot) be computed? Computability Theory What can (cannot) be computed efficiently? Complexity Theory

29. Three major topics What is a computer? Automata Theory What can (cannot) be computed? Computability Theory What can (cannot) be computed efficiently? Complexity Theory

31. Key notion: tractable vs. intractable problems

32. Complexity Theory Key notion: tractable vs. intractable problems HARD EASY

33. Complexity Theory A problem is a general question: Description of parameters Description of solution E.g. Input: Given a list of integers Can you sort the list in ascending order?

34. Complexity Theory An algorithm is a step-by-step procedure: A recipe A computer program A mathematical object

35. Complexity Theory We want the most efficient algorithm as a function of problem size: Fastest (mostly) Most economical with memory (sometimes)

36. Example: Traveling Salesman Problem Given A set of m cities A set of inter-city distances Find a permutation of cities which makes a tour of minimum length

37. Example: TSP instance a c b d 5 10 9 6 3 9 (not drawn to scale)

38. Example: TSP instance a c b d 5 10 9 6 3 9 A solution: The tour a,b,d,c,a has length 27

39. Problem Size What is an appropriate measure of problem size? m the number of nodes? m(m+1)/2 distances?

40. Problem Encoding Use an encoding of the problem Alphabet of symbols Strings: abcd//10/5/9//6/9//3

41. Measures Problem size: length of the encoding Time complexity: how long an algorithm takes, as a function of problem size

42. Time Complexity What is tractable? A function f(n) is O(g(n)) whenever f(n) ≤ c. g(n) for n > some constant

43. Time Complexity A polynomial-time algorithm is one whose time complexity is O(p(n)) for some polynomial p(n) e.g. O(n 2 )

44. Time Complexity An exponential-time algorithm is one whose time complexity cannot be bounded by a polynomial e.g. O(n log n )

45. Tractability Basic distinction Polynomial time means tractable Exponential time means intractable

46. Tractability 103060 n.00001 sec.00003 sec.00006 sec n2n2.00001 sec.00009 sec.00036 sec n3n3.00001 sec.027 sec.216 sec 2n2n.001 sec17.9 min336 centuries 3n3n 0.59 sec6.5 years1.3 10 13 centuries Execution time

47. Effect of Speed-Ups Present problem size 100 times faster 1000 times faster nN1100N11000N1 n2n2 N210N231.6N2 n3n3 N34.64N310N3 2n2n N4N4+6.64N4+9.97 3n3n N5N5+4.19N5+6.29 Let us wait for faster hardware!

48. Back to TSP Your boss says: “Get me an efficient traveling-salesman algorithm, or else!” What are you going to do?

49. Response “Yes Ma’am, expect it this afternoon!”

50. Response “Yes Ma’am, expect it this afternoon!” Problem is All known algorithms (essentially) check all possible paths Exhaustive checking is exponential! … so best of luck!

51. Response “hah!, I will prove that no polynomial algorithm is possible”

52. Response “hah!, I will prove that no polynomial algorithm is possible” Problem is Proving intractability is very hard Many important problems have No known tractable algorithms No known proof for intractability

53. Response “I can’t find an efficient algorithm, I guess I’m just a pathetic loser!”

54. Response “I can’t find an efficient algorithm, I guess I’m just a pathetic loser!” Bad for job security

55. Perfect Response “The problem is NP-Complete. I cant find an efficient algorithm, but neither can any of these famous people …!” Advantage is The problem is “just as hard” as other problems smart people can’t solve efficiently. So it would no good to fire you and fire someone else to do the job

56. Perfect Response “Would you settle for a pretty good algorithm, but that does not give you the optimal solution?”

57. Perfect Response “Would you settle for a pretty good algorithm, but that does not give you the optimal solution?” Intractability isn’t the end of the story Find an approximate solution Use randomization Parallelism can help

58. Computability Theory

59. Complexity theory divides problems into two classes Tractable Intractable Computability theory divides problems into two classes Decidable Un-decidable

60. Computability Theory: important discovery In the first half of the 20 th century, mathematicians like Kurt Goedel, Alan Turing, and Alonzo Church discovered that some problems do not have a computable solution!

61. Example of undecidable problem Is a mathematical statement true or false? N.B. what could be more amenable to automation? But, no computer algorithm can perform this task!

62. Example of undecidable problem Is a mathematical statement true or false? N.B. what could be more amenable to automation? But, no computer algorithm can perform this task! These results require theoretical models for computers  Automata Theory  construction of real computers

63. Another example Hilbert’s tenth problem (1900): Find a finite algorithm that decides whether a polynomial has an integer root.

64. Another example A polynomial is a sum of terms, each term is a product of variables and constants (coefficients) A root of a polynomial is an assignment of values to the variables such the value of the polynomial is 0 x=2 and y=2 is a root for 5x +15y=25 There is no such algorithm (1970)!

65. More example of undecidable problems Does a program run forever? Is a program correct? Are two programs equivalent? Is a program optimal? …

66. Automata Theory

67. Deals with the definitions and properties of mathematical models of computation Several models Finite automata: is used in text processing, compilers, and hardware design Context-free grammar: is used in programming language and artificial intelligence (language processing)

68. Automata Theory Complexity and Computability theory require a precise definition of a computer Thus Automata theory is the best starting point to begin the study of the theory of computation!

69. Mathematical Notations and Terminology

70. Outline Sets Functions and relations Graphs Strings and languages Boolean logic Proofs

71. Sets Sets are defined by its members E.g. {1,2,4} or {blue, green, red} Order is not important

72. Set Cardinality Sets can be finite such as {1, 2} infinite such as {x| x is even} i.e. the set of even naturals If A is finite then its cardinality (or size) denoted by |A| is the number of its elements The empty Ø set has cardinality 0

73. Set operations Membership x Є A means that x is a member of the set A Subset A ⊆ B iff for every x, x Є A implies x Є A Equality A=B means that for x, x Є A iff x Є B Union {a,b} U {c,d} = {a,b,c,d} Intersection {a,b} ∩ {a,c} = {a} Difference {a,b} - {b,c} = {a}

74. Set Operations A and B are disjoint iff A ∩ B = Ø The power set of A, P(A) = {B | B ⊆ A } E.g., A= {1, 2} and P(A)={Ø, {1}, {2}, {1,2}} Cartesian Product A X B ={{a,b} | a Є A and b Є B} E.g. {1,2} x {3,4} = {(1,3), (1,4), (2,3), (2,4)}

75. Relations and functions A k-ary relation R on A 1,…,A k is a subset of A 1 x A 2 X … X A k A binary relation on A is a subset of A x A

76. Example of a relation R= {(a, b), (a, c), (b, d), (d, b), (d, d)} ab c d

77. Example of a relation R= {(a, b), (a, c), (b, d), (d, b), (d, d)} a b c d a b c d

78. Properties of binary relations Reflexive (a,a) Є R for each a Є A a

79. Properties of binary relations Symmetric If (a,b) Є R then (b,a) Є R ab ba then If

80. Properties of binary relations Transitive If (a,b) Є R and (b,c) Є R then (a,c) Є R ab And IfThen bc ac

81. Equivalence relation An equivalence relation is a relation that is Reflexive Symmetric Transitive

82. Examples Transitive Not symmetric Not reflexive

83. Examples Not Transitive Symmetric Not reflexive

84. Examples Not Transitive Symmetric Reflexive

85. Examples Transitive Symmetric Reflexive

86. Partial Functions A partial function f: S  T satisfies f ⊆ S X T For all s Є S there is at most one t Є T with (s, t) Є f Note that (s, t) Є f we write f(s)=t

87. Partial function example f:{a, b, c, d}  {a, b, c, d} f= {(a, b), (b, d), (d, d)} a b c d a b c d

88. Total Functions A partial function f: S  T satisfies f ⊆ S X T For all s Є S there is exactly one t Є T with (s, t) Є f Note that (s, t) Є f we write f(s)=t and we say that t is the image of s under f S is called the domain of f T is called the range of f

89. Total function example f:{a, b, c, d}  {a, b, c, d} f= {(a, b), (b, d), (c,d), (d, d)} a b c d a b c d

90. Injections A function f:S  T is injective if distinct elements in its domain S have distinct images For all a,b Є S. a ≠ b  f(a) ≠ f(b)

91. Injections A function f:S  T is injective if distinct elements in its domain S have distinct images For all a,b Є S. a ≠ b  f(a) ≠ f(b) 123123 123123 123123 123123

92. Surjections A function f:S  T is surjective if every b in T is the image of some a in S For all b Є T. there exists a Є S: f(a) =b

93. Surjections A function f:S  T is surjective if every b in T is the image of some a in S For all b Є T. there exists a Є S: f(a) =b 123123 123123 123123 123123

94. Bijections A function f:S  T is bijective iff it is injective and surjective. Also called a 1-1 correspondence

95. Bijections A function f:S  T is bijective iff it is injective and surjective. Also called a 1-1 correspondence 1212 123123 123123 123123

96. Relations

97. Partial functions Relations

98. Partial functions Total functions Relations

99. Injections Partial functions Total functions Relations

100. Injections Surjections Partial functions Total functions Relations

101. Injections Surjections Partial functions Total functions Relations Bijections

102. Graphs Graph G=, where V is the set of nodes or vertices E is the set of edges Degree of a vertex is the number of edges including that vertex

103. Graphs Graph G=, where V is the set of nodes or vertices E is the set of edges 1 2 4 5 3 a d c b

104. Graph Subgraph

105. Graph Path A path in a graph is a sequence of nodes connected by edge A simple path is a path that doesn’t repeat any node path

106. 1 2 4 5 3 a d c b Fully connected graph A graph is connected if every two nodes have a path between them Not fully connected graph

107. cycle Cycle A cycle is a path that starts and ends at the same node A simple cycle is a cycle that doesn’t repeat any node except the first and the last

108. tree root leaves A graph is a tree if it is connected and has no simple cycles The nodes of degree 1 are called leaves There is a specially designated node called a root

109. Directed Graphs Graph G= is directed if it has arrows instead of edges V is the set of nodes or vertices E is the set of directed edges a d c b

110. Directed Graphs The number of arrows pointing from a particular node is the outdegree The number of arrows pointing to a particular node is the indegree a d c b

111. Strings and languages An alphabet is a finite set of symbols A string over an alphabet is a finite sequence of symbols from the alphabet The length of a string is the number of symbols in the string The empty string is denoted by ε and has length 0

112. Strings and languages Reverse: abcd vs. dcba Substring: bc is a substring of abcd Concatenation: of ab and cd is abcd x k is x...x, k times A language is a set of strings

113. Boolean logic Boolean logic is a mathematical system built around the two values TRUE (1) and FLASE (0) Foundation of digital electronics and computer design TRUE and FALSE are called the Boolean values

114. Boolean operations Negation: not(1)=0 and NOT(0)=1 Conjunction: 1 AND 0 = 0 1 AND 1 = 1 0 AND 0 = 0 0 AND 1 = 0

115. Boolean expressions Use Boolean operations for combining simple statements Let P stand for the truth value of “The sun is shining” Let Q stand for the truth value of “Today is Tuesday” “P AND Q” stands for “The sun is shining and Today is Tuesday” “P OR Q” stands for “The sun is shining or Today is Tuesday” P and Q are called operands of the operation

116. More logic operations Equality: P ↔ Q is 1 iff both of its operands have the same truth value Implication: P  Q is equivalent NOT(P) OR Q

117. Distributive law P AND (Q OR R) equals (P AND Q) OR (P AND R) P OR (Q AND R) equals (P OR Q) AND (P OR R)

118. Definition Definitions describe the objects and the notions that we use. Simple or complex Precise and concise When defining some object we must make clear what constitutes that object and what does not!

119. Proofs Proofs determine the truth or falsity of a mathematical statement Finding proofs isn’t always easy There is no recipe … but there are some guidelines Actually the more proofs you do the better you get at it!

120. Proofs: Guidelines Use examples and pictures Be patient Come back to it Be neat Be concise Use a strategy I’ll attempt a proof by induction strategy It seems that I can find a counter-example easily etc

121. Proof types Proof by construction Proof by contradiction Proof by induction … Or a combination

122. Proof by construction Many theorems state that a particular type of objects exist. Proof by construction demonstrates how to build such an object To show that there exists an O(nlog n) sorting algorithm, we can construct the quick- sort algorithm

123. Proof by contradiction Proof by contradiction: we assume that the theorem is false and then show that this assumption leads to a contradiction

124. Proof by contradiction Prove by contradiction that the square root of 2 is an irrational number N.B. A number is rational if it is a fraction m/n where m and n are integers. A number is irrational iff it is not rational

125. Proof by contradiction Let us assume that sqrt(2) is rational This means there exists relatively prime integers m and n such that sqrt(2)= m/n Note that m and n cannot both be even Now rewrite sqrt(2)= m/n as n sqrt(2) = m

126. Proof by contradiction Let us take the square (n sqrt(2)) 2 = m 2 n 2 2 = m 2 Thus m 2 is even and so is m (= 2 k) So we get n 2 2 = (2k) 2 =4 k 2

127. Proof by contradiction So we get n 2 2 = (2k) 2 =4 k 2 Which simplifies to n 2 =2k 2 Thus n 2 is even and so is n This is a contradiction! QED

128. Proof by induction To prove P(n) Base case: prove P(0) Induction hypothesis: assume P(k) holds for some k≤n Induction step: Given the induction hypothesis show that P(n+1) holds

129. Proof by induction Prove by induction 1+2+…..+n = n(n+1)/2 Any volunteers?

130. Conclusions Introduction Course overview Complexity Theory Computability Theory Automata Theory Formal background