Donghyun (David) Kim Department of Mathematics and Computer Science North Carolina Central University 1 Chapter 7 Time Complexity Some slides are in courtesy of Prof. Verma at Univ. of Houston, Max Alekseyev at Univ. of South Carolina Spring 2016 CISG 5115 Theory of Computation

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Chapter 7.1 Measuring Complexity 2

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Running Time 3

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Highest-Order Term Informally speaking, the highest-order term of a function f ( n ) is a term that grows with n most quickly as compared to the other terms. For example, the function has four terms and the highest-order term is 6 n 3. 4

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Asymptotic Analysis Often the exact running time of an algorithm is a complex expression. The exact running time may depends on minor inessential details such as a precise definition of a TM, while there are many different but equivalent definitions of a TM. In most cases, to determine whether an algorithm is “fast” or to compare it to other algorithms, it is enough to know just the highest-order term of its time complexity. As the size of input grows, the contribution of the lower-order terms in the running time becomes negligible as compared to the highest-order term. Finding the highest-order term of an algorithm's time complexity is called asymptotic analysis. 5

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Asymptotic Upper Bound 6

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Examples Usually for logarithms we need to specify a base, such as log 2 n. But since for any fixed base b > 1, there is an identity we can write log 2 n = O (log b n ) or simply log 2 n = O (log n ) (as the base of logarithm does not matter). Q: Why the inequality b > 1 is important? log 2 log 2 n = O (loglog n ) because for any base b > 1: Similarly, and However, The upper bound is not uniquely defined: for example, we can say or or or, whichever is the most suitable for our needs. 7

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Polynomial vs. Exponential Bounds We will distinguish between algorithms with polynomial running time of the form n c (which is the same as n O (1) or 2 O (log n ) from algorithms with exponential running time of the form. Often we will deal with integer inputs and outputs written in the binary form (i.e., over the alphabet {0, 1}). The length n of a binary representation of an integer m is bounded by Note that the running time is always viewed as a function of the length of the input. In terms of an input m (if it is an integer), running time is polynomial if it is (log m ) O (1) but not m O (1) (the latter may be exponential in log m ). 8

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Small-o Notation 9

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Complexity Classes 10

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Complexity in Other Models 11

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Chapter 7.2 The Class P 12

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Polynomial vs. Exponential Running Time 13

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Exhaustive Search Algorithms 14

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Exhaustive Search Algorithms – cont’ However, existence of an exponential-time algorithm does not mean that the problem cannot be solved in polynomial time. Usually designing a polynomial-time algorithm requires deeper understanding of the problem. In particular, for a number of decades it was not known whether it is possible to solve the primality testing problem in polynomial time. This uncertainty was resolved only in 2002 by Agrawal, Kayal, and Saxena who gave rst polynomial-time algorithms for primality testing (now called AKS test). 15

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Why Polynomial? An algorithm is usually supposed to read its input. Hence, the “minimal” complexity of a non-trivial algorithm is proportional to n (where n is the length of the input). We often combine existing algorithms (e.g., use them as subroutines) to get new ones. All reasonable deterministic computational models are polynomially equivalent – any one of them can simulate another with only a polynomial increase in running time. (next page) 16

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Properties of Polynomials 17

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Complexity Classes 18

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University The Class P From now on, we will be interested in principal possibility to solve problems in polynomial time. Namely, we will try to determine whether a given language can be decided in polynomial time. If so, we will say that this language belongs to the class P. Definition P is the class of languages that are decidable in polynomial time on a deterministic single-tape Turing machine. In other words, The class P plays a central role in theory of computation since: P is invariant for all “reasonable” models of deterministic computations. P roughly corresponds to the class of problems that can be realistically solved on a computer. 19

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Examples of Problems in P PATH ={, G is a directed graph that has a directed path from s to t.} RELPRIME ={, x and y are relatively prime.} Every context-free language is a member of P. 20

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Chapter 7.3 The Class NP 21

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University “Hard” Problems Form Complexity Classes For some problems, it is possible to optimize brute- force solution and obtain a polynomial running time solution (e.g., via dynamic programming) But for some other problems, there is no known simple way to obtain polynomial solution (and it is even not known if a polynomial time solution exists) However, these supposedly “hard” problems are not independent. Many of them form classes such that a polynomial solution to one problem from a class can be used to solve every other problem from the same class in polynomial time (thy are linked!). 22

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Hamiltonian Path A Hamiltonian path in a directed graph G is a directed path that goes through each node in G exactly once. HAMPATH = { | G is a directed path graph with a Hamiltonian path from s to t } It is easy to construct an exponential time solution to HAMPATH (Q: how?) No polynomial time algorithm is known for HAMPATH 23

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Polynomial Verifiability Despite that nobody knows how to find a Hamiltonian path in a given graph efficiently, once such a path is found, it is very easy to verify it is indeed a Hamiltonian path. This property is called polynomial verifiability. Q: Prove that HAMPATH is polynomial verifiable. Q: Prove that the following problem is polynomial verifiable: COMPOSITES = {x | x = pq, for integers p, q > 1} Note that some problems may not be polynomial verifiable. For example, it is not clear how to prove that is polynomial verifiable (i.e., to prove that there is no Hamiltonian path in a given graph). 24

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Polynomial Verifiability – cont’ 25

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Polynomial Verifiability – cont’ 26

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Class NP Definition NP is the class of languages that have polynomial time verifiers. Problems in NP are called NP-problems. The term NP stands for nondeterministic polynomial time due to the following theorem: Theorem A language is in NP if and only if it is decided by some nondeterministic polynomial time Turing machine. (see Theorem 7.20 on p. 266 in Sipser) Proof Idea Let A in NP and show that A is decided by a polynomial time NTM N. Assume A is decided by a polynomial time NTM N and construct a polynomial time verifier V. 27

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Nondeterministic Time Complexity Definition NTIME ( t ( n )) = { L | L is a language decided by a O ( t ( n )) time nondeterministic Turing machine} As a corollary from the previous theorem, we have 28

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Examples of NP-problems 29

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University P vs. NP 30

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Chapter 7.4 NP -completeness 31

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University NP -complete Problems For some problems in NP, there is no simple way to obtain a solution in polynomial time (and it is even not known if it does exist or not) However, these supposedly “hard” problems are not independent. Many of them form classes such that a polynomial time solution to one problem from a class can be used to solve every other problem from the same class in polynomial time. The “hardest” problems in NP are called NP -complete. Finding a polynomial time algorithm for an NP - complete problem would imply a polynomial time algorithm for any other problem in NP. 32

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Importance of NP -complete Problems 33

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Boolean Expressions and Satisfiability

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Boolean Expressions and Satisfiability – cont’ 35

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Satisfiability Problem (SAT) 36

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Polynomial Time Computable Functions 37

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Polynomial Time Reducibility 38

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Polynomial Time Reducibility 39

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Theorems 40

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University NP -complete Problems 41

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University 3 Satisfiability Problem ( 3SAT ) 42

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University 3SAT is NP -complete – cont’ 43

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University Chapter 7.5 Additional NP-complete Problems 44

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University VERTEX-COVER is NP - complete 45

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University CLIQUE is NP -complete 46

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University SET-COVER is NP -complete 47

Spring 2016 CISG 5115 Department of Mathematics and Physics Donghyun (David) Kim North Carolina Central University UHAMPATH is NP -complete 48