2. Problem Complexity and NP-Complete Problems Presented by Ming-Tsung Hsu

3. “I can’t find an efficient algorithm, I guess I’m just too dumb”

4. “I can’t find an efficient algorithm, because no such algorithm is possible”

5. “I can’t find an efficient algorithm, but neither can all these famous people”

6. Problem Complexity

7. Overview Complexity theory Part of the theory of computation Dealing with the resources required to solve a given problem –time (how many steps it takes to solve a problem) –space (how much memory it takes) Computational difficulty of computable functions

8. Relations between Problems, Algorithms, and Programs Problem Algorithm Program.... A single "problem" is an entire set of related questions, where each question is a finite-length string A particular question is called an instance

9. Time Complexity Number of steps that it takes to solve an instance of the problem –Function of the size of the input –Using the most efficient algorithm Exact number of steps will depend on exactly what machine or language is being used –To avoid that problem, generally use Big O notation

10. Time Complexity (cont’d) Worst-case –an upper bound on the running time for any input Average-case –we shall assume that all inputs of a given size are equally likely Best-case –to get the lower bound

11. Time Complexity (cont’d) Sequential search in a list of size n –worst-case : n times –best-case : 1 times –average-case :

12. Asymptotic Notation - O-notation (Big O, Order of) Asymptotic upper bound Definition: For a given function g(n), denoted by O(g(n)) is the set of functions: O(g(n)) = {there exist positive constants c and n 0 such that f(n) = n 0 }

13. Asymptotic Notation - Ω-notation Asymptotic lower bound Definition: For a given function g(n), denoted by Ω(g(n)) is the set of functions:  (g(n)) = {there exist positive constants c and n 0 such that f(n) >= cg(n) for all n >= n 0 }

14. Asymptotic Notation –  -notation Definition: For a given function g(n), denoted by  (g(n)) is the set of functions:  (g(n)) = {there exist positive constants c 1, c 2, and n 0 such that c 1 g(n) = n 0 }

15. Cost and Complexity Algorithm complexity can be expressed in Order notation, e.g. “at what rate does work grow with N?”: –O(1)Constant –O(logN)Sub-linear –O(N)Linear –O(NlogN)Nearly linear –O(N 2 )Quadratic –O(X N )Exponential But, for a given problem, how do we know if a better algorithm is possible?

16. Practical Complexities 10 9 instructions/second computer

17. Impractical Complexities 10 9 instructions/second computer

18. Faster Computer Vs Better Algorithm Algorithmic improvement more useful than hardware improvement. E.g. 2 n to n 3

19. The Problem of Sorting For example, in discussing the problem of sorting: Algorithms: –Bubble-sort – O(N 2 ) –Merge-sort – O(N Log N) –Quick-sort – O(N Log N) Can we do better than O(N Log N)?  What is the problem complexity?

20. Algorithm vs. Problem Complexity Algorithm complexity is defined by analysis of an algorithm Problem complexity is defined by –An upper bound – defined by an algorithm (worst case) –A lower bound – defined by a proof (A lot of problems are still unknown)

21. The Upper Bound Defined by an algorithm Defines that we know we can do at least this good Perhaps we can do better Lowered by a better algorithm –“For problem X, the best algorithm was O(N 3 ), but my new algorithm is O(N 2 ).”

22. The Lower Bound Defined by a proof Defines that we know we can do no better than this It may be worse Raised by a better proof –“For problem X, the strongest proof showed that it required O(N), but my new, stronger proof shows that it requires at least O(N 2 ).” –Might get harder and harder

23. Upper and Lower Bounds The Upper bound is the best algorithmic solution that has been found for a problem –“What’s the best that we know we can do?” The Lower bound is the best solution that is theoretically possible. –“What cost can we prove is necessary?”

24. Changing the Bounds Upper bound Lowered by better algorithm Lower bound Raised by better proof

26. Closed Problems The upper and lower bounds are identical Upper bound Lower bound The inherent complexity of problem

27. Closed Problems (cont’d) Better algorithms are still possible Better algorithms will not provide an improvement detectable by “Big O” Better algorithms can improve the constant costs hidden in “Big O” characterizations

28. Open Problems The upper and lower bounds differ. Upper bound Lowered by better algorithm Lower bound Raised by better proof Unknown Who Falls Short?

29. Open Problems (cont’d) D. Harel. Algorithmics: The Spirit of Computing. Addison-Wesley, 2nd edition,1992 “... if a problem gives rise to a... gap, the deficiency is not in the problem but in our knowledge about it. We have failed either in finding the best algorithm for it or in proving that a better one does not exist, or in both”

30. Examples - Closed Problem: Searching an unordered list of n items –Upper bound: O(n) comparisons (from linear search) –Lower bound: O(n) comparisons –No gap; so we know the problem complexity: O(n) Problem: Searching an ordered list of n items –Upper bound: O(log n) comparisons (from binary search) –Lower bound: O(log n) comparisons –No gap; so we know the problem complexity: O(log n)

31. Examples - Closed (cont’d) Problem: Sorting n arbitrary elements –Upper bound: O(n × log n) comparisons (from, e.g., merge sort) –Lower bound: O(n × log n) comparisons –No gap; so we know the problem complexity: O(n × log n) Problem: Towers of Hanoi for n disks (n > 1) –Upper bound: O(2 n ) moves –Lower bound: O(2 n ) moves –No gap; so we know the problem complexity: O(2 n )

32. Examples - Open Problem: Multiplication of two integers, where n is the number of digits –Upper bound: O(nlognloglogn) –Lower bound: O(n) –There’s a gap (but only a small one) Problem: Finding the minimum spanning tree in a graph of n edges and m vertices. –Upper bound: O(nlogn) or O(n+mlogm) –Lower bound: O(n) –There’s a gap (but only a small one) Do not be fooled by the last two examples into thinking that all gaps are small!

33. Tractable vs. Intractable Problems are tractable if the upper bounds and lower bounds have only polynomial factors –O (log N) –O (N) –O (N K ) where K is a constant Problems are intractable if the upper bounds and lower bounds have an exponential factor (are solvable in theory, but can't be solved in practice) –O (N!) –O (N N ) –O (2 N )

34. Terminology Polynomial algorithms are reasonable Polynomial problems are tractable Exponential algorithms are unreasonable Exponential problems are intractable

35. Terminology Tractable Intractable Reasonable Unreasonable ProblemsAlgorithms Polynomial Exponential

36. NP-Complete Problems

37. Definitions of P, NP A decision problem is a problem where the answer is always “YES/NO” An optimization problem is a problem can be many possible solutions. Each solution has a value, and we wish to fond a solution with the optimal (maximum or minimum) value We can re-cast an optimization problem to a decision problem by using loop EX: Find Max Clique in a graph  Is there a clique which size is k=n, n-1, …?

38. Definitions (cont’d) A deterministic algorithm is an algorithm which, in informal terms, behaves predictably. If it runs on a particular input, it will always produce the same correct output, and the underlying machine will always pass through the same sequence of states A non-deterministic algorithm has two stages: 1. Guessing (in nondeterministic polynomial time with correct guessing) Stage 2. Verification (in deterministic polynomial time) Stage

39. Definitions (cont’d) EX: A: (3, 11, 2, 5, 8, 16, …, 200 ), Is the “x=5” in A? deterministic algorithm for i=1 to n if A(i) = x then print (i) and return true next return false non-deterministic algorithm j  choice (1:n) if A(j) = x then print (j); success endif print (‘0’); failure

40. Certificates Returning true: in order to show that the solution can be made, we only have to show one solution that works –This is called a certificate. Returning false: in order to show that the solution cannot be made, we must test all solution.

41. Oracles If we could make the ‘right decision’ at all decision points, then we can determine whether a solution is possible very quickly! –If the found solution is valid, then True –If the found solution is invalid, then False If we could find the certificates quickly, NPC problems would become tractable – O(N) This (magic) process that can always make the right guess is called an Oracle.

42. Determinism vs. Nondeterminism Nondeterministic algorithms produce an answer by a series of “correct guesses” Deterministic algorithms (like those that a computer executes) make decisions based on information.

43. Definitions (cont’d) The complexity class P is the set of decision problems that can be solved by a deterministic machine (algorithm) in polynomial time The complexity class NP is the set of decision problems that can be solved by a non-deterministic machine (algorithm) in polynomial time Since deterministic algorithms are just a special case of deterministic ones  P ⊆ NP The big question: Does P = NP?

44. Definitions (cont’d)

45. Problems that Cross the Line What if a problem has: –An exponential upper bound –A polynomial lower bound We have only found exponential algorithms, so it appears to be intractable. But... we can’t prove that an exponential solution is needed, we can’t prove that a polynomial algorithm cannot be developed, so we can’t say the problem is intractable...

46. Reduction A problem P can be reduced to another problem Q if any instance of P can be rephrased to an instance of Q, the solution to which provides a solution to the instance of P –This rephrasing is called a transformation If P is polynomial-time reducible to Q, we denote this P ∝ Q Intuitively: If P reduces in polynomial time to Q, P is “no harder to solve” than Q EX: Maximum ∝ Sorting but Sorting cannot be reduced to Maximum

47. The Boolean Satisfiability Problem (SAT) An instance of the problem is a Boolean expression written using only “AND”, “OR”, “NOT”, “variables”, and “parentheses” Question: given the expression, is there some assignment of “TRUE” and “FALSE” values to the variables will make the entire expression true? Clause: “OR” together a group of literals Conjunctive normal form (CNF): formulas that are a conjunction (AND) of clauses SAT ∈ NP O(2 n m) (n: # of literals, m: # of clauses)

48. SAT (cont’d) EX: CNF: a1 V a2 V a3 (clause1) & ā1 (clause2)  (0, 0, 1) satisfied & ā2 (clause3)

49. Cook’s Theorem NP = P iff SAT ∈ P pf: (  ) ∵ SAT ∈ NP  SAT ∈ P (  ) Difficult (every NP problem can be reduced to SAT problem)

50. NP-Complete and NP-Hard A problem P is NP-Complete if P ∈ NP and every NP problem can be reduced to P –P ∈ NP and X ∝ P for all X ∈ NP –If any NP-Complete problem can be solved in polynomial time, then NP = P –SAT problem is an NP-complete problem A problem P is NP-Hard if every NP problem can be reduced to P –We say P is NP-Complete if P is NP-Hard and P  NP

51. NP-Complete “NP-Complete” comes from: – Nondeterministic Algorithm in Polynomial time – Complete - “Solve one, Solve them all” There are more NP-Complete problems than provably intractable problems.

52. NP-Hard “NP-Hard” comes from: – Nondeterministic Algorithm in Polynomial time – Hard - “as "hard" as any problem in NP”

53. NP-Complete and NP-Hard (cont’d) NP NPC NPH

54. Example NP-Complete Problems Path-Finding (Traveling salesman) Map coloring Scheduling and Matching (bin packing) 2-D arrangement problems Planning problems (pert planning) Clique

55. Traveling Salesman

56. 5-Clique

57. Map Coloring

58. The Knapsack Problem The 0-1 knapsack problem: –A thief must choose among n items, where the ith item worth v i dollars and weighs w i pounds –Carrying at most W pounds, maximize value –EX: B = {13,17,19,21,24,27,36,42}, S = 93, Find X=(X1,X2,…,X8), where Xi=0 or 1, i=1,2,3,…,8 such that ΣXiBi = S A variation, the fractional knapsack problem: –Thief can take fractions of items –Think of items in 0-1 problem as gold ingots, in fractional problem as buckets of gold dust

59. Class Scheduling Problem With N teachers with certain hour restrictions M classes to be scheduled, can we: –Schedule all the classes –Make sure that no two teachers teach the same class at the same time –No teacher is scheduled to teach two classes at once

60. Proving NP-Completeness Show that the problem A is in NP. (i.e. Show that a certificate can be verified in polynomial time) Assume A is not NP complete Show how to convert an existing NPC problem into the problem A (in polynomial time) –P is NP-Complete –P ∝ A –Proving that reduction is polynomial  A is NP-Complete If we can do it we’ve done the proof! If we can turn an existing NP-complete problem into our problem in polynomial time... 

61. 3-SAT is NPC ? 3-SAT is NP SAT ∝ 3-SAT (by construct sets of unsatisfiable clauses) –Clause contains only one literal L i Q1 V Q2 L i V Q1 V Q2 & Ǭ1 V Q2 & L i V Ǭ1 V Q2 & Q1 V Ǭ2  & L i V Q1 V Ǭ2 & Ǭ1 V Ǭ2 & L i V Ǭ1 V Ǭ2 –Clause contains two literals L i and L j Q1 L i V L j V Q1 & Ǭ1  & L i V L j V Ǭ1

62. 3-SAT is NPC? (cont’d) –Clause contains three literals  do nothing –Clause contains more than three literals (L1, L2, …, L n ) –Reduction is O(mn) (m: # of clauses, n: # of literals)

63. 3-SAT is NPC? (cont’d) Example:

64. 3-SAT is NPC? (cont’d) L1L2L3L4L5Y1Y2Y3Y4Y5 1111000000 1100000001 1101000000 1100100000 1101100000 1111100000 1000000001 1001000000 1000100000 1001100000 L1L2L3L4L5 11110 11000 11010 11001 11011 11111 10000 10010 10001 10011

65. Clique Decision Problem (CDP) is NPC Construct a graph G = (V,E) from such that G will have a clique of size at least m iff F is satisfiable (SAT ∝ CDP) Construction Rule –A literal in a clause is a vertex (node) in G –A edge is a intersection of two clauses An assignment of F –(a) One literal is true then the clause is true –(b) Ā & A does not exist  Add edges between any two nodes exclusive of nodes are in the same clause and nodes are exclusive

66. CDP (cont’d) Example: F = (a1 V a2 V a3) & ( ā1 V a2 V ā3 ) & ( ā2) and G constructed by F –F is satisfied by (1,0,0) or (0,0,1)  G has a clique of size at least 3 –m = 3  F is satisfied ā2, 3 a1, 1 ā1, 2 a2, 1 ā3, 2 a3, 1 a2, 2

67. CDP (cont’d) Proof: support F has m clauses (  ) F is satisfiable  exists an assignment the literals in this assignment forms a clique with size equal m (a complete subgraph of G with degree m-1) (  )If we can find a clique C with size equal m Every clause has exactly one literal in C Any two vertices in C are jointed and each edge is the intersection of two clauses  Every clause is true by literals in C  F is satisfied

68. Become Famous! To get famous in a hurry, for any NP- Complete problem: –Raise the lower bound (via a stronger proof) –Lower the upper bound (via a better algorithm) They’ll be naming buildings after you before you are dead!

69. Decidable vs. Undecidable Problems

70. Decidable Problems We now have three categories: –Tractable problems –NP-Complete problems –Intractable problems All of the above have algorithmic solutions, even if impractical.

71. Undecidable Problems No algorithmic solution exists –Regardless of cost –These problems aren’t computable –No answer can be obtained in finite amount of time

72. The Unbounded Tiling Problem What if we took the tiling puzzle and removed the bounds: –For a given number (T) of different kinds of tiles –Can we arrive at an arrangement that will fill any size area? By removing the bounds, this problem becomes undecidable.

73. The Halting Problem Given an algorithm A and an input I, will the algorithm reach a stopping place? loop exitif (x = 1) if (even(x)) then x <- x div 2 else x <- 3 * x + 1 endloop In general, we cannot solve this problem in finite time.

74. Partially and Highly Undecidable Most undecidable problems have finite certificates. These are partially decidable. Some undecidable problems do not have finite certificates even with an oracle. These are called highly undecidable.

75. A Hierarchy of Problems For a given problem, is there or will there ever be an algorithmic solution? ProblemIn Theory In Application Highly Undecidable NO NO Partially Undecidable NO NO Intractable YES NO Tractable YES YES

76. Thanks