1. Algorithm Design and Analysis (ADA)
, Semester
17. P and NP, and Beyond
Objective
look at the complexity classes P, NP, NP-Complete, NP-Hard, and Undecidable problems

2. Overview A Decision Problem Polynomial Time Solvable (P)
Nondeterministic Polynomial (NP)
Nondeterminism
P = NP : the BIG Question
NP-Complete
The Circuit-SAT Problem
NPC and Reducability
NP-Hard
Approximation Algorithms
Decidable Problems
Undecidable Problems
Computational Difficulty
Non-Technical P and NP

3. 1. A Decision Problem
The problems in the P, NP, NP-Complete complexity classes (sets) are all defined as decision problems
A decision problem is one where the solution is written as a yes or no answer
Example:
Is 29 a prime number?

4. Converting to a Decision Problem
0/1 Knapsack is an optimization problem whose goal is to optimize (maximize) the number of items in a knapsack; the result is numerical.
The decision problem version of 0/1 Knapsack determines if a certain number of items is possible without exceeding the knapsack's capacity; the result is yes or no.
Usually, the translation from an optimization problem into a decision problem is easy since the required result is simpler.

5. 2. Polynomial Time Solvable (P)
The P set contains problems that are solvable in polynomial-time
the algorithm has running time O(nk) for some constant k.
Polynomial times: O(n2), O(n3), O(1), O(n log n)
Not polynomial times: O(2n), O(nn), O(n!)
e.g. Testing if a number is prime runs in O(n) time
P problems are called tractable because of their polynomial running time.

6. P Examples

7. 3. Nondeterministic Polynomial (NP)
NP is the class of problems that
can be checked by a polynomial running time algorithm
but can (or might) not be solved in polynomial time
Most people believe that the NP problems are not polynomial time solvable (written as P ≠ NP)
this means "cross out the might"
Algorithms to solve NP problems require exponential time, or greater
they are called intractable

8. P and NP as Sets NP P P = NP or P ≠ NP Easily checkable problems
e.g. prime factorization NP P
Easily solvable problems
e.g. multiplication, sorting,
prime testing
P = NP
or P ≠ NP

9. Computational Difficulty Line=?
computational
difficulty
(or hardness) P NP

10. 3.1. Factorization is in NP
Prime Factorization is the decomposition of a composite number into prime divisors.
24 = 2 x 2 x 2 x 3
91 = 7 x 13
This problem is in NP, not P
It is not possible to find (solve) a prime factorization of a number in polynomial time
But given a number and a set of prime numbers, we can check if those numbers are a factorization by multiplication, which is a polynomial time operation

11. A Bigger Example
? x ? =
Answer is: X

12. Factoring in the Real World
Spread over 2 years, and finishing in 2009, several researchers factored a 232-digit number utilizing hundreds of machines.

13. 3.2. HC: Another NP Example
Is there a Hamiltonian Cycle in a graph G ?
A hamiltonian cycle of an undirected graph is a simple cycle that contains every vertex exactly once.
find a HC

14. Finding a HC
How would an algorithm find an hamiltonian cycle in a graph G?
One solution is to list all the permutations of G's vertices and check each permutation to see if it is a hamiltonian cycle.
What is the running time of this algorithm?
there are |V|! possible permutations, which is an exponential running time
So the hamiltonian cycle problem is not polynomial time solvable.

15. Checking a HC But what about checking a possible solution?
You are given a set of vertices that are meant to form an hamiltonian cycle
Check whether the set is a permutation of the graph's vertices and whether each of the consecutive edges along the cycle exists in the graph
Checking needs polynomial running time

16. Deciding whether a Graph G has a Hamiltonian Cycle is not polynomial time solvable but is polynomial time checkable, so is in NP.
We'll see later that it is also in the NP-Complete (NPC) set

17. 4. Nondeterminism A deterministic algorithm behaves predictably.
When it runs on a particular input, it always produces the same output, and passes through the same sequence of states.
A nondeterministic algorithm has two stages:
1. Guessing (in nondeterministic polynomial time by generating guesses that are always correct)
The (magic) process that always make the right guess is called an Oracle.
2. Verification/checking (in deterministic polynomial time)

18. Example (Searching)
Is x in the array A: (3, 11, 2, 5, 8, 16, …, 200 )?
Deterministic algorithm
for i=1 to n if (A[i] == x) then print i; return true return false
Nondeterministic algorithm
j = choice (1:n) // choice is always correct if (A[j] == x) then print j; return true return false // since j must be correct if A[] has x

19. Oracle as Path Finder
Another way of thinking of an Oracle is in terms of navigating through a search graph of choices
at each choice point the Oracle will always choose the correct branch which will lead eventually to the correct solution

20. P and NP and Determinism
P is the set of decision problems that can be solved by a deterministic algorithm in polynomial time
NP is the set of decision problems that can be solved by a nondeterministic algorithm in polynomial time
Deterministic algorithms are a 'simple' case of nondeterministic algorithms, and so P ⊆ NP
simple because no oracle is needed

21. No one knows how to write a nondeterministic polynomial algorithm, since it depends on a magical oracle always making correct guesses.
So in the real world, the solving part of every NP problem has exponential running time, not polynomial.
But if someone discovered/implemented an oracle, then NP would becomes polynomial, and P = NP
this is the biggest unsolved question in computing

22. 5. P = NP : the BIG Question
On balance it is probably false (i.e. P ≠ NP)
Why?
you can't engineer perfect luck
and
generating a solution (along with evidence) is harder than checking a possible solution

23. One Possible Oracle
One possible way to write an oracle would be to use a parallel computer to create an infinite number of processes/threads
have one thread work on each possible guess at the same time
the "infinite" part is the problem
Perhaps solvable by using quantum computing or DNA computing (?)
(but probably not)

24. Clay Millennium Prize Problems
P = NP Problem
Riemann Hypothesis
Birch and Swinnerton-Dyer Conjecture
Navier-Stokes Problem
Poincaré Conjecture 
solved by Grigori Perelman; declined the award in 2010
Hodge Conjecture
Yang-Mills Theory
each one is worth US $1,000,000

25. Short Descriptions P = NP Problem Riemann Hypothesis
no need, I hope :)
Riemann Hypothesis
involves the zeta function distribution of the primes
Birch and Swinnerton-Dyer Conjecture
concerns elliptic (~cubic) curves, their rational coordinates, and the zeta function; wide uses (e.g. in data comms)
Navier-Stokes Problem
solve the equation for calculating fluid flow

26. Poincaré Conjecture Hodge Conjecture
distinguish between 3D topological shapes (manifolds) in 4D space; involves shrinking loops around a shape to a point
Hodge Conjecture
reshape topological shapes (spaces) made with algebra (polynomial equations) and calculus (integration) techniques so only algebra is needed to define them
a solution would strongly link topology, analysis and algebraic geometry

27. Yang-Mills Theory (and the Mass gap Hypothesis)
the Yang-Mills equations allow the accurate calculation of many fundamental physical constants in electromagnetism, strong and weak nuclear forces
very useful
but there's no solid mathematical theory for it
also need to prove the Mass gap hypothesis which explains why the strong nuclear force don't reach far outside the nucleus

28. Two Clay Institute videos
Riemann hypothesis, Birch and Swinnerton-Dyer Conjecture, P vs NP
Poincaré conjecture, Hodge conjecture, Quantum Yang-Mills problem, Navier-Stokes problem
A popular maths book about the problems:
The Millennium Problem Keith J. Devlin Basic Book, 2003
(17 min video):

29. 6. NP-Complete same hardness or harder
A problem is in the NP-Complete (NPC) set if it is in NP and is at least as hard as other problems in NP.
Some NPC problems:
Hamiltonian cycle
Path-Finding (Traveling salesman)
Cliques
Map (graph, vertex) coloring
subset sum
Knapsack problem
Scheduling
Many, many more ...
same hardness or harder

30. Computational Difficulty=?
computational
difficulty
NP-complete P
Hamiltonian Cycle, TSP, Knapsack, Tetris, ... NP

31. Drawing P, NP, NPC as Sets P ≠ NP P = NP (= NPC)
all problems together in one P set

32. 6.1. Travelling Salesman (TSP)2 2 1 1 3 4 1 3 4 5 4 2 1 2 2 2 2 1 1 1
The salesperson must make a minimum cost circuit, visiting each city exactly once.

33. A solution (distance = 23):1 3 1 1 4 1 3 5 4 4 2 1 2 2 2 2 1 1 1

34. TSPs (and variants) have enormous practical importance
Lots of research into good approximation algorithms
Recently made famous as a DNA computing problem

35. Clique
Set of vertices such that they all connect to each other.

36. K-Cliques
A K-clique is a set of K nodes connected by all the possible K(K-1)/2 edges between them
This graph contains a 3-clique and a 4-clique

37. K-Cliques Given: (G, k) Decision problem: Does G contain a k-clique?
Brute Force
Try out all {n choose k} possible locations for the k clique

38. 6.3. Map Coloring
Can a graph be colored with k colors such that no adjacent vertices are the same color?

39. 6.4. Subset-Sum
Given a set of integers, does there exist a subset that adds up to some target T?
Example: given the set {−7, −3, −2, 5, 8}, is there a non-empty subset that adds up to 0.
Yes: {−3, −2, 5}
Can also be thought of as a special case of the knapsack problem

40. 6.5. The Knapsack Problem The 0-1 knapsack problem: x1 x2 x4 x3 x5
A thief must choose among n items, where the ith item is worth vi dollars and weighs wi pounds
Carry at most W kg, but make the value maximum
Example : 5 items: x1 – x5, values = (4, 2, 2, 1, 10} weights = {12, 2, 1, 1, 4}, W = 15, maximize total value x1 x2 x4 x3 x5

41. Pseudocode n items 2n subsets O(n) for Running time is O(n2n)
def knapsack(items, maxweight):
best = {}
bestvalue = 0
for s in allPossibleSubsets(items):
value = 0
weight = 0
for item in s:
value += item.value
weight += item.weight
if weight <= maxweight:
if value > bestvalue:
best = s
bestvalue = value
return best
2n subsets
O(n) for
each one
Running time is O(n2n)

42. The fractional knapsack problem:
The bounded knapsack problem removes the restriction that there is only one of each item, but restricts the number of copies of each kind of item
The fractional knapsack problem:
the thief can take fractions of items
e.g. the thief is presented with buckets of gold dust, sugar, spices, flour, etc.

43. 6.6. Class Scheduling Problem
We have N teachers with certain time restrictions, and M classes to be scheduled. Can we:
Schedule all the classes?
Make sure that no two teachers teach the same class at the same time?
No teacher is scheduled to teach two classes at once?

44. 6.7. Why study NPC? Most interesting problems are NPC.
If you can show that a problem is NPC then:
You can spend your time developing an approximation algorithm rather than searching for a fast algorithm that solves the problem exactly (see later)
or you can look for a special case of the problem, which is polynomial

45. Impact of NPC Main intellectual export of CS to other disciplines
6,000 citations per year (title, abstract, keywords).
more than "compiler", "operating system", "database"
Broad applicability and classification power
"Captures vast domains of computational, scientific, mathematical endeavors, and seems to roughly delimit what mathematicians and scientists had been aspiring to compute feasibly." [Papadimitriou 1995]

46. More NPC Problems There are over 3000 known NPC problems.
Many of the major ones are listed at:
The list is divided into several categories:
Graph theory, Network design, Sets and partitions, Storage and retrieval, Sequencing and scheduling, Mathematical programming, Algebra and number theory, Games and puzzles, and Logic

47. Another way of classifying NPC problems:
Packing problems: SET-PACKING, INDEPENDENT SET
Covering problems: SET-COVER, VERTEX-COVER
Constraint satisfaction problems: SAT, 3-SAT
Sequencing problems: HAMILTONIAN-CYCLE, TSP
Partitioning problems: 3D-MATCHING 3-COLOR
Numerical problems: SUBSET-SUM, KNAPSACK

48. Hard vs Easy
There's often not much difference in appearance between easy problems and hard ones:
Hard Problems
(NP-Complete)
Easy Problems
(in P)
SAT, 3SAT
2SAT
Traveling Salesman Problem
Minimum Spanning Tree
3D Matching
Bipartite Matching
Knapsack
Fractional Knapsack

49. 7. The Circuit-SAT Problem
This was (almost) the first problem to be proved NPC
Cook 1971, Levin 1973
actually Cook proved SAT was in NPC, but Circuit-SAT is almost the same, and is the example used in CLRS
What did Cook do?
Remember: a problem is in NPC if
it is in NP and
it is at least as hard as other problems in NP

50. What is Circuit-SAT?
The Circuit-Satisfiability Problem (Circuit-SAT) is a combinational circuit composed of AND, OR, and NOT gates.
A circuit is satisfiable if there exists a set of boolean input values that makes the output of the circuit equal to 1.

51. A Circuit
Logic Gates 1 1 1
NOT 1 1 OR 1 1 1
AND

52. Satisfiablity Examples
No 1 is
possible
(a) Unsatisfiable
(a) Satisfiable
Circuit (a) is satisfiable since <x1, x2, x3> = <1, 1, 0> makes the output 1.

53. Part 1. Circuit-SAT is NP
If there are n inputs, then there are 2n possible assignments that need to be tried out in order to find the solution
solving is O(2n), exponential running time
Testing a solution is done by evaluating the circuit's logical expression which can be done in linear running time
testing is O(n), polynomial running time

54. Part 2. Circuit-SAT is NPC
Cook’s Theorem
it proves (Circuit-) SAT is NPC from first principles, based on its implementation of the Turing machine
All other problems have been shown to be NPC by reducing (transforming) Circuit-SAT into those problems (either directly or indirectly)

55. What's a Turing Machine?
A Turing machine is a theoretical device composed of an infinitely long tape with printed symbols representing instructions.
The tape can move backwards and forwards in the machine, which can read the instructions and write results back onto the tape.
sensor to read,
write, or erase
machine with
finite number
of states
symbols on tape
moving tape

56. Invented in 1936 by Alan Turing.
Despite its simplicity, a Turing machine can simulate the logic of any computing machine or algorithm.
i.e. a function is algorithmically computable if and only if it is computable by a Turing machine
called the Church–Turing thesis (hypothesis)

57. Algorithms & Turing Machines
Any algorithm can be implemented as a Turing machine
this is how "computable" is formally defined
Every problem in NP can be transformed into a program running on a (non-deterministic) Turing machine. (Church-Turing)
"non-deterministic" means that the finite state machine used by the Turing machine is non-deterministic

58. Back to Cook's Theorem
Cook's Theorem argues that every feature of the Turing machine, including how instructions are executed, can be written as logical formulas in a satisfiability (SAT) problem. (cook 1)
So any a program executing on a Turing machine can be transformed into a logical formula of satisfiability. (cook 2)

59. Combining Church-Turing and cook 2 means that every problem in NP can be transformed into a logical formula of satisfiability. (cook 3)
This means that the satisfiability problem is at least as hard as any problem in NP (cook 4)
By definition(see slide 49), a problem is in NPC if it is in NP and is at least as hard as other problems in NP
so (Circuit-) SAT is in NPC

60. Cook's Theorem as Pictures
TM's tape
. . .
. . . A x y
solution
data y
problem x
an algorithm A for
checking the solution
data y against problem x in
polynomial time
Turing Machine TM
for executing A on x
and y in polynomial
time
polynomial
time
translation
YES/NO
TM ≈ simulation of
a simple computer

61. . . . . . . binary inputs (...101011000...) representing A, x and y
TM's tape
. . .
. . . A x y
Turing Machine TM
for executing A on x
and solution data y
in polynomial time
polynomial
time
translation
Circuit-SAT C
TM ≈ simulation of
a simple computer
C ≈ a simple real computer
output of 1 means A
has checked the y against x

62. Informal Explanation
The Turing Machine runs the program A (run-time software) against the input x (program) and the solution data y (program's input).
This Turing machine can be implemented as a Circuit-SAT 'computer' made up of AND, OR, and NOT gates.
Since Circuit-SAT can implement any problem (i.e. the A, x, and y) then it is as least as hard (complex) as any other problem, and so is NP-Complete.

63. The Turing Machine in Action
M is the Turing
Machine, or its
equivalent
logical formula
in Circuit-SAT

64. Everything is polynomial O(nk), where n is the size of the input:
The checking software (A) executes through a series of states (configurations) until it outputs the satisfiability of y (solution data) against the problem x
this is reported as a 1 (YES) or 0 (NO) written to the working storage area
Everything is polynomial O(nk), where n is the size of the input:
the tape length
A's running time
the number of configurations

65. 8. NPC and Reducability
A formal defintion of NPC requires the reducability of one problem into another.
If we can reduce (tranform) a problem S into a problem L in polynomial time, then S is not polynomial factor harder than L
this is written as : S ≤p L
Harder means "bigger running time"

66. A problem L to be proved NPC
Is a Problem 'L' in NPC?
First show that the problem L is in NP
show that it can be solved in exponential time
Show that it can be checked in polynomial time
Show how to reduce (transform) an existing NPC problem S into L (in polynomial time)
S is NPC
S ≤p L
Then L is NPC
A known NPC problem S
A problem L to be proved NPC
reduce

67. This reduction means that the existing NPC problem (S) is no harder than the new problem L
harder means "bigger running time"
This hardness 'link' between NPC problems means that all NPC problems have a similar (or smaller) hardness, and that all NP problems are less hard.
If a researcher proves that any NPC problem can be solved in polynomial time, then all problems in NPC and NP must also have polynomial running time algorithms.
in other words P = NP

68. Proving Problems are NPC
use
reductions

69. 8.1. The Satisfiability Problem (SAT)
Given a Boolean expression on n variables, can we assign values such that the expression is TRUE?
Example:
(x1 x2)  ((x1  x3)  x4)) x2
A satisfying truth assignment:
<x1, x2, x3, x4 > = <0, 0, 1, 1>

70. Reduction Graphically
with additional 'gate' variables
φ = x10 ∧ (x4 ↔ ¬ x3) ∧ (x5 ↔(x1∨ x2)) ∧ (x6 ↔ ¬ x4) ∧ (x7 ↔(x1∧ x2 ∧ x4)) ∧ (x8 ↔(x5 ∨ x6)) ∧(x9 ↔(x6 ∨ x7)) ∧ (x10 ↔ (x7 ∧ x8 ∧ x9))

71. Is SAT a NPC Problem? SAT is an NP problem.
Prove that Circuit-SAT ≤P SAT
remember the left-hand-side must be an existing NPC problem
For each wire xi in circuit C, the formula φ has a variable xi.
The operation of a gate is expressed as a formula with associated variables,
e.g., x10 ↔ (x7 ∧ x8 ∧ x9).

72. Circuit C is satisfiable ⇔ formula φ is satisfiable.
φ = AND of the circuit-output variable with the conjunction (∧) of clauses describing the operation of each gate,
e.g.,
φ = x10 ∧ (x4 ↔ ¬ x3) ∧ (x5 ↔(x1∨ x2)) ∧ (x6 ↔ ¬ x4) ∧ (x7 ↔(x1∧ x2 ∧ x4)) ∧ (x8 ↔(x5 ∨ x6)) ∧(x9 ↔(x6 ∨ x7)) ∧ (x10 ↔ (x7 ∧ x8 ∧ x9))
Circuit C is satisfiable ⇔ formula φ is satisfiable.
Given a circuit C, it takes polynomial time to construct φ .

73. 8.2. The 3-SAT Problem
3SAT: the Satisfiability of boolean formulas in 3-conjunctive normal form (3-CNF).
3SAT is NP
Show that 3SAT is NPC by proving SAT ≤P 3SAT.
Four stages to the transformation.
1. Construct a binary “parse” tree for input formula φ and introduce a variable yi for the output of each internal node.

74. Conjunctive Normal Form (CNF)
A Boolean formula is in conjunctive normal form if it is an AND of clauses, each of which is an OR of literals
e.g. (x1  x2)  (x1  x3  x4)  (x5)
3-CNF: each clause has exactly 3 distinct literals
e.g. (x1  x2  x3)  (x1  x3  x4)  (x5  x3  x4)
Note: the formula is true if at least one literal in each clause is true

75. 3-SAT Parse Tree φ = ((x1→ x2) ∨ ¬ ((¬ x1 ↔ x3) ∨ x4)) ∧ ¬ x2
y variables are new

76. Stages 2 and 3
2. Rewrite φ as the AND of the root variable and a conjunction of clauses describing the operation of each node:
φ' = y1 ∧ (y1 ↔(y2 ∧ ¬ x2)) ∧ (y2 ↔(y3 ∨ y4)) ∧ (y3 ↔(x1→x2)) ∧ (y4 ↔ ¬ y5) ∧ (y5 ↔ (y6 ∨ x4)) ∧ (y6 ↔(¬ x1 ↔ x3))
parent node  children

77. 3. Convert each ith sub-clause in φ' into conjunctive normal form.
Several sub-steps are required for each φ'i:
create a truth table
use the truth table to create a disjunctive normal form (DNF), called ¬φ'i
use DeMorgan's laws to change the DNF to conjunctive normal form (CNF), calling the result φ''i

78. Sub-Steps Example φ'1 = (y1 ↔(y2 ∧ ¬ x2))
The truth table for this clause:
used by
DNF

79. Create the disjunctive normal form (DNF) ¬φ'1 using the 0 result rows of the truth table:
¬φ'1 = (y1 ∧ y2 ∧ x2) ∨ (y1 ∧ ¬ y2 ∧ x2) ∨ (y1 ∧ ¬ y2 ∧ ¬x2) ∨ (¬ y1 ∧ y2 ∧ ¬ x2)
Now use DeMorgan's laws to make ¬φ'1 positive (now called φ''1), and change the DNF right-hand side to CNF:
φ''1 = ¬ (¬ φ'1) = (¬ y1 ∨ ¬ y2 ∨ ¬ x2) ∧ (¬ y1 ∨ y2 ∨ ¬ x2) ∧ (¬ y1 ∨ y2 ∨ x2) ∧ (y1 ∨ ¬ y2 ∨ x2)

80. DNF and DeMorgan's
DNF: A Boolean formula is in disjunctive normal form if it is an ORof clauses, each of which is an AND of literals
e.g. (x1  x2)  (x1 x3  x4)  (x5)
DeMorgan's Laws:

81. Stage 4
4. Pad out each clause Ci so it has 3 distinct literals. The result is φ'''.
Ci has 3 distinct literals: no padding required
Ci has 2 distinct literals; add 1 dummy: Ci = (l1 ∨ l2) = (l1 ∨ l2 ∨ p) ∧ (l1 ∨ l2 ∨ ¬p)
Ci has 1 literal; add 2 dummies: Ci = l = (l ∨ p ∨ q) ∧ (l ∨ ¬ p ∨ q) ∧ (l ∨ p ∨ ¬ q) ∧ (l ∨ ¬ p ∨ ¬ q)

82. Claim: The 3-CNF formula φ''' is satisfiable ⇔ φ is satisfiable.
All transformations can be done in polynomial time
O(m*n) (m = no. of clauses , n = no. of literals)

83. From SAT to 3-SAT reduction
The Y literals are the added dummies to pad out the clauses.

84. Why Bother with 3-SAT?
The 3-SAT problem is relatively easy to reduce to others problems
Thus by proving 3-SAT is NPC we can prove that many other problems are NPC
see the reduction tree on slide 62

85. 8.3. The Clique Problem
A clique in G= (V, E) is a complete subgraph of G.
For a positive integer k ≤ |V|, is there a clique V’ ⊆ V of size ≥ k?
Clique is NP.
Show that Clique is NPC by using 3SAT ≤P Clique

86. Example Transformation
A 3-CNF such as:
will be transformed into the graph shown on the next slide.
Each 3-element sub-clause becomes a triple of nodes.
Nodes are linked by edges:
if they are not in the same triple
if they are not opposites (inconsistent)

87. Graph, G
G can be computed from φ in polynomial time.

88. Are φ and G the Same? 1. φ is satisfiable ⇒ G has a clique of size k
3SAT φ is satisfiable is there's at least 1 literal in each clause that is true in:
For example: x2 = 0, x3 = 1 and x1 can be 0 or 1
This corresponds to picking x2 and x3 in each graph triple, which makes a 3-clique.

89. 2. G has a clique of size k ⇒ φ is satisfiable
Selecting a k-clique means that one node is selected in each graph triple.
This is the same as making one literal in each clause of φ equal to true.

90. 9. NP-Hard
A NP-Hard problem is at least as hard as the hardest problems in NP (i.e. NPC problems).
A NP-Hard problem does not have a polynomial time checking algorithm
this makes most NP-hard problems harder to solve than NP problems

91. For example, the optimization version of Traveling Salesman where we need to find an actual schedule is harder than the decision version of traveling salesman where we just need to determine whether a schedule with length <= k exists or not.
The same is true for the optimization version of the Knapsack problem vs its decision problem version.

92. Computational Difficulty
NP-hard =?
computational
difficulty
NP-complete P
Tetris, etc.
chess
halting
problem NP

93. Even if P = NP, NP-Hard can be harder
Drawing P, NP, etc. as Sets
Even if P = NP, NP-Hard can be harder

94. NP-Hard: a Bad Name
NP-hard problems are not all NP, despite having NP in the name.
There are problems that are NP-hard but not NP-complete
e.g. chess, the halting problem

95. Coping with NPC/NP-Hard Problems
Approximation algorithms:
guarantee to be a fixed percentage away from the optimum
Pseudo-polynomial time algorithms:
e.g., dynamic programming for the 0-1 Knapsack problem
Probabilistic algorithms:
assume some probabilistic distribution of the data
Randomized algorithms:
use randomness to get a faster running time, and allow the algorithm to fail with some small probability
e.g. Monte Carlo method, Genetic algorithms

96. Exponential algorithms / Branch and Bound / Exhaustive search:
Restriction: work on special cases of the original problem which run better
Exponential algorithms / Branch and Bound / Exhaustive search:
feasible only when the problem size is small.
Local search:
simulated annealing (hill climbing)
Heuristics
use "rules of thumb" that have no formal guarantee of performance

97. Exhaustive Search v.s. Branch and Bound

98. Simulated Annealing
Use probabilities to jump around in a large search space looking for a good enough solution called a local optimum.
By contrast, the actual best solution is the global optimum.

99. 10. Approximation Algorithms
An approximation algorithm is one that returns a near-optimal solution.
The quality of the approximation can be measured by comparing the 'cost' of the optimal solution against the approximate one using ratios
called the approximation ratio, ρ(n)
n is the input size
Cost can be any aspect of the algorithm
e.g. number of nodes, edge distance, running time, space

100. Calculating ρ(n) C = cost of the approximation algorithm
C* = cost of an optimal solution
In some applications, we wish to maximize the 'cost', in others we want to minimize it.
In maximization problems, we use C*/C to get a measure of how much bigger C* is than C
In minimization problems, we use C/C* to get a measure of how much smaller C* is than C.

101. Approximation (Cost) Ratio ρ(n):
ρ(n) is always ≥ 1
Bigger values for ρ(n), mean worse approaximations
An approximate algorithm that has a ratio == ρ(n) is called a ρ(n)-approximation algorithm.

102. 10.1. Approx. Algorithm for TSP
Approx-TSP-Tour(G)
select a vertex r ∈ V[G] to be a “root” vertex
grow a minimum spanning tree (MST) for G from root r using Prim()
create a list L of vertices visited in a preorder tree walk over the MST
return Hamiltonian cycle using L for its order
Time complexity: O(V log V).

103. Approx-TSP Example an integer unit grid Undirected
Graph
(b) MST created with Prim()

104. Preorder walk: a, b, c, b, h, b, a, d, e, f, e, g, e, d, a
(c) Preorder walk
over the MST
A preorder walk that lists a vertex only when it is first encountered, as indicated by the dot next to each vertex:
a, b, c, h, d, e, f, g
(d) Tour using
the preorder
MST.
Cost of tour: ~19.074
(by measuring squares)

105. What are We Comparing?
We want to compare an optimal tour with a tour using a preorder MST. For this example:
Tour using the preorder MST
Optimal Tour
Cost of tour: ~19.074
Cost of tour: ~14.715

106. Calculating the Approximation Ratio
Let H* denote an optimal tour, T the minimal spanning tree, W a full walk over the graph, and H our preorder MST approximation. Cost aim is to minimize the distance.
The cost of the minimal spanning tree is a lower bound for the cost of the optimal tour:
c(T) ≤ c(H*) (1)
An example of a full walk W is shown in (c) on the previous slide, which involves traversing every edge twice. This means:
c(W) ≤ 2*c(T) (2)

107. Combining equations (1) and (2):
c(W) ≤ 2*c(H*) (3)
The preorder MST, H, is much shorter than the full walk (see (d) on slide 98), so:
c(H) ≤ c(W) (4)
Combining equations (3) and (4):
c(H) ≤ 2*c(H*)

108. This means that the cost of H (the preorder MST) is a factor of 2 greater than the cost of H* (the optimal tour), and so ρ(n) ≥ 2
the minimal distance ratio is C/C* = c(H)/c(H*) = 2
We can say that the preorder MST version of TSP is a 2-approximation algorithm
which indicates that it is a good approximation

109. 10.2. Greedy Algorithms
Greedy algorithms progress by making locally optimal decisions
Greedy algorithms (such as Prim's) are not guaranteed to find the best (globally optimal) answer, but they often return good approximations.

110. Greedy Knapsack Algorithm
def knapsack_greedy(items, maxweight):
result = []
weight = 0
while True
weightleft = maxweight - weight
bestitem = None
for item in items //grab the biggest that fits
if item.weight <= weightleft
and (bestitem == None
or item.value > bestitem.value):
bestitem = item
if bestitem == None: break
else
result.append (bestitem)
weight += bestitem.weight
return result
Running Time
is O(n2)

111. Is Greedy Knapsack Optimal?
No (but close). Proof by counterexample:
Consider the input
items = {<“gold”, 100, 1 >, <“platinum”, 110, 3>
<“silver”, 80, 2 >}
maxweight = 3
Greedy algorithm picks {<“platinum”>}
value = 110, but {<“gold”>, “silver”>} has weight <= 3 and value = 180

112. 11. Decidable Problems
Decidable problems are often split into three categories:
Tractable problems (polynomial)
NPC problems
Intractable problems (exponential)
sometimes grouped in the set EXP
All of the above have algorithmic solutions, even if they have impractical running times.
All of these are placed in a group called R
R = {set of problems solvable in finite time}

113. 12. Undecidable Problems
An undecidable problem is a decision problem for which it is impossible to construct a single algorithm that always leads to a correct yes-or-no answer, or might never give any answer at all.
In summary:
the answer may be wrong, or
no answer may appear, no matter how long you wait
No correct algorithm can be written for an undecidable problem.

114. Computational Difficulty
NP-hard
EXP-hard =? =?
computational
difficulty
NP-complete
EXP-complete P
tetris, etc
chess
halting
problem NP
EXP R
undecidable
decidable

115. 12.1. The Halting Problem
Decide whether a given program with some input finishes, or continues to run forever.
The halting problem is undecidable.
this means that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist

116. Isn't Detecting Halting Easy?
No. It's very difficult for a human to decide if a real-size program will actually finish.
Here's a very simple bit of code. Does it finish for all inputs?
read n;
while(n != 1) {
if (even(n))
n = n/2;
else
n = 3*n + 1;
print n; }

117. These sequences are referred to as hailstone sequences.
The maths behind this function goes by various names, including the Collatz conjecture.
The conjecture is that no matter what positive number you start with, you will always eventually reach 1.
Examples
start with n = 6, the sequence is 6, 3, 10, 5, 16, 8, 4, 2, 1.
n = 11, sequence is 11, 34, 17, 52, 26, 13, 40, 20, 10, 5, 16, 8, 4, 2, 1
These sequences are referred to as hailstone sequences.

118. The sequence for n = 27, loops 111 times, climbing to over 9000 before descending to 1

119. The conjecture is a famous unsolved problem in Mathematics.
Paul Erdős said "Mathematics is not yet ripe for such problems" but offered $500 for its solution.
The Wikipedia page is very informative

120. Why the Halting Problem is Undecidable
Let's assume there is a halt(String p, String i) function which implements the halting problem.
It takes two inputs, an encoded representation of a program as text and the program's input data (also as text).
halt() return true or false depending on whether or not p will halt when executed on the i input.
p halts on i
true p
halt() or i
false
p does not
halt on i

121. Examples returns true returns false
halts(“def sqr(x): return x * x", "sqr(4)")
returns true
halts(“def fac(n):
if (n == 0) return 1
else return n * fac(n-1)", "fac(10)”)
halts(“def foo(n): while (true) { n = n + 1}", "foo(4)")
returns false

122. Now for Trouble Let's create a 2nd function called trouble(String q).
It creates two copies of its input q, and passes them as arguments to halt().
trouble(q) calls halt(q, q)
This means that q is used as both the program and its data in halt()

123. Implementing trouble()
If halt(q,q) outputs false, then trouble() returns true and finishes.
If halt(q,q) outputs true, then trouble() goes into an unending loop (and thus does not finish).
halt(q,q) is
false
true; and finishes q
trouble() or
never
finishes
halt(q,q) is
true

124. boolean trouble(String q) { if (
boolean trouble(String q) { if (!halt(q, q)) return true; else { // halt(q, q) is true while (true) { // loop forever // do nothing } return false; // execution never gets here

125. Double Trouble
The input string q to trouble() can be anything, so let's encode the trouble() function itself as a string called t, and use that as input.
What is the result of trouble(t)?
There are two possibilities: either trouble() returns true (i.e. halts), or loops forever (it does not halt). Let's look at both cases.

126. Two Cases
1. trouble(t) halts which means that halt(t,t) returned false.
But halt() is saying that trouble(t) does not halt. Contradiction.
2. trouble(t) does not halt which means that halt(t,t) returned true.
But halt() is saying that trouble(t) does halt. Contradiction

127. Contradictions are Deadly
In both cases, there's a logical contradiction because of halt().
This means that it is possible to construct an program/input pair for halt() which it processes incorrectly.
This means that there is no halt() algorithm which consistently works for all inputs.
the Halting Problem is undecidable

128. 12.2. Wang Tiling
In 1961, Wang conjectured that if a finite set of tiles can tile the plane, then there exists a periodic tiling
i.e. a sort of wallpaper pattern is possible
But in 1966, Robert Berger proved that no such algorithm exists
he used the undecidability of the halting problem to imply the undecidability of this tiling problem
'Pratically' this means that a finite set of Wang tiles that tiles the plane can only do so aperiodically
i.e. the pattern never repeats

129. Example
13 Wang tiles that tile aperiodically:
Aperiodic example:

130. Uses
Wang tiles have become a popular tool for procedural synthesis of textures, heightfields, and other large and nonrepeating data sets.
e.g. for decorating 3D gaming scenes
A small set of precomputed or hand-made source tiles can be assembled very cheaply without too obvious repetitions and without periodicity.

131. 12.3. Other Undecidable Problems
A good list can be found at:
The problem categories are mathematical, as you'd expect:
logic, abstract machines, matrices, combinatorial group theory, topology, analysis, others

132. 13. Computational Difficulty

133. What!
Adding memory space and randomization to running time creates new complexity classes
PSPACE is the set of all decision problems that can be solved using a polynomial amount of space
EXPSPACE uses an exponential amount of space
BPP ( bounded-error probabilistic polynomial time)uses a probabilistic algorithm in polynomial time, with an error probability of at most 1/3 for all instances
NPSPACE is equivalent to PSPACE, because a deterministic Turing machine can simulate a nondeterministic Turing machine without needing much more space.

134. co-NP is the class of problems for which polynomial checking times are possible for counterexamples of the original NP problem
A counterexample is where the decision problem is negated, and you find a YES/NO answer for that question.

135. Complexity Classes P, NP, NPC, NP-Hard, EXP, R, EXP-Hard, ...
PSPACE, NPSPACE, EXPSPACE, ...
quite a few complexity classes (sets)
is that all of them?
No, the "Complexity Zoo" website currently lists 495 classes
But the "petting zoo" only lists 16.

136. 14. Non-Technical P and NP
Video talk "Beyond Computation: The P vs NP Problem" Michael Sipser, Dept. of Maths, MIT Oct. 2006

137. The Complexity of Songs Donald Knuth (1977) CACM, 1984, 27 (4) pp
joke article about complexity
analyzes the very low complexity of pop songs
knuth_song_complexity.pdf