1. Lectures on NP-hard problems and Approximation algorithms
COMP 523: Advanced Algorithmic Techniques
Lecturer: Dariusz Kowalski
NP-hard problems and Approximation algorithms

2. NP-hard problems and Approximation algorithms
Overview
Previous lectures:
Greedy algorithms
Dynamic programming
Network flows
These lectures:
NP-hard problems
Approximation algorithms
NP-hard problems and Approximation algorithms

3. NP-hard problems and Approximation algorithms
P versus NP
Decision problem: problem for which the answer must be either YES or NO
Polynomial time algorithm: there is a constant c such that the algorithm solves the problem in time O(nc) for every input of size n
P (polynomial time) - class of decision problems for which there is a polynomial time deterministic algorithm solving the problem
NP (nondeterministic polynomial time) - class of decision problems for which there is a certifier which can check a witness in polynomial time
NP-hard problems and Approximation algorithms

4. Certifying in polynomial time
Representation of decision problem: set of inputs which are correct (and should be answered YES, while others should be answered NO)
An efficient certifier B for problem X:
B is in P
s is in X iff B(s,t) = YES for some t of size polynomial in s
Example:
Problem: is there a clique of size k in a given graph with n nodes?
Certifier: if a given graph has a clique of size k then given
this clique as the second parameter we can answer YES
NP-hard problems and Approximation algorithms

5. Computing an efficient certifier
How to compute witnesses for an efficient
certifier for a given problem?
Fact:
If the witnesses for the efficient certifier can be
found in polynomial time then the problem is in P.
Conclusion:
P is included in NP
Open question:
P = NP ?
NP-hard problems and Approximation algorithms

6. Polynomial reductions
Example: decision problem
if there exists a perfect matching in a bipartite graph
can be reduced to network flow problem in polynomial time (by adding source, target and directing the edges)
Other problems for undirected graphs (in NP and not known to be in P):
Independent Set of nodes
Vertex cover
Set Cover
NP-hard problems and Approximation algorithms

7. Polynomial reductions
Definition:
Problem X is polynomial-time reducible to problem Y, or
problem Y is at least as hard as problem X, iff
problem X can be solved by an algorithm which works in
polynomial time and uses polynomial number of calls to the
black box solving problem Y.
Notation: X P Y
Transitivity Property: if X P Y and Y P Z then X P Z
NP-hard problems and Approximation algorithms

8. Independent Set to Vertex Cover
Independent Set: given a graph G of n nodes and parameter k,
is there a set of k nodes such that none two of them are connected by
an edge?
Vertex Cover: given a graph G of n nodes and parameter k,
is there a set of k nodes such that every edge has at least one
end selected?
Polynomial Reduction:
Solve Vertex-Cover(n-k) for the same graph
Proof:
Set S of size n - k is a vertex cover set in G iff
There is no edge between remaining k nodes iff
Set of k remaining nodes is independent in G
NP-hard problems and Approximation algorithms

9. Vertex Cover to Set Cover
Vertex Cover: given a graph G of n nodes and parameter k, is there a set of k
nodes such that every edge has at least one end among selected nodes?
Set Cover: given n nodes, m sets which cover the set of nodes, and parameter k,
is there a family of k sets that covers all n nodes?
Polynomial Reduction:
Let each edge from the graph correspond to the node for SC system
For each vertex in the graph create a set of incident edges to SC system
Solve Set-Cover(m,n,k) for the created SC system with m nodes and n sets
Proof:
Each node-edge is covered by at least one set-vertex since each node is covered.
This covering is minimal.
NP-hard problems and Approximation algorithms

10. NP-completeness and NP-hardness
NP-complete: class of problems X such that every
problem from NP is polynomial-time reducible to X.
Optimization problems: problems where the
answer is a number (maximum/minimum possible)
Each optimization problem has its decision version, e.g.,
Find a maximum Independent Set
Is there an Independent Set of size k?
NP-hard: class of optimization problems X such that its
decision version is NP-complete.
Example: having solution for decision version of
Independent Set problem, we can probe a parameter k,
starting from k = 1 , to find the size of the maximum independent set
NP-hard problems and Approximation algorithms

11. Approximation algorithms
Having an NP-hard problem, we do not know at this
moment any polynomial-time algorithm solving the
problem (exact solution)
How to find an almost optimal solution?
Approximation algorithm with ratio a > 1 gives a solution
A such that
OPT  A  a  OPT for a min-optimization problems
(1/a)  OPT  A  OPT for a max-optimization problems
where OPT is the optimal solution.
NP-hard problems and Approximation algorithms

12. NP-hard problems and Approximation algorithms
2-approximation for VC
Minimum Vertex Cover - NP-hard problem
(maximum is trivially n)
Algorithm:
Initialize set C to an empty set
While there are remaining edges:
Choose an edge {v,w} with the largest degree, where degree of an edge is a sum of degrees of its ends v,w in a current graph G
Put v,w to C
Remove all the edges adjacent to nodes v,w from graph G
Output: witness set C and its size
NP-hard problems and Approximation algorithms

13. Analysis of 2-approximation for VC
Correctness:
Each edge is removed only after one of its ends is chosen
to set C, so each edge is covered
Termination:
In each iteration we remove at least one edge from the
graph, and there are less than n2 edges
Approximation ratio 2:
For each edge {v,w} selected at the beginning of an iteration at least
one end must be in min-VC, and we selected two, so set C is at most
twice bigger than the min-VC
Time complexity: O(m + n)
Exercise
NP-hard problems and Approximation algorithms

14. NP-hard problems and Approximation algorithms
Approximation for SC
Minimum Set Cover - NP-hard problem
(maximum is trivially m)
Greedy Algorithm:
Initialize set C to empty set
While there are uncovered nodes:
Choose a set F which covers the largest number of uncovered nodes
Put F to C
Remove all nodes covered by F
Output: witness set C and its size
NP-hard problems and Approximation algorithms

15. Analysis of approximation for SC
Correctness:
Each node is marked as covered when we put the set covering it to set C.
The algorithm stops when all nodes are covered.
Termination:
In each iteration we cover at least one new node, and there are n nodes.
Approximation ratio log n:
Let Si be the set selected to C in ith iteration, and denote by si the number of uncovered nodes covered by Si; OPT be the minimum covering set
Let cv = 1/ si for each node v which was covered by Si for the first time
The following holds: |C| = v cv
For every set S = Si : vS cv  H(|S|) (H(i)=ji1/j denotes harmonic number)
|C| = v cv  SOPT vS cv  H(n) SOPT1 = H(n) |OPT|  |OPT| log n
Time and memory complexities:
O(M + n), where M is the sum of cardinalities of sets
Exercise
NP-hard problems and Approximation algorithms

16. NP-hard problems and Approximation algorithms
Conclusions
Decision problems P and NP-complete
Polynomial-time reduction
Optimization problems in NP-hard
Maximum Independent Set
Minimum Vertex Cover
Minimum Set Cover
Approximation algorithms - polynomial time
Min-VC with ratio 2
Min-SC with ratio log n
NP-hard problems and Approximation algorithms

17. Textbook and Questions
READING:
Chapters 8 and 11, Sections 8.1, 8.2, 8.3, 11.3, 11.4
EXERCISES:
What is the time and memory complexities of min-VC approximation algorithm with ratio 2 and min-SC algorithm?
Consider a modification of min-VC algorithm: choose a node which covers the largest number of uncovered edges. Is it a 2-approximation algorithm?
Having a 2-approximation algorithm for min-VC, is it easy to modify it to be a 2-approximation algorithm for max-IS (since there is a simple polynomial-time reduction between these two problems)?
NP-hard problems and Approximation algorithms

18. NP-hard problems and Approximation algorithms
Overview
Previous lectures:
NP-hard problems and approximation algorithms
Graph problems (IS, VC)
Set problem (SC)
This lecture:
NP-hard numerical problems and their approximation
Numerical Knapsack problem
Weighted Independent Set
NP-hard problems and Approximation algorithms

19. NP-hard problems and Approximation algorithms
Knapsack problem
Input: set of n items, each represented by its weight wi and value vi ; thresholds W and V
Decision problem: is there a set of items of total weight at most W and total value V ?
Optimization problem: find a set of items with
total weight at most W , and
maximum possible value
Assumptions:
weights and values are positive integers
each weight is at most W
NP-hard problems and Approximation algorithms

20. NP-hardness of knapsack
Knapsack is NP-hard problem, but
there exists pseudo-polynomial algorithm (complexity is polynomial in terms of values)
Typical numerical polynomial algorithm: polynomial in logarithm from the maximum values (longest representation)
Existence of pseudo-polynomial solution often yields very good approximation schemes
NP-hard problems and Approximation algorithms

21. Dynamic pseudo-polynomial optimization algorithm
Let v* be the maximum (integer) value of an item.
Consider any order of objects.
Let OPT(i,v) denote the minimum possible total weight of a subset of items 1,2,…,i which has total value v
Dynamic formula for i = 0,1,…,n-1 and v = 0,1,…,nv* :
OPT(i+1,v) =
= min{ OPT(i,v) , wi+1 + OPT(i,max{0,v-vi+1})}
Formula OPT does not provide direct solution for our problem, but can be easily adapted: maximum value of knapsack is the maximum value v such that OPT(n,v)  W
NP-hard problems and Approximation algorithms

22. NP-hard problems and Approximation algorithms
Dynamic algorithm
Initialize array M[0…n,0…nv*] for storing OPT(i,v)
Fill positions M[,0] and M[0,] with zeros
For i = 0,1,…,n-1
For v = 0,1,…,nv*
M[i+1,v] :=
= min{ M[i,v] , wi+1 + M[i,max{0,v-vi+1}] }
Go through the whole array M and find the maximum value v such that M[n,v]  W
NP-hard problems and Approximation algorithms

23. NP-hard problems and Approximation algorithms
Complexities
Time: O(n2v*)
Initializing array M : O(n2v*)
Iterating loop: O(n2v*)
Searching for maximum v : O(n2v*)
Memory: O(n2v*)
NP-hard problems and Approximation algorithms

24. Polynomial approximation algorithm
Fix b = (/(2n)) v*
Set (by rounding up) xi = [vi/b]
Solve knapsack problem for values xi and weights wi using dynamic program
Return set of computed items and its total value in terms of the sum of vi’s
NP-hard problems and Approximation algorithms

25. NP-hard problems and Approximation algorithms
Analysis
PTAS: polynomial time approximation scheme - for any fixed positive  it produces (1+)-approximation in polynomial time (but  is hidden in big Oh)
Time: O(n2x*) = O(n3/)
Approximation: (1+)
NP-hard problems and Approximation algorithms

26. Analysis of approximation ratio
Recall notation:
b = (/(2n)) v*
xi = [vi/b]
Approximation: (1+)
Let S denote the set of items returned by the algorithm
vi  bxi  vi + b  iS bxi - b|S|  iS vi
iS bxi  v* = 2nb/  (2/ -1)nb  iS vi
iOPT vi  iOPT bxi  iS bxi  b|S|+iS (bxi - b) 
b(2/ -1)n + iS vi   iS vi + iS vi = (1+) iS vi
NP-hard problems and Approximation algorithms

27. Weighted Independent Set
Optimization problem:
Weighted Independent Set: given graph G of n valued nodes, find an independent set of maximum value (set of nodes such that none two of them are connected by an edge)
Even for values 1 problem remains NP-hard, which is not the case for knapsack problem! WIS problem is an example of strong NP-hard problem, and no PTAS is known for it
NP-hard problems and Approximation algorithms

28. NP-hard problems and Approximation algorithms
Conclusions
Optimization numerical problem in NP-hard
Maximum Knapsack
Weighted Independent Set
PTAS in time O(n3) for Knapsack, based on dynamic programming
NP-hard problems and Approximation algorithms

29. Textbook and Questions
Chapters 6 and 11, Sections 6.4, 11.8
Is it possible to design an efficient Knapsack algorithm based on dynamic programming for the case where weights are small (values can be arbitrarily large)
How to implement arithmetical operations: + - * / and rounding, each in time proportional to at most square of the length of the longest number? What are the complexity formulas?
NP-hard problems and Approximation algorithms

30. NP-hard problems and Approximation algorithms
Overview
Previous lectures:
NP-hard problems
Approximation algorithms
Greedy (VC and SC)
Dynamic Programming (Knapsack)
This lecture:
Approximation through integer programming
NP-hard problems and Approximation algorithms

31. NP-hard problems and Approximation algorithms
Vertex Cover
Weighted Vertex Cover: (weights are in nodes)
Decision problem:
given weighted graph G of n nodes and parameter k,
is there a set of nodes with total weight k such that every edge has at least one end in this set?
Optimization problem:
given weighted graph G of n nodes, what is the minimum total weight of a set such that every edge has at least one end in this set?
NP-hard problems and Approximation algorithms

32. Approximation algorithms
Having an NP-hard problem, we do not know in this
moment any polynomial-time algorithm solving the
problem (exact solution)
How to find almost optimal solution?
Approximation algorithm with ratio a > 1 gives a solution
A such that
OPT  A  a  OPT for a min-optimization problems
OPT/a  A  OPT for a max-optimization problems
where OPT is an optimal solution.
NP-hard problems and Approximation algorithms

33. NP-hard problems and Approximation algorithms
2-approximation for VC
Minimum Vertex Cover - NP-hard problem even for all weights = 1
(maximum is trivially n)
Algorithm: (for all weights equal to 1)
Initialize set C to empty set
While there are remaining edges:
Choose an edge {v,w} (with the largest degree, where degree of an edge is a sum of degrees of its ends v,w in a current graph G )
Put v,w to C
Remove all the edges adjacent to nodes v,w from graph G
Output: witness set C and its size
NP-hard problems and Approximation algorithms

34. NP-hard problems and Approximation algorithms
Integer Programming
Represent the problem as Integer Programming
Relax the problem to Linear Programming
Solve Linear Programming
Round the solution to get integers
NP-hard problems and Approximation algorithms

35. Integer and linear programs
Set of constraints (linear equations):
x1 , x2  0
x1 + 2x2  6
2x1 + x2  6
Function to minimize (linear):
4x1 + 3x2
Linear programming:
variables are real numbers
there are polynomial time algorithms solving it (e.g., interior point method - by N. Karmarkar in 1984); simplex method is not polynomial
Integer programming:
variables are integers
problem is NP-hard
NP-hard problems and Approximation algorithms

36. NP-hard problems and Approximation algorithms
VC as Integer Program
Set of constraints :
xi  {0,1} for every node i
xi + xj  1 for every pair {i,j}  E
Function to minimize:
i xiwi
Example: x1 , x2 , x3 , x4  {0,1}
x1 + x3  1 , x1 + x4  1 , x2 + x4  1 , x2 + x3  1
Minimize: x1 + x2 + x3 + x4 x1 x4 x2 x3
NP-hard problems and Approximation algorithms

37. Relaxation to Linear Program
Set of constraints :
yi  [0,1] for every node i
yi + yj  1 for every pair {i,j}  E
Function to minimize:
i yiwi
Example: y1 , y2 , y3 , y4  [0,1]
y1 + y3  1 , y1 + y4  1 , y2 + y4  1 , y2 + y3  1
Minimize: y1 + y2 + y3 + y4 y1 y4 y2 y3
NP-hard problems and Approximation algorithms

38. Rounding the linear program solution
Obtained exact Linear Program solution yi  [0,1] for every node i
satisfying
yi + yj  1 for every pair {i,j}  E
How to obtain a (approximate?) solution for Integer Program?
Rounding: for every node i
xi = 1 iff yi  1/2 (otherwise xi = 0)
Example: y1 , y2 , y3 , y4 = 1/2
x1 , x2 , x3 , x4 = 1
Optimum solution (minimum) e.g.:
x1 , x2 = 1, x3 , x4 = 0 x1 x4 x2 x3
NP-hard problems and Approximation algorithms

39. NP-hard problems and Approximation algorithms
Analysis
Correctness: since each xi  {0,1} and each edge is guarded by constraint xi + xj  1 which is satisfied also after rounding
Time: time for solving linear program plus O(m+n)
Approximation:
Each xi is at most twice as large as yi hence the weighted sum of xi is also at most twice bigger than the weighted sum of yi
Example: y1 , y2 , y3 , y4 = 1/2
x1 , x2 , x3 , x4 = 1
Optimum solution (minimum) e.g.:
x1 , x2 = 1, x3 , x4 = 0 x1 x4 x2 x3
NP-hard problems and Approximation algorithms

40. NP-hard problems and Approximation algorithms
Conclusions
Decision problems P and NP-complete
Polynomial-time reduction
Optimization problems in NP-hard
Maximum Independent Set
Minimum Vertex Cover
Minimum Set Cover
Maximum Knapsack
Approximation algorithms - polynomial time
Greedy (VC, SC)
Dynamic program (Knapsack)
Integer and Linear programs (weighted VC)
NP-hard problems and Approximation algorithms

41. Textbook and Questions
READING:
Chapter 11, Section 11.6
EXERCISES:
Could we solve Weighted VC by modification of greedy algorithm solving (pure) VC?
What approximation we get if we apply randomized rounding, i.e.,
xi = 1 with probability yj (otherwise xi = 0)
Traveling Salesman Problem : Section 8.5
TSP can not be approximated with a constant unless P=NP
Constant approximation of TSP problem under the assumption that the weights satisfy metric conditions (symmetric weights satisfying triangle inequality)
NP-hard problems and Approximation algorithms