Exact Exponential-Time Algorithms

Exact Exponential-Time Algorithms
Algorithms and Networks 2017/2018 Johan M. M. van Rooij Hans L. Bodlaender TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAA

Polynomial solution algorithms
Solution quality Computation time Optimal Bound on quality Good solution, no quality guarantee Polynomial Polynomial solution algorithms Approximation algorithms Construction heuristics Super polynomial and/or no guarantee Exact algorithms: Tree search Dynamic programming Integer linear programming …… Hybrid algorithms Column generation without complete branch-and-price Meta heuristics: Local search Genetic algorithms I borrowed this slide from Marjan van den Akker.

Polynomial solution algorithms
Solution quality Computation time Optimal Bound on quality Good solution, no quality guarantee Polynomial Polynomial solution algorithms Approximation algorithms Construction heuristics Super polynomial and/or no guarantee Exact algorithms: Tree search Dynamic programming Integer linear programming …… Hybrid algorithms Column generation without complete branch-and-price Meta heuristics: Local search Genetic algorithms Algorithms and networks Algorithms and networks Explore the fundamentals of what can algorithmically be done on hard problems. Focus on what can be done while still obtaining optimal solution quality.

What to do if a problem is NP-complete?
Solve on special cases. Heuristics and approximations. Algorithms that are fast on average. Good exponential-time algorithms. … Exact exponential-time algorithms is one of many options. Practical applications exist. If the instance is small, and a simple exponential-time algorithm exist, then why not use it? Sometimes useful as a quick way to solve small subproblems. If one needs the guarantee that an optimal solution is always to be found one has little other choice.

Exact exponential-time algorithms in this course
Four or Five Lectures Introduction to exponential-time algorithms. Overview of some techniques. Inclusion/Exclusion. Measure-and-conquer analysis. Complexity theory of parameterised and exact algorithms (end of the course; exact algorithms part if time allows). Many ideas from exact exponential-time algorithms are also applicable to or are related to parameterised algorithms.

Today: lecture one. Introduction to exponential-time algorithms.
Techniques in this lecture: Clever enumeration. Dynamic programming. Problems: Maximum/Maximal Independent Set. Graph Colouring / k-Colouring. Travelling Salesman Problem.

Exponential-time algorithms – Algorithms and Networks
INTRODUCTION

Good exponential-time algorithms
Algorithms with a running time of O( cn p(n) ) c a constant p() a polynomial Notation: O*(cn) Smaller c helps a lot! O*(f(n)): hides polynomial factors, i.e., O*(f(n)) = O(p(n)*f(n)) for some polynomial p.

Good exponential-time algorithms improve upon the trivial algorithm
What would be a trivial algorithm for the following problem? Maximum Independent Set: Instance: graph G = (V,E), integer k. Question: does G have an independent set of size ¸ k? An Independent set is a subset of the vertices s.t. no two vertices are adjacent. What would we consider a trivial algorithm in general? For problems in NP (e.g. NP-Complete problems), the answer involves the definition of P and NP: ... I give the answer after a short recap on P and NP.

P and NP A decision problem belongs to the class P if there is a algorithm solving the problem with a running time that is polynomial in the input size. A decision problem belongs to the class NP if: Any solution y leading to ‘yes' can be encoded in polynomial space with respect to the size of the input x. Checking whether a given solution leads to `yes' can be done in polynomial time with respect to the size of x and y. Slightly more formal: Problem Q belongs to the class NP, if there exists a polynomial-time 2-argument algorithm A, such that: For each instance i, i is a yes-instance to Q, if and only if, there is a polynomial-size certificate c for which A(i,c) = true. Hier bij stilstaan: check of de stof in algorithms for decision support over gekomen is.

Certificates What are natural certificates for the following problems? Maximum Independent Set: Instance: graph G = (V,E), integer k. Question: does G have an independent set of size ¸ k? Travelling Salesman Problem: Instance: n vertices (cities) with distance between every pair of vertices, integer k. Question: Is there shortest (simple) cycle that visits every city of total length · k? Dit interactief doen! En wat are certificates for Travelling Salesman Problem? 1 2 3 4 5 11

Good exponential-time algorithms improve upon the trivial algorithm
What is a trivial algorithm? For problems in NP (e.g. NP-Complete problems), the answer lies in the definition of NP: Choose a definition for the problem as part op NP, i.e., choose a way to encode certificates. Trivial algorithm: brute force enumerate all certificates and check whether there one proves that we have a ‘Yes’ instance. Exponential part of the running time of the trivial algorithm corresponds to the number of possible certificates. We consider trivial algorithms those that enumerate natural certificates. Sometimes one can use a smarter encoding than the natural encoding to make the certificates smaller. Thus: formally trivial is defined after defining the encoding. Question: what is the number of possible certificates for: Max Independent Set? k-Sat? TSP?

Good exponential-time algorithms improve upon a trivial algorithm
There is a lot of choice in defining these certificates, so there can be multiple natural ‘trivial algorithms’. Satisfiability: O*( 2n ). Independent set / vertex cover: O*( 2n ). TSP: O*( n! ) = O*( 2n log n ) or O*( 2m ). Choices often depend on what parameter of ‘certificate size’ you use (nr of nodes, nr of edges, etc.) The goal of exact exponential-time algorithms is to improve upon this brute force approach as much as possible. In this lecture, we will do so using: Clever enumeration. Dynamic programming

Time and space Exact-exponential time algorithms are often compared on two properties: Running time. Space usage. Exponential time is a resources we often have. As long as the input is small and the algorithm is fast enough. Just keep you computer running all night (or worse). Exponential space can be a much bigger problem. One can easily use all your computers memory. Algorithms that require a lot of disk swapping often are too slow for practical usage. In mijn opinie zou parallellisatie hier even belangrijk moeten zijn.

CLEVER ENUMERATION

Enumeration for maximum independent set
Instance: graph G = (V,E), integer k. Question: does G have an independent set of size ¸ k? Algorithm: If |V| = 0, return 0. Choose a vertex v 2 V of minimum degree. For v and each neighbour of v: u 2 N[v] Solve the subproblem on G\N[u] (G without u and its neighbours). Return 1 + the maximum size solution found.

Analysis of enumeration for maximum independent set
Algorithm: If |V| = 0, return 0. Choose a vertex v 2 V of minimum degree. For v and each neighbour of v: u 2 N[v] Solve the subproblem on G\N[u] (G without u and its neighbours). Return 1 + the maximum size solution found. We generate d(v) + 1 subproblems, each with at least d(v) + 1 vertices removed. T(n) · s £ T( n – s ). Where T(n) is the number of leaves of the seach tree on a graph with n vertices, and s = d(v) + 1. One can show T(n) = O*(sn/s). sn/s is maximal w.r.t. s if s = 3. So: T(n) = O*(3n/3) = O( n ). This improves the trivial O*( 2n ) algorithm. Question to students: why did I omit the * in O(1.4423n) , i.e., why not O*(1.4423n) -> lazy and not necessary. Why is this faster than brute force? We omit lots of possible brute-force solutions.

Maximum and maximal independent sets
Maximum independent set: Maximum size over all independent sets in G. Maximal independent set: Maximal in the sense that we cannot add a vertex to the set. We will use a slight modification of the algorithm to enumerate all maximal independent sets. Because each leaf of the search tree contains at most one maximal independent set, this also proves a bound on the number of maximal independent sets in a graph. Corollary: Any graph contains at most O(1.4423n) maximal independent sets. Algorithm of the corollary on next slide!

The enumeration algorithm
Algorithm (for enumerating maximal independent sets): If |V| = 0, return ;. Choose a vertex v 2 V of minimum degree. For v and each neighbour of v: u 2 N[v] Solve the subproblem on G\N[u] (G without u and its neighbours), add u to the generated solutions. Return all maximal independent sets found. All maximal independent sets can be enumerated in O*(3n/3) = O( n ) time. Each leaf of the search tree at most one maximal independent set: so any graph has at most O*(3n/3) = O(1.4423n) maximal independent sets. This bound is tight: consider a collection of triangles. Deze relatie tussen enumeratie en looptijden zien we vaker voor complexe problemen.

Graph colouring 1 2 3 Graph Colouring k-Colouring
Given: Graph G=(V,E), integer k. Question: Can we colour the vertices with k colours, such that for all edges {v,w} in E, the colour of v differs from the colour of w. k-Colouring Given: Graph G=(V,E). 1 2 3 Q: can we colour this graph with 2 colours? Q: can we colour this graph with 4 colours?

Graph colouring 1-Colouring is easy in O(n) time.
2-Colouring is easy in O(n+m) time. k-Colouring is NP-Complete for k ¸ 3. Applications: Scheduling Frequency assignment ... Practical problems are usually more complex variants of the standard graph colouring problem. Discuss: How to do 2 colouring? NP-completeness? Hoe relateert dit zich tot de 4-colour theorem?

3-Colouring O*(3n) is trivial or can we do this faster?
G is 3-colourable, if and only if, there is a set of vertices S with: S is an independent set. G[V-S] is 2-colourable. Algorithm: enumerate all sets, and test these properties. 2n tests of O(n+m) time each. So depending on how we define certificates, 3-colouring is ‘trivial’ to solve in O*(3n) or O*(2n) time. Discuss certificates before 2nd click.

3-Colouring Lawler, 1976: G is 3-colourable, if and only if, there is a set of vertices S with: S is a maximal independent set. G[V-S] is 2-colourable. Enumerating all maximal independent sets in O*(3n/3) = O(1.4423n) time. Thus O(1.4423n) time algorithm for 3-colouring. Schiermeyer, 1994; O(1.398n) time. Beigel, Eppstein, 1995: O(1.3446n) time. Beigel, Eppstein, 2005: O(1.3289n) time.

4-Colouring in O*(2n) time
Lawler, 1976: G is 4-colourable, if and only if, we can partition the vertices in two sets X and Y such that: G[X] and G[Y] are both 2-colorable. Enumerate all partitions in O*(2n) time For each, check both halves in O(n+m) time. k-colouring is ‘trivial’ to solve in O*( dk/2en ) time. Enumerate all partitions into dk/2e disjoint sets. Check that each partition is 2-colourable (if k is odd, for all but one partition). Certificates again. What is trivial here for k-colouring?

Faster 4-colouring Using 3-Colouring:
Enumerate all maximal independent sets S. For each, check 3-colourability of G[V-S]. 1.4423n * n = n. Better: there is always a colour with at least n/4 vertices. Enumerate all m.i.s. S with at least n/4 vertices. 1.4423n * n/4 = n. Byskov, 2004: O(1.7504n) time. Fomin, Gaspers, Saurabh, 2007: O(1.7272n) time.

More Techniques for Exact Exponential-Time Algorithms
Algorithms and Networks 2017/2018 Johan M. M. van Rooij Hans L. Bodlaender Short on time: skip improvement for sort and search. 15 min per technique!

Content of this lecture
Five Lectures Introduction to exponential-time algorithms. Overview of some techniques. Dynamic Programming Branching algorithms (divide and conquer). Meet in the middle (a.k.a. Sort and Search). Local search – will probably fall off due to time restrictions. Inclusion/Exclusion. Measure-and-conquer analysis. Complexity theory of parameterised and exact algorithms (end of the course). Why these? Useful also in other contexts, e.g., FPT algorithms (later on).

DYNAMIC PROGRAMMING

Graph colouring with dynamic programming
Consider the graph colouring problem, not k-colouring. (The number of colours is no longer fixed). Lawler, 1976: using DP for solving graph colouring. C(G) = minS is m.i.s. in G 1 + X(G[V-S]) Tabulate chromatic number of G[W] over all subsets W: In increasing size. Using formula above. Total time: 2n * n = n. Total space: 2n. Time usage on blackboard or interactive.

Graph colouring Lawler 1976: O(2.4423n) (improved analysis) .
Eppstein, 2003: O(2.4151n). Byskov, 2004: O(2.4023n). All using O*(2n) memory. Improvements on DP method. Björklund, Husfeld, 2005: O(2.3236n). Björklund, Husfeld, 2006: Inclusion/Exclusion: O(2n). The last algorithm will be in the next lecture! 2.4423n tijd kun je bewijzen op blackboard via binomium van Newton.

Held-Karp algorithm for Travelling Salesman Problem
Take one starting vertex s arbitrarily. For set of vertices S µ V\{s} , vertex v Î S, let: B(S,v) = minimum length of a path, that: Starts in s. Visits s and all vertices in S (and no other vertices). Ends in v. Algorithm: B({v},v) = d(s,v). For j = 2 to |V|-1: For all sets S, |S| = j, for all v Î S: B(S,v) = minwÎS\{v}{ B(S\{v},w}) + d(w,v) } Return min{ B(V\{s},v) + d(v,s) | v Î V }. 1 2 3 4 5 11 This algorithm should be familiar (algorithms course, algorithms for decision support course).

Held-Karp algorithm for TSP
1 2 3 4 5 11 Algorithm: B({v},v) = d(s,v). For j = 2 to |V|-1: For all sets S, |S| = j, for all v Î S: B(S,v) = minwÎS\{v}{ B(S\{v},w}) + d(w,v) } Return min{ B(V\{s},v) + d(v,s) | v Î V }. This improves the trivial O*( n! ) algorithm. Time: O(n22n) Space: O(n2n)

BRANCHING ALGORITHMS (DIVIDE AND CONQUER)
Exponential-time algorithms – Algorithms and Networks BRANCHING ALGORITHMS (DIVIDE AND CONQUER)

Branching algorithm: divide and conquer
Algorithm that splits the current problem instance into multiple easier subinstances that are solved recursively. Earlier example: maximum/maximal independent set algorithm. Let T(n) be the number of subproblems generated on an instance of size n. Analysis of branching algorithm generating k subproblems, where subproblem i is di smaller than the original instance: T( n ) · T( n – d1 ) + T( n – d2 ) + … + T( n – dk ) + 1 T( 1 ) = 1 Solution of this recurrence is O*( an ) where a is the solution to: xn – xn-d1 – xn-d2 - … - xn-dk = 0. a often denoted as ¿( d1, d2, …, dk ). Different from algorithms course one does not use the Master Theorem here to analyse recurrences. Not of the form T(n) < a T(n/b) + f(n)

Branching algorithm for k-SAT
k-SAT (k-Satisfiability): Instance: set of clauses C (logical formula in CNF format) where each clause contains at most k literals. Question: does there exist a satisfying assignment to the variables in C, i.e., a truth assignment such that each clause has at least one literal set to true? Trivial algorithm: O*(2n). Why is this the trivial algorithm?

Branching algorithm for k-SAT
Algorithm (for k-SAT formula F): If F contains the empty clause: return false. If F is the empty formula: return true. Choose the smallest clause ( l1, l2, ..., lc ) from F. Recursively solve: F with l1 = true. F with l1 = false, l2 = true. ... F with l1 = false, l2 = false, ..., lc-1 = false, lc = true. Return true if and only if any recursive call returned true.

Branching algorithm for k-SAT: analysis
Algorithm is correct since in each clause at least one literal must be set to true. Recurrence relation describing the algorithm: T( n ) · T( n – 1 ) + T( n – 2 ) + … + T( n – c ) Worst case: T( n ) · T( n – 1 ) + T( n – 2 ) + … + T( n – k ) For different values of k, this leads to: k = 3: ¿( 1,2,3 ) < O( n ). k = 4: ¿( 1,2,3,4 ) < O( n ). k = 5: ¿( 1,2,3,4,5 ) < O( n ).

Reduction rules for independent set
Often, branching algorithms use reduction rules. Rules that simplify the instance without branching. For Maximum Independent Set we can formulate the following rules for any v 2 V: Reduction rule 1: if v has degree 0, put v in the solution set and recurse on G-v. Reduction rule 2: if v has degree 1, then put v in the solution set. Suppose v has neighbor w. Recurse on G – {v,w}. If v has degree 1, then there is always a maximum independent set containing v. Reduction rule 3: if all vertices of G have degree at most two, solve the problem directly. (Easy in O(n) time.)

A faster algorithm New branching rule: Algorithm: Analysis:
Take vertex v of maximum degree Take best of two recursive steps: v not in solution: recurse on G – {v} v in solution: recurse on G – N[v]; add 1 to solution size. Algorithm: Exhaustively apply all reduction rules. Apply branching rule. Analysis: T(n) £ T(n – 1) + T(n – 4), as v has degree at least 3, we lose at least 4 vertices in the second case. O*( ¿(1,4)n ) = O(1.3803n). This is faster than the O(1.4423n) enumeration algorithm.

Maximum independent set: final remarks
More detailed analysis gives better bounds. Many possibilities for improvement. Many papers with huge case analyses exist. Long time best known: O(1.1844n) (Robson, 2001) Extensive, computer generated case analysis! Includes memorization (DP) Recently improved (I think), often discussions on correctness of such partially computer generated algorithms. The measure and conquer technique allows for better analysis of branch and reduce algorithms (Fomin, Grandoni, Kratsch, 2005) Much simpler algorithm and only slightly slower compared to Robson. See lecture 4 on exact exponential-time algorithms.

MEET IN THE MIDDLE (OR SORT AND SEARCH)
Exponential-time algorithms – Algorithms and Networks MEET IN THE MIDDLE (OR SORT AND SEARCH)

Subset Sum Subset sum problem: Example: Example 2:
Given: set of positive integers S = {a1,a2,..,an}, integer t. Question: Is there a subset of S with total sum t? Example: S = {1,2,3,8,12,15,27}. t = 34 Example 2: t = 33 Example 1: not possible: is too little, yet to big. Without 27, need 15 (rest too small), but 15 without 12 too small and to big, too small. Example 2: , or

Sort and Search: algorithm
We split S into two subsets S1 and S2: S1 = {a1,a2,…,an/2}, S2 = {an/2+1,an/2+2,…,an}. For S1 and S2 we compute 2n/2 size tables containing all sums that can be made using elements from S1 and S2 respectively. Sort the two tables in O(2n/2 log(2n/2)) = O*(2n/2) time. Start with the smallest value in S1 and the largest value in S2 and repeat these steps: Move one value further in S1 if the sum is too small. Move one value back in S2 if the sum is too large. Output ‘Yes’ if the sum is exactly t. Output ‘No’ if we finished searching S1 and S2. Ask students: why correct? Running time?

The knapsack problem Knapsack We split S into two subsets S1 and S2:
Given: Set S of items, each with integer value v and integer weight w, integers W and V. Question: is there a subset of S of weight no more than W, with total value at least V? We split S into two subsets S1 and S2: S1 = {s1, s2,…, sn/2}, S2 = {sn/2+1, sn/2+2,…,sn}.

Sort and Search: knapsack
Algorithm: For S1 and S2 we compute 2n/2 size tables containing all subsets of items that can be picked from S1 or S2. Sort the sets S1 and S2 by their weight. Remove all dominated subsets. Dominated subsets: subsets for which there exists another subset with more (or equal) value and less (or equal) weight. S1 and S2 are now sorted increasingly on weight and value. Start with the smallest item in S1 and the largest item in S2 and keep track of the highest value combination found: Update the highest value if the combination of the subset from S1 and the subset from S2 is not too large and more valuable as the current most valuable solution. Move one value further in S1 if the sum of the sizes is too small. Move one value back in S2 if the sum of the sizes is is too large.

Sort and search space improvement
Both algorithms can be improved to use O*(2n/2) time and O*(2n/4) space (Schroeppel, Shamir, 1981). Compare to O*(2n/2) time and space. Algorithm: Split S into four subsets S1, S2, S3, S4 of equal size. For each subset create a sorted table with all possible sums that we can make using elements from that set. We list all possible sums from elements from S1 [ S2 in increasing order and all possible sums from elements from S3 [ S4 in decreasing order. How? Next Slides. Combining values from the increasing list on S1 [ S2 and the decreasing list on S3 [ S4 is identical to the previous algorithms. When short in time: skip this!

Sort and search space improvement: analysis
Algorithm uses O*(2n/4) space: Four tables of all values one can make using numbers in Si. Algorithm uses O*(2n/2) time: If we can list all values from S1 [ S2 in increasing order within this amount of time, without using extra space. Similarly for all values from S3 [ S4 in decreasing order. Why is this correct

Sort and search space improvement: listing the numbers
How to list all elements from S1 [ S2 in increasing order? Select the smallest element e from S1. Make a priority queue with all e + s2 with all elements s2 2 S2. Whenever we need the next element we take the smallest element from the priority queue e + s2 and replace it by e’+s2 (where e’ is the next biggest element after e in S1). Then, return the next smallest element of the priority queue. Priority queue holds tuples (e,si) with priority e + si. The priorities cause the numbers to be listed in increasing order. The tuple representation allow (e,si) to be replaced by (e,si+1), causing all possible numbers from S1 [ S2 to be generated. 2n/2 updates, each in O*(log(2n/4)) time gives O*(2n/2) time.

LOCAL SEARCH

Local search Use exponential time to search a ‘local’ neighbourhood of possible solutions. Consider 3-Satisfiability. For two truth assignment a1 and a2 that assign true/false to the variables x1, ..., xn, we define the Hamming distance to be the number of variables on which a1 and a2 differ. Lemma: Given any truth assignment a and a 3-SAT formula F, we can decide in O*(3d) time whether there is an truth assign-ment that satisfies F at Hamming distance at most d from a.

A simple local search algorithm for 3-Satisfiability
Any truth assignment lies either at Hamming distance at most n/2 from the all-true or the all-false assignment. Algorithm: Apply the local search algorithm and look within the Hamming-Balls of size n/2 from the all-true and all-false assignments. Running time: O*(3n/2) = O(1.7321n) This already improves the O(1.8393n)-time branching algorithm we saw earlier.

Randomised local search
We improve the previous local search algorithm for 3-SAT using randomisation. Monte-Carlo Algorithm: If it outputs ‘Yes’ it is correct. If it outputs ‘No’ there is a probability that the algorithm is wrong. Algorithm: Choose uniformly at random a truth assignment a. Search the Hamming ball of size n/4 centred around a for a satisfying assignment. If found: output ‘Yes’. If not found, repeat from 1 (a fixed large number of times). Output ‘No’.

On the error probability
Lemma 1: When performing repeated trials of an experiment with success probability p, the expected number of repetitions until the first success is 1/p. Lemma 2: (Markov’s inequality) Let X be a non-negative random variable and let c >0, then: Corollary: The repeated application of a Monto-Carlo algorithm with one-sided error probability p, either until successful, or at most 2/p tries, has an overall error probability of at most ½. Proof: use Markov’s inequality with c = 2/p. Does anyone see why the corollary is true? Let X be the number of repetitions until succes. En wat moet er gebeuren voor faal kans ¼? En 1/1024? E Vaak genoeg herhalen dus!

Randomised local search: running time
Given that a satisfying assignment exist, we find it in a Hamming ball of size n/4 with probability at least p: With 200/p repetitions: Error probability at most (½)100. Running time: O*( 200¢1/p ¢3n/4) = O*(1.5n). By Stirling’s approximation the sum over the binomials is roughly (256/27)n/4. Check details yourself! 1/p = 2n/bionamial_sum

Concluding remarks on local search for 3-SAT
The O*(1.5n) algorithm can be derandomised. Family of O(1/p) balls that together cover the entire search space (covering code). Using faster algorithms to search the Hamming ball, this can be improved to O(1.465n) [Scheder]. Significantly faster algorithms can be obtained by: Not brute-force searching the Hamming ball, But, performing a random walk through the Hamming ball. Schönings algorithm: O(1.334n). Later improved to O(1.323n) by Rolf.

CONCLUSION

Conclusion We discussed three or four other techniques in exact exponential-time algorithms: Dynamic programming Graph colouring, travelling salesman problem. Branching algorithms For k-satisfiability and maximum independent set. Sort and Search (a.k.a. Split and List). For subset sum and knapsack. Local search. For algorithms for 3-satisfiability.

Questions? Research on exact exponential-time algorithms is relatively recent. Several interesting open problems. Today we considered: Clever enumeration. Dynamic programming.

Inclusion/Exclusion Algorithms and Networks 2017/2018
Johan M. M. van Rooij Hans L. Bodlaender Copy question on integers to blackboard: nice reference when proving the theorems.

Five Lectures Introduction to exponential-time algorithms. Overview of some techniques. Inclusion/Exclusion. Measure-and-conquer analysis. Complexity theory of parameterised and exact algorithms (end of the course). Many ideas from exact exponential-time algorithms are also applicable to or are related to parameterised algorithms.

Question...? How many integers exist in the range [1,2,...,10.000] with the property that they are: Not divisible by 2. Not divisible by 3. Not divisible by 5. So: 1,7,11,13,17,19,23,29,31,... If I would let the student puzzle for 5 minutes, I guess some proportion of them would give the correct answer. But, let us do this in a step-by-step fashion. Copy question to blackboard so that

Inclusion/exclusion and a Venn diagram
Px: integers divisible by x. |P2| = 5.000 |P3| = 3.333 |P5| = 2.000 |P2| - |P3| - |P5| = -333 We subtracted too many! How many too many? P2 P5 P3 1 2 2 3 1 1 2

|P2 Å P3| = 1.666 |P2 Å P5| = 1.000 |P3 Å P5| = 666 Divisible by 6, 10, 15. |P2| - |P3| - |P5| + |P2 Å P3| + |P2 Å P5| + |P3 Å P5| = 2.999 We added too many! How many? P2 P5 P3 1 1 1 1 1 1

How many integers exist in the range [1,2,...,10.000] with the property that they are: Not divisible by 2. Not divisible by 3. Not divisible by 5. |P2 Å P3 Å P5| = 333 |P2| - |P3| - |P5| + |P2 Å P3| + |P2 Å P5| + |P3 Å P5| - |P2 Å P3 Å P5| = 2.666 P2 P5 P3 1 1 1 1 1 1 This is the principe of inclusion exclusion. Copy |P2| - |P3| - |P5| + |P2 and P3| + |P2 and P5| + |P3 and P5| - |P2 and P3 and P5| to blackboard. 1

What are we going to do in this lecture?
Apply the principle of inclusion/exclusion for faster algorithms. The inclusion/exclusion formula. Algorithmic applications: Graph colouring. Hamilton cycle. Steiner tree (if time allows).

INCLUSION/EXCLUSION FORMULA
Exponential-time algorithms – Algorithms and Networks INCLUSION/EXCLUSION FORMULA

The inclusion/exclusion formula in its general form
Let N be a collection of objects (anything). Let 1,2, ...,n be a series of requirements on objects. Finally, let for a subset R µ {1,2,...,n}, N(R) be the number of objects in N that do not satisfy the requirements in R. N(;) = |N| N({1,2...n}) counts objects that satisfy no requirement at all. Then, the number of objects X that satisfy all requirements is: In the counting integers example: N = [1,2,....,10.000] Requirements: integer is not divisible by x, for x = 2, 3, 5. Click: link to counting example. Next: proof.

Intuition behind the inclusion/exclusion formula
If we count all objects in N, we also count some wrong objects that do not satisfy all properties. So, subtract from this number all objects that do not satisfy property 1. Also subtract from this number all objects that do not satisfy property 2. Etc. However, we now subtract too many, as objects that do not satisfy two or more properties are subtracted twice. So, add for all pairs of properties, the number of objects that satisfy both properties. But, then what happens to objects that avoid 3 properties? Continue, and note that the parity tells if we add or subtract… This gives the formula of the previous slide.

Proof of the inclusion/exclusion formula
The number of objects X that satisfy all requirements is: Proof: An object that has all properties is only counted with R=;. Consider an object that does not have the properties in P µ {1,2,...,n} is counted once for all S µ P: By the binomial theorem. Click: link to counting example. Next: proof.

The inclusion/exclusion formula: alternative proof (that I find more intuitive)
See the formula as a branching algorithm branching on a requirement: required = optional – forbidden ‘Not divisible by x’ = ‘can be divisible by x’ – ‘divisible by x’ The full branching tree has 2|X| leaves. Each corresponding to a set R. Requirements not in R are ‘optional’ and can be violated. Requirements in R are ‘forbidden’ and must be violated. On the blackboard: example “How many integers 1,2,..., not divisible by 2,3, 5.”

INCLUSION/EXCLUSION FOR GRAPH COLOURING
Exponential-time algorithms – Algorithms and Networks INCLUSION/EXCLUSION FOR GRAPH COLOURING

Graph colouring with inclusion/exclusion
Björklund and Husfeld, and independently Koivisto, 2006. O*(2n) time algorithm for graph colouring. Covering G with independent sets. Objects to count: Sequences (I1,…,Ik) where each Ii is an independent set. Requirements: For every vertex v we require that v is contained in at least one independent set Ii. Let ck(G) be the number of ways we can cover all vertices in G with k independent sets, where the independent sets may be overlapping, or even the same. Lemma: G is k-colorable, if and only if ck(G) > 0.

Expressing ck in s Let ck(G) be the number of ways we can cover all vertices in G with k independent sets, where the independent sets may be overlapping, or even the same. Let for S µ V, s(S) be the number of independent sets in G[S]. Objects to count: Sequences (I1,…,Ik) where each Ii is an independent set. Requirements: For every vertex v we require that v is contained in at least one independent set Ii. Click: small I/E formula as a reminder. This works, if we can compute s(v\X) fast.

Counting independent sets
Let for SµV, s(S) be the number of independent sets in G[S]. s(S) = s( S\{v} ) + s( S\N[v] ) for some v 2 S. s(;) = 1. We can compute all values s(S) in O*(2n) time. Use bottom-up dynamic programming and store all values. The algorithm: Compute all s(S) for all S µ V by dynamic programming. Compute values ck(G) with the inclusion/exclusion formula. Take the smallest k for which ck(G) > 0. O*(2n) time and space

A note on the algorithm Polynomial space algorithm is also possible.
Then we computing s(X) each time again. This requires additional exponential time. Running time depends on how fast we count independent sets. O(1.4423n) time enumeration, then O(2.4423n) time total. Gaspers and Lee 2016: O(2.2356n). Lots of techniques: Measure and conquer. Potentials. Compound measures. Branching to separate the graph. Out of scope for this lecture...

INCLUSION/EXCLUSION FOR HAMILTON CYCLE
Exponential-time algorithms – Algorithms and Networks INCLUSION/EXCLUSION FOR HAMILTON CYCLE

Space improvement for Hamiltonian cycle
Dynamic programming algorithm for Hamiltonian cycle or TSP uses: O( n22n ) time. O( n2n ) space. In practice, space becomes a problem before time does. We now give an algorithm for Hamiltonian cycle that uses: O(n32n) time. O(n) space. This algorithm counts Hamiltonian cycles using the principle of inclusion/exclusion.

Counting (non-)Hamiltonian cycles
Computing/counting tours is much easier if we do not care about which cities are visited. This uses exponential space in the DP algorithm. Define: Walks[ vi, k ] = the number of ways to travel from v1 to vi traversing k times an edge. We do not care whether nodes/edges are visited (or twice). Using Dynamic Programming: Walks[ vi, 0 ] = 1 if i = 0 Walks[ vi, 0 ] = 0 otherwise Walks[ vi, k ] = ∑vj ∈ N(vi) Walks[ vj, k – 1 ] Walks[ v1, n ] counts all length n walks that return in v1. This requires O(n3) time and O(n) space. In het Nederlands noem ik dit wel eens de dronken handelsreiziger: tel het aantal mogelijke mogelijke manieren waarop hij rond zwalkt. O(n) space.... Waarom? Slechts 2 waarden van k hoeven bijgehouden te worden!

Apply the formula to every city: the inclusion/exclusion formula
Let CycleWalks( V’ ) be the number of walks of length n from v1 to v1 using only vertices in V’, then: Formula requires 2n times the counting of cyclic walks through a graph: O(n32n) time. O(n) space.

INCLUSION/EXCLUSION FOR STEINER TREE
Exponential-time algorithms – Algorithms and Networks INCLUSION/EXCLUSION FOR STEINER TREE

The Steiner tree problem
Let G = (V,E) be an undirected graph, and let N µ V be a set of terminals. A Steiner tree is a tree T = (V’,E’) in G connecting all terminals in N: V’ µ V, E’ µ E, N µ V’ We use k=|N|. Streiner tree problem: Given: an undirected graph G = (V,E), a terminal set N µ V, and an integer t. Question: is there a Steiner tree consisting of at most t edges in G.

Some background on the algorithm
Algorithm invented by Jesper Nederlof. Just after he finished his Master thesis supervised by Hans (and a little bit by me). Master thesis on Inclusion/Exclusion algorithms. O*(2k) time and polynomial space algorithm. n is the number of vertices (disappears in de O*). k is the number of terminals.

Using the inclusion/exclusion formula for Steiner tree (problematic version)
One possible approach: Objects: trees in the graph G. Requirements: contain every terminal. Then we need to compute 2k times the number of trees in a subgraph of G. For each W µ N, compute trees in G[V\W]. However, counting trees is difficult: Hard to keep track of which vertices are already in the tree. Compare to Hamiltonian Cycle: We want something that looks like a walk, so that we do not need to remember where we have been.

Branching walks Definition: Branching walk in G=(V,E) is a tuple (T,Á): Ordered tree T. Mapping Á from nodes of T to nodes of G, s.t. for any edge {u,v} in the tree T we have that {Á(u),Á(v)} 2 E. The length of a branching walk is the number of edges in T. When r is the root of T, we say that the branching walk starts in Á(r) 2 V. For n 2 T, we say that the branching walk visits the vertices Á(n) 2 V. Some examples on the blackboard...

Branching walks and steiner tree
Definition: Branching walk in G=(V,E) is a tuple (T,Á): Ordered tree T. Mapping Á from nodes of T to nodes of G, s.t. for any edge {u,v} in the tree T we have that {Á(u),Á(v)} 2 E. Lemma: Let s 2 N a terminal. There exists a Steiner tree T in G with at most c edges, if and only if, there exists a branching walk of length at most c starting in s visiting all terminals N.

Using the inclusion/exclusion formula for Steiner tree
Approach: Objects: branching walks from some s 2 N of length c in the graph G. Requirements: contain every terminal in N\{s}. We need to compute 2k-1 times the number of branching walks of length c in a subgraph of G. For each W µ N\{s}, compute branching walks from s in G[V\W]. Next: how do we count branching walks? Dynamic programming (similar to ordinary walks).

Counting branching walks
Let BW(v,j) be the number of branching walks of length j starting in v in G[W]. BW(v,0) = 1 for any vertex v. BW(v,j) = u2(N(v)ÅW) j1 + j2 = j-1 BW(u,j1) BW(v,j2) j2 = 0 covers the case where we do not branch / split up and walk to vertex u. Otherwise, a subtree of size j1 is created from neighbour u, while a new tree of size j2 is added starting in v. This splits off one branch, and can be repeated to split of more branches. We can compute BW(v,j) for j = 0,1,2,....,t. All in polynomial time.

Putting it all together
Algorithm: Choose any s 2 N. For t = 1, 2, … Use the inclusion/exclusion formula to count the number of branching walks from s of length t visiting all terminals N. This results in 2k-1 times counting branching walks from s of length c in G[V\W]. If this number is non-zero: stop the algorithm and output that the smallest Steiner tree has size t.

P2 P5 P3 Conclusion Today we discussed:
The inclusion/exclusion formula. Algorithmic applications: Graph colouring. Hamilton cycle. Steiner tree. Many more applications. Different fields of math and/of algorithms. FPT algorithms. Fast subset convolution. P2 P5 P3

Measure-and-Conquer Analysis
Algorithms and Networks 2017/2018 Johan M. M. van Rooij Hans L. Bodlaender Dominating set and set cover + simple algorithm: 20 min Measure 1+2: 15 min. Measure and Conquer: 15 min. TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAA

Five Lectures Introduction to exponential-time algorithms. Overview of some techniques. Inclusion/Exclusion. Measure-and-conquer analysis. Complexity theory of parameterised and exact algorithms (end of the course). Many ideas from exact exponential-time algorithms are also applicable to or are related to parameterised algorithms.

This lecture ‘Measure and conquer’ is a technique for better analyses of branching algorithms. Contents of this lecture: Dominating set and set cover. A simple algorithm for set cover and for dominating set. Introducing measures: a better analysis of the same algorithm. Measure and conquer. Emphasise: we will not focus on creating better algorithms, just on analysing the better.

SET COVER and DOMINATING SET
Exponential-time algorithms – Algorithms and Networks SET COVER and DOMINATING SET

Set cover Given a collection of sets S over a universe U, a set cover is a subcollection C of S such that X2CX = U. Example: U = {1,2,3,4,5,6,7} S = { {1}, {1,2}, {3,4}, {5,6}, {1,5,6}, {2,4,7}, {1,6,7} } Set Cover C = { {1,5,6}, {2,4,7}, {3,4} } Set Cover (problem): Given: a collection of sets S over a universe U, and an integer k. Question: does there exists a subcollection of S of size at most k that is a set cover?

Algorithms for set cover
For a set cover instance (S,U): Let n be the number of sets: n = |S|. Let m be the number of elements: m = |U|. Let d be the dimension (d = n + m). Algorithms: Brute force: O*(2n) time and polynomial space. Dynamic programming: O*(2m) time and space. How? Inclusion/exclusion: O*(2m) time and polynomial space.

Dominating set A dominating set is a subset D of the vertices V such that N[D]=V. Dominating Set (problem): Given: a graph G=(V,E) and an integer k. Question: does there exists a dominating set of size at most k in G?

Algorithms for dominating set
Dominating set is NP-complete. Algorithms beating the trivial O*(2n)-time algorithm using polynomial space: 2004: Fomin, Kratsch, Woeginger O(1.9327n). 2004: Grandoni* O(1.9053n). 2004: Randerath and Schiermeyer O(1.8899n). 2005: Fomin, Grandoni, Kratsch* O(1.5263n). 2009: van Rooij (PhD Thesis) O(1.4969n). 2011: Iwata* O(1.4864n). (results with a * are improved using exponential space) Fomin, Grandoni and Kratsch introduced measure and conquer. Their algorithm is (almost) the same as Grandoni’s.

Dominating Set as a set cover problem
We can model an instance of dominating set as a set cover problem: A set for every vertex (choose the vertex). An element for every vertex (dominate the vertex). Incedence graph of set cover instance: Bipartite graph with a vertex for every set S and element e and an edge if and only if e2S. Dominating set on n vertices becomes a set cover instance of dimension d. b c d e f h a g a b c d e f g h Dominating set on such bipartite graphs representing set cover instances is also called red-blue dominating set.

A simple algorithm

A branching algorithm for set cover
Algorithm (for set cover instances (S,U)) Exhaustively apply the following reduction rules to (S,U): Subsets rule: if S contains two sets X1, X2 such that X1 µ X2, then remove X1 from S. Unique elements rule: if U contains an element that occurs in only one set X in S, then add X to the solution and let U := U\X. Take a set X in S of maximum size. If |X| · 2, solve the problem using maximum matching. Branch into two subproblems: X not in solution: remove X from S and recurse. X in the solution: add X to the solution, let U := U\X and recurse. Note: when we remove elements from U, we (implicitly) also remove these elements from all sets in S. Discuss correctness of reduction rules. Maximum matching: example on the blackboard! Clearly correct right?

Analysis of running time
Reduction rules can be applied in polynomial time. Branching rule generates subproblems: Take X branch: remove 1 set, at least 3 elements. Discard X branch: remove 1 set. Let d be the dimension of (S,U) = d = |S| + |U|. Let T(d) be the number of subproblems generated by the algorithm starting with a problem of dimension d. Worst case recurrence relation: T( d ) · T( d-1 ) + T( d-4 ) T( 1 ) = 1 Running time O*( ¿(1,4)d )=O(1.3803d). ¿(1,4) is unique positive real soltuion to: 1 - x-1 - x-4 = 0 Ask for the running time in terms of d.... Before second click.

What does this mean? a b c d e f g h
We have an O(1.3803d) time algorithm for set cover. Not comparable to earlier results expressed in n or m. What does this mean? Why is this interesting? Algorithm for dominating set! Set cover instance obtained from dominating set has d = 2n. Running time: O( n) = O(1.9053n). b c d e f h a g a b c d e f g h Why did they not think of this before 2004?

INTRODUCING MEASURES

Reconsider the algorithm for a moment
Algorithm (for set cover instances (S,U)) Exhaustively apply the following reduction rules to (S,U): Subsets rule: if S contains two sets X1, X2 such that X1 µ X2, then remove X1 from S. Unique elements rule: if U contains an element that occurs in only one set X in S, then add X to the solution and let U := U\X. Take a set X in S of maximum size. If |X| · 2, solve the problem using maximum matching. Branch into two subproblems: X not in solution: remove X from S and recurse. X in the solution: add X to the solution, let U := U\X and recurse. Note: when we remove elements from U, we (implicitly) also remove these elements from all sets in S. Discuss: if we compare sets of different sizes, do they have the same role here? What of sets of size 2? They are ‘almost removed’ as sets of size 1 are removed, and if all sets have size 2 then we are done!

Some observations Observation on sets of size 2:
These sets are ‘almost done’. If they lose an element they are removed. If all sets have size at most 2, then we solve the problem in polynomial time. What would happen if we define the dimension (or some other measure of instance size) not to count these for full? Note that a similar observation can be made on elements of frequency 1 or 2 (but we ignore this for the moment): Frequency one elements cause reduction rules to fire (‘done’). Frequency two elements are ‘almost done’.

A different measure What would happen if we define the dimension (or some other measure of instance size) not to count these for full? Let us try this, using a measure ¹. Define: Observe that ¹ · d. Hence, any set cover algorithm running in O*(®¹) runs in O*(®d). This gives a dominating set algorithm running in O*(®2n). Next step: Analyse the algorithm using the measure ¹. Note that sets of size 1 do not exist: they are always subsets or unique elements.

Analysis using a different measure
Define: Reduction rules can still be applied in polynomial time. For the branching rule, we consider two cases: |S| ¸ 4. Take branch: remove 1 large set, at least 4 elements. measure decreases by at least 5. Discard branch: remove 1 large set. measure decreases by at least 1. |S| = 3. Take branch: remove 1 large set, at least 3 elements. measure decreases by at least for removing the elements from other sets.

Different measure, different running time!
Let T(¹) be the number of subproblems generated by the algorithm starting with a problem of measure ¹. Recurrence relation: T( ¹ ) · T( ¹ – 5 ) + T( ¹ – 1 ) ¿( 5, 1 ) < T( ¹ ) · T( ¹ – 5.5 ) + T( ¹ – 1 ) ¿( 5.5, 1 ) < T( 0 ) = 1 Clearly, the algorithm runs in O*( ¿(5,1)¹ ) · O(1.3248d). Improvement compared to the last analysis: O(1.3803d). For dominating set O(1.9053n) is improved to O(1.7551n). Big improvement without modifying the algorithm!

LET US TRY THAT AGAIN

Let us try that again! Why did we choose precisely this measure?
No real reason. Maybe because it was the most simple thing to try. Let us try something else! With different weights for sets than for elements.

A different measure Let us try something else! Define:
Use weight w(i) for the measure of a set of size i. And weight v(i) for the measure of an element of frequency i. Define: Note that sets of size less than 2 and elements of frequency 1 are directly removed by the reduction rules.

A (partial) analysis on the blackboard
Consider several cases: |S| ¸ 4, S does not contain a frequency 2 element. |S| ¸ 5 |S| = 4 |S| ¸ 4, S contains a frequency 2 element. |S| ¸ 5, |S| = 3, with ei elements of frequency i for: All e2, e3, e>3 with values 0, 1, 2 and 3, such that their sum is 3. We need a method to quickly compute ¿ function values. E.g., excel with solver adding.

Different measure, different running time!
Over all cases, the worst case is ¿ (6, 1) < For set cover: Original analysis: O(1.3803d). First measure: O(1.3248d). Second measure: O(1.2852d). For dominating set: Original analysis: O(1.9053n). First measure: O(1.7551n). Second measure: O(1.6518n). Again improvement without modifying the algorithm! Shall we try yet another measure?

MEASURE AND CONQUER

Measure and Conquer Measure and conquer is a more structural approach to trying different measures. Define a measure using weight functions, e.g.: Define an analysis using this measure in which the weights remain variables. Now we have a numerical optimisation problem: Find the weights leading to the best running time. Finite problem if we set v(i)=w(i)=1 for i¸p for some integer p. Often, one also needs to define some constraints one the weights to prevent the number of cases in the analysis from exploding.

An analysis using the weight functions
Define: Cases: All 3 · |S| · p+1, all (e2,e3,e4,...,e>p+1) with sum equal to |S|. Recurrences: Where:

An analysis using the weight functions
Using p=6 (or larger) one can obtain T(¹) · ¹ Using these weights: Running time for dominating set: O( n) = O(1.5259n).

On solving the numerical problems
One needs to optimise quasi-convex functions. The set of weights with solution value ® or smaller is convex. This means that there are no local optima. Gradient descent based hill climbing suffices. For small instances one can use the solver in excel. For large instance one needs to implement a smart gradient descent. When formulating the problem slightly differently, the numerical optimisation can also be translated into a convex optimisation problems for which large scale solvers exist.

CONCLUSION

Conclusion We have seen how to use measures to get better bounds on the running time of a branching algorithm. Measure and conquer is a structural approach to this. Several improvements possible to what we have seen: Weights for sets and elements do both not need to converge to one. Several new reduction rules possible if one analyses the resulting worst-case recurrences better. Add memorisation (never solve the same subproblem twice). Switch to counting set covers/dominating sets and use two branching rules: one branching on sets and one branching on elements (based on inclusion/exclusion).

Conclusion 2 We have seen how to use measures to get better bounds on the running time of a branching algorithm. Measure and conquer is a structural approach to this. Combination with other analysis techniques (outside the scope of this course) makes this approach even stronger. Potentials (Iwata, and earlier Robson). Compound measures (Wahlstrom). Bottom-up using average degrees (Bourgeious et al. (me)) Separate, measure and conquer (Gaspers and Sorkin).

Exact Exponential-Time Algorithms

Similar presentations

Presentation on theme: "Exact Exponential-Time Algorithms"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Exact Exponential-Time Algorithms

Similar presentations

Presentation on theme: "Exact Exponential-Time Algorithms"— Presentation transcript:

Similar presentations

About project

Feedback