1. Introduction to Algorithms
Lecture 3 – Divide and Conquer

2. The Methodology Divide-and-Conquer is a useful paradigm. Main idea
Divide the problem into k > 2 sub-problems.
Solve the sub-problems.
Combine the solutions of the sub-problems to arrive at a solution to the original problem.

3. The Methodology Divide-and-Conquer is a useful paradigm. Main idea
Divide the problem into k > 2 sub-problems.
Solve the sub-problems, usually recursively.
Combine the solutions of the sub-problems to arrive at a solution to the original problem.

4. The Methodology
When using recursion to solve the subproblems, the analysis for running time is also natural.
Let the size of the original problem be n and T(n) be the time
D(n) : Time taken to divide the problem. Let there be k subproblems of sizes n1, n2, …,nk.
To solve the ith subproblem, the time taken would be T(ni).
C(n): Time taken to combine the solutions.

5. Examples
Merge Sort
Division is easy. D(n) = O(1). We get two subproblems of size n/2 each.
Each subproblems take T(n/2) time.
Combine step takes O(n) time.
Recurrence relation is T(n) = 2T(n/2) + O(n)
Solution using Master’s theorem is T(n) = O(nlog n).

6. Examples Quick sort (assuming a ``good’’ pivot)
Divide : Uses the partition algorithm. Take time O(n).
Solving the subproblems is now 2T(n/2)
Combine: Nothing to do here.., so O(1).
Recurrence relation: T(n) = 2T(n/2) + O(n).
Solution using recurrence relation again is T(n) = O(nlog n)

7. Examples The selection algorithm from previous lecture.
The subproblem sizes are not equal.
No combination step, but an O(n) time to divide.

8. A Complete Example
Let us consider the multiplication of two square matrices A and B.
Suppose we know just to multiply scalar values.
And we know the scheme of matrix multiplication.

9. Matrix Multiplication
Can use divide and conquer as follows.
Divide the matrices into 4 parts.
B00
B01
A00
A01
C00
C01 X =
B10
B11
A10
A11
C10
C11 A B C
C00 = A00 x B00 + A10 x B01
C00 = A00 x B00 + A10 x B01
C00 = A00 x B00 + A10 x B01
C00 = A00 x B00 + A10 x B01

10. Matrix Multiplication
The recurrence relation can be obtained as follows.
Divide time is O(1).
There are four subproblems. And each subproblem is of size n/4.
Combine time is O(n2).
The recurrence relation is
T(n) = 8T(n/2) + O(n2)
The solution is T(n) = O(n3).

11. A Better Algorithm Surprisingly, you can do better than O(n3).
Recently, there is an O(n2.23) algorithm.
In fact, the lower bound is still open.
We will get somewhere close to that.

12. A Slight Detour - Motivation
Consider two complex numbers a+ib, and c+id that we need to multiply.
The result is (ac-bd) + i(ad+bc).
Requires four multiplications and two additions.
Suppose that multiplications are expensive compared to additions.
Can we save on one multiplication?
Can use more additions or subtractions.

13. Motivation
Consider the products P1 = ac, P2 = bd, and P3 = (a+b)(c+d).
Now, the result of multiplication is (P1-P2) + i(P3-P2-P1).
We have 3 multiplications and 5 additions/ subtractions.
A similar idea can be used for matrix multiplication too.

14. Motivations
Question: Verify quickly with two complex numbers of your choice.

15. Motivation
Is there a practical reason to use the new approach even if multiplication is as expensive as an addition/subtraction?

16. A Better Algorithm
Notice that matrix addition is easier than matrix multiplication.
So, we can trade-off some matrix multiplications for matrix additions, and subtractions.

17. A Better Algorithm
Let A11, A12, A21, and A22 be the four submatrices of A, each of size n/2xn/2.
Let B11, B12, B21, and B22 be the four submatrices of B, each of size n/2xn/2.
We need,
C11 = A11 x B11 + A12 x B21
C12 = A11 x B12 + A12 x B22
C21 = A21 x B11 + A22 x B21
C22 = A21 x B12 + A22 x B22

18. A Better Algorithm
Strassen computes each of the eight AixBj using linera combinations of the following seven matrix products.
P1 = A11(B12 − B22)
P2 = (A11 + A12)B22
P3 = (A21 + A22)B11
P4 = A22(B21 − B11)
P5 = (A11 + A22)(B11 + B22)
P6 = (A12 − A22)(B21 + B22)
P7 = (A11 − A21)(B11 + B12)

19. Strassen’s Algorithm Question: Verify that C11 =P5 + P4 − P2 + P6

20. A Better Algorithm
Asymptotically, Strassen’s algorithm works better than the O(n3) standard algorithm.
In reality, notice that one has to perform 14 matrix additions and seven matrix products.
So, in practice, the method may be slow due to the large constants involved in the O notation.

21. A Better Algorithm
Strassen saves one multiplication by introducing more additions.
Thus, the recurrence relation would be
T(n) = 7T(n/2) + O(n2).
Question: Solve for T(n) using Master’s theorem.

22. A Better Algorithm
Strassen saves one multiplication by introducing more additions.
Thus, the recurrence relation would be
T(n) = 7T(n/2) + O(n2).
Using Master’s theorem, the solution now is T(n) = O(nlog2 7).
Notice that log2 7 < 3. Hence, the algorithm beats the n3 time for the earlier algorithm.

23. Yet Another Example We will consider sorting again.
But this time as a network/hardware for sorting.
There are some nice principles along the way.

24. Comparators
A comparator is a hardware element that takes two inputs and produces two outputs.
We assume that a comparator operates in O(1) time.
Can connect comparators to form a network.
Min(x,y) x
Comparator y
Max(x,y)

25. A Network

26. Sorting Network
A comparator network that takes n inputs and produces a sorted rearrangement of the inputs.

27. Sorting Network Size of the network : The number of comparators used.
Depth of a network is defined as follows.
Let the depth of a input wire be 0.
The depth of a comparator with inputs as depth dx and dy will be max(dx, dy) + 1.
The depth of a network is the maximum depth of a comparator in the network.

28. How to Build a Sorting Network
Bitonic Sequence: A sequence that monotonically increases and then decreases monotonically.
More formally, has two indices i and j such that A(i: j-1) is monotonically increasing, and A(j:i-1) is monotonically decreasing.
Corresponds to a shifting of A so as to get the intuitive definition.
Example: 11, 16, 20, 23, 34, 17, 9, 5, 1
We first construct a sorting network that can sort input sequences that are bitonic.
We then show how to use such a network to sort arbitrary sequences.

29. Sorting Network – Bitonic Sequences
Suppose that a sequence X is bitonic.
Let L(X) = {min{xi, xn/2+i}, 0 ≤ i ≤ n/2-1}.
Let R(X) = {max{xi, xn/2+i}, 0 ≤ i ≤ n/2-1}.
Then both L(X) and R(X) are bitonic, and every element of L(X) has a value at most the value of any element of R(X).

30. How to Use the Above Theorem
What the above theorem says is that a bitonic sequence can be “divided” into two bitonic sequences of equal length, with
One sequence containing elements that are all smaller than the other.
Can be applied recursively to divide sequences further into smaller subsequences.
Eventually, sorts the input bitonic sequence.

31. In Pictures
We have that any of min(a1,a5) to min(a4,a8) is SMALLER than any of the other four.

32. Put Them Together..

33. Bitonic Sorter Has a depth of O(log n) Has O(nlog n) comparators.
But does not work for arbitrary sequences.
Also, have used a property that is yet to be proved.

34. Sorting Network – Bitonic Sequences
Suppose that a sequence X is bitonic.
Let L(X) = {min{xi, xn/2+i}, 0 ≤ i ≤ n/2-1}.
Let R(X) = {max{xi, xn/2+i}, 0 ≤ i ≤ n/2-1}.
Then both L(X) and R(X) are bitonic, and every element of L(X) has a value at most the value of any element of R(X).
Can be shown by the unique crossover property.

35. Unique Crossover Property

36. Unique Crossover Property
Let X be a bitonic sequence. The crossover property states that there exists an index i such that:
for any a in {x0, x1, …, xi-1} and for any b in {xn/2+i, xn/2+i-1 , …, xn/2}, it holds that a ≤ b.
for any a in {xi, xi+1, …, xn/2-1} and for any b in {xn/2+i, xn/2+i+1, …, xn-1} , it holds that a > b .
or any a in {x0, x1, …, xi-1} and for any b in {xi, xi+1, …, xn/2-1} , it holds that a ≤ b , and
for any a in {xn/2, xn/2+1, …, xn/2+i-1} and for any b in {xn/2+i, xn/2+i+1, …, xn-1}, it holds that a ≥ b.

37. Unique Crossover Property
The proof is much simpler than the statement.
Let i be the smallest index such that xi > xn/2+i with 0 ≤ i ≤ n/2-1.
It then holds that for any 0 ≤ j ≤ i-1, x0 ≤ x1 ≤ ... ≤ xi-1 ≤ xn/2+i-1 ≤ xn/2+i-2 ≤... ≤ xn/2.
Question: Why?

38. Unique Crossover Property
The proof is much simpler than the statement.
Let i be the smallest index such that xi > xn/2+i with 0 ≤ i ≤ n/2-1.
It then holds that for any 0 ≤ j ≤ i-1, x0 ≤ x1 ≤ ... ≤ xi-1 ≤ xn/2+i-1 ≤ xn/2+i-2 ≤... ≤ xn/2.
Question: Why?
So property (1) holds.
Question: Verify the other three properties.

39. From Bitonic to Arbitrary
Recall that every 1, or 2 element sequence is also bitonic.
So, if we can create bitonic sequences of larger length from these shorter bitonic sequences, we can use the earlier network.
Question: If X and Y are two sorted sequences, how to create a bitonic sequence with elements of X and Y.

40. From Bitonic to Arbitrary
Recall that every 1, or 2 element sequence is also bitonic.
So, if we can create bitonic sequences of larger length from these shorter bitonic sequences, we can use the earlier network.
Question: If X and Y are two sorted sequences, how to create a bitonic sequence with elements of X and Y.
Answer: Concatenate X with Reverse(Y).

41. The Overall Sorting Network
Built in a bottom-up fashion as follows.
Consider sorting n/2 bitonic sequences of length 2 each.
Each uses the network designed earlier.
Each produces a bitonic sequence of length 4.
Now, sort pairs of bitonic sequences each of length 4.
Continue building up to two bitonic sequence of length n/2 each.
Combine the two sequences to a single bitonic sequence of n elements.
Finally, use the bitonic sort network.

43. Another Example – The Closest Pair
Consider a set of n points in k-dimensions.
Let the distance between any two points be their Euclidean distance.
The closest pair problem is to find a pair of points whose pairwise distance is the smallest.

44. The Closest Pair Problem
For any k, one can always compute the n(n-1)/2 pairwise distances, and take the smallest.
This takes time O(kn2).
The k refers to the time taken to compute the distance in a k-dimensional space.
We will show today using divide and conquer that better solutions can be designed.

45. In One Dimension If k = 1, all points are on a line.
One solution is to sort the points, and compute adjacent distances.
This takes O(nlog n) time.
Turns out that this is also the best one can achieve.
Unfortunately, the solution does not extend to more than one dimension.

46. In One Dimension
Let us design a generic algorithm using the divide and conquer strategy that also extends to any dimension (with minimal changes).
The divide step intuitively seems to be to find the closest pair in the first n/2 points, and the next n/2 points.
Since the subproblems can be solved recursively, we just have to focus on the combine step.

47. In One Dimension

48. In One Dimension
Let dl be the shortest distance in the first subproblem.
Let dr be the shortest distance in the second subproblem.
The closest pair is either the pair at distance dl from the first subproblem, or at distance dr from the second subproblem,

49. In One Dimension
Let dl be the shortest distance in the first subproblem.
Let dr be the shortest distance in the second subproblem.
The closest pair is either the pair at distance dl from the first subproblem, or at distance dr from the second subproblem,
Or a pair that crosses the subproblems at a distance less than min{dl, dr}.
How to quickly find if such a pair exists?

50. In One Dimension In the one dimensional case, this is rather easy.
Find the largest point from the first subproblem and the smallest point from the second subproblem. Call these as x and y.
If y – x < min{dl, dr}, then x and y are the closest pair.
Otherwise, the closest pair is the one with mutual distance min{dl, dr}.

51. In One Dimension The algorithm is as follows: Algorithm ClosestPair(A)
Begin
1. Find the median point m.
2. Find the closest pair in the points that are
less than m, recursively.
3. Find the closest pair in the points that are
more than m, recursively.
4. Compute dl, dr from Solutions to
problems in Steps 2 and 3 respectively.
5. Find the pair, and the distance d, with one
point less than m and the other more than m.
6. Return the closest pair from Steps 2, 3, and 5.
End.

52. In One Dimension
To analyze, notice that each subproblem takes time T(n/2).
The combine step takes O(n) time
Why?
The recurrence relation is T(n) = 2T(n/2) + O(n).
The solution is Q(nlog n), according to Masters theorem.

53. Extend to Two Dimensions
Problem 1: How to divide so that each subproblem has n/2 points.
Problem 2: How to combine the solutions to the subproblems.

54. Two Dimensions
Can pick the median m of the x-coordinates and draw a vertical line y = m.
Points to the left and the right of m define the subproblems.

55. Two Dimensions
Can now find the closest points in each subproblem and denote the distances by dl and dr as earlier.

56. Two Dimensions
How many pairs do we have to consider during the combine step.

57. Two Dimensions
How many pairs do we have to consider during the combine step.
Ans: n2/4

58. Two Dimensions So, the recurrence now becomes T(n) = 2T(n/2) + O(n2).
With T(n) = Q(n2).
No savings accrued with the divide and conquer approach.
Unless there is a clever combine step.

59. Two Dimensions
Notice that while there may be as many as n2/4 pairs that are yet uncompared, there is some structure in them.
Also, dl and dr throw additional light.
Need to consider pairs that are at a distance less than d* = min{dl, dr}.
Intuition: As we move along increasing y coordinates of points on one side, the distance from a fixed point P increases.

60. Two Dimensions Fix a point P on one side of the line y = m.
How many points can pair up with P to have a distance (with P) at most d*, and their pairwise distance at least d*.
Why is the second condition true?

61. Two Dimensions Can approximate (Upper bound) the answer as follows.
Consider a rectangle of height 2d* and width d* with at most six points on the boundary.
Actually six is quite an over count.

62. Two Dimensions Call these as the friends of P.
For each P to the left of the line y = m, there are six friends.
Can find these six distances in O(n) time.
With some care, we can actually find these friends for all P in O(n) time.
So, the recurrence now is T(n) = 2T(n/2) + O(n), with a solutio of Q(nlog n).

63. Two Dimensions The clever step explained:
Project all points onto the line y = m.
Sort the points to the left of line and the right of the line by their y-coordinate.
For each P on the left, find all the points whose projection to y = m is at a distance at most d* from P.
To get these sorted lists, can pre-sort once.

64. Higher Dimensions The same solution extends to higher dimensions.
Notice that the combine step is operating in one dimension smaller that the input dimension.
However, the runtime will be O(nlogd-1 n).
Can be improved to O(nlog n) with more ideas.

65. Another Example – Skyline Points
A point P dominates a point Q if the x- and the y-coordinates of P are larger than that of Q.
Points that are NOT dominated by any other points are said to be maximal points, or skyline points

66. Skyline Points
Given a set of n points in a two dimensional space, find all the skyline (maximal) points.
Use divide and conquer.

67. Skyline Points
Divide the points into two equal sized sets.

68. Skyline Points
Solve each part recursively.

69. Skyline Points
Combine the solutions (skylines)

70. Skyline Points Slight asymptotic improvements possible.
Solve the right subproblem first,
Filter the points in the left subproblem that are dominated by skyline points in the right subproblem.
Solve only the remaining part of the left subproblem.