Basic Algorithms

5.2 Divide-and-Conquer

The idea of divide and conquer is that we solve a large problem by solving a number of smaller problems and then we apply this idea recursively.

• A common example would be that to solve a single problem of size N we need to solve two problems each of size approximately N/2.
  
  
• When we apply the recursion again, we wind up with four problems each of size approximately N/4.
  
  
• In addition to the time required to solve the subproblems, we need to include the time to split the original problem into pieces and the time required to combine the solutions to the subproblems into a solution for the original problem.

5.2.1 Divide-and-Conquer Recurrence Equations

From the description above we see that the complexity of a divide and conquer solution has three parts.

1. The time required to split the problem into subproblems.
2. The time required to solve all the subproblems.
3. The time required to combine the subproblem solutions

Let T(N) be the time required to solve an instance of the problem having time N. The time required to split the problem and combine the subproblems is typically a function of N, say f(N).

More interesting is the time required to solve the subproblems. If the problem has been split in half then the time required for each subproblem is T(N/2).

Since the total time required includes splitting, solving both subproblems, and combining we get.

    T(N) = 2T(N/2)+f(N)

Very often the splitting and combining are fast, specifically linear in N. Then we get

    T(N) = 2T(N/2)+bN

What if N is not divisible by 2? We should be using floor or ceiling or something, but we won't. Instead we will be assuming for recurrences like this one that N is a power of two so that we can keep dividing by 2 and get an integer. (The general case is not more difficult; but is more tedious)

But that is crazy! There is no integer that can be divided by 2 forever and still give an integer! At some point we will get to 1, the so called base case. But when N=1, the problem is almost always trivial and has a O(1) solution. So we write either

           b              if N = 1
    T(N) = 
           2T(N/2)+bN   if N > 1

    T(1) = b
    T(N) = 2T(N/2)+bN   if N > 1

No we will now see three techniques that, when cleverly applied, can solve a number of problems. We will also see a theorem that, when its conditions are met, gives a solution without our being clever.

The Iterative Substitution Method

Also called ``plug and chug'' since we plug the equation into itself and chug along.

    T(N) = 2T(N/2)+bN
         = 2[    ]+bN
         = 2[2T((N/2)/2)+b(N/2)]+bN
         = 4T(N/4)+2bN    now do it again
         = 8T(N/8)+3bN

A flash of inspiration is now needed. When the smoke clears we get

    T(N) = 2iT(N/2i)+ibN

When i=log(N), N/2ⁱ=1 and we have the base case. This gives the final result

    T(N) = 2log(N)T(N/2log(N))+ibN
         = N    T(1)      +log(N)bN
         = Nb +log(N)bN

The Recursion Tree

The idea is similar but we use a visual approach.

We already studied the recursion tree when we analyzed merge-sort. Let's look at it again.

The diagram shows the various subproblems that are executed, with their sizes. (It is not always true that we can divide the problem so evenly as shown.) Then we show that the splitting and combining that occur at each node (plus calling the recursive routine) only take time linear in the number of elements. For each level of the tree the number of elements is N so the time for all the nodes on that level is Θ(N) and we need to find the height of the tree. When the tree is split so evenly the sizes of all the nodes on each level go down by a factor of two so we reach a node with size 1 in logN levels (assuming N is a power of 2). Thus T(N) is Θ(Nlog(N)).

The Guess and Test Method

This method is really only useful after you have practice in recurrences. The idea is that, when confronted with a new problem, you recognize that it is similar to a problem you have seen the solution to previously and you guess that the solution to the new problem is similar to the old.

But we don't have enough experience for this to be very useful.

The Master Method

In this section, we apply the heavy artillery. The following theorem, which we will not prove, enables us to solve some problems by just plugging in.

We will only be considering complexities of the form

    T(1) = c
    T(N) = aT(N/b)+f(N)   if N > 1

The idea is that we have done some sort of divide and conquer where there are a subproblems of size at most N/b. As mentioned earlier f(N) accounts for the time to divide the problem into subproblems and to combine the subproblem solutions.

Theorem [The Master Theorem]: Let f(N) and T(N) be as above.

1. (f is small) If there is a constant ε>0 such that f(N) is O(N^log_ba/n^ε), then T(N) is Θ(N^log_ba).
2. (f is medium) If there is a constant k≥0 such that f(N) is θ(N^log_ba(logN)^k), then T(N) is θ(N^log_ba(logN)^k+1).
3. (f is large) If there are constants ε>0 and &delta<1 such that f(N) is Ω(N^log_baN^ε) and af(N/b)≤δf(N), then T(N) is Θ(f(N)).

Proof: Not given.

Remarks:

1. When we say f is small/medium/large we are comparing it to the mysterious N^log_ba, which is the star of the show.
   
   
2. So when f is small, T is polynomial; when f is medium, T is polynomial times a log power; and when f is big, T is f.
   
   
3. N^log_ba/N^ε can also be written as N^log_ba-ε and N^log_baN^ε can also be written as N^log_ba+ε.

Now we can solve some problems easily and will do two serious problems after that.

Example: T(N) = 4T(N/2)+N.

• a=4, b=2, and f(N)=N
• So N^log_ba = N^log₂4 = N².
• f is small compared to this. Specifically, f(N) = N = N^2-1; so we are in case 1 with ε=1.
• Hence T(N) = Θ(N^log_ba) = Θ(N²).

Example: T(N) = 2T(N/2) = Nlog(N)

• a=2, b=2, and f(N)=Nlog(N)
• So we have case 2 of the master theorem with k=1.
• Hence T(N) = Θ(N(logN)²).

Example: T(N) = T(N/4) + 2N

• a=1, b=4, and f(N)=2N
• log_ba = log₄1 = 0
• N^log_b = N⁰ = 1
• We have case 3 since f(N) is Ω(N^0+ε) for &epsilon=1 and af(N/b) = (1)2(N/4) = N/2 = (1/4)f(N)
• Hence T(N) = Θ(N)

Example: T(N) = 9T(N/3) + N^2.5

• a=9, b=3, f(N)=N^2.5
• log_ba = 2, so N^log_b = N²
• We have case 3 since f(N) is Ω(N^0+ε) for &epsilon=.5 and af(N/b)=9f(N/3)= 9(N/3)^2.5=(1/3)^0.5f(n).
• Hence T(n) is O(N^2.5)

Homework: R-5.4

5.2.2 Integer Multiplication

We want to multiply big integers.

When we multiply without machine help, we use as knowledge the times tables for all 1 digit numbers and then do the standard 5th-grade algorithm that requires Θ(N²) steps for N digit numbers. On a computer the known tables are for all 32-bit numbers (or 64-bit on some machines). If we want to multiply two 32N-bit numbers on the computer we could write the 5-th grade algorithm in software (using 32-bit numbers as ``digits'') and again compute the result in Θ(N²) time.

To make the wording easier lets assume we only know how to multiply 1-bit numbers and we want to multiply two N-bit numbers X and Y. Using the 5th grade algorithm we can do this in time Θ(N²). We want to go faster.

If we simply divide the bits in half (the high order and the low order), we see that to multiply X and Y we need to compute 4 sub-products Xhi*Yhi+Xhi*Ylo+Xlo*Yhi+Xlo*Ylo. (Note that when I write the products above I don't include the powers of 2 needed. This corresponds to writing the numbers in the correct column with the 5th grade algorithm.)

Since addition of K-bit numbers is Θ(K) and our multiplications are only of N/2-bit values we get

   T(N) = 4T(N/2)+cn

We compute Xhi*Yhi, Xlo*Ylo, and the miraculous (Xhi-Xlo)*(Ylo-Yhi). The miracle multiplication produces the two remaining terms we need but has added to this two other terms. But those are just the (negative of) the two terms we did compute so we can add them back and they cancel.

The summary is that T(N) = 3T(N/2)+cN, which by the master theorem (case 1) gives T(N) is Θ(N^log₂3N). But log₂3<1.585 so we get the surprising

Theorem: We can multiple two N-bit numbers in O(N^1.585) time.

5.2.3 Matrix Multiplication

If you thought the integer multiplication involved pulling a rabbit out of our hat, get ready.

This algorithm was a sensation when it was discovered by Strassen. The standard algorithm for multiply two NxN matrices is Θ(N³). We want to do better

First try. Assume N is a power of 2 and break each matrix into 4 parts as shown on the right. Then X = AE+BG and similarly for Y, Z, and W. This gives 8 multiplications of half size matrices plus Θ(N²) scalar addition so

    T(N) = 8T(N/2) + bN2

But strassen found the way! He (somehow, I don't know how) decided to consider the following 7 (not 8) multiplications of half size matrices.

S1 = A(F-H)       S2 = (A+B)H       S3 = (C+D)E       S4 = D(G-E)
S5 = (A+D)(E+H)   S6 = (B-D)(G+H)   S7 = (A-C)(E+F)

Now we can compute X, Y, Z, and W from the S's

X = S5+S6+S4-S2     Y = S1+S2    Z = S3+S4    W = S1-S7-S3+S5

This computation shows that

    T(N) = 7T(N/2) + bN2

Thus the master theorem now gives

Theorem(Strassen): We can multiply two NxN matrices in time O(n^log7).

Remarks:

1. log7<2.808 so we can multiply NxN matrices in time O(N^2.808), which is o(N³). Amazing.
2. There are even better algorithms, which are more complicated. I think they break the matrix into more pieces. The current best is O(N^2.376).