Lecture 8--divide and conquer: binary search. Lower bounds. 10/11/97

================================================================

8.1. The divide and conquer technique. We can describe the binary search algorithm for a sorted array, presented in lecture 6, in pseudocode as follows:

     While we have not found X and there are more array items to examine

          compare X with the item in the middle of the remaining items.

          If X = this item, we are done

          else

               if X > this item, throw away the bottom half of the list

               else throw away the top half of the list

This algorithm is a very simple application of the general algorithm strategy known as "divide and conquer." More generally, we can state this method as follows:

Divide and conquer for a problem of size n:

1. Divide the problem into k smaller subproblems, each of size m_j, for 1 <= j <= k..

2. Solve each of the k subproblems, using divide and conquer recursively, in time T_j , 1 <= j <= k.

3. Put these results back together to find an answer to the original problem.

This technique has a straightforward recursive implementation, where step 2 becomes a sequence of k recursive calls, one to solve each of the smaller subproblems.

Divide and conquer is useful when the smaller subproblems are "easier" to solve than the original one, in terms of time and/or space usage.

In some cases it is more efficient to use divide and conquer until the smaller problems reach a certain size and then to use a more direct method on these problems. We will see an example of this in the quicksort algorithm, which we will analyze later.

Example 1. Binary search. For binary search we have:

1. We divide the original array into k = 2 smaller arrays, each of size approximately n / 2.

2. For one of these arrays there is no work to be done, since we know X cannot be in it. Suppose, for example, X > the middle element. Then T₁ = 0 and T₂ = T( n/2).

3. Since we "throw away" one of the two smaller sets each time, step 3 is trivial in this case.

Note that this way of looking at the problem leads immediately to the recurrence relation

T(n) = T ( |_ n / 2 _| ) + c, T(1) = d, for constants c and d,

and we have seen that this can be solved to give T(n) = "capital theta" (log₂ n).

Example 2. Another example of "divide and conquer" is the procedure for searching a binary search tree for the presence of an item X. Beginning with the root node, we examine the node we are at for X. If X is not at this node, we know we can ignore one branch of the tree as we continue our search. How quickly the algorithm finishes, in the worst case, depends on how close to minimal the height of the tree is.

Example 3. Later we will look at the mergesort algorithm in detail. But we have already seen the pseudocode for this algorithm in the programming assignment. This is also an example of the use of divide and conquer:

Step 1. Divide the array of size n into 2 arrays, each of size n / 2.

Step 2. Mergeort each of these arrays; each can be mergesorted in time T( n / 2).

Step 3. Merge the sorted arrays.

Exercise 8.1.

a. Based on the pseudocode in the programming assignment, explain why step 3 in mergesort takes time "capital theta" (n).

b. Explain why the time T(n) to sort an array of n items using mergesort can be expressed as

T(n) = 2T(n / 2) + dn for a constant d.

c. Solve the recurrence relation in step b.

Exercise 8.2 What other sort in the programming assignment uses the divide and conquer technique? Give the appropriate recurrence relation for this sort if possible.

Exercise 8.3. Suppose we have an algorithm for solving a problem P of size n which uses divide and conquer in the following way:

Divide the problem into a problems, each of size approximately n / a , where a is a fixed integer > 1. Solve each of these problems recursively. Then put the results back together, in time dn, for a fixed constant d, to solve the original problem.

a. Give a recurrence relation for the time T(n) to solve the problem. You may assume T(1) = d.

b. Derive a closed form expression for this recurrence relation.

Exercise 8.4. (grad) In Algorithms 2 we will discuss ways to speed up computer arithmetic. In these problems we will be working at the bit level, i.e., our inputs will generally be two integers, each representable in n bits. We shall see that in general addition is "faster" than multiplication, i.e., we can add two n-bit numbers much more quickly than we can find their product. This fact has led to many algorithms to try to speed up multiplication. One of these is a "recursive" algorithm, which can be described as follows:

Suppose X and Y are 2 2n-bit (positive) integers with

X = 2ⁿx₁ + x₂ ,

Y = 2ⁿy₁ + y₂,

i.e., x₁ and y₁ represent the high-order bits of X and Y, and x₂ and y₂ represent the low-order bits. Then we can form the product in two different ways:

m1: XY = 2²ⁿx₁y₁ + 2ⁿ(x₁y₂ + x₂y₁) + x₂y₂

m2: XY = 2ⁿ(x₁ + x₂ ) ( y₁ + y₂ ) + ( 2²ⁿ - 2ⁿ ) x₁y₁ - (2ⁿ - 1) x₂y₂.

a. Give a recurrence relation for the number of multiplications needed to compute XY according to m1 and a recurrence relation for the number of multiplications needed to compute XY according to m2. Do not count time for additions, subtractions, or multiplying by a power of 2 (i.e., a shift).

b. Give a closed form solution for each of the recurrence relations derived in a.

8.2. Lower bounds. When we have an algorithm A to solve a problem P and when we can compute an upper bound on the resources required for A, as a function of the problem size n, we are essentially computing an upper bound on the time it takes to solve P. For example, when we developed the sequential search algorithm to look for an item X in an n-element array, our analysis showed that sequential search could be implemented in time proportional to n and with space usage proportional to n. So we could say that it was possible to solve the problem of finding out whether or not an element X was in an n-element set in time and space O(n). When we developed the binary search algorithm, we showed that for an ordered set, this same problem could be solved in time O(log2n) and space O(n). A related problem is to ask "is this the best we can do"? That is, are these lower bounds on the time and space required, i.e., will it always take time which is growing as log2n and space which is growing as n? It is easy to see that we need at least "capital omega" (n) space, since we need to store n elements. But what about time? To answer this question we cannot just give an algorithm, since we want to answer it "no matter what the algorithm is".

In general it is not possible to answer this question--it is not well defined enough. But we can give the following result:

Theorem. Suppose we have an array L and an algorithm A which searches L for a given element X. Suppose we count only the number of comparisons which A does (i.e., we do not count any time A spends moving items around in the array, for example). Then the number of comparisons done by A, for some input, will be at least |_ log₂ n _| + 1.

Corollary. Since we have shown that binary search does |_ log_2i n _| + 1 comparisons in the worst case, binary search is an optimal algorithm for the problem of determining whether or not an element X is an a list L, if we count only the comparisons which the algorithm does.

Proof of theorem. We construct a binary tree which is called a "decision tree" for A and represents the number of comparisons A does for a given input list L. We assume the algorithm A stops if A has compared X to L[j] for all j and stops when X = L[i] for some i. We label the root of the tree with i, where i is the index of the first entry in the tree which A examines. Now if a node is labeled j, then we construct and label its children as follows: Label the left child of the node with the index of the next list element A will compare with X if X < L[j]. If A stops in this case, then the node has no left child. Similarly label the right child with the index of the next list element A will compare with X if X > L[j], and if A stops in this case there is no right child. Now for a given input the comparisons A will make looking for X will trace out a path in this binary tree. The number of comparisons is the number or nodes on the path, so the most comparisons A can ever do, in any case, is the length of the longest path in a tree which can be constructed by the above rules. Note also that when A stops the path ends, i.e., we are at a leaf node.

Now the number of nodes in the tree, N, will be >= n. For suppose that this is not so, i.e., suppose that N < n. Then there must exist a node r, 0 <= r <= n-1 (since we are C or C++ programmers) for which no node in the tree has label r. But then we can "fool" algorithm A by constructing two input lists, L1 and L2, which have L1[i] = L2[i} for 0 <= i <= n -1, I <> r, and L1[r] = X, L2[r] <> X (and also, X does not appear in any other position in L1 or L2). Because node r is not a label in the decision tree, A never compares X with L1[r] or L2[r}. So either for L1 or for L2 A must give the wrong output. Thus we must conclude that there must be a node labeled r, i.e., that N >= n.

Now by our results on trees proved earlier we know that the tree must have depth d which

satisfies

d >= |_ log ₂ n _| .

Therefore A must visit at least |_ log ₂ n _| + 1 nodes.

(Note that the decision tree for A might have MORE than n nodes. We have only shown that it has at least n nodes).

Exercise 8.5. Suppose n = 10. Then the decision tree for any algorithm A to search an array with 10 elements will have at least 10 nodes and there will be at least one node labeled i for 0 <= i <= 9.

a. Show the decision tree for binary search.

b. Show the decision tree for sequential search.