Lecture 4--Asymptotic growth of functions; induction; recursion. 10/3/97

=============================================================================

Section 4.1. Asymptotic growth of functions.

In the formal analysis of algorithms, we try to measure how the resources required to solve a problem are related to the problem size. For example, if we are sorting a list L of keys, the size of the problem is the number of keys in L. If we are trying to find the shortest path between two vertices in a graph G, the size of the problem is actually an ordered pair (N_v,N_e), where N_v is the number of vertices in G and N_e is the number of edges in G. Typically we are trying to find out how the resources needed to solve the problem grow with problem size. For example, we want to make statements about the resources required to sort a list of length n, where n is any integer >= 1.

Now suppose we have a problem P of size n and two algorithms, A1, and A2, that will give a solution to P. Suppose we know that algorithm A1 requires us to execute cn steps and algorithm A2 takes dn² steps. If c = 1000 and d = 2, then for small values of n, A2 is faster. In fact, A2 is better whenever

2n <= 1000n or

n <= 500.

But as n gets larger A1 will clearly be faster. We say that the time taken by A2 grows faster than the time taken by A1, or that the time taken by A2 is of a larger order than the time taken by A1, so "asymptotically" A1 is faster.

To express this kind of relationship between growth rates of functions, computer science has borrowed notation originally developed for problems in analytic number theory. For computer science purposes we will need to define four types of relationships between functions. These will be denoted by the letters O, "capital omega", "capital theta", and o. The first three will be the most important for elementary algorithm analysis.

Definitions. We assume f and g are positive functions defined on the positive integers (these are the conditions we are interested here--the domain of positive integers represents problem sizes and the function values represent resources such as time units and memory cells).

1. f = O(g) means that the rate of growth of f is essentially slower than the rate of growth of g. We sometimes say that the order of g is greater thatn the order of f. Formally, f = O(g) if there exists a constant c > 0 such that for all n >= 1.

f(n) <= cg(n).

Examples. If g(n) = 3n², then f = O(g) if f is any one of the functions:

n² ; 1000n² ; 8n² + 10n ; 16nlog₂n ; 50000n ; e^-n ; 84n^2-x, any x > 0.

2. f = "capital omega"(g) means that the rate of growth of f is essentially at least as fast as the rate of growth of g. Formally, f = "capital omega"(g) if there is a constant c >= 0 such that for all n >= 1.

f(n) >= cg(n).

Examples. Again let us take g = 3n². Then f = "capital omega"(g) if f is any one of the functions:

n² ; 1000n² ; 8n² + 10n ; 16n²log₂n ; 50000n³ ; eⁿ ; 84n^2+x, any x > 0.

3. f = "capital theta"(g) means that f and g grow at the same rate, or are of the same order. Formally, f = "capital theta"(g) if there exist constants c and d such that for all n >= 1.

cf(n) <= g(n) <= df(n).

Examples. For g = 3n², the functions n², 1000n², and 8n² + 10n are all "capital theta"(g).

(4. f = o(g) if lim_n->oo(f(n) / g(n)) = 0. For example, if g = 3n², then f = o(g) if f is any one of the functions

16nlog₂n ; e^-n ; 84n^2-x , any x > 0.

In analysis of algorithms, this relationship is used less often than 1-3.)

The relations between functions which we can express with the above notation allow us to discuss the relative growth rates of two functions without being specific about the constants involved. As we have already mentioned, in practice these constants often cannot be ignored.

In some cases, two functions f and g can be compared according to the above definitions except for a finite set of values of n. Thus we could rewrite the above definitions to be slightly more general by replacing the phrase "for all n >= 1" with the phrase "for all n sufficiently large".

Some authors replace the equal sign in the above relations by the mathematical symbol for set membership, writing, for example, f "is a member of" O(g) instead of f = O(g). While this is technically correct, it limits the usefulness of the notation in doing calculations, so we will keep to the notation as it was originally developed.

Exercise 4.1. List the following functions from smallest to largest order. If two functions are of the same order, make that clear:

log₂n, eⁿ, ln(n), 2ⁿ, n!, n^1-x for any x > 0, log₇n, log₇n², n, n^1+x for any x >: 0,

(log₂n)^k for any integer k >= 1, nlog₂n, nln(n), log₂log₂n.

Exercise 4.2. Suppose Algorithm 1 requires nlog₂n steps, algorithm 2 requires n² steps, and algorithm 3 requires 2ⁿ steps to solve a given problem P. Suppose we are running each algorithm on a computer which executes one instruction in 10 nanoseconds. How long will it take each algorithm to solve problem P for each value n = 10^k, 1 <= k <= 7?.

Exercise 4.3. Prove: a. f = O(g) <=> g = "capital omega"(f).

b. f = "capital theta"(g) <=> g = "capital theta"(f).

c. If p(n) is a polynomial of degree k with leading coefficient a_k > 0, show

that p(n) = O(n^k).

d. (grad) repeat part c with O replaced by "capital theta".

Exercise 4.4 (grad). Prove that if f₁ = O(g₁) and f₂ = O(g₂) then f₁ + f₂ = O(max(g₁,g₂)).

Section 4.2. Induction. The proof technique known as induction is often useful in analyzing the bahavior of an algorithm. Recall that a proof by induction has three steps:

Step 1: Prove that an expression holds for a fixed value N₀ of n (often N₀ = 0 or N₀ = 1).

Step 2: Prove that if the expression is true for n <= N then it is also true for n = N + 1.

Step 3: Conclude from steps 1 and 2 that the expression is true for all N >= N₀.

We give two example of proofs by induction here.

Example 1. "SUM"_(0<=i<=N)i = N(N+1) / 2 .

Proof. Step 1. The equation is true for N = 0. It is also true for N = 1, since the l.h.s. =

1 and the r.h.s. = 1.

Step 2. Assume "SUM"_(0<=i<=M)i = M(M+1) / 2 for M <= N..

Then "SUM"_(0<=i<=N+1)i = "SUM"_(0<=i<=N)i + (N+1)

= N(N+1) / 2 + (N+1)

= (N+1)(N / 2 + 1)

= (N+1)(N+2)/2

= (N+1)(N+1+1)/2.

Step 3. Based on steps 1 and 2 we conclude the equation is true for all N >= 0.

Example 2. We can use this technique to prove several useful facts about binary trees. We use the standard definitions for binary tree, root, leaf, internal node, etc. Recall that the level of the root is defined to be 0, while the level of any other node is defined to be 1 + the level of its parent. The depth of the tree is the maximum level of any node in the tree. A complete binary tree is one in which all internal nodes have two children and all leaves are at the same level.

Useful facts about binary trees:

T1. A binary tree has at most 2^l nodes at level l.

T2. A binary tree with height h has at most 2^h+1 -1 nodes.

T3. A binary tree with n nodes has height at least |_ log₂n_|.

Proof of T1. Suppose T is a binary tree with (nonempty) levels 0,1, .....N..

Step 1. For l = 0, T has exactly one node, so T1 is true for l = 0.

Step 2. Assume T has at most 2^l nodes at level l, 0 <= l < N. Since every node at level l can have at most 2 children, the number of nodes at level l+1 is at most

2*(number of nodes at level l) <= 2*(2^l) = 2^l+1.

So we have completed step 2.

Step 3. Since we have verified the necessary assumptions in Steps 1 and 2, we have proved statement T1.

Note that T2 and T3 can also be proved by induction.

Exercise 4.5. Prove "SUM" _0<=i<=N i² = N(N+1)(2N+1) / 6.

Exercise 4.6. (grad). Prove T2.

Section 4.3. Recursion. For development and understanding it is often useful to express an algorithm in a recursive form. For a production program it may be more efficient to rewrite the program to do sequential iteration. This amounts to managing the stack directly rather than leaving the management to function calls. When a recursive program is rewritten using sequential iteration, the stacking mechanism can sometimes be removed entirely. In some cases the use of recursion may lead to an inefficient implementation, and for such problems sequential iteration should always be used.

Exercise 4.7. Write recursive and nonrecursive versions (in pseudocode) to calculate n!. What is the relative efficiency of your algorithms in terms of number of statements executed? Which is preferred?

Exercise 4.8. Explain why computing the nth Fibonacci number Fib(n) recursively is not a

good idea. (Recall that Fib(0) = 0, Fib(1) = 1, Fib(n) = Fib(n-1) + Fib(n-2) for n >= 2.)