Lecture 10. Sorting; O(n²) sorts. 10/17/97.

10.1. Brief review. Before starting our discussion of sorting, we review what we have covered so far. Please make sure you understand the following concepts:

Useful tools:

--make sure you know the techniques to answer all the questions on the diagnostic test.

--know the meaning of O, "capital theta", "capital omega", and o. Know how to decide how two functions f and g are related according to this notation.

--know how to do a proof by induction.

--know how to "guess" a closed form for a recurrence relation (like those covered in the notes) and how to prove that your guess is correct using induction.

--be able to rewrite a recursive algorithm in iterative form and vice versa.

--properties of binary trees.

Algorithm design:

--understand the "divide and conquer" method.

Algorithm analysis:

--understand the informal "sequential machine" model we are analyzing our algorithms with respect to

--be able to analyze the time and space requirements of simple algorithms either by direct counting or by deriving an appropriate recurrence relation which can be put in closed form.

--know the major important ways to evaluate the efficiency and appropriateness of an algorithm:

----------execution time (O, "capital theta", "capital omega" results)

----------space requirements

----------actual execution time ("what are the constants"?)

----------difficulty of coding versus number of times the program will be run (including deciding whether to remove recursion from the program)

----------any special characteristics of the data which will be input

Simple algorithms:

--finding min, max element in a list, e.g.

Searching:

--relation of searching to data structure the data is in--array, linked list, binary tree; is the data sorted or unsorted?

--linear search--worst case and average behavior

--binary search (sorted array)--worst case and average behavior

--lower bound on the number of comparisons required to search a list of elements

--order statistics--worst case behavior

10.2. Sorting. Recall that we mentioned early in the course that the problems of searching and sorting are two of the most common that need to be solved in computer science. We have looked at searching. Now we will look at some of the fundamental techniques for sorting.

As in the searching problem we can use different data structures to store our data. For example, we can store the data in a linked list. The most natural sort to use in this case in insertion sort, which we will discuss here.

Exercise 10.1. Assume L is a linked list with entries S containing two fields, S.info and S.next. Assume the data stored in the info fields contains a key allowing two data items to be compared.

a. Write pseudocode to do insertion sort on L. Assume you have pointers head and current. Make your algorithm recursive.

b. Derive a recurrence relation for T(n) the time it takes to sort L if L contains n entries.

c. Find a closed form for T(n).

We can also consider using a tree structure to sort our data. If the data is already stored in a tree but is unsorted, this would be an unpleasant task. A more natural problem would involve constructing a binary search tree, i.e., sorting the data as it is added to the tree.

Exercise 10.2. Suppose we construct a binary search tree to hold the integers 0, 1, 2, ..., n-1, which are read in IN THAT ORDER. (Assume if the next item is > the item stored at a node you go right, otherwise you go left). a. How many steps will it take to add item i, 0 <= i <= n-1?

b. What is the time to construct this tree?

Exercise 10.3. (grad). Give an order for the integers 0, ..., n-1 so that if the integers are read in the order you have given the binary search tree constructed will be of minimal height. What is the time to construct the tree in this case?

For the remainder of our discussion of sorting we will assume that our data is stored in an array A = A[0], ... , A[n-1]. We will discuss three type of sorting algorithms:

1. "Internal sorts", where the entire array fits in main memory. we look at two subclasses of internal sorts:

1a. "Comparison sorts", where the sorting procedure is done using comparisons between list elements.

1b. "Bucket sorts", where the structure of the data items being sorted is used instead of comparisons between list items.

2. "External sorts", where the array is too large to fit entirely in memory. we will need to use some strategy which allows us to sort only part of the array at one time.

10.3. Simple comparison sorts.

There are several sorts that fall into this category. Typically they use a strategy which requires a double loop structure:

(1) for each item in the array (or for each array position)

(2) find where that item goes (or find the item that goes in that position)

Typically step 1 (the outer loop) is iterated n times, "for i = 0 to n-1" while step 2 (the inner loop) is a loop of comparisons iterated i times, "for j = 0 to i" . Therefore the time T(n) required to do the sort will be proportional to the number of comparisons done and will be expressible as

T(n) = "capital theta" ( "SUM"_{0 <= i <= n-1} ("SUM" _{0 <= j <= i} c)) = "SUM"_{0 <= i <= n-1} (ci) )

= "capital theta" ( cn²).

Some sorting algorithms which fit this model include bubble sort, insertion sort, and selection sort. Your programming assignment includes code for insertion sort and a modified form of bubble sort.

Insertion sort. The pseudocode in your programming assignment shows how to implement this algorithm by moving items within the array. (Above you implemented the same algorithm recursively for a linked list).

Example. Suppose the array A contains 10 integers, 6,3,8,1,4, 2,7,9,0,5

index = 1: set x = A[1] = 3 and j = index-1 = 0

while j >= 0 and A[j] > x

A[0] > 3 so set A[1] = A[0] = 6 and j = j - 1 = -1.

Now j < 0 so assign x to A[j+1] = A[0]

Array now contains 3,6,8,1,4,2,7,9,0,5

index = 2: set x = A[2] = 8 and j = index - 1 = 1.

while j >= 0 and A[j] > x

A[1] = 3 is not greater than 8 so inner loop is not executed at all

Set A[2] = x--i.e., no change.

index = 3: set x = A[3] = 1 and j = index - 1 = 2.

while j >= 0 and A[j] > x

A[2] = 8 > 1 so set A[3] = 8 and j = 1

A[1] = 6 > 1 so set A[2] = 6 and j = 0

A[0] = 3 > 1 so set A[1] = 3 and j = -1.

Now j < 0 so set A[0] = 1.

Now A contains 1,3,6,8,4,2,7,9,0,5.

So, for each value of index, A[index] is placed in its proper position within the range of positions 0,1, ... , index.

We can prove by induction that insertion sort is a correct algorithm.

Exercise 10.4.

a. Show that the maximum number of comparisons between array elements in the "while" loop is i-1, when index = I.

b. Show that the total number of comparisons done, in the worst case, is n(n-1) / 2.

c. Give a convincing argument that insertion sort will take TOTAL time "capital theta"(n²) in the worst case.

d. Give an example of an array A with n elements which is a "worst case" for insertion sort.

We can also compute the average behavior of the insertion sort algorithm. It turns out to be n² / 4, so this is not a very interesting result. But we present it here because it is fairly straightforward and will help with our understanding of the more complex but also more interesting case of quicksort.

First we assume that all the array elements are distinct. (If some elements are equal, a similar proof will work, but the details are more complicated). Now what are the inputs to our algorithm? We will be given a sequence of n distinct items from an ordered set. For definiteness let us assume we are sorting integers. Then the values of the integers do not matter, since we are only doing comparisons; only the order of the input matters. So there are n! different possible inputs. We will assume that each of these is equally likely to be input.

Now for a fixed x = A[i] it is possible that x will be placed in any one of i + 1 positions ( 0 up to i) during the execution of the while loop. Since the algorithm has not previously looked at x, it is equally likely that x belongs in any one of these positions, i.e., x belongs in any one of these positions with probability 1 / (i+1). Now the number of comparisons done if x belongs in position j can be seen to be:

j:        0    1    2 . . .  i-1    i

compares: i    i    3  . . . 2      1

{ "SUM" _{0 <= j <= i-1} (1/(i+1)) * (j+1)} + (1/(i+1)) * i

= 1 / (i+1) { "SUM"_{1 <= j <= i} j } + (1/(i+1)) *i

= 1 / (i+1) * (i * (i + 1) / 2) + (1/(i+1)) * i

= i / 2 + 1 + 1 / (i + 1)

now summing over i gives

Average number of comparisons to sort using insertion sort =

"SUM" _{1 <= i <= n-1} (i / 2 + 1 + 1 / (i + 1) ) = "capital theta" (n²).

Bubble sort. In bubble sort, each element, starting with element 0, is swapped with its neighbor if the two are not in the proper order. So on the first pass the largest element "bubbles" to the last position, on the second pass the second largest element gets into the correct position, etc. Bubble sort requires "capital theta" (n²) time in the worst case, and on the average. Bubble sort is easy to program and so is a useful sorting algorithm for small arrays.

If any element is out of order, then there will be at least one swap done in the inner loop. The modification uses this fact by stopping the sorting process when no element is moved in one pass. So if an array is "almost sorted", it will be completely sorted in only a few passes of bubble sort. Thus modified bubble sort can finish in linear time ("capital theta"(n)) if the array has only a few records out of position. In the worst case and on the average, though, modified bubble sort also requires time "capital theta'(n²).

Exercise 10.5. Prove that the number of comparisons done in the worst case for modified bubble sort is "capital theta" (n²). Give an example of an array of n integers for which this worst case time is achieved.

Exercise 10.6.

a. For the pseudocode for insertion sort in the programming assignment, derive reasonable constants a,b,c to express the (worst case) running time of each algorithm in the form an² + bn + c

b. Repeat part a for modified bubble sort.