Basic Algorithms: Lecture 11

================ Start Lecture #11 ================

Notes. Vote on midterm date. It can be either next tuesday or next thurs.
The winner was next thursday 21 Oct.
Final exam is 21 dec 10am. Room TBA.
End of note

Problem Set #2, Problem 4.
Write

Algorithm compareLeaves(T1, T2)
    Input: Two binary trees T1 and T2

compareLeaves prints "fewer" if T1 has fewer leaves than T2; prints "equal" if they have the same number of leaves; and prints "less" if T1 has more leaves.

Hint: First write

Algorithm countLeaves(T)
    Input: A binary tree T.

Another hint countLeaves will probably begin with

    countLeaves1(T, T.root(), 0)

(the third parameter is the sum so far).
End of Problem Set #2, Problem 2.

Problem Set #2 assigned. Due in 2 weeks. which is after the midterm.

A linked structure for binary trees

Represent each node by a quadruple.

A reference to the parent (null if we are at the root).
A reference to the left child (null if at a leaf).
A reference to the right child (null if at a leaf).
The element().

Once again the algorithms are all O(1) except for positions() and elements(), which are Θ(n).

The space is Θ(n) which is much better that for the vector implementation. The constant is larger however since three pointers are stored for each position rather than one index.

A linked structure for general trees

The only difference is that we don't know how many children each node has. We could store k child pointers and say that we cannot process a tree having more than k children with the same parent.

We don't like limiting the number of children in this way. Moreover, if we choose k moderate, say k=10. We are limited to 10-ary trees and for 3-ary trees most of the space is wasted.

So instead of storing the child references in the node, we store just one reference to a container. The container has references to the children. Imagine implementing the container as an extendable array.

Since a node v contains an arbitrary number of children, say C_v, the complexity of the children(v) iterator is Θ(C_v).

2.4 Priority Queues and Heaps

2.4.1 The Priority Queue Abstract Data Type

Up to now we have not considered elements that must be retrieved in a fixed order. But often in practice we assign a priority to each item and want the most important (highest priority) item first. (For some reason that I don't know, low numbers are often used to represent high priority.)

For example consider processor scheduling from Operating Systems (202). The simplest scheduling policy is FCFS for which a queue of ready processors is appropriate. But if we want SJF (short job first) then we want to extract the ready process that has the smallest remaining time. Hence a FIFO queue is not appropriate.

For a non-computer example,consider managing your todo list. When you get another item to add, you decide on its importance (priority) and then insert the item into the todo list. When it comes time to perform an item, you want to remove the highest priority item. Again the behavior is not FIFO.

Ordering properties and comparators

To return items in order, we must know when one item is less than another. For real numbers this is of course obvious.

We assume that each item has a key on which the priority is to be based. For the SJF example given above, the key is the time remaining. For the todo example, the key is the importance.

We assume the existence of an order relation (often called a total order) written ≤ satisfying for all keys s, t, and u.

"Fully defined" Either s≤t or t≤s.
Reflexive: s≤s
Antisymmetric: s≤t and t≤s implies t=s.
Transitive: s≤t and t≤u implies s≤u.

Remark: For the complex numbers no such ordering exists that extends the natural ordering on the reals and imaginaries. This is unofficial (not part of 310).

Is it OK to define s≤t for all s and t?
No. That would not be antisymmetric.

Definition: A priority queue is a container of elements each of which has an associated key supporting the following methods.

InsertItem(k,e): Insert an element e with key k.
removeMin(): Return and remove an element with a minimal key, i.e., a key ≤ the key of any other element.
minElement(): Return an element with minimal key.
minKey(): Return a minimal key.

Comparators

Users may choose different comparison functions for the same data. For example, if the keys are the longitude and latitude of cities, one user may be interested in comparing longitudes and another latitudes. So we consider a general comparator containing methods.

isLess(a,b)
isLessOrEqual(a,b)
isEqual(a,b)
isGreater(a,b)
isGreaterOrEqual(a,b)
isComparable(a,b)

2.4.2 PQ-Sort, Selection-Sort, and Insertion-Sort

Given a priority queue it is trivial to sort a collection of elements. Just insert them and then do removeMin to get them in order. Written formally this is

Algorithm PQ-Sort(C,P)
   Input: an n element sequence C and an empty priority queue P
   Output: C with the elements sorted

   while not C.isEmpty() do
      e←C.removeFirst()
      P.insertItem(e,e)  // We are sorting on the element itself.

   while not P.isEmpty()
      C.insertLast(P.removeMin())

So whenever we give an implementation of a priority queue, we are also giving a sorting algorithm. Two obvious implementations of a priority queue give well known (but slow) sorts. A non-obvious implementation gives a fast sort. We begin with the obvious.

Implementing a priority queue with an unordered sequence

So insertItem() takes Θ(1) time and hence takes Θ(N) to insert all n items of C. But remove min, requires we go through the entire list. This requires time Θ(k) when there are k items in the list. Hence to remove all the items requires Θ(n+(n-1)+...+1) = Θ(N²) time.

This sorting algorithm is normally called selection sort since the dominant step is selecting the minimum each time.

Implementing a priority queue with an ordered sequence

Now removeMin() is simple since it is just removeFirst(), which is Θ(1). (We store the sequence in wrap-around, i.e. mod N, style.) But insertItem is Θ(k) when there are k items already in the priority queue since after finding the correct location in which to insert, one must slide the remaining elements over.

This sorting algorithm is normally called insertion sort since the dominant effort is inserting each element.

2.4.3 The Heap Data Structure

We now consider the non-obvious implementation of a priority queue that gives a fast sort (and a fast priority queue). The idea is to use a tree to store the elements, with the smallest element stored in the root. We will store elements only in the internal nodes (i.e., the leaves are not used). One could imagine an implementation in which the leaves are not even implemented since they are not used. We follow the book and draw internal nodes as circles and leaves (i.e., external nodes) as squares.
Since the priority queue algorithm will perform steps with complexity Θ(height of tree), we want to keep the height small. The way to do this is to fully use each level.

Definition: A binary tree of height h is complete if the levels 0,...,h-1 contain the maximum number of elements and on level h-1 all the internal nodes are to the left of all the leaves.

Remarks:

Level i has 2ⁱ nodes, for 0≤i≤h-1.
Level h contains only leaves.
Levels less than h-1 contain no leaves.
So really, for all levels, internal nodes are to the left of leaves.

Definition: A tree storing a key at each internal node satisfies the heap-order property if, for every node v other than the root, the key at v is no smaller than the key at v's parent.

Definition: A heap is a complete binary tree satisfying the heap order property.

Definition: The last node of a heap is the right most internal node in level h-1. In the diagrams above the last nodes are pink.

Remark: As written the ``last node'' is really the last internal node. However, we actually don't use the leaves to store keys so in some sense ``last node'' is the last (significant) node.

Homework: R-2.11, R-2.14

Implementing a Priority Queue with a Heap

With a heap it is clear where the minimum is located, namely at the root. We will also use a reference to the last node since insertions will occur at the first node after last.

Theorem: A heap storing n keys has height ⌈log(n+1)⌉

Proof:

Let h be the height of the tree and note that n is the number of internal nodes.
Levels 0 through h-2 are full and contain only internal nodes (which contain keys).
This gives 1+2+4+...+2^h-2 keys.
Homework: Show this is 2^h-1-1 keys.
Level h-1 contains between 1 and 2^h-1 internal nodes each containing a key.
Level h contains no internal nodes.
So (2^h-1-1)+1 ≤ n ≤ (2^h-1-1)+2^h-1
So 2^h-1 ≤ n ≤ 2^h-1
So 2^h-1 < n+1 ≤ 2^h
So h-1 < log(n+1) ≤ h
So ⌈log(n+1)&rceil = h

Corollary: If we can implement insert and removeMin in time Θ(height), we will have implemented the priority queue operations in logarithmic time (our goal).

Illustrate the theorem with the diagrams above.

The Vector Representation of a Heap

Since we know that a heap is complete, it is efficient to use the vector representation of a binary tree. We can actually not bother with the leaves since we don't ever use them. We call the last node w (remember that is the last internal node). Its index in the vector representation is n, the number of keys in the heap. We call the first leaf z; its index is n+1. Node z is where we will insert a new element and is called the insertion position.

Insertion

This looks trivial. Since we know n, we can find n+1 and hence the reference to node z in O(1) time. But there is a problem; the result might not be a heap since the new key inserted at z might be less than the key stored at u the parent of z. Reminiscent of bubble sort, we need to bubble the value in z up to the correct location.

Up-Heap Bubbling

We compare key(z) with key(u) and swap the items if necessary. In the diagram on the right we added 45 and then had to swap it with 70. But now 45 is still less than its parent so we need to swap again. At worst we need to go all the way up to the root. But that is only Θ(log(n)) as desired. Let's slow down and see that this really works.

We had a heap before we inserted the new element.
When we insert the new element it can ruin the heap because it is too small (i.e. smaller than its parent),
So the blue node is the problem.
Since blue is smaller than its parent, we swap them. Call the parent the victim.
Two nodes have been swapped we have four things to check: For each of these two nodes we must see that it is not larger than its new children and not smaller than its new parent.
1. The blue node is definitely not larger than its new children: One child is the victim, which we know is larger than the blue. The other child was not smaller than the victim so is surely not smaller than the blue.
2. The blue node might be smaller than its new parent. Indeed in the diagram on the right it is. That is why we have to keep bubbling up.
3. Before we did the insert, we had a heap so at that point the victim was not larger than all its descendents. But after the swap, all of the children of the victim were descendents of the victim before.
4. The victim is definitely not smaller than its new parent, which is the blue.

Great. It works (i.e., is a heap) and there can only be O(log(n)) swaps because that is the height of the tree.

But wait! What I showed is that it only takes O(n) steps. Is each step O(1)?

Comparing is clearly O(1) and swapping two fixed elements is also O(1). Finding the parent of a node is easy (integer divide the vector index by 2). Finally, it is trivial to find the new index for the insertion point (just increase the insertion point by 1).

Allan Gottlieb