Let's extend our sorting study from keys that are integers to keys that are pairs of integers. The first question to ask is, given two keys (k,m) and (k',m'), which is larger? Note that (k,m) is just the key; an item would be written ((k,m),e).
Definition: The lexicographical (dictionary) ordering on pairs of integers is defined by declaring (k,m) < (k',m') if either
Note that this really is dictionary order:
canary < eagle < egret < heron
10 < 11 < 12 < 2
Algorithm radix-sort-on-pairs
input: A sequence S of N items with keys
pairs of integers in the range [0,N)
Write elements of S as ((k,m),e)
output: Sequence S lexicographically sorted on the keys
bucket-sort(S) using m as the key
bucket-sort(S) using k as the key
Do an example of radix sorting on pairs.
Do an incorrect sort by starting with the most significant element of the pair.
Do an incorrect sort by using an individual sort that is not stable.
What if the keys are triples or in general d-tuples?
The answer is ...
Homework: R-4.15
Theorem: Let S be a sequence of N items each of which has a key (k1,k2,...kd), where each ki is in integer in the range [0,R). We can sort S lexicographically in time Θ(d(N+R)) using radix-sort.
Insertion sort or bubble sort are not suitable for general sorting of large problems because their running time is quadratic in N, the number of items. For small problems, when time is not an issue, these are attractive, because they are so simple. Also if the input is almost sorted, insertion sort is fast since it can be implemented in a way that is Θ(N+A), where A is the number of inversions, (i.e., the number of pairs out of order).
Heap-sort is a fine general-purpose sort with complexity Θ(Nlog(N)), which is optimal for comparison-based sorting. Also heap-sort can be executed in place (i.e., without much extra memory beyond the data to be sorted). (The coverage of in-place sorting was ``unofficial'' in this course.) If the in-place version of heap-sort fits in memory (i.e., if the data is less than the size of memory), heap-sort is very good.
Merge-sort is another optimal Θ(Nlog(N)) sort. It is not easy to do in place, so is inferior for problems that can fit in memory. However, it is quite good when the problem is too large to fit in memory and must be done ``out-of-core''. We didn't discuss this issue, but the merges can be done with two input and one output file (this is not trivial to do well, you want to utilize the available memory in the most efficient manner).
Quick-sort is hard to evaluate. The version with the fast median algorithm is fine theoretically (worst case again Θ(Nlog(N)), but is not used because of large constant factors in the fast median. Randomized quick-sort has a low expected time, but a poor (quadratic) worst-case time. It can be done in place, and is quite fast on average in that case, often the fastest. But the quadratic worst case is a fear (and a non-starter for many real-time applications).
Bucket and radix sort are wonderful when they apply, i.e., when the keys are integers in a modest range (R a small multiple of N). For radix sort with d-tuples the complexity is Θ(d(N+R)) so if d(N+R) is o(Nlog(N)), radix sort is asymptotically faster than any comparison based sort (e.g., heap-, insertion-, merge-, or quick-sort).
Selection means the ability to find the kth smallest element. Sorting will do it, but there are faster (comparison-based) methods. One example problem is finding the median (N/2 th smallest).
It is not too hard (but not easy) to implement selection with linear expected time. The surprising and difficult result is that there is a version with linear worst-case time.
The idea is to prune away parts of the set that cannot contain the desired element. This is easy to do as seen in the next algorithm. The less easy part is to show that it takes O(n) expected time. The hard part is to modify the algorithm so that it takes O(n) worst case time.
Algorithm quickSelect(S,k)
Input: A sequence S of n elements and an integer k in [1,n]
Output: The kth smallest element of S
if n=1 the return the (only) element in S
pick a random element x of X
divide S into 3 sequences
L, the elements of S that are less than x
E, the elements of S that are equal to x
G, the elements of S that are greater than x
{ Now we reduce the search to one of these three sets }
if k≤|L| then return quickSelect(L,k)
if k>|L|+|E| then return quickSelect(G,k-|L|+|E|
return x { We want an element in E; all are equal to x }
The greedy method is applied to maximization/minimization problems. The idea is to at each decision point choose the configuration that maximizes/minimizes the objective function so far. Clearly this does not lead to the global max/min for all problems, but it does for a number of problems.
This chapter does not make a good case for the greedy method. The method is used to solve simple variants of standard problems, but the the standard variants are not solved with the greedy method. There are better examples, for example the minimal spanning tree and shortest path graph problems. The two algorithms chosen for this section, fractional knapsack and task scheduling, were (presumably) chosen because they are simple and natural to solve with the greedy method.
In the knapsack problem we have a knapsack of a fixed capacity (say W pounds) and different items i each with a given weight wi and a given benefit bi. We want to put items into the knapsack so as to maximize the benefit subject to the constraint that the sum of the weights must be less than W.
The knapsack problem is actually rather difficult in the normal case where one must either put an item in the knapsack or not. However, in this section, in order to illustrate greedy algorithms, we consider a much simpler variation in which we can take a portion, say xi≤wi, of an item and get a proportional part of the benefit. This is called the ``fractional knapsack problem'' since we can take a fraction of an item. (The more common knapsack problem is called the ``0-1 knapsack problem'' since we must either take all (1) or none (0) of an item).
More formally, for each item i we choose an amount xi (0≤xi≤wi) that we will place in the knapsack. We are subject to the constraint that the sum of the xi is no more than W since that is all the knapsack can hold.
We desire to maximize the total benefit. Since, for item i, we only put xi in the knapsack, we don't get the full benefit. Specifically we get benefit (xi/wi)bi.
But now this is easy!
Why doesn't this work for the normal knapsack problem when we must take all of an item or none of it?
algorithm FractionalKnapsack(S,W):
Input: Set S of items i with weight wi and benefit bi all positive.
Knapsack capacity W>0.
Output: Amount xi of i that maximizes the total benefit without
exceeding the capacity.
for each i in S do
xi ← 0 { for items not chosen in next phase }
vi ← bi/wi { the value of item i "per pound" }
w ← W { remaining capacity in knapsack }
while w > 0 and S is not empty
remove from S an item of maximal value { greedy choice }
xi ← min(wi,w) { can't carry more than w more }
w ← w-xi
FractionalKnapsack has time complexity O(NlogN) where N is the number of items in S.
Homework: R-5.1
Allan Gottlieb