The CAS Learning Center is offering tutoring services
for students needing help in 002, 004, 201, or 310.
You can announce this in your classes. The website for
the CAS Learning Center is http://www.nyu.edu/cas/clc/.
The schedule is listed on the site.
End of Note
Often we have a data structure supporting a number of different operations that will each be applied many times. Sometimes the worst case time complexity (i.e., the longest amount of time) of a sequence of n operations is significantly less than n times the worst case complexity of one operations. We give an example very soon.
If we divide the running time of the sequence by the number of operations performed we get the average time for each operation in the sequence, which is called the amortized running time.
Why amortized?
Because the cost of the occasional expensive application is
amortized over the numerous cheap application (I think).
Example:: (From the book.) The clearable table. This is essentially an array. The table is initially empty (i.e., has size zero). We want to support three operations.
The natural implementation is to use a large array A and an integer N indicating the current size of A. More precisely A is (always) of size B (Big) and N indicates the extent of A that is currently in use.
Declare A[B] a large integer array, initially all zero
n an integer, initially zero
Algorithm add(e)
A[n] ← e
n ← n+1
Algorithm get(i)
return A[i]
Algorithm clear()
for i ← 0 to n-1
A[i] ← 0
n ← 0
We are ignoring a number of error cases. For example, it is an error to issue Get(5) if only two entries have been put into the clearable table.
The question we want to answer is, given an initially empty clearable table A, what is the worst-case (i.e., largest) time required for a sequence of K operations (K large). The amortized cost is this total divided by K.
Add(e) requires an array reference, an addition, and two assignments for a total cost of 4.
Get(i) requires a reference and a return; cost is 2.
Clear() requires an assignment to i, n+1 tests, n additions to i, n array references, n assignments to A, and one assignment to n for a total cost of 4n+3.
So Add is Θ(1), Get is &Theta(1);, and clear is Θ(n).
We start with a size zero table and assume we perform k (legal) operations. Question: What is the worst-case running time for all k operations? Once we know the answer, the amortized time is this answer divided by k. It represents the (worst case) average cost of each operation.
A difficulty is that we are only given k, the number of operations. We do not know what is the worst case sequence of operations. For example, given k=1000, which sequence of 1000 operations is the most expensive? Since we don't know the sequence we don't know the value of n when a clear is done.
One upper bound that is easy to compute goes as follows. Since we execute k operations, n is never bigger than k and hence clear never costs more than 4k+3. Since get(i) and add(e) cost less than 4k+3, we see that no single operation can cost more than 4k+3 so k operations cannot cost more than 4k2+3k and the average cost per operation cannot be more than (4k2+3k)/k or 4k+3.
Hence the total cost for k operations is O(k2) and the amortized cost is O(k).
Since we might be overestimating the cost of some operations we only get big-Oh values not Theta values.
Everything we have said so far is correct; in particular the amortized cost is O(k). Of course it also is O(k2) and also O(k50), etc. We naturally prefer the smallest possible upper bound.
The problem with our analysis so far is that our upper bound is too big. The amortized cost is actually O(1) (since it is clearly ω(1), it is also θ(1)).
The reason we have too large an upper bound is that, once we (correctly) determined that 4k+3 is the most one operation could cost, we asserted that k(4k+3) is the most k operations could cost.
It is true that if A is the most one operation can cost, then k operations cannot cost more than kA.
However it is NOT true that if there is one operation that can cost A, then there is a sequence of k operations that cost as much as kA.
The point is that you cannot always get a sequence of k operations such that each one costs as much as the most expensive operation possible.
Consider the sequence consisting of k-1 add(e) operations followed by one Clear(). Each add(e) costs 4, for a total of 4k-4. Since there are k-1 add operations preceding the clear, n=k-1 when the clear() is invoked. Hence the clear costs 4(k-1)+3=4k-1 and the total cost of the sequence is (4k-4)+(4k-1)=8k-5.
So, for this particular sequence, the amortized cost is (8k-5)/k=8-5/k, which is Θ(1).
But this doesn't look like the most expensive sequence since it has only one clear and clear is the expensive operation. To be definite let's assume k=1000. So we know that the sequence of 999 adds followed by one clear costs a total of 8(1000)-5=7995. Let's try to make a more expensive sequence.
What if we try two clears, say one in the middle and one at the end? That is, 499 adds, one clear, 499 adds, one clear. The 998 adds cost a total of 4(998)=3992. The first clear costs 4(499)+3=1999 (only 499 adds precede the clear so n=499 when the clear is invoked).
Although it is true that there are 998 adds preceding the second clear, 499 of them have already been processed by the first clear and n=499 (not 998) when the second clear is invoked. Thus this clear also costs 1999 and the entire sequence costs 3992 for the adds and 2(1999) for the two clears, for a grand total of 7990. This is less than the cost of our first sequence.
But maybe the problem is that we should have put both clears at the end where they are the most expensive, i.e. 998 adds, clear, clear. The 998 adds still cost 3992. The first clear is expensive, but now the second one is very cheap since n=0!
In fact no matter what sequence of k operations you construct the total number of adds cannot exceed k and thus and this means that all the clears combined can clear no more than k values.
Since the cost of a clear is 4n+3 when n items are present, we see that the total of all the n's in all the clears cannot exceed k. Hence all the clears combined have complexity O(k) (since we found a sequence, the first one considered, where the clear costs 4k+2, the complexity is θ(k)).
Hence, the amortized time for each operation in the clearable table ADT (abstract data type) is O(1), in fact Θ(1). That is, the worst case total time for k operations is Θ(k).
Why?
Note that we first found an upper bound on the complexity (i.e., big-Oh) and then a lower bound (Ω). Together this gave Θ.
Overcharge for cheap operations and undercharge expensive so that the excess charged for the cheap (the profit) covers the undercharge (the loss). This is called in accounting an amortization schedule.
Assume the get(i) and add(e) really cost one ``cyber-dollar'', i.e., there is a constant B so that they each take fewer than B primitive operations and we let a ``cyber-dollar'' be B. Similarly, assume that clear() costs P cyber-dollars when the table has P elements in it.
We charge 2 cyber-dollars for every operation. So we have a profit of 1 cyber-dollar on each add(e) and we see that the profit is enough to cover next clear() since if we clear P entries, we had P add(e)s.
All operations cost 2 cyber-dollars so k operations cost 2k. Since we have just seen that the real cost is no more than the cyber-dollars spent, the total cost is O(k) and the amortized cost is O(1). Since every operation has cost Ω(1), the amortized cost is Θ(1).
Very similar to the accounting method. Instead of banking money,
you increase the potential energy. I don't believe we will use this
method so we are skipping it. If you like physics more than
accounting, you might prefer it.
1.5.2 Analyzing an Extendable Array Implementation
We want to let the size of an array grow dynamically (i.e., during execution). The implementation is quite simple. Copy the old array into a new one twice the size. Specifically, on an array overflow instead of signaling an error perform the following steps (assume the array is A and the current size is N)
The cost of this growing operation is Θ(N).
Theorem: Given an extendable array A that is initially empty and of size N, the amortized time to perform n add(e) operations is Θ(1).
Proof: Assume one cyber dollar is enough for an add w/o the grow and that N cyber-dollars are enough to grow from N to 2N. Charge 2 cyber dollars for each add; so a profit of 1 for each add w/o growing. When you must do a grow, you had N adds so have N dollars banked. Hence the amortized cost is O(1). Since Omega;(1) is obvious, we get Θ(1).
Alternate Proof that the amortized time is O(1). Note that amortized time is O(1) means that the total time is O(N). The new proof is a two step procedure
For step one we note that when the N add operations are complete the size of the array will be FS<.2N, with FS a power of 2. Let FS=2k So the total size used is TS=1+2+4+8+...+FS=∑2i (i from 0 to k). We already proved that this is (2k+1-1)/(2-1)=2k+1-1=2FS-1<4N as desired.
An easier way to see that the sum is 2k+1-1 is to write 1+2+4+8+...2k in binary, in which case we get (for k=5)
1
10
100
1000
10000
+100000
------
111111 = 1000000-1 = 25+1-1
The second part is clear. For each cell the algorithm only does a bounded number of operations. The cell is allocated, a value is copied in to the cell, and a value is copied out of the cell (and into another cell).
Allan Gottlieb