Very similar to induction. Assume we have a loop with controlling variable i. For example a "for i←0 to n-1". We then associate with the loop a statement S(j) depending on j such that
I favor having array and loop indexes starting at zero. However, here it causes us some grief. We must remember that iteration j occurs when i=j-1.
Example:: Recall the countPositives algorithm
Algorithm countPositives
Input: Non-negative integer n and an integer array A of size n.
Output: The number of positive elements in A
pos ← 0
for i ← 0 to n-1 do
if A[i] > 0 then
pos ← pos + 1
return pos
Let S(j) be "pos equals the number of positive values in the first j elements of A".
Just before the loop starts S(0) is true vacuously. Indeed that is the purpose of the first statement in the algorithm.
Assume S(j-1) is true before iteration j, then iteration j (i.e., i=j-1) checks A[j-1] which is the jth element and updates pos accordingly. Hence S(j) is true after iteration j finishes.
Hence we conclude that S(n) is true when iteration n concludes, i.e. when the loop terminates. Thus pos is the correct value to return.
Skipped for now.
We trivially improved innerProduct (same asymptotic complexity before and after). Now we will see a real improvement. For simplicity I do a slightly simpler algorithm, prefix sums.
Algorithm partialSumsSlow
Input: Positive integer n and a real array A of size n
Output: A real array B of size n with B[i]=A[0]+…+A[i]
for i ← 0 to n-1 do
s ← 0
for j ← 0 to i do
s ← s + A[j]
B[i] ← s
return B
The update of s is performed 1+2+…+n times. Hence the running time is Ω(1+2+…+n)=&Omega(n2). In fact it is easy to see that the time is &Theta(n2).
Algorithm partialSumsFast
Input: Positive integer n and a real array A of size n
Output: A real array B of size n with B[i]=A[0]+…+A[i]
s ← 0
for i ← 0 to n-1 do
s ← s + A[i]
B[i] ← s
return B
We just have a single loop and each statement inside is O(1), so the algorithm is O(n) (in fact Θ(n)).
Homework: Write partialSumsFastNoTemps, which is also Θ(n) time but avoids the use of s (it still uses i so my name is not great).
Often we have a data structure supporting a number of different operations that will each be applied many times. Sometimes the worst case time complexity (i.e., the longest amount of time) of a sequence of n operations is significantly less than n times the worst case complexity of one operations. We give an example very soon.
If we divide the running time of the sequence by the number of operations performed we get the average time for each operation in the sequence, which is called the amortized running time.
Why amortized?
Because the cost of the occasional expensive application is
amortized over the numerous cheap application (I think).
Example:: (From the book.) The clearable table. This is essentially an array. The table is initially empty (i.e., has size zero). We want to support three operations.
The obvious implementation is to use a large array A and an integer s indicating the current size of A. More precisely A is (always) of size N (large) and s indicates the extent of A that is currently in use.
We are ignoring a number of error cases. For example, it is an error to issue Get(5) if only two entries have been put into the clearable table.
We start with a size zero table and assume we perform n (legal) operations. Question: What is the worst-case running time for all n operations? Once we know the answer, the amortized time is this answer divided by n.
One possibility is that the sequence consists of n-1 add(e) operations followed by one Clear(). The Clear() takes Θ(n), which is the worst-case time for any operation (assuming n operations in total). Since there are n operations and the worst-case is Θ(n) for one of them, we might think that the worst-case sequence would take Θ(n2).
But this is wrong.
It is easy to see that Add(e) and Get(i) are Θ(1).
The total time for all the Clear() operations in any sequence of n operation is O(n) since in total O(n) entries were cleared (since at most n entries were added).
Hence, the amortized time for each operation in the clearable ADT (abstract data type) is O(1), in fact Θ(1).
Why?
Note that we first found an upper bound on the complexity (i.e., big-Oh) and then a lower bound (Ω). Together this gave Θ.
Overcharge for cheap operations and undercharge expensive so that the excess charged for the cheap (the profit) covers the undercharge (the loss). This is called in accounting an amortization schedule.
Assume the get(i) and add(e) really cost one ``cyber-dollar'', i.e., there is a constant K so that they each take fewer than K primitive operations and we let a ``cyber-dollar'' be K. Similarly, assume that clear() costs P cyber-dollars when the table has P elements in it.
We charge 2 cyber-dollars for every operation. So we have a profit of 1 on each add(e) and we see that the profit is enough to cover next clear() since if we clear P entries, we had P add(e)s.
All operations cost 2 cyber-dollars so n operations cost 2n. Since we have just seen that the real cost is no more than the cyber-dollars spent, the total cost is O(n) and the amortized cost is O(1). Since every operation has cost Ω(1), the amortized cost is Θ(1).
Very similar to the accounting method. Instead of banking money,
you increase the potential energy. I don't believe we will use this
method so we are skipping it. If you like physics more than
accounting, you might prefer it.
1.5.2 Analyzing an Extendable Array Implementation
We want to let the size of an array grow dynamically (i.e., during execution). The implementation is quite simple. Copy the old array into a new one twice the size. Specifically, on an array overflow instead of signaling an error perform the following steps (assume the array is A and the current size is N)
The cost of this growing operation is Θ(N).
Theorem: Given an extendable array A that is initially empty and of size N, the amortized time to perform n add(e) operations is Θ(1).
Proof: Assume one cyber dollar is enough for an add w/o the grow and that N cyber-dollars are enough to grow from N to 2N. Charge 2 cyber dollars for each add; so a profit of 1 for each add w/o growing. When you must do a grow, you had N adds so have N dollars banked. Hence the amortized cost is O(1). Since Omega;(1) is obvious, we get Θ(1).