V22.0436 - Prof. Grishman

Lecture 26:  Multiprocessors

Trend towards increasing numbers of processors on a single machine, either on a single chip (multi-core) or a multiple chips in a system.  Central question is how these processors (and the processes running on them) communicate ... applications will vary widely in the amount of communication required.

Communication (text, sec. 7.3 and 7.4)

Shared-memory multiprocessors

Message passing multiprocessors

Info on NYU high-performance clusters.

Comparing the models:  shared memory vs message passing

Task:  computing sum of 100,000 numbers using 100 processors (text, 639-643)

Shared memory model (text, p. 639- 640)

global array:  float A[100000];
local variable Pn:  process (thread) number
global array:  float sum[100];

/* each processor sums 1000 numbers */
sum[Pn] = 0;

for (int i = 1000*Pn; i < 1000*(Pn+1); i++)
    sum[Pn] += A[i];
/* then we reduce these 100 sums to a total in sum[0] */
half = 100;
repeat
    synch();
    if (half%2 != 0 && Pn == 0)
        sum[0] += sum[half+1];
    half /= 2;
    if (Pn < half) sum[Pn] += sum[Pn+half];
until (half == 1);

Message passing model (text, p. 643)

Assume each processor has an array float AN[1000];
As before, special variable Pn = processor number

/* each processor sums the 1000 numbers in its local memory */
sum = 0;
for (int i = 0; i < 1000; i++)
    sum += AN[i];
/* now we combine the sums across processors */
limit = 100;
half = 100;
repeat
    half = (half + 1) / 2;
    if (Pn >= half && Pn < limit) send (Pn - half, sum);
    if (Pn < (limit/2)) sum += receive();
    limit = half;
until (half == 1);

Networks (sec'n 7.8)

Alternative topologies differ in
Common topologies
Multi-stage networks