Parallel Computing                       
Prof. Mohamed Zahran   (aka Prof. Z)
mzahran AT cs DOT nyu DOT edu
M/W 2:00-3:15 pm  Location: KIMM 803
Office Hours (WWH 320): Tue 2-4pm

It is all about parallelism!! 


Welcome students! ... to the Parallel Computing course, edition Spring 2016.  I will keep updating this page regularly. If you have questions related to that course feel free to email me. 

Most of us have learned to program a single microprocessor (single core) using a high-level programming language like C/C++, Java, ... This is called sequential programming. We feel very comfortable with this because we think in sequential way and give the machine statements to be executed in sequence. However, this has to change. A microprocessor with single core no longer exists in almost all computers we are using today (including your tablets and smart phones). Most of our devices are now multicore processors. A multicore processor contains several core (called CPUs or cores) on-chip. To make the best use of these multicore chips we need to program them in-parallel. Sequential programming, for all platforms from smartphones to supercomputers, is falling out of fashion and taking back-seat to parallel programming.

How to think in parallel? How to write code in parallel to make the best use of the underlying hardware? How is that new hardware different from the traditional one? What will the future be for the software and hardware? This is the topic of this course.

Here is some basic information about this course:

Midterm exam:  March 21st
Final exam:  May 16th - 2pm

Mailing List

Sign up for the Mailman mailing list for the course, if the system has not already signed you.
You can do so by clicking here. Please follow the mailing list etiquette. 
  • Use the Reply command to contribute to the current thread, but NOT to start another topic.
  • If quoting a previous message,  try to trim off irrelevant parts.
  • Use a descriptive Subject: field when starting a new topic.
  • Do not use one message to ask two unrelated questions.

  • Lecture Notes and Schedule 

    11/25Why Parallel Computing?
    • 1.1
    21/27Parallel Hardware: Basics
    32/1Parallel Hardware: Advanced
    • 1.2
    • 1.3.4
    42/3Parallel Software: Basics
    • 3.1
    • 3.2.1
    • 2.2
    • hw1 assigned
    52/8Parallel Software: Advanced
    62/10Performance Analysis
    • 1.4

    2/15No class: President's day
    72/17MPI I
    • 5.3-5.4 (neglect p. 245)
    • 5.6
    82/22MPI II
    • 5.11.3
    • 5.11.6
    82/24MPI II (cont'd)
    92/29MPI III
    • 5.11.1
    • 5.11.2
    • lab 1 assigned
    103/2MPI IV
    113/7MPI Last Touch

    3/14 No class: Spring Recess

    3/16No class Spring Recess
    3/21Midterm Exam
    123/23OpenMP I
    • 4.1-4.2
    133/28OpenMP II
    • skim 4.3
    • lab 2 assigned
    143/30OpenMP III
    • 4.4.3
    154/4OpenMP IV
    164/6OpenMP Last Touch
    • 4.6 (neglect listing 4.23 material)
    174/11GPU - Intro
    184/13CUDA  I
    • 6.1-6.2
    194/18CUDA II
    204/20CUDA III
    • 6.7.1 (till end of p.435 only)
    • hw3 assigned
    204/25CUDA III (cont'd)
    • lab3 assigned
    214/27CUDA IV
    225/2CUDA V
    235/4CUDA - Last Touch

    midterm sp 16 sol



    Before doing any MPI programming, do the following once you login onto your CIMS account:
    ssh to one of the computational nodes (e.g. crunchy1, crunchy3, crunchy4, crunchy5, and  crunchy6 ... No crunchy2!)
    Type, the following:
         module load openmpi-x86_64

    lab 1 you will also need this file that containts the skeleton (gs.c), reference program(gsref), and inputs (small.txt, medium.txt, large.txt, and huge.txt)  
    due  March 14th (by midnight)

    lab 2  - due Apr 6th 11:59pm

    Before running your firsr CUDA program, you need to do the following:

    login to your CIMS account
    Then, ssh to cuda1 (or any cudax where x is 1 to 5)

    module load mpi/mpich-x86_64
    cp -r /usr/local/cuda/samples ~/samples
    cd samples
    cd bin/x86_64/linux/release

    After that, each time you need to do CUDA programming, you need only  to do step1.

    lab 3: you will also need this file - due May 4th

    Homework Assignments 

    Interesting Links (Geeky stuff )

    If you have an interesting link, please email it to the instructor and it will find its way to this page.

    Top500 Supercomputers in the world
    Future Chips (targeting both software and hardware folks interested in parallel programming)

    Designing and Building Parallel Programs

    The trouble with multicore

    Introduction to parallel computing

    The Landscape of Parallel Computing

    EPCC benchmarks for OpenMP