Parallel Computing                       
Prof. Mohamed Zahran   (aka Prof. Z)
mzahran AT cs DOT nyu DOT edu
M/W 2:00-3:15 pm  Location: WWH 317
Office Hours (WWH 320): Tue 2-4pm

It is all about parallelism!! 


Welcome students! ... to the Parallel Computing course, edition Spring 2015.  I will keep updating this page regularly. If you have questions related to that course feel free to email me. 

Most of us have learned to program a single microprocessor using a high-level programming language like C/C++, Java, ... This is called sequential programming. We feel very comfortable with this because we think in sequential way and give the machine statements to be executed in sequence. However, this has to change. A single microprocessor (or single core) no longer exists in almost all computers we are using today (including your tablets and smart phones). Most of our devices are now multicore processors. A multicore processor contains several core (called CPUs or cores) on-chip. To make the best use of these multicore chips we need to program them in-parallel. Sequential programming, for all platforms from smartphones to supercomputers, is falling out of fashion and taking back-seat to parallel programming.

How to think in parallel? How to write code in parallel to make the best use of the underlying hardware? How is that new hardware different from the traditional one? What will the future be for the software and hardware? This is the topic of this course.

Here is some basic information about this course:

Midterm exam:  March 23rd
Final exam:  May 18th, 2pm

Mailing List

Sign up for the Mailman mailing list for the course, if the system has not already signed you.
You can do so by clicking here. Please follow the mailing list etiquette. 
  • Use the Reply command to contribute to the current thread, but NOT to start another topic.
  • If quoting a previous message,  try to trim off irrelevant parts.
  • Use a descriptive Subject: field when starting a new topic.
  • Do not use one message to ask two unrelated questions.
  • Do NOT make the mistake of sending your completed project assignment to the mailing list!

  • Lecture Notes

    11/26Why Parallel Computing?1.1-1.4
    21/28Parallel Hardware: Basics2.1-2.2
    32/2Parallel Hardware: Advanced2.3
    • hw1 assigned
    42/4Parallel Hardware: Examples
    52/9Parallel Software: Basics2.4, 2.7
    62/11Parallel Software: Advanced

    2/16No class - President's day
    72/18Performance Analysis2.6
    82/23MPI - I3.1
    92/25MPI - II3.2, 3.3, 3.4.1-3.4.5
    • code: 1  2  3  
    103/2MPI - III3.4.6-3.5
    • code: 1  2  
    • hw2 assigned 
    • lab1 assigned
    113/4MPI - IV3.7
    123/9MPI - Last Touch
    133/11Revision    Sol
    midterm Spring 14  Sol

    3/16 No class - Spring recess

    3/18No class - Spring recess

    3/23Midterm Exam
    143/25OpenMP -I5.1-5.2
    153/30OpenMP - II5.3, 5.4, 5.5
    164/1OpenMP - III  &  OpenMP - IV5.6, 5.7, 5.8
    • lab 2 assigned
    174/6OpenMP- Last Touch5.9
    • code: 1  2  3  
    184/8GPUs - Intro
    194/13CUDA - I
    • Chapters 1 and 2 here
               are good intro
    204/15CUDA - II
    • hw3 assigned
    214/20CUDA - III
    224/22CUDA - IV
    234/27CUDA - V
    • lab 3 assigned
    244/29CUDA - VI
    255/4CUDA - Last Touch  
    265/6CUDA - Last Touch  
    Practice Problems  sol
    Final Spring 2014  sol

    midterm solutions


    Before doing any MPI programming, do the following once you login onto your CIMS account:
    ssh to one of the computational nodes (e.g. crunchy1, crunchy3, crunchy4, crunchy5, and  crunchy6 ... No crunchy2!)
    Type, the following:
         module load openmpi-x86_64
    Before running your firsr CUDA program, you need to do the following:
    1. logging to your CIMS account
    2. ssh to cuda1  (or cuda3 or cuda4)
    3. type:  module load mpich2
    4. cp -r /usr/local/pkg/cuda/current/sdk ~/nvidia_sdk
    5. cd ~/nvidia_sdk
    6. make
    7. cd bin/linux/release
    8. ./deviceQuery
    9. ./bandwidthTest
    After that, each time you need to do CUDA programming, you need onlu to do steps 1 and 2.

    Homework Assignment 

    Interesting Links (Geeky stuff )

    If you have an interesting link, please email it to the instructor and it will find its way to this page.

    Top500 Supercomputers in the world
    Future Chips (targeting both software and hardware folks interested in parallel programming)

    Designing and Building Parallel Programs

    Multicore info

    The trouble with multicore

    Computer Architecture

    Introduction to parallel computing

    The Landscape of Parallel Computing

    EPCC benchmarks for OpenMP