Realtime and Big Data Analytics

CSCI-GA.3033-006

 

NYU Courant Institute of Mathematical Sciences

Computer Science Department, Graduate Division

Fall 2014

 


 

General Information

 

Lecturer: Suzanne McIntosh (mcintosh@cs.nyu.edu)

 

Office Hours: Evenings by appointment in WWH 328, and after class.

 

Semester: Fall 2014

 

Room: CIWW (Courant Institute, Warren Weaver Hall) room 1302

 

Day and Time: Thursday, 7:10-9:00 pm

 


Prerequisites

 

Prerequisites: CSCI-GA 2250 or equivalent Operating Systems course; programming experience in C/C++ or Java for assignments and final project; CSCI-GA 2262, CSCI-GA 2620, or undergraduate course in networks. A familiarity with databases will be useful.

 


Texts

 

Hadoop: The Definitive Guide, by Tom White

Hadoop Operations, by Eric Sammer (optional)

Programming Pig, by Alan Gates (optional)

 


Tools

Cloudera Distribution for Apache Hadoop (CDH) Fully configured QuickStart VM is available at:

http://www.cloudera.com/content/support/en/downloads/download-components/download-products.html?productID=F6mO278Rvo

 


Description

 

This course introduces the architectures and technologies at the foundation of the Big Data movement. These technologies have facilitated scalable management and processing of vast quantities of data collected through realtime and near realtime sensing. We explore tools enabling the acquisition of data in the social domain and the fusion of those data when in flight and at rest. 

 

The material covered in this course aligns with the prevailing state of the art in Big Data technologies, which continues to be a rapidly evolving landscape as new technologies emerge and existing ones evolve and mature.

 

Students are required to complete weekly reading and/or programming assignments and demonstrate mastery of course topics by designing, developing, and demonstrating an analytics project of their choosing. Class time will be set aside for project proposal and final demo.

 


Acknowledgement

 

We are grateful to Amazon for supporting this course through the Amazon Web Services (AWS) in Education Grant. The AWS in Education grant enables students to use AWS to develop Big Data Analytics projects using Hadoop.

 


Grading

 

Grades are based on the following approximate weighting:

 

Readings, lab assignments, class participation

25%

Midterm

25%

Final

20%

Project

30%

 


Syllabus (tentative)

Class

Topic

1

Introduction to Hadoop and Big Data

2

Distributed File Systems, HDFS, MapReduce

3

HDFS and MapReduce Architecture

4

Introduction to Pig, Analytics Examples

5

Project Tee-up, Realtime Systems and Intro. To Flume

6

New Alternatives to Traditional Database Systems and Access Methods, NoSQL

7

Midterm Exam

8

Hadoop in the Cloud I, Project Team meetings

9

Hadoop in the Cloud II

10

Realtime and Big Data in The Cloud: Autonomic Systems

11

Realtime and Big Data in The Cloud: Distributed Coordination

12

Fault Tolerance, YARN

13

ZooKeeper, Project Demo Day

14

Cluster Performance, Final Exam Review

15

Final Exam