Statistical Natural Language Processing Fall 2016

Course#: CSCI-GA.3033-008
Instructor: Slav Petrov
Lecture: Tuesdays 5:10-7:00PM, Warren Weaver Hall Room 512
Mailing List: Piazza
Office hours: By appointment

Class Summary:

In this course we will examine some of the core tasks in natural language processing, starting with simple word-based models for text classification and building up to rich, structured models for syntactic parsing and machine translation. In each case we will first cover classic approaches, then review recent research progress and finally discuss with the help of guest lecturers (from Google) how to design efficient systems for practical user applications. There will be a focus on corpus-driven methods that make use of supervised and unsupervised machine learning methods and algorithms. We will explore statistical approaches based on graphical models and neural networks.

There will be five course assignments, which will be updated this year to include more neural network modeling, and a final project. In the assignments, you will construct basic systems for core NLP tasks and then improve them through a cycle of error analysis and model redesign. The assignments make up 40% of your final grade. For the final project you will be able to choose a single topic or application and investigate it in greater depth. The final project will be 30% of your final grade. The remaining 30% will come from class participation.

The class is open to graduate as well as undergraduate students.

Textbooks:

The primary text book for this course will be:
Jurafsky and Martin, Speech and Language Processing, Second Edition
Make sure to get the purple second edition of the book and not the white first edition.

The following book is useful as secondary literature, especially since it is available online:
Manning and Schuetze, Foundations of Statistical Natural Language Processing

Additional readings will come from recent research papers.

Syllabus:

Date Topic Textbook Reading Recent Papers Reading Assignments
9/6 Introduction &
Language Modeling
Jurafsky & Martin Chapter 4 (also Ch. 1)
(or Manning & Schuetze Chapters 1-3, 6)
MT Tutorial, LM Overview,
Large LMs, Neural LM
Assignment 1 out
9/13 (Text) Classification via
Naive Bayes, Maximum Entropy
and Neural Networks
Mitchell Chapter 3,
Jurafsky & Martin Chapter 7
Classification Tutorial,
MaxEnt Tutorial,
NN Primer, Neural Network Tutorial
Assignment 1 due,
Assignment 2 out
9/20 Neural Networks for Classification;
Part-of-Speech Tagging
Jurafsky & Martin Chapter 9 (old Ch. 5)
(or Manning & Schuetze Chapter 10)
NN Guide, TnT Tagger
9/27 Advanced Part-of-Speech
Tagging
Jurafsky & Martin Chapter 8 (old Ch. 6)
(or Manning & Schuetze Chapter 9)
Merialdo '94, Bilingual POS Induction,
CRFs, BiLSTM POS Tagging
Assignment 2 due,
Assignment 3 out
10/4 Word Alignments Jurafsky & Martin Chapter 25
(or Manning & Schuetze Chapter 13)
MT Tutorial, Overview, IBM Models,
HMM-Alignments, Agreement
10/11 Phrase-Based Translation
Chapter 5&6 available online Decoding, Phrases, Moses Assignment 3 due,
Assignment 4 out
10/18 Lexical Semantics Jurafsky & Martin Chapters 19, 20 Frege, word2vec, word2vec Explained,
Syntactic Embeddings, NNs for NLP
10/25 Constituency Parsing Jurafsky & Martin Chapters 12, 13, 14
(or Manning & Schuetze Chapters 3, 11, 12),
Unlexicalized, Lexicalized,
Latent Variable, Neural
Assignment 4 due
11/1 Dependency Parsing Dependency Parsing Chapters 1-4
(available from NYU network)
MST Parsing, Vine Pruning,
Shift-Reduce Parsing, NN Parsing
Project Proposal due,
Assignment 5 out
11/8 Compositional Semantics Jurafsky & Martin Chapter 22, 28
(old Ch. 17, 18)
Joint, ILP for SRL Framenet Parsing,
Compositional, Question Answering
11/15 Deep Learning and
Recurrent Neural Networks
Video Day 1, Video Day 2 Assignment 5 due
11/22 Hierarchical, Syntax-Based and
Neural Machine Translation
Long: Neural MT Tutorial at ACL '16,
Short: Systran Blogpost
Hiero, GHKM, BBN Neural Joint Models,
GNMT, Multilingual GNMT
11/29 Sentiment Analysis,
Summarization,
Outlook
Pang & Lee Sentiment Analysis Book,
Jurafsky & Martin Chapter 23.3 - 23.7
Sentiment: Aspects, Lexicons, Summarization
Summarization: Query, N-Gram, Topical
Outlook: Deep Learnng Tsunami
12/6 Final Presentations How to give a talk,
How to (not) write a paper
Project due 12/18
(at midnight)

Assignments:

There will be five assignments in which you will build systems for various NLP tasks. The assignments will be updated this year to include more neural network modeling. You will be provided with some code as a starting point and will need to implement the core learning and inference components. For the final project you can choose any NLP topic - even if it was not covered in the class. The scope of the project should be comparable to that of two assignments. Working in groups is encouraged for the final project. More guidelines and advice will be provided in class.

You will be submitting the predictions that your system makes and the results will be automatically posted to the leadeboard below.

Leaderboard:

Rank Handle #1
WER
#2
Accuracy
#3
Accuracy
#4
AER
#5
Spearman

The leadeboard is based on code from Wang Ling and Chris Dyer, which in turn is based on this course.