G22-3033-002, Fall 2004:
Machine Learning and Pattern Recognition

Graduate Course on machine learning, pattern recognition, neural nets, statistical modeling.

Instructor: Yann LeCun, 715 Broadway, Room 1220, x83283, yann [ a t ] cs.nyu.edu (note: new room number).

Teaching Assistant: Sumit Chopra, Warren Weaver Hall, Room 1106, x8-3242, sumit [ a t ] cs.nyu.edu.

Classes: Tuesdays 5:00-6:50PM, Room 102, Warren Weaver Hall.

Office Hours for Prof. LeCun: Wednesdays 2:00-4:00 PM

Office Hours for the TA: Wednesdays 5:00-7:00 PM

Course Description

This course will be an updated version of G22-3033-014 taught in Spring 2004.

This course will cover a wide variety of topics in machine learning, pattern recognition, statistical modeling, and neural computation. The course will cover the mathematical methods and theoretical aspects, but will primarily focus on algorithmic and practical issues.

Machine Learning and Pattern Recognition methods are at the core of many recent advances in "intelligent computing". Current applications include machine perception (vision, audition, speech recognition), control (process control, robotics), data mining, time-series prediction (e.g. in finance), natural language processing, text mining and text classification, bio-informatics, neural modeling, computational models of biological processes, and many other areas.

Who Can Take This Course?

This course can be useful all graduate students who would want to use or develop statistical modeling methods. This includes students in CS (AI, Vision, Graphics), Math (System Modeling), CNS (Computational Neuroscience, Brain Imaging), Finance (Financial modeling and prediction), Psychology (Vision, Linguistics), Biology (Computational Biology, Genomics, Bio-informatics), and Medicine (Bio-Statistics, Epidemiology).

Topics Treated

The topics studied in the course include:

• the basics of inductive inference, learning, and generalization.
• linear classifiers: perceptron, LMS, logistic regression.
• non-linear classifiers with linear parameterizations: basis-function methods, boosting, support vector machines.
• multilayer neural networks, backpropagation
• heterogeneous learning systems
• graph-based models for sequences: hidden Markov models, finite-state transducers, recurrent networks.
• unsupervised learning: density estimation, clustering, and dimensionality reduction methods.
• introduction to graphical models and factor graphs
• approximate inference, sampling.
• optimization methods in learning: gradient-based methods, second-order methods, Expectation-Maximization.
• objective functions: maximum likelihood, maximum a-posteriori, discriminative criteria, maximum margin.
• the bias-variance dilemma, regularization, model selection.
• applications in vision, speech, language, forecasting, and biological modeling.

This course will be a (much updated) re-run of G22.3033.014 taught in Spring 2004. Please visit the site of the Spring 2004 edition of this course to have a look at the schedule and source material.

Evaluations

The best way (some would say the only way) to understand an algorithm is to implement it and apply it. Building working systems is also a lot more fun, more creative, and more relevant than taking formal exams.

Therefore students will be evaluated primarily (almost exclusively) on programming projects given on a 2 week cycle, and on a final project.

Prerequisites

Linear algebra, vector calculus, elementary statistics and probability theory. Good programming ability is a must: most assignements will consist in implementing algorithms studied in class.

The course will include a short tutorial on the Lush language, a simple interpreted language for numerical applications.

Lush can be downloaded and installed on Linux, Mac, and Windows (under Cygwin). See Chris Poultney's notes on installing Lush under Cygwin.

Lush is available on the CIMS Sun workstations available for student use.

Programming projects may be implemented in any language, (C, C++, Java, Matlab, Lisp, Python,...) but the use of a high-level interpreted language with good numerical support and and good support for vector/matrix algebra is highly recommended (Lush, Matlab, Octave...). Some assignments require the use of an object-oriented language.

Also, for most assignments, a code squeleton in Lush will be provided.

Mailing List

Register to the course's mailing list.

Text Books

Richard O. Duda, Peter E. Hart, David G. Stork: "Pattern Classification" Wiley-Interscience; 2nd edition, October 2000.

I will not follow this book very closely. In particular, much of the material covered in the second half of the course cannot be found in the above book. I will refer to research papers and lectures notes for those topics.

Either one of the following books is also recommended, but not absolutely required (you can get a copy from the library):

• T. Hastie, R. Tibshirani, and J. Friedman: "Elements of Statistical Learning", Springer-Verlag, 2001.
• C. Bishop: "Neural Networks for Pattern Recognition", Oxford University Press, 1996. [quite good, if a little dated].

Other Books of Interest

• S. Haykin: "Neural Networks, a comprehensive foundation", Prentice Hall, 1999 (second edition).
• Tom Mitchell: "Machine Learning", McGraw Hill, 1997.

Links

Code

• Lush: A simple language for quick implementation of, and experimentation with, numerical algorithms (for Linux, Mac, and Windows/Cygwin). Many algorithms described in this course are implemented in the Lush library. Lush is available on the department's Sun machines that are freely accessible to NYU graduate students. See Chris Poultney's notes on installing Lush under Cygwin.
• Torch: A C++ library for machine learning.

Lush is installed on the department's PCs. It will soon be available on the Sun network as well.

Papers

• Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-Based Learning Applied to Document Recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998. [PS.GZ] [DjVu]
• Y. LeCun, L. Bottou, G. Orr, and K. Muller, "Efficient BackProp," in Neural Networks: Tricks of the trade, (G. Orr and Muller K., eds.), 1998. [PS.GZ] [DjVu]
• P. Simard, Y. LeCun, J. Denker, and B. Victorri, "Transformation Invariance in Pattern Recognition, Tangent Distance and Tangent Propagation," in Neural Networks: Tricks of the trade, (G. Orr and Muller K., eds.), 1998. [PS.GZ] [DjVu]

Publications, Journals

• Web site of the textbook The Elements of Statistical Learning: Data Mining, Inference, and Prediction, by Trevor Hastie, Robert Tibshirani, and Jerome Friedman.
• Nips Online: Volume 0-13 of the Neural Information Processing Systems Conference.
• Journal of Machine Learning Research.
• Neural Computation.

Conference Sites

• Neural Information Processing Systems Conference.

Datasets

• Datasets from the textbook "Elements of Statistical Learning".
• The MNIST Data: a database of hand-written digits: 60,000 training samples and 10,000 test samples. This data is in a Lush-readable format.

Demos and Pretty Pictures

Demo of Convolutional Nets for handwriting recognition.

Object Recognition with Convolutional Neural Nets