Machine Learning and Pattern Recognition: Schedule

This page contains the schedule, slide from the lectures, lecture notes, reading lists, assigments, and web links.

I urge you to download the DjVu viewer and view the DjVu version of the documents below. They display faster, are higher quality, and have generally smaller file sizes than the PS and PDF.

Full-text search is provided for the entire collection of slides and papers. Click here to search

You can have a look at the schedule and class material for the version of this course taught during the Fall 2005 semester, but be warned that the new edition is significantly different.

WARNING: The schedule below is almost certainly going to change.

09/05: Introduction and basic concepts

Subjects treated: Intro, types of learning, nearest neighbor, how biology does it, linear classifier, perceptron learning procedure, linear regression,

Slides: [DjVu | PDF]

Recommended Reading:

• Bishop, Chapter 1.
• Refresher on random variables and probabilites by Andrew Moore: (slides 1-27) [DjVu | PDF]
• Refresher on joint probabilities, Bayes theorem by Chris Willams: [DjVu | PDF]
• Refresher on statistics and probabilities by Sam Roweis: [DjVu | PS]
• If you are interested in the early history of self-organizing systems and cybernetics, have a look at this book available from the Internet Archive's Million Book Project: Self-Organizing Systems, proceedings of a 1959 conference edited by Yovits and Cameron (DjVu viewer required for full text).

09/12: Energy-Based Models, Loss Functions, Linear Machines

Subjects treated: Energy-based learning, minimum-energy inference, loss functions. Linear machines: least square, perceptron, logistic regression.

Slides: [DjVu | PDF]

Recommended Reading:

• Bishop, Chapter 2 and 4.

09/19: Basis Functions, Kernel Trick, Regularization, Generalization

Subjects treated: Energy-based models, minimum-energy machines, loss functions. Linear machines: perceptron, logistic regression. Linearly parameterized classifiers: Polynomial classifiers, basis function expansion, RBFs, Kernel-based expansion.

Slides Basis Functions, Kernel Trick: [DjVu | PDF]

Slides Regularization, Generalization: [DjVu | PDF]

Homework Assignements: Linear Classifier: implementing the Perceptron Algorithm, MSE Classifier (linear regression), Logistic Regression. Details and datasets below:

• Download this tar.gz archive. It contains the datasets and the homework description.
• Decompress it with "tar xvfz hw-linear.tgz" on Unix/Linux or with Winzip in Windows.
• The file README.txt contains the questions and instructions.
• Most the of the necessary Lush code is provided.
• Due Date is Tuesday October 03th, before the lecture.

09/26: Gradient-Based Learning I, Multi-Module Architectures and Back-Propagation, Regularization

Subjects treated: Multi-Module learning machines. Vector modules and switches. Multilayer neural nets. Backpropagation Learning.

Slides Multi-Module Learning machines, backprop: [DjVu | PDF]

Slides Special modules: [DjVu | PDF]

Recommended Reading: Bishop, Chapter 5.

10/03: Gradient-Based Learning II: Special Modules and Architectures

10/10: Convolutional Nets, Image Recognition

Required Reading:

If you haven't read it already: Gradient-based Learning Applied to Document Recognition by LeCun, Bottou, Bengio, and Haffner; pages 1 to the first column of page 18: [ DjVu | .pdf ]

Optional Reading: Fu-Jie Huang, Yann LeCun, Leon Bottou: "Learning Methods for Generic Object Recognition with Invariance to Pose and Lighting.", Proc. CVPR 2004. [DJVU, PDF, .PS.GZ,

Homework Assignements: Neural Nets and Backpropagation:

Click on this links to get the homework: hw-backprop.tgz.

Due Date is Tuesday October 31th, before the lecture.

10/17: Efficient Optimization

Required Reading: Efficient Backprop, by LeCun, Bottou, Orr, and Muller: [ DjVu | .pdf ]

10/24: Probabilistic Learning, MLE, MAP, Bayesian Learning

Slides Review of Probability and Statistics: [DjVu | PDF]

Slides Bayesian Learning: [DjVu | PDF]

10/31: Intro to unsupervised Learning

Slides Unsupervised Learning, PCA, K-Means: [DjVu | PDF]

11/07: More on unsupervised Learning, Latent Variables, EM

Slides Latent variables: [DjVu | PDF]

Slides EM, Mixture of Gaussians: [DjVu | PDF]

11/14: Learning Theory, Bagging, Boosting, VC-Dim

Slides Latent variables: [DjVu | PDF]

Homework Assignements: K-Means and Mixture of Gaussians:

Click on this links to get the homework: hw-backprop.tgz.

Due Date is Tuesday November 28, before the lecture.

11/21: Intro to Graphical Models

Suggested Reading:

• Read corresponding chapter in Bishop.
• David Mackay's book
• Information Theory, Inference, and Learning Algorithms. (available for free download from Mackay's website in PDF and DjVu).

11/28: Structured Outputs: HMMs, Graph Transformer Networks

Suggested Reading:

• Read parts Gradient-based Learning Applied to Document Recognition by LeCun, Bottou, Bengio, and Haffner; pages 18 (part IV) to the end. [ DjVu | .pdf ]
• Read chapter on HMM in Bishop.

The projects descriptions are here

Due date is December 30th (or December 23rd if you are graduating this semester).

12/05: Kernel Methods and Support Vector Machines

Lecture given by Sumit Chopra and Fu Jie Huang.

• Slides Constrained Optimization, Lagrange Duality: [DjVu | PDF]
• Slides Support Vector Machines: [DjVu | PDF]