V22-0480-006, Spring 2005:
Introduction to Machine Learning and Pattern Recognition

Machine Learning and Pattern Recognition methods are at the core of many recent advances in "intelligent computing". Current applications include machine perception (vision, audition, speech recognition), control (process control, robotics), data mining, time-series prediction (e.g. in finance), natural language processing, text mining and text classification, bio-informatics, neural modeling, computational models of biological processes, and many other areas.

The only formal pre-requisites are familiarity with computer programming and linear algebra, but the course relies heavily on such mathematical tools as probability and statistics, multi-variate calculus, and function optimization. The basic mathematical concepts will be introduced when needed, but the students will be expected to assimilate a non-trivial amount of new mathematical concepts in a fairly short time.

Therefore, the course is primarily for students at the senior and junior levels.

• the basics of inductive inference, learning, and generalization.
• linear classifiers: Perceptron, LMS, logistic regression.
• non-linear classifiers with linear parameterizations: basis-function methods, boosting, Support Vector Machines.
• multilayer neural networks, backpropagation
• heterogeneous learning systems
• Energy-based models, loss functions
• optimization methods in learning: gradient-based methods, second-order methods.
• Probabilistic models, Bayesian methods, the Expectation-Maximization algorithm.
• graph-based models for sequences: Hidden Markov Models, finite-state transducers, recurrent networks.
• unsupervised learning: density estimation, clustering, and dimensionality reduction methods.
• introduction to graphical models and factor graphs
• approximate inference, sampling.
• Learning theory, the bias-variance dilemma, regularization, model selection.
• applications in computer vision, handwriting recognition, speech recognition, robotics, natural language processing, financial forecasting, biological modeling...

This course will be a adapted version of the graduate course G22-3033-002 taught in Fall 2004. Please visit the site of the Fall 2004 Graduate-level version of this course to have a look at the schedule and source material.

Good programming ability is a must: many assignements will consist in implementing algorithms studied in class. For most programming assignement, a skeleton code in the Lush language will be provided. Lush is an interpreted language that makes it easy to quickly implement numerical algorithms.

The course will include a short tutorial on Lush.

Lush can be downloaded and installed on Linux, Mac, and Windows (under Cygwin). See Chris Poultney's notes on installing Lush under Cygwin.

Lush is available on the CIMS Sun workstations.

Programming projects may be implemented in any language, (C, C++, Java, Matlab, Lisp, Python,...) but the use of a high-level interpreted language with good numerical support and and good support for vector/matrix algebra is highly recommended (Lush, Matlab, Octave...). Some assignments require the use of an object-oriented language.