V22-0480-002, Spring 2007:
Machine Learning and Pattern Recognition

This course will cover a wide variety of topics in machine learning, pattern recognition, statistical modeling, and neural computation. The course will cover the mathematical methods and theoretical aspects, but will primarily focus on algorithmic and practical issues.

Machine Learning and Pattern Recognition methods are at the core of many recent advances in "intelligent computing". Current applications include machine perception (vision, audition, speech recognition), control (process control, robotics), data mining, time-series prediction (e.g. in finance), natural language processing, text mining and text classification, bio-informatics, neural modeling, computational models of biological processes, and many other areas.

The only formal pre-requisites are familiarity with computer programming and linear algebra, but the course relies heavily on such mathematical tools as probability and statistics, multi-variate calculus, and function optimization. The basic mathematical concepts will be introduced when needed, but students will be expected to assimilate a non-trivial amount of mathematical concepts in a fairly short time.

Therefore, the course is primarily for students at the senior and junior levels.

• the basics of inductive inference, learning, and generalization.
• linear classifiers: Perceptron, LMS, logistic regression.
• non-linear classifiers with linear parameterizations: basis-function methods, boosting, Support Vector Machines.
• multilayer neural networks, backpropagation
• heterogeneous learning systems
• Energy-based models, loss functions
• optimization methods in learning: gradient-based methods, second-order methods.
• Probabilistic models, Bayesian methods, the Expectation-Maximization algorithm.
• graph-based models for sequences: Hidden Markov Models, finite-state transducers, recurrent networks.
• unsupervised learning: density estimation, clustering, and dimensionality reduction methods.
• introduction to graphical models and factor graphs
• approximate inference, sampling.
• Learning theory, the bias-variance dilemma, regularization, model selection.
• applications in computer vision, handwriting recognition, speech recognition, robotics, natural language processing, financial forecasting, biological modeling...

The course will include a short tutorial on the Lush language, a simple interpreted language for numerical applications.

Lush can be downloaded and installed on Linux, Mac OS-X, and Windows (under Cygwin). See Chris Poultney's notes on installing Lush under Cygwin.

Lush is available on the CIMS Sun workstations available for student use.

Programming projects may be implemented in any language, (C, C++, Java, Matlab, Lisp, Python,...) but the use of a high-level interpreted language with good numerical support and and good support for vector/matrix algebra is highly recommended (Lush, Matlab, Octave...). Some assignments require the use of an object-oriented language.

Also, for most assignments, a code squeleton in Lush will be provided.

NOTE: This is not Bishop's previous book called Neural Networks and Pattern Recognition;

The textbook can be used for reference, but I will not follow it very closely.

The following books can also be used for complementary material (you can get copies from the library):

• Richard O. Duda, Peter E. Hart, David G. Stork: "Pattern Classification" Wiley-Interscience; 2nd edition, October 2000.
• T. Hastie, R. Tibshirani, and J. Friedman: "Elements of Statistical Learning", Springer-Verlag, 2001.
• Ethem Alpaydin: Introduction to Machine Learning, MIT Press, October 2004.

Other Books of Interest

• C. Bishop: "Neural Networks for Pattern Recognition", Oxford University Press, 1996. Pretty much superseded by Bishop's lates book.
• S. Haykin: "Neural Networks, a comprehensive foundation", Prentice Hall, 1999 (second edition).
• Tom Mitchell: "Machine Learning", McGraw Hill, 1997.

automatic obstacle avoidance

visual navigation