CSC2515 Fall 2005 - Weekly and Other Readings

There is no required textbook for the class.
Two recommended books that cover similar material are Hastie, Tibshirani, Friedman Elements of Statistical Learning and MacKay, Info Theory, Inference, and Learning Algorithms which is freely available online.

I will be handing out class notes as we go along.

Some classic papers will be assigned as weekly readings.

We will also be covering material similar to a variety of chapters from a few other books which I will point out in class.

September 13
L.G. Valiant, A Theory of the Learnable [pdf, 9pages]
September 20
R.A. Fisher, The Use of Multiple Measurements in Taxonomic Problems [pdf, 10pages]
September 26
Pedro Domingos, Michael Pazzani, On the Optimality of the Simple Bayesian Classifier [pdf, 28pages]
October 4
Michael Jordan, Why the logistic function? [pdf , ps.gz, 13pages]
October 11
Rumelhart, Hinton and Williams, Learning representation by backpropagating errors, (Nature, 1986). [pdf, 4pages]
October 18
Michael I. Jordan and Robert A. Jacobs (1994), Hierarchical Mixtures of Experts and the EM Algorithm [pdf , ps.gz, 36pages]
October 25
C.K.Chow and C.N. Liu, Approximating discrete probability distributions with dependence trees [pdf, 6 pages]
November 1
Geoff Hinton and Radford Neal, A View of the EM Algorithm, Learning in Graphical Models (1998), [pdf , ps.gz, 14pages]
November 8
Zoubin Ghahramani and Geoff Hinton, The EM algorithm for Mixtures of Factor Analyzers [ps.gz, pdf, 8 pages]
November 15
Alan Poritz, Hidden Markov Models: A guided tour., ICASSP 1988. [pdf, 7pages]
November 22
Lawrence Saul, Fernando Pereira, Aggregate and mixed-order Markov models for statistical language processing., EMNLP 1997. [pdf, 9pages]
November 29
Rob Shapire, The Strength of Weak Learnability., Machine Learning 1990. [pdf, 31pages]
December 6
Corinna Cortes and Vladimir Vapnik, Support Vector Networks, Machine Learning 20(3): 273-297 (1995) [ps.gz, 31pages]

Probability and Statistics Review [ps.gz].
Some useful matrix identities and gaussian identities.
Andrew Moore at CMU has a tutorials page with many excellent mini-tutorials on various statistical machine learning topics.
In particular, you might want to check out his tutorials on probability and density, and on Gaussian and Bayesian classifiers.
A short MATLAB tutorial.
Numerical Recipes in C has some useful things on optimization and computing with real numbers, although some of their algorithms I wouldn't recommend actually using. It is available in pdf on the web here. Chapters 10 and and 15 might be of interest.

Geman, Bienstock, Doursat, Neural Networks and the Bias/Variance Dilemma. Neural Computation (1992), [pdf 58 pages]
Sam Roweis and Zoubin Ghahramani, A Unifying Review of Linear Gaussian Models, Neural Compuation (1999), [pdf 41pages]
David Mackay, Maximum Likelihood and Covariant Algorithms for ICA [ps.gz, 15 pages]
Joel Max, Quantizing for Minimum Distortion, IRE Transactions on Information Theory, March 1960. [pdf, 6 pages]
Stuart Lloyd, Least Squares Quantization in PCM, IEEE Transactions on Information Theory 28(2), March 1982. [pdf, 9 pages]
Fix, Hodges, Nonparametric Discrimination: Consistency Properties, 1951. [pdf, 21 pages]
Marina Meila, An accelerated Chow and Liu algorithm [ps.gz, 12 pages]
Robert Tibshirani Regression shrinkage and selection via the lasso [pdf , ps.gz, 28pages]
An article from Scientific American on Stein's Paradox. Another paper on this topic.
Golub, Heath, Wahaba, Generalized Cross Validation, Technometrics 1979. [pdf]
The voted perceptron algorithm is introduced in this paper by Freund and Schapire.
The very cool Winnow algorithm is introduced by Littlestone in this paper.
Some papers on decision trees: Quinlan's original paper, a review paper by Murthy, a paper by Chou on approximately optimal partitioning.
A couple of suggestions for how to improve Fisher's discriminant.