README file for Deblurring code used in SIGGRAPH '06 paper: "Removing Camera Shake from a Single Image" by R.Fergus, B.Singh, A. Hertzmann, S. T. Roweis, W. T. Freeman. Distribution code Version 1.0 -- 10/23/2006 by Rob Fergus (fergus at csail dot mit dot edu) Copyright 2006, Massachusetts Institute of Technology. IMPORTANT DISCLAIMER: The source code is messy research-grade code. There are limited comments through out and lots of strange obsolete sections and so on as a result of the organic development process. If you are having problems understanding what the code is doing, please take some time to work through it yourself and understand what is going on. Try reading the various references in Section 3 below, hopefully this should offer some enlightenment. The algorithm is not yet sufficiently robust to be a tool that can used by anyone. You may have to run the algorithm several times on an image, trying different subwindows etc. before you get a sharp image out. Experience of using the algorithm certainly does help in selecting the various input parameters. We are working on improving the algorithm to bring it up to commerical-grade robustness. Please remember that blind deconvolution of images is a very challenging problem that has been around for several decades. I don't want to be mean but I'd really appreciate it if you could only email me if you are really stuck. I will try and get back to you, but I may take some time as I am busy with other stuff. -------------------------------------------------------------------------------- ¦ Contents ¦ -------------------------------------------------------------------------------- 1. Legal blurb regarding the patent filed on this algorithm. 2. Acknowledgements 3. Academic use 4. Installing the code 5. Overall structure of algorithm 6. Detailed breakdown of parameter script 7. Hints and tips for using the algorithm. ------------------------------------------------------------------------------- ¦ 1. Legal blurb ¦ ------------------------------------------------------------------------------- Begin patent blurb from MIT TLO 1. A patent has been filed for this algorithm. 2. It may be used for non-commercial use only. 3. If you wish to use it commercially, please contact the MIT Technology Licensing Office (tlo@mit.edu): Technology Licensing Office Massachusetts Institute of Technology Five Cambridge Center, Kendall Square Room NE25-230 Cambridge, MA 02142-1493 Tel: +1 617-253-6966 Fax: +1 617-258-6790 End patent blurb from MIT TLO ------------------------------------------------------------------------------- ¦ 2. Acknowledgements ¦ ------------------------------------------------------------------------------- We'd like to acknowledge a variety of people for their help on this project: - David Mackay and James Miskin for graciously allowing us to distribute modified versions of their Variational deblurring algorithm. - Yair Weiss for the Poisson reconstruction code. - Antonio Torralba, Don Geman, Marshall Tappen, Fredo Durand and others for their useful insights and suggestions. - Daniel Dardani and the MIT TLO for their assistance in patenting and licensing issues regarding this algorithm. - Various funding agencies ------------------------------------------------------------------------------- ¦ 3. Academic use of the code ¦ ------------------------------------------------------------------------------- If you use this code to generate results that you use in an academic publication you should include the following citations in your paper (included in Bibtex format for your convenience): - The Fergus et al. SIGGRAPH 06 paper: @article{Fergus06, author = "Fergus, R. and Singh, B. and Hertzmann, A. and Roweis, S. T. and Freeman, W.T.", title = "Removing Camera Shake From A Single Photograph", volume = "25", issue = "3", pages = "787--794", journal = "ACM Transactions on Graphics, {SIGGRAPH} 2006 Conference Proceedings, Boston, {MA}", publisher = "ACM Press", year = "2006" } - At least one of the following Miskin and Mackay papers: @incollection{Miskin2000, author = {James Miskin and David J. C. MacKay}, title = {Ensemble {L}earning for {B}lind {I}mage {S}eparation and {D}econvolution}, booktitle = {Adv.~in {I}ndependent {C}omponent {A}nalysis}, editor = {M. Girolani}, publisher = {Springer-Verlag}, year = {2000} } @phdthesis {Miskin2000a, author = "Miskin, James W.", title = {Ensemble Learning for Independent Component Analysis}, school = {University of Cambridge}, year = {2000}, month = {December}, url = {http://www.inference.phy.cam.ac.uk/jwm1003/} } @misc{Miskin_software, author = {James Miskin}, title = {Train Ensemble Library}, howpublished = "\url{http://www.inference.phy.cam.ac.uk/jwm1003/train_ensemble.tar.gz}", year = 2000 } ------------------------------------------------------------------------------- ¦ 4. Installing the code ¦ ------------------------------------------------------------------------------- 4.1 Platform issues / requirements - The current implementation is purely in Matlab, so an installation of Matlab, along with the Image Processing Toolbox will be needed. I've tested the code under Matlab 7.2 and it checks out OK., but it should work OK under Matlab 6.5 although the included .mat files are V7 format. But you should also read comment 8.6. - If you haven't got Matlab then I'm afraid you can't use the code. Try getting hold of the Matlab student version. - It should work under Windows/Mac/Linux, although I've only tried it under Linux. - As for hardware issues, any modern machine should be fine. The memory requirements of the kernel inference part are minimal, although running Matlab's deconvlucy on large (>6 megapixel) images can require a lot (>1Gb) memory. The inference is slow, so a fast CPU is a good idea. I don't think it can be parallelized easily. - Hard disk usage: the .mat files generated for each image can be quite large due the unnecessary storage of all variables. So you might want to have a Gig or two free if you want to run on lots of images. 4.2 Unpacking and installing the code - The code, prior files and demo images and results are all in one .zip file, deblurring.zip (available from http://people.csail.mit.edu/fergus) - Download this and unzip it somewhere sensible. You should see the subdirs: code/ images/ priors/ results/. - Ensure that the code/ subdirectory is in your Matlab path. 4.3 Files not included in the .zip file - There are certain options in the code (currently disabled) that require additional software packages. These are: - Eero Simoncelli's matlabPyrTools routines http://www.cns.nyu.edu/~lcv/softare.html ------------------------------------------------------------------------------- ¦ 5. Overall structure of the algorithm ¦ ------------------------------------------------------------------------------- 5.1 Directory structure There are 4 subdirectories. code/ holds all the Matlab source code (it should be in your Matlab path). priors/ holds the .mat files specifying the parameters of the mixture of Gaussians priors on image gradients. images/ holds the original images to be deblurred. Three example images are provided - add your own in to the same directory. results/ holds the main input and output files for the algorithm, detailed below. 5.2 File structure 5.2.1 code/ The top-level function is deblur.m This is called with the name of an image script (passed as a string). This runs the multi-scale inference procedure to infer the blur kernel. See section 5.3 for a more detailed explanation. The core inference algorithm uses source code adapted from James Miskin and David Mackay who have kindly permitted the redistribution of the modified versions of their code. All their files start with 'train_'. They are very dense and most likely unintelligible until you have read the relevant parts of James Miskin's thesis (see Bibtex reference in section 3.) Please see the header comments in each file for more details. fiddle_lucy3.m is the routine that runs Richardson-Lucy and can be set to save the resulting deblurred output to disk. See comments at top of file for the various options. initialize_parameters2.m is the routine the initializes the gradient image and kernel estimate at each scale. move_level.m does the upsampling after inference at each scale. GaussianMixtures1D.m, mix_exponentials.m are used to estimate the prior parameters via EM. They are called by estimate_priors2.m (this file is a real mess, sorry). plotgray.m is a useful utility to inspect progress during inference. reconsEdge3.m and invDel2.m are two files written by Yair Weiss that do a Poisson reconstruction from image gradients. Thanks to him for letting me use it. The remaining files are just various utilities. 5.2.2 priors/ linear_street_4.mat - default priors file. 4 component mixtures of Gaussians prior estimated from a typical street scene image using linearized intensities. linear_whiteboard_4.mat - alternate priors file. 4 component mixtures of Gaussians prior estimated from an image of a lab whiteboard with text/equations on it, image using linearized intensities. 5.2.3 images/ ian1.jpg - blurry bird picture used in SIGGRAPH 06 presentation lyndsey2.jpg - blurry statue picture used in SIGGRAPH 06 presentation 5.2.4 results/ ian1.m - image script file for ian1.jpg ian1_final.jpg - deblurred output from ian1.jpg ian1_kernel.eps - inferred kernel from ian1.jpg lyndsey2.m - image script file for lyndsey2.jpg lyndsey2_final.jpg - deblurred output from lyndsey2.jpg lyndsey2_kernel.eps - inferred kernel from lyndsey2.jpg When running the code you will see tmp_