Computational Photography - CSCI-GA.3033-012
|
Semester: Fall 2013. Time and Location: Thursday 5:00-6:50pm, Room 1221, 715 Broadway. TA: To be decided Instructor: Rob Fergus Office hours: Thursday 4-5pm, Room 1226, 12th floor, 715 Broadway. |
 |
Overview
Computational Photography is an exciting new area at the intersection of Computer Graphics and Computer Vision. Through the use of computation, its goal is to move beyond the limitations of conventional photography to produce enhanced and novel imagery of the world around us. The main focus of the course will be on software-based methods for producing visually compelling pictures. However, it will also cover novel camera designs, for which computation is integral to their operation. The course will explain the principles behind many of the advanced tools that can be found in Adobe Photoshop, although the use of this package itself is outside the scope of the course. The course will be suitable for advanced undergraduates, masters and PhD students. A reasonable knowledge of linear algebra is required and familiarity with Matlab is desirable. Assessment will be through coursework and a course project.
Schedule
| Date | Time | Topics | Relevant Papers | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|||||||||||
|
|
|
|
|||||||||
|
|
|||||||||||
|
|
|
|
|||||||||
|
|
|
|
|
||||||||
|
|
|||||||||||
|
|
|
|
| ||||||||
|
|
|||||||||||
|
|
|
|
|
||||||||
|
|
|||||||||||
|
|
|
|
|
||||||||
|
|
|
|
|
||||||||
|
|
|||||||||||
|
|
|
|
|
||||||||
|
|
|||||||||||
|
|
|
|
|
||||||||
|
|
|
|
|
||||||||
|
|
|||||||||||
|
|
|
|
|
| ||||||||
|
|
|||||||||||
|
|
|
|
|
||||||||
|
|
|
|
|
||||||||
|
|
|
|
|
||||||||
|
|
|||||||||||
|
|
|
|
|
||||||||
|
|
|||||||||||
|
|
|
|
|
||||||||
|
|
|||||||||||
|
|
|
|
|
||||||||
|
|
|
|
|
||||||||
|
|
|||||||||||
|
|
|||||||||||
|
|
|||||||||||
|
|
|
|
|||||||||
|
|
|||||||||||
|
|
|
|
|
||||||||
|
|
|
|
|
||||||||
Acknowledgements
The instuctor would like to thank the large number of people who have made their slides available for use in this course. In particular, I'd like to thank Fredo Durand and Bill Freeman for letting me use their wonderful slides. Thanks also to: Eero Simoncelli, Ravi Ramamoorthi, Rick Szeliski, Steve Seitz, whose material I've used (and credited) in the above slides.Course project ideas
1. Camera spectral response calibration . This project involves determining the sensitivity vs wavelength curve for each color channel in the camera. This is a hardware-oriented project. The software component involves the use of simple least-squares minimization techniques in Matlab. The DSLR camera will be provided.
2. Seam Carving implementation . The initial part of this project involves re-implementing the Seam Carving paper by Avidan and Shamir PDF. This is a software-only project, requiring a C-based implementation to give real-time performance. The latter part of the project involves applying this method to a variety of settings and imagery, such as text images, web pages, Possible
3. 2-D Panoramas . This project involves the re-implementation the panorama creation paper of Brown et al PDF. This is a software-only project that can be performed in Matlab. Executables will be provided for certain parts of the algorithm, such as feature detection.
4. Creation of David Hockney Joiners. . The artist David Hockney has created images, known as "joiners", that consist of a collection of photos of scene, taken from a wide range of viewpoints, arranged in a collage to give a photo-collage yielding a non-realistic but compelling representation of the 3-D scene. Creating these currently involves a lot of manual alignment and artistic skill. The project would involve creating a tool that would let a user create these images much more quickly, while retaining artistic control over the process. This is a software-only project which an artistic and human-compute-interface flavor.
5. Hot or not. . This project explores ways to parametrize our notion of beauty. First, a large collection of images of people from websites such as Hot Or Not should be collected, along with metadata such as viewers ratings, hair color, eye color etc. After alignment of the faces, you should try and build simple models that allow a user to traverse the space of faces in a logical manner, e.g. blonder hair, prettier, thinner face etc. Some machine learning may be required, so is more suitable for someone who took Prof. LeCun's machine learning class last semester.
6. Cleaning up surveillance footage. . Surveillance cameras are now a common feature of many big cities in the US and Europe. However, the quality of footage that the record varies enormously. Older cameras record at modest resolutions and are further compromised by heavy compression artifacts. This project involves taking some real surveillance camera footage of a crime taking place and trying to remove the compression artifacts that prevent clear identification of the suspect. The key to this will be modeling the compression process and using image priors to infer what the uncompressed image would have looked like. This is a software-only project that can be performed in Matlab.
7. Image Analogies. . Implement the Image Analogies paper by Aaron Hertzmann (former NYU student) and co-authors. See the project page here
Additional Material
Matlab guides
Matlab tutorial by Hany Farid and Eero Simoncelli Link
A more comprehensive Matlab tutorial by David Griffiths Link
Further documentation on Matlab can be found here Link
Books
Forsyth, David A., and Ponce, J. Computer Vision: A Modern Approach, Prentice Hall, 2003.
Goldberg, Norman. Camera Technology: the Dark Side of the Lens, Academic Press, 2002.
Hartley, R. and Zisserman, A. Multiple View Geometry in Computer Vision, Academic Press, 2002.
Palmer, Stephen E. Vision Science: Photos to Phenomenology, MIT Press, 1999.
Ray, Sidney F. Applied Photographic Optics, Focal Press, 3/e, 2002.
Strang, Gilbert. Linear Algebra and Its Applications 2/e, Academic Press, 1980.
Wandell, Brian A. Foundations of Vision, Sinauer, 1995.
Further References
Adelson, Edward H. and Bergen, James R.
The Plenoptic Function and the Elements of Early Vision.
In Computational Models of Visual Processing,
ed. M. Landy and J. A. Movshon, MIT Press, 1991, 3-20.
Adelson, Edward H. and Wang, John Y. A.
Single Lens Stereo with a Plenoptic Camera,
IEEE PAMI, Vol 14, No. 2, 1992
Avidan, Shai and Shamir, Ariel.
Seam Carving for Content-Aware Image Resizing,
ACM Transactions on Graphics 26(3), SIGGRAPH 2007.
Brown, M. and Szeliski, R. and Winder, S.
Multi-image matching using multi-scale oriented patches,
IEEE Computer Society Conference on Computer Vision and Pattern Recognition,
2005, 510-517.
Burt, P. J., and Adelson, E. H.
The Laplacian Pyramid as a Compact Image Code,
IEEE Transactions On Communications, Vol. COM-31(4):532-540, 1983.
Cathey, W. T. and Dowski, E.
A new paradigm for imaging systems
Applied Optics, 1995.
Criminisi, A., and Perez, P. and Toyama, K.
Object Removal by Exemplar-based Inpainting,
IEEE Computer Society Conference on Computer Vision and Pattern Recognition,
2003.
Debevic, Paul E. and Malik, Jitendra
Recovering High Dynamic Range Radiance Maps from Photographs,
ACM SIGGRAPH, 1997.
Dowski, E. and Cathey, W. T.
Single-lens single-image incoherent
passive-ranging systems
Applied Optics, 1994.
Efros, Alexei A., and Leung, Thomas K.
Texture Synthesis by Non-parametric Sampling,
IEEE ICCV, 1999.
Efros, Alexei A, and Freeman, William T.
Image Quilting for Texture Synthesis and Transfer,
Proceedings of SIGGRAPH'01, Los Angeles, California, August, 2001.
Fergus, R. and Singh, B. and Hertzmann, A. and Roweis, S. T. and Freeman, W.T.
Removing Camera Shake From A Single Photograph,
ACM SIGGRAPH, 2006.
Field, D. What is the goal of sensory coding?
Neural Computation, 1994.
Fischler, M. and Bolles, R.
Random sample consensus: a paradigm for model fitting with
applications to image analysis and automated cartography,
Communications of the ACM, 24(6):381-395, 1981.
Kolmogorov, V. and Zabih, R. What Energy Functions can be Minimized
via Graph Cuts?, PAMI Feb 2004.
Kundur, D. and Hatzinakos, D., Blind Image Deconvolution IEEE
Signal Processign, May 1996.
Levin, A.Blind Motion Deblurring using Image Statistics,
NIPS 2006
Levin, A. and Fergus, R. and Durand, F. and Freeman, W. T.Image and Depth from a Conventional Camera with a Coded Aperture,
ACM SIGGRAPH 2007
Levoy, M. and Hanrahan, P.
Light Field Rendering
ACM SIGGRAPH, 1996.
Lowe, David G.
Distinctive image features from scale-invariant keypoints,
International Journal of Computer Vision, 60(2):91-110, 2004.
Lucas, Bruce D., and Kanade, Takeo.
An iterative image registration technique with an application to
stereo vision,
Proceedings of the Image Understanding Workshop, 121-130, 1981.
Ng, R. and Levoy, M. and Bredif, M. and Duval, G. and Horowitz, M. and Hanrahan, P.
Light Field Photography with a Hand-Held Plenoptic Camera,
Stanford University Computer Science Tech Report CSTR 2005-02, 2005.
Portilla, J. and Strela, V. and Wainwright, M. and Simoncelli, E. P.
Image denoising using scale mixtures of Gaussians in the wavelet domain
IEEE TIP 2003.
Simoncelli, E. P.
Statistical modeling of photographic images
Chapter 4, Handbook of Image and Video Processing, 2005.
Raskar, R. and Agrawal, A. and Tubmlin, J.
Coded Exposure Photography: Motion Deblurring using Fluttered
Shutter
ACM SIGGRAPH 2006.
Roth, S. and Black, M. J.,
Fields of Experts: A framework for learning image priors
CVPR 2005.