Computational photography 1059791 225174816 2008-07-12T08:08:44Z Chia-Kai Liang 1772972 {{external links}} ''Computational imaging'' refers to any image formation method that involves a digital computer. In contrast with digital imaging, which refers simply to representation of image data using symbols, computational imaging integrates sensing and data processing. Computational imaging is under development by a large and diverse community, as represented in a series of Optical Society of America conferences beginning with the 2001 Integrated Computational Imaging Systems topical meeting and continuing through the 2005, 2007 and 2009 Computational Optical Sensing and Imaging meetings. '''Computational photography''' refers broadly to computational imaging techniques that enhance or extend the capabilities of [[digital photography]]. The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera. The term was first used by [[Steve Mann]], and possibly others, to describe their own research. Its current definition, which stems from a 2004 course at Stanford University and a 2005 symposium at MIT (see links below), has evolved to cover a number of subject areas in [[computer graphics]], [[computer vision]], and applied [[optics]]. These areas are given below, organized according to a taxonomy proposed by [http://www1.cs.columbia.edu/~nayar/ Shree Nayar]. Within each area is a list of techniques, and for each technique one or two representative papers or books are cited. Deliberately omitted from the taxonomy are [[image processing]] (see also [[digital image processing]]) techniques applied to traditionally captured images in order to produce better images. Examples of such techniques are [[image scaling]], dynamic range compression (i.e. [[tone mapping]]), [[color management]], image completion (a.k.a. inpainting or hole filling), [[image compression]], [[digital watermarking]], and artistic image effects. Also omitted are techniques that produce [[3D scanner|range data]], [[voxel|volume data]], [[3D model]]s, [[light field|4D light fields]], 4D, 6D, or 8D [[BRDF]]s, or other high-dimensional image-based representations. ==Computational illumination== Controlling photographic illumination in a structured fashion, then processing the captured images, to create new images. The applications include image-based relighting, image enhancement, image deblurring, geometry/material recovery and so forth. * Flash/no-flash imaging ** Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., Toyama, K. (2004). [http://research.microsoft.com/projects/FlashNoFlash/ "Digital Photography with Flash and No-flash Image pairs"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH), Vol. 23, No. 3, pp. 664-672. ** Eisemann, E., Durand, F. (2004). [http://people.csail.mit.edu/fredo/PUBLI/flash/index.htm "Flash Photography Enhancement via Intrinsic Relighting"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH), Vol. 23, No. 3, pp. 673-678. ** Agrawal, A., Raskar, R., Nayar, S.K., Li, Y. (2005). [http://www.umiacs.umd.edu/~aagrawal/sig05/Gradient_Projection.html "Removing photography artifacts using gradient projection and flash-exposure sampling"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH), Vol. 24, No. 3, pp. 828-835. * Multi-flash imaging ** Raskar, R., Tan, K.-h., Feris, R., Yu, J., Turk, M., (2004). [http://www.merl.com/people/raskar/NprCamera/ " Non-photorealistic Camera: Depth Edge Detection and Stylized Rendering using Multi-Flash Imaging"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH), Vol. 23, No. 3, pp. 679-688. * Different exposures imaging ** Yuan L., Sun J., Quan L., Shum H.-Y. (2007). [http://portal.acm.org/citation.cfm?id=1276377.1276379"Image deblurring with blurred/noisy image pairs"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH), Vol. 26, No. 3, July 2007. * Image-based Relighting ** Nimeroff J., Simoncelli E., Dorsey J. (1994). [http://www.cns.nyu.edu/ftp/eero/nimeroff94a.pdf "Efficient Re-rendering of Naturally Illuminated Environments"], 5th Annual Eurographics Workshop on Rendering, Darmstadt Germany, June 13-15 ** Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., Sagar, M. (2000). [http://www.debevec.org/Research/LS/ "Acquiring the Reflectance Field of a Human Face"], ''Proc. ACM SIGGRAPH'', ACM Press, pp. 145-156. ** Malzbender, T., Gelb, D., Wolters, H. (2001). [http://www.hpl.hp.com/research/ptm/ "Polynomial Texture maps"], ''Proc. ACM SIGGRAPH'', ACM Press, pp. 519-528. ** Matusik W., Loper M., Pfister H.-P. (2004). [http://people.csail.mit.edu/wojciech/ReflFunctions/index.html "Progressively-Refined Reflectance Functions from natural Illumination"], Rendering Techniques 2004, pp. 299-308. ** Masselus, V., Peers, P., Dutré, P., Willems, Y.D. (2003). [http://portal.acm.org/citation.cfm?id=882315&dl=ACM&coll=&CFID=15151515&CFTOKEN=6184618 "Relighting with 4D Incident Light Fields"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH), Vol. 22, No. 3, pp. 613-620 ** Schechner, Y., Nayar., S., Belhumeur, P. (2003). [http://portal.acm.org/citation.cfm?id=946723&dl=ACM&coll=&CFID=15151515&CFTOKEN=6184618 "A Theory of Multiplexed Illumination"], ''Proc. ICCV 2003'', p. 808. * Other uses of structured illumination ** Levoy, M., Chen, B., Vaish, V., Horowitz, M., McDowall, I., Bolas, M. (2004). [http://graphics.stanford.edu/papers/confocal/ "Synthetic Aperture Confocal Imaging"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH), Vol. 23, No. 3, pp. 825-834. ** Sen, P., Chen, B., Garg, G., Marschner, S., Horowitz, M., Levoy, M., Lensch, H. (2005). [http://graphics.stanford.edu/papers/dual_photography/ "Dual Photography"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH), Vol. 24, No. 3, 2005, pp. 745-755. ** Nayar, S.K., Krishnan, G., Grossberg, M.D., Raskar, R. (2006). [http://portal.acm.org/citation.cfm?id=1141911.1141977&coll=&dl=ACM&idx=J778&part=periodical&WantType=periodical&title=ACM%20Transactions%20on%20Graphics%20(TOG)&CFID=15151515&CFTOKEN=6184618 "Fast Separation of Direct and Global Components of a Scene using High Frequency Illumination"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH), Vol. 25, No. 3, pp. 935-944. ** Zhang, L. Nayar, S. K. (2006).[http://www1.cs.columbia.edu/CAVE/projects/proj_def/proj_def_cc.php "Projection Defocus Analysis for Scene Capture and Image Display"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH) Jul, (2006) ==Computational optics== Capture of optically coded images, followed by computational decoding to produce new images. Coded aperture imaging was mainly applied in astronomy or X-ray imaging to boost the image quality. Instead of a single pin-hole, a pinhole pattern is applied in imaging, and [[deconvolution]] is performed to recover the image. In [http://www.merl.com/people/raskar/deblur/ coded exposure imaging], the on/off state of the shutter is coded to modify the kernel of [[motion blur]]. In this way motion deblurring becomes a [[well-conditioned problem]]. Similarly, in a lens based coded aperture, the aperture can be modified by inserting a [http://www.merl.com/people/raskar/Mask/ broadband mask]. Thus, out of focus deblurring becomes a [[well-conditioned problem]]. The coded aperture can also improve the quality in light field acquisition using Hadamard transform optics. * Coded aperture imaging ** Zand, J. (1996). [http://lheawww.gsfc.nasa.gov/docs/cai/coded_intr.html "Coded Aperture Imaging in High Energy Astronomy"], NASA Laboratory for High Energy Astrophysics (LHEA) at NASA's GSFC. ** Levin, A. , Fergus, R., Durand, F., Freeman, B. (2007) [http://groups.csail.mit.edu/graphics/CodedAperture/ " Image and Depth from a Conventional Camera with a Coded Aperture"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH) ** Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., Tumblin, J. (2007) [http://www.merl.com/people/raskar/Mask/ "Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH) ** Liang, C.K., Lin, T.H., Wong, B.Y., Liu, C., Chen, H. H. (2008). [http://mpac.ee.ntu.edu.tw/~chiakai/pap/ "Programmable Aperture Photography:Multiplexed Light Field Acquisition"], ''Proc. ACM SIGGRAPH''. * Coded exposure imaging ** Raskar, .R., Agrawal, A., Tumblin, J. (2006). [http://www.merl.com/people/raskar/deblur/ "Coded Exposure Photography: Motion Deblurring using Fluttered Shutter"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH), Vol. 25, No. 3, pp. 795-804. ** Agrawal, A., Raskar, .R. (2007). [http://www.merl.com/people/agrawal/cvpr07/SuperRes.html "Resolving Objects at Higher Resolution using a Single Motion Blurred Image"], ''IEEE Conference on Computer Vision and Pattern Recognition (CVPR)'' * [[Light field|Light field photography]] ** Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., Levoy, M. (2005). [http://graphics.stanford.edu/papers/CameraArray/ "High Performance Imaging Using Large Camera Arrays"], ''ACM Trans. on Graphics'' (Proc. SIGGRAPH), Vol 24, No 3, pp. 765-776. ** Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., Hanrahan, P. (2005). [http://graphics.stanford.edu/papers/lfcamera/ "Light Field Photography with a Hand-Held Plenoptic Camera"], Stanford Tech Report CTSR 2005-02, April, 2005. ** Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., Tumblin, J. (2007) [http://www.merl.com/people/raskar/Mask/ "Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH) * [[Catadioptric|Catadioptric imaging]] ** Baker, S., Nayar, S. (1999). [http://portal.acm.org/citation.cfm?coll=GUIDE&dl=GUIDE&id=339364 "A Theory of Single-Viewpoint Catadioptric Image Formation"], ''International Journal of Computer Vision'', Vol. 35, Issue 2, pp. 175-196. ** Kuthirummal, S., Nayar, S. K. (2006). [http://www1.cs.columbia.edu/CAVE/projects/rad_im/rad_im.php "Multiview Radial Catadioptric Imaging for Scene Capture"]''ACM Trans. on Graphics'' (Proc. of ACM SIGGRAPH). * [[Wavefront coding]] ** Cathey, W.T., Dowski, E.R. (2002). [http://www.cdm-optics.com/site/publications.php "New Paradigm for Imaging Systems"], ''Applied Optics'', Vol. 41, No. 29, 10 October 2002. * Compressive imaging ** [http://www.dsp.ece.rice.edu/cs/ Compressed Sensing Group], Kelly Lab, Rice University. ==Computational processing== Processing of non-optically coded images to produce new images. * [[Panorama|Panorama mosaicing]] ** Steve Mann and R. W. Picard "[http://www.eyetap.org/papers/docs/mann94virtual.pdf Virtual bellows: constructing high-quality images from video].", In Proceedings of the IEEE First International Conference on Image Processing, Austin, Texas, November 13-16, 1994 [Adobe PDF] ** Steve Mann and R.W. Picard "[http://www.eyetap.org/papers/docs/video_orbits.pdf Video orbits of the projective group a simple approach to featureless estimation of parameters] ", IEEE Transactions on Image Processing, pp. 1281 - 1295, Vol. 6, Iss. 9, 1997 ** Szeliski, R., Shum, H.-Y. (1997). [http://portal.acm.org/citation.cfm?id=258861&dl=GUIDE&coll=GUIDE "Creating full view panoramic image mosaics and environment maps"], ''Proc. ACM SIGGRAPH'', ACM Press, pp. 251-258. * [[Matte|Matte extraction]] ** Joshi, N., Matusik W., Avidan S. (2006). [http://graphics.ucsd.edu/papers/camera_array_matting/ "Natural Video Matting using Camera Arrays"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH). ** Chuang, Y.-Y., Curless, B., Salesin, D.H., Szeliski, R. (2001). [http://www.csie.ntu.edu.tw/~cyy/publications/index.html "Video Matting of Complex Scenes"], ''ACM Transactions on Graphics'' (Proc. SIGGRAPH), Vol. 21, No. 3, pp. 243-248. * [[Interactive Digital Photomontage|Digital photomontage]] (see also [[photomontage]]) ** Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., Cohen, M. (2004). [http://grail.cs.washington.edu/projects/photomontage/ "Interactive digital photomontage"], ''Proc. ACM SIGGRAPH'', ACM Press, pp. 294-302. * [[Super-resolution]] ** Baker, S., Kanade, T. (2002). [http://portal.acm.org/citation.cfm?coll=GUIDE&dl=GUIDE&id=628815 "Limits on super-resolution and how to break them"], ''IEEE Trans. PAMI'', Vol. 24, No. 8, August, 2002. * [[High dynamic range imaging]] ** S. Mann. Compositing multiple pictures of the same scene. In Proceedings of the 46th Annual IS&T Conference, Cambridge, Massachusetts, May 9-14 1993. The Society of Imaging Science and Technology. ** Mann, S. (1994). [http://citeseer.ist.psu.edu/mann95being.html "Being 'undigital' with digital cameras: Extending dynamic range by combining differently exposed pictures"], MIT Media Laboratory Perceptual Computing Section Technical Report TR-323, also appears in ''Proc. IS&T 46th Annual Conference, May 1995, pp. 422-428. ** Debevec, P.E., Malik, J. (1997). [http://www.debevec.org/Research/HDR/ "Recovering high dynamic range radiance maps from photographs" ], ''Proc. ACM SIGGRAPH'', ACM Press, pp. 369-378. ** Steve Mann, "[http://www.eyetap.org/papers/docs/comparametric.pdf Comparametric Equations with Practical Applications in Quantigraphic Image Processing]", IEEE Transactions on Image Processing, Vol 9, No.8, August, 2000 * All-focus imaging ** Haeberli, P., [http://www.graficaobscura.com/depth/index.html "A Multifocus Method for Controlling Depth of Field"], ''Graphica Obscura'', October, 1994. ==Computational sensors== Detectors that combine sensing and processing, typically in hardware. * [[Visual prosthetic|Artificial retinas]] * [http://dragon.larc.nasa.gov/background/retpubs.html Retinex sensors] * [http://isl.stanford.edu/groups/elgamal/nav_publications.html High dynamic range sensors] ==Early work in computer vision== Although computational photography is a currently popular buzzword, many of its techniques first appeared in the computer vision literature, either under other names or within papers aimed at 3D shape analysis. A few examples are: * Image-based relighting ** Hallinan, P.W. 1994. [http://ieeexplore.ieee.org/xpls/abs_all.jsp?tp=&arnumber=323941&isnumber=7716 "A low-dimensional representation of human faces for arbitrary lighting conditions"], ''Proc. CVPR 1994'', pp. 995-999. ** Belhumeur, P.N., Kriegman, D.J. 1998. [http://portal.acm.org/citation.cfm?id=294987&dl=ACM&coll=&CFID=15151515&CFTOKEN=6184618 "What Is the Set of Images of an Object under All Possible Illumination Conditions?"], ''International Journal of Computer Vision (IJCV)'', Vol. 28, Issue 3, p. 245-260. * All-focus imaging ** Subbarao, M., Choi, T. 1995. [http://portal.acm.org/citation.cfm?id=203385&dl=ACM&coll=&CFID=15151515&CFTOKEN=6184618 "Accurate recovery of three-dimensional shape from image focus"], ''IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI)'', Vol. 17, Issue 3, pp. 266 - 274. ==External links== ===Overviews=== * [http://graphics.stanford.edu/papers/lfphoto/comp-photo-articles/start-e.pdf Special issue on Computational Photography], IEEE Computer, August 2006. * Raskar, R., Tumblin, J., [http://www.amazon.com/dp/1568813139/ ''Computational Photography''], A.K. Peters. In press. ===Symposia=== * [http://photo.csail.mit.edu/ Symposium on Computational Photography and Video] (MIT, May 23-25, 2005) * [http://www.eecs.berkeley.edu/Research/Projects/CS/vision/workshop01/conference.html Workshop on the Convergence of Graphics, Vision, and Video] (UC Berkeley, May 28-30, 2001) * [http://graphics.stanford.edu/workshops/ibr98/ Workshop Image-Based Modeling and Rendering] (Stanford, March 23-25, 1998) ===Tools=== * [http://www.hp.com/idealab/us/en/relight.html Interactive Relighting Tool ] ===Courses=== * Carnegie Mellon's 15-463, Computational Photography (Alexei Efros) [http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/463.html 2005] [http://graphics.cs.cmu.edu/courses/15-463/2006_fall/www/463.html 2006] [http://graphics.cs.cmu.edu/courses/15-463/2007_fall/www/463.html 2007] * Georgia Tech's CS 4475 (Computational Photography) [http://www-static.cc.gatech.edu/classes/AY2005/cs4803cp_summer/ 2005] [http://www-static.cc.gatech.edu/classes/AY2006/cs4475_summer/ 2006] ([http://www.cc.gatech.edu/~irfan/ Irfan Essa]) * [http://www.merl.com/people/raskar/photo/course/ Northeastern University's CSG242] (Ramesh Raskar) * [http://groups.csail.mit.edu/graphics/classes/CompPhoto06/ MIT's 6.098/6.882] (Fredo Durand and Bill Freeman) * [http://www.merl.com/people/raskar/photo/ SIGGRAPH 2005-07 courses on Computational Photography] (Ramesh Raskar and Jack Tumblin) * Stanford's CS 448A (Marc Levoy) [http://graphics.stanford.edu/courses/cs448-04-spring/ 2004] [http://graphics.stanford.edu/courses/cs448a-06-winter/ 2006] [http://graphics.stanford.edu/courses/cs448a-08-spring/ 2008] * [http://www.csie.ntu.edu.tw/~cyy/courses/vfx/06spring/news/ NTU's Digital Visual Effects] (Yung-Yu Chuang) * [http://wearcam.org/ece516 University of Toronto's Intelligent Image Processing] [[Category:Digital photography]]