Emerging theories and technologies on computational imaging



Computational imaging describes the whole imaging process from the perspective of light transport and information transmission, features traditional optical computing capabilities, and assists in breaking through the limitations of visual information recording. Progress in computational imaging promotes the development of diverse basic and applied disciplines. In this review, we provide an overview of the fundamental principles and methods in computational imaging, the history of this field, and the important roles that it plays in the development of science. We review the most recent and promising advances in computational imaging, from the perspective of different dimensions of visual signals, including spatial dimension, temporal dimension, angular dimension, spectral dimension, and phase. We also discuss some topics worth studying for future developments in computational imaging.

Key words

Computational imaging Multi-scale and multi-dimensional Super-resolution Femto-photography 3D reconstruction Hyperspectral imaging 

CLC number



  1. Assion, A., Baumert, T., Bergt, M., et al., 1998. Control of chemical reactions by feedback-optimized phase-shaped femtosecond laser pulses. Science, 282(5390):919–922. https://doi.org/10.1126/science.282.5390.919CrossRefGoogle Scholar
  2. Backman, V., Wallace, M.B., Perelman, L., et al., 2000. Detection of preinvasive cancer cells. Nature, 406(6791): 35–36. https://doi.org/10.1038/35017638CrossRefGoogle Scholar
  3. Bao, J., Bawendi, M.G., 2015. A colloidal quantum dot spectrometer. Nature, 523(7558):67–70. https://doi.org/10.1038/nature14576CrossRefGoogle Scholar
  4. Bifano, T., 2011. Adaptive imaging: MEMS deformable mirrors. Nat. Photon., 5(1):21–23. https://doi.org/10.1038/nphoton.2010.297CrossRefGoogle Scholar
  5. Bina, M., Magatti, D., Molteni, M., et al., 2013. Backscattering differential ghost imaging in turbid media. Phys. Rev. Lett., 110(8), Article 083901. https://doi.org/10.1103/PhysRevLett.110.083901Google Scholar
  6. Brady, D., Gehm, M., Stack, R., et al., 2012. Multiscale gigapixel photography. Nature, 486(7403):386–389. https://doi.org/10.1038/nature11150CrossRefGoogle Scholar
  7. Brenner, D.J., Hall, E.J., 2007. Computed tomography—an increasing source of radiation exposure. New Engl. J. Med., 357:2277–2284. https://doi.org/10.1056/NEJMra072149CrossRefGoogle Scholar
  8. Candès, E.J., Romberg, J., Tao, T., 2006. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inform. Theory, 52(2):489–509. https://doi.org/10.1109/TIT.2005.862083MathSciNetMATHCrossRefGoogle Scholar
  9. Chaigne, T., Katz, O., Boccara, A.C., et al., 2014. Controlling light in scattering media non-invasively using the photoacoustic transmission matrix. Nat. Photon., 8(1):58–64. https://doi.org/10.1038/nphoton.2013.307CrossRefGoogle Scholar
  10. Chakrabarti, A., Zickler, T., 2011. Statistics of real-world hyperspectral images. IEEE Conf. on Computer Vision and Pattern Recognition, p.193–200. https://doi.org/10.1109/CVPR.2011.5995660Google Scholar
  11. Chao, T.H., Zhou, H., Xia, X., et al., 2005. Near IR electrooptic imaging Fourier transform spectrometer. Proc. Optical Pattern Recognition, p.163–172. https://doi.org/10.1117/12.607968CrossRefGoogle Scholar
  12. Charles, A.S., Olshausen, B.A., Rozell, C.J., 2011. Learning sparse codes for hyperspectral imagery. IEEE J. Sel. Topics Signal Process., 5(5):963–978. https://doi.org/10.1109/JSTSP.2011.2149497CrossRefGoogle Scholar
  13. Choi, W., Fang-Yen, C., Badizadegan, K.R., et al., 2007. Tomographic phase microscopy. Nat. Meth., 4(9):717–719. https://doi.org/10.1038/NMETH1078CrossRefGoogle Scholar
  14. Cotte, Y., Toy, F., Jourdain, P., et al., 2013. Marker-free phase nanoscopy. Nat. Photon., 7(2):113–117. https://doi.org/10.1038/nphoton.2012.329CrossRefGoogle Scholar
  15. Cuche, E., Bevilacqua, F., Depeursinge, C., 1999. Digital holography for quantitative phase-contrast imaging. Opt. Lett., 24(5):291–293. https://doi.org/10.1364/OL.24.000291CrossRefGoogle Scholar
  16. Delalieux, S., Auwerkerken, A., Verstraeten, W.W., et al., 2009. Hyperspectral reflectance and fluorescence imaging to detect scab induced stress in apple leaves. Remote Sens., 1(4):858–874. https://doi.org/10.3390/rs1040858CrossRefGoogle Scholar
  17. Descour, M., Dereniak, E., 1995. Computed-tomography imaging spectrometer: experimental calibration and reconstruction results. Appl. Opt., 34(22):4817–4826. https://doi.org/10.1364/AO.34.004817CrossRefGoogle Scholar
  18. Diaspro, A., Chirico, G., Collini, M., 2005. Two-photon fluorescence excitation and related techniques in biological microscopy. Q. Rev. Biophys., 38(02):97–166. https://doi.org/10.1017/S0033583505004129CrossRefGoogle Scholar
  19. Ding, W., Wang, Y., Chen, H., et al., 2014. Plasmonic nanocavity organic light-emitting diode with significantly enhanced light extraction, contrast, viewing angle, brightness, and low-glare. Adv. Funct. Mater., 24(40):6329–6339. https://doi.org/10.1017/S0033583505004129CrossRefGoogle Scholar
  20. Ferguson, R., Phillips, W., 1967. High-resolution nuclear magnetic resonance spectroscopy. Science, 157(3786): 257–267. https://doi.org/10.1126/science.157.3786.257CrossRefGoogle Scholar
  21. Fienup, J.R., 1982. Phase retrieval algorithms: a comparison. Appl. Opt., 21(15):2758–2769. https://doi.org/10.1364/AO.21.002758CrossRefGoogle Scholar
  22. Fienup, J.R., 2013. Phase retrieval algorithms: a personal tour [invited]. Appl. Opt., 52(1):45–56. https://doi.org/10.1364/AO.52.000045CrossRefGoogle Scholar
  23. Frenkel, K.A., 2010. Panning for science. Science, 330(6005):748–749. https://doi.org/10.1126/science.330.6005.748CrossRefGoogle Scholar
  24. Gatti, A., Brambilla, E., Bache, M., et al., 2004. Ghost imaging with thermal light: comparing entanglement and classical correlation. Phys. Rev. Lett., 93(9), Article 093602. https://doi.org/10.1103/PhysRevLett.93.093602Google Scholar
  25. Gebbie, H., 1961. Molecular emission spectroscopy from 2μ to 12μ by a michelson interferometer. Nature, 191:264–265. https://doi.org/10.1038/191264a0CrossRefGoogle Scholar
  26. Goda, K., Tsia, K., Jalali, B., 2009. Serial time-encoded amplified imaging for real-time observation of fast dynamic phenomena. Nature, 458(7242):1145–1149. https://doi.org/10.1038/nature07980CrossRefGoogle Scholar
  27. Greenbaum, A., Luo, W., Su, T.W., et al., 2012. Imaging without lenses: achievements and remaining challenges of wide-field on-chip microscopy. Nat. Meth., 9(9):889–895. https://doi.org/10.1038/nmeth.2114CrossRefGoogle Scholar
  28. Gustafsson, M.G., 2005. Nonlinear structured-illumination microscopy: wide-field fluorescence imaging with theoretically unlimited resolution. PNAS, 102(37):13081–13086. https://doi.org/10.1073/pnas.0406877102CrossRefGoogle Scholar
  29. Heide, F., Hullin, M.B., Gregson, J., et al., 2013. Low-budget transient imaging using photonic mixer devices. ACM Trans. Graph., 32(4), Article ai]45. https://doi.org/10.1145/2461912.2461945Google Scholar
  30. Hein, B., Willig, K.I., Hell, S.W., 2008. Stimulated emission depletion (STED) nanoscopy of a fluorescent protein-labeled organelle inside a living cell. PNAS, 105(38):14271–14276. https://doi.org/10.1073/pnas.0807705105CrossRefGoogle Scholar
  31. Hell, S.W., Wichmann, J., 1994. Breaking the diffraction resolution limit by stimulated emission: stimulatedemission- depletion fluorescence microscopy. Opt. Lett., 19(11):780–782. https://doi.org/10.1364/OL.19.000780CrossRefGoogle Scholar
  32. Helmchen, F., Denk, W., 2005. Deep tissue two-photon microscopy. Nat. Meth., 2(12):932–940. https://doi.org/10.1038/nmeth818CrossRefGoogle Scholar
  33. Hess, S.T., Girirajan, T.P., Mason, M.D., 2006. Ultrahigh resolution imaging by fluorescence photoactivation localization microscopy. Biophys. J., 91(11):4258–4272. https://doi.org/10.1529/biophysj.106.091116CrossRefGoogle Scholar
  34. Horton, N.G., Wang, K., Kobat, D., et al., 2013. In vivo three-photon microscopy of subcortical structures within an intact mouse brain. Nat. Photon., 7(3):205–209. https://doi.org/10.1038/nphoton.2012.336CrossRefGoogle Scholar
  35. Howard, S.S., Straub, A., Horton, N.G., et al., 2013. Frequency-multiplexed in vivo multiphoton phosphorescence lifetime microscopy. Nat. Photon., 7(1):33–37. https://doi.org/0.1038/nphoton.2012.307CrossRefGoogle Scholar
  36. Jahr, W., Schmid, B., Schmied, C., et al., 2015. Hyperspectral light sheet microscopy. Nat. Commun., 6, Article 7990. https://doi.org/10.1038/ncomms8990Google Scholar
  37. Ji, N., Milkie, D.E., Betzig, E., 2010. Adaptive optics via pupil segmentation for high-resolution imaging in biological tissues. Nat. Meth., 7(2):141–147. https://doi.org/10.1038/nmeth.1411CrossRefGoogle Scholar
  38. Kester, R.T., Bedard, N., Gao, L., et al., 2011. Real-time snapshot hyperspectral imaging endoscope. J. Biomed. Opt., 16(5), Article 056005. https://doi.org/10.1117/1.3574756Google Scholar
  39. Kim, T., Zhou, R., Mir, M., et al., 2014. White-light diffraction tomography of unlabelled live cells. Nat. Photon., 8(3):256–263. https://doi.org/10.1038/nphoton.2013.350CrossRefGoogle Scholar
  40. Levoy, M., Hanrahan, P., 1996. Light field rendering. Proc. 23rd Annual Conf. on Computer Graphics and Interactive Techniques, p.31–42. https://doi.org/10.1145/237170.237199Google Scholar
  41. Levoy, M., Ng, R., Adams, A., et al., 2006. Light field microscopy. ACM Trans. Graph., 25(3):924–934. https://doi.org/10.1145/1141911.1141976CrossRefGoogle Scholar
  42. Lin, X., Liu, Y., Wu, J., et al., 2014. Spatial-spectral encoded compressive hyperspectral imaging. ACM Trans. Graph., 33(6), Article 233. https://doi.org/10.1145/2661229.2661262Google Scholar
  43. Lin, X., Wu, J., Zheng, G., et al., 2015. Camera array based light field microscopy. Biomed. Opt. Expr., 6(9):3179–3189. https://doi.org/10.1364/BODA.2015.JT3A.48CrossRefGoogle Scholar
  44. Ma, C., Cao, X., Tong, X., et al., 2014. Acquisition of high spatial and spectral resolution video with a hybrid camera system. Int. J. Comput. Vis., 110(2):141–155. https://doi.org/10.1007/s11263-013-0690-4CrossRefGoogle Scholar
  45. Manley, S., Gillette, J.M., Patterson, G.H., et al., 2008. High-density mapping of single-molecule trajectories with photoactivated localization microscopy. Nat. Meth., 5(2):155–157. https://doi.org/10.1038/nmeth.1176CrossRefGoogle Scholar
  46. Marks, D.L., Son, H.S., Kim, J., et al., 2012. Engineering a gigapixel monocentric multiscale camera. Opt. Eng., 51(8), Article 083202. https://doi.org/10.1117/1.OE.51.8.083202Google Scholar
  47. Morris, P.A., Aspden, R.S., Bell, J.E., et al., 2015. Imaging with a small number of photons. Nat. Commun., 6, Article 5913. https://doi.org/10.1038/ncomms6913Google Scholar
  48. Nakagawa, K., Iwasaki, A., Oishi, Y., et al., 2014. Sequentially timed all-optical mapping photography (STAMP). Nat. Photon., 8(9):695–700. https://doi.org/10.1038/nphoton.2014.163CrossRefGoogle Scholar
  49. Neifeld, M.A., Shankar, P., 2003. Feature-specific imaging. Appl. Opt., 42(17):3379–3389. https://doi.org/10.1364/AO.42.003379CrossRefGoogle Scholar
  50. Ng, R., Levoy, M., Brédif, M., et al., 2005. Light field photography with a hand-held plenoptic camera. Comput. Sci. Techn. Rep., 2(11):1–11.Google Scholar
  51. Orth, A., Tomaszewski, M.J., Ghosh, R.N., et al., 2015. Gigapixel multispectral microscopy. Optica, 2(7):654–662. https://doi.org/10.1364/OPTICA.2.000654CrossRefGoogle Scholar
  52. Pal, H., Neifeld, M., 2003. Multispectral principal component imaging. Opt. Expr., 11(18):2118–2125. https://doi.org/10.1364/OE.11.002118CrossRefGoogle Scholar
  53. Popescu, G., Deflores, L.P., Vaughan, J.C., et al., 2004. Fourier phase microscopy for investigation of biological structures and dynamics. Opt. Lett., 29(21):2503–2505. https://doi.org/10.1364/OL.29.002503CrossRefGoogle Scholar
  54. Prevedel, R., Yoon, Y.G., Hoffmann, M., et al., 2014. Simultaneous whole-animal 3D imaging of neuronal activity using light-field microscopy. Nat. Meth., 11(7):727–730. https://doi.org/10.1038/nmeth.2964CrossRefGoogle Scholar
  55. Rust, M.J., Bates, M., Zhuang, X., 2006. Sub-diffractionlimit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Meth., 3(10):793–796. https://doi.org/10.1038/nmeth929CrossRefGoogle Scholar
  56. Ryle, M., 1972. The 5-km radio telescope at Cambridge. Nature, 239:435–438. https://doi.org/10.1038/239435a0CrossRefGoogle Scholar
  57. Schermelleh, L., Heintzmann, R., Leonhardt, H., 2010. A guide to super-resolution fluorescence microscopy. J. Cell Biol., 190(2):165–175. https://doi.org/10.1083/jcb.201002018CrossRefGoogle Scholar
  58. Stoklasa, B., Motka, L., Rehacek, J., et al., 2014. Wavefront sensing reveals optical coherence. Nat. Commun., 5, Article 3275. https://doi.org/10.1038/ncomms4275Google Scholar
  59. Strack, R., 2016. Highly multiplexed imaging. Nat. Meth., 13(1), Article 35. https://doi.org/10.1038/nmeth.3706Google Scholar
  60. Suo, J., Bian, L., Chen, F., et al., 2014. Bispectral coding: compressive and high-quality acquisition of fluorescence and reflectance. Opt. Expr., 22(2):1697–1712. https://doi.org/10.1364/OE.22.001697CrossRefGoogle Scholar
  61. Teague, M.R., 1983. Deterministic phase retrieval: a green’s function solution. JOSA, 73(11):1434–1441. https://doi.org/10.1364/JOSA.73.001434CrossRefGoogle Scholar
  62. van Tilbeurgh, H., Egloff, M., Martinez, C., et al., 1993. Interfacial activation of the lipase-procolipase complex by mixed micelles revealed by X-ray crystallography. Nature, 362(6423):814–820. https://doi.org/10.1038/362814a0CrossRefGoogle Scholar
  63. Vellekoop, I., Lagendijk, A., Mosk, A., 2010. Exploiting disorder for perfect focusing. Nat. Photon., 4(5):320–322. https://doi.org/10.1038/nphoton.2010.3CrossRefGoogle Scholar
  64. Velten, A., Willwacher, T., Gupta, O., et al., 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nat. Commun., 3, Article ai]745. https://doi.org/10.1038/ncomms1747Google Scholar
  65. Velten, A., Wu, D., Jarabo, A., et al., 2013. Femtophotography: capturing and visualizing the propagation of light. ACM Trans. Graph., 32(4), Article ai]44. https://doi.org/10.1145/2461912.2461928Google Scholar
  66. Waller, L., Kou, S.S., Sheppard, C.J., et al., 2010a. Phase from chromatic aberrations. Opt. Expr., 18(22):22817–22825. https://doi.org/10.1364/OE.18.022817CrossRefGoogle Scholar
  67. Waller, L., Tian, L., Barbastathis, G., 2010b. Transport of intensity phase-amplitude imaging with higher order intensity derivatives. Opt. Expr., 18(12):12552–12561. https://doi.org/10.1364/OE.18.012552CrossRefGoogle Scholar
  68. Waller, L., Situ, G., Fleischer, J.W., 2012. Phase-space measurement and coherence synthesis of optical beams. Nat. Photon., 6(7):474–479. https://doi.org/10.1038/nphoton.2012.144CrossRefGoogle Scholar
  69. Wang, L.V., Hu, S., 2012. Photoacoustic tomography: in vivo imaging from organelles to organs. Science, 335(6075):1458–1462. https://doi.org/10.1126/science.1216210CrossRefGoogle Scholar
  70. Wilburn, B., Joshi, N., Vaish, V., et al., 2004. Highspeed videography using a dense camera array. Proc. IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, p.294–301. https://doi.org/10.1109/CVPR.2004.1315176Google Scholar
  71. Willett, R., Gehm, M.E., Brady, D.J., 2007. Multiscale reconstruction for computational spectral imaging. Proc. Electronic Imaging, Article 64980L. https://doi.org/10.1117/12.715711Google Scholar
  72. Wong, G., 2009. Snapshot hyperspectral imaging and practical applications. J. Phys., 178(1), Article 012048. https://doi.org/10.1088/1742-6596/178/1/012048Google Scholar
  73. Zernike, F., 1955. How I discovered phase contrast. Science, 121(3141):345–349.CrossRefGoogle Scholar
  74. Zheng, G., Horstmeyer, R., Yang, C., 2013. Wide-field, high-resolution Fourier ptychographic microscopy. Nat. Photon., 7(9):739–745. https://doi.org/10.1038/nphoton.2013.187CrossRefGoogle Scholar

Copyright information

© Zhejiang University and Springer-Verlag GmbH Germany, part of Springer Nature 2017

Authors and Affiliations

  1. 1.Department of AutomationTsinghua UniversityBeijingChina
  2. 2.Zhejiang Future Technology InstituteJiaxingChina

Personalised recommendations