Temporal Compressive Sensing for Video

  • Patrick LlullEmail author
  • Xin Yuan
  • Xuejun Liao
  • Jianbo Yang
  • David Kittle
  • Lawrence Carin
  • Guillermo Sapiro
  • David J. Brady
Part of the Applied and Numerical Harmonic Analysis book series (ANHA)


Video camera architects must design cameras capable of high-quality, dynamic event capture, while adhering to power and communications constraints. Though modern imagers are capable of both simultaneous spatial and temporal resolutions at micrometer and microsecond scales, the power required to sample at these rates is undesirable. The field of compressive sensing (CS) has recently suggested a solution to this design challenge. By exploiting physical-layer compression strategies, one may overlay the original scene with a coding sequence to sample at sub-Nyquist rates with virtually no additional power requirement. The underlying scene may be later estimated without significant loss of fidelity. In this chapter, we cover a variety of such strategies taken to improve an imager’s temporal resolution. Highlighting a new low-power acquisition paradigm, we show how a video sequence of high temporal resolution may be reconstructed from a single video frame taken with a low-framerate camera.


Gaussian Mixture Model Compressive Sense Transmission Function International Standard Organization Spatial Light Modulator 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The research, results, and theory presented here were supported by the Knowledge Enhanced Compressive Measurement Program at the Defense Advanced Research Projects Agency, grant N660011114002. Additional support from the ONR, NGA, ARO, and NSF is acknowledged.


  1. 1.
    Agrawal, A., Gupta, M., Veeraraghavan, A., Narasimhan S.G.: Optimal coded sampling for temporal super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 599–606, 2010Google Scholar
  2. 2.
    Aharon, M., Elad, M., Bruckstein, A.: K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54(11), 4311–4322 (2006)CrossRefGoogle Scholar
  3. 3.
    Arguello, H., Rueda, H.F., Arce, G.R.: Spatial super-resolution in code aperture spectral imaging. In: SPIE Defense, Security, and Sensing. International Society for Optics and Photonics, pp. 83650A–83650A, 2012Google Scholar
  4. 4.
    Baraniuk, R.G.: Single-pixel imaging via compressive sampling. IEEE Signal Process. Mag. 25(2), 83–91 (2008)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Bioucas-Dias, J.M., Figueiredo, M.A.T.: A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process. 16(12), 2992–3004 (2007)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Brady D.J.: Optical Imaging and Spectroscopy. Wiley-Interscience, New York (2009)CrossRefGoogle Scholar
  7. 7.
    Brady, D.J., Hagen, N.: Multiscale lens design. Opt. Express 17(13), 10659–10674 (2009)CrossRefGoogle Scholar
  8. 8.
    Brady, D.J., Feldman, M., Pitsianis, N., Guo, J.P., Portnoy, A., Fiddy, M.: Compressive optical montage photography. In: International Society for Optics and Photonics, Optics & Photonics 2005, pp. 590708–590708, 2005Google Scholar
  9. 9.
    Brady, D.J., Gehm, M.E., Stack, R.A., Marks, D.L., Kittle, D.S., Golish, D.R., Vera, E.M., Feller, D.: Multiscale gigapixel photography. Nature 486(7403), 386–389 (2012)CrossRefGoogle Scholar
  10. 10.
    Cai, J.-F., Osher, S., Shen Z.: Linearized bregman iterations for compressed sensing. Math. Comput. 78(267), 1515–1536 (2009)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Chen, M., Silva, J., Paisley, J., Wang, C., Dunson, D., Carin, L.: Compressive sensing on manifolds using a nonparametric mixture of factor analyzers: algorithm and performance bounds. IEEE Trans. Signal Process. 58(12), 6140–6155 (2010)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Elad, M., Aharon, M.: Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15(12), 3736–3745 (2006)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Gehm, M.E., John, R., Brady, D.J., Willett, R.M., Schulz T.J., et al.: Single-shot compressive spectral imaging with a dual-disperser architecture. Opt. Express 15(21), 14013–14027 (2007)Google Scholar
  16. 16.
    Goldstein, T., Xu, L., Kelly, K.F., Baraniuk, R.: The stone transform: Multi-resolution image enhancement and real-time compressive video. arXiv preprint arXiv:1311.3405 (2013)Google Scholar
  17. 17.
    Hitomi, Y., Gu, J., Gupta, M., Mitsunaga, T., Nayar, S.K.: Video from a single coded exposure photograph using a learned over-complete dictionary. In: IEEE International Conference on Computer Vision (ICCV), 2011 pp. 287–294, 2011Google Scholar
  18. 18.
    Holloway, J., Sankaranarayanan, A.C., Veeraraghavan, A., Tambe, S.: Flutter shutter video camera for compressive sensing of videos. In: IEEE International Conference on Computational Photography (ICCP), pp. 1–9, 2012Google Scholar
  19. 19.
    Hunt, J., Driscoll. T., Mrozack, A., Lipworth, G., Reynolds, M., Brady, D., Smith, D.R.: Metamaterial apertures for computational imaging. Science 339(6117), 310–313 ( 2013)Google Scholar
  20. 20.
    Ji, S., Xue, Y., Carin, L.: Bayesian compressive sensing. IEEE Trans. Signal Process. 56(6), 2346–2356 (2008)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Kittle, D., Choi, K., Wagadarikar, A., Brady, D.J.: Multiframe image estimation for coded aperture snapshot spectral imagers. Appl. Opt. 49(36), 6824–6833 (2010)CrossRefGoogle Scholar
  22. 22.
    Kleinfelder, S., Lim, S.H., Liu, X., Gamal, A.El.: A 10,000 frames/s 0.18 μm cmos digital pixel sensor with pixel-level memory. ISSCC Dig. Tech. Papers 88–89 (2001)Google Scholar
  23. 23.
    Liao, X., Li, H., Carin, L.: Generalized alternating projection for weighted- 2, 1 minimization with applications to model-based compressive sensing. SIAM J. Imag. Sci. 7(2), 797–823 (2014)Google Scholar
  24. 24.
    Llull, P., Liao, X., Yuan, X., Yang, J., Kittle, D., Carin, L., Sapiro, G., Brady, D.J.: Coded aperture compressive temporal imaging. Opt. Express 21(9), 10526–10545 (2013)CrossRefGoogle Scholar
  25. 25.
    Llull, P., Yuan, X., Liao, X., Yang J., Carin, L., Sapiro, G., Brady, D.J.: Compressive extended depth of field using image space coding. In: Computational Optical Sensing and Imaging, pp. CM2D–3. Optical Society of America, 2014Google Scholar
  26. 26.
    MacCabe, K., Krishnamurthy, K., Chawla, A., Marks, D., Samei, E., Brady, D.J.: Pencil beam coded aperture x-ray scatter imaging. Opt. Express 20(15), 16310–16320 (2012)CrossRefGoogle Scholar
  27. 27.
    MacCabe, K.P., Holmgren, A.D., Tornai, M.P., Brady, D.J.: Snapshot 2d tomography via coded aperture x-ray scatter imaging. Appl. Opt. 52(19), 4582–4589 (2013)CrossRefGoogle Scholar
  28. 28.
    Mairal, J., Elad, M., Sapiro, G.: Sparse representation for color image restoration. IEEE Trans. Image Process. 17, 53–69 (2008)MathSciNetCrossRefGoogle Scholar
  29. 29.
    McCain, S.T., Gehm, M.E., Wang, Y., Pitsianis, N.P., Brady, D.J.: Coded aperture raman spectroscopy for quantitative measurements of ethanol in a tissue phantom. Appl. Spectrosc. 60(6), 663–671 (2006)CrossRefGoogle Scholar
  30. 30.
    Osher, S., Mao, Y., Dong, B., Yin, W.: Fast linearized bregman iteration for compressive sensing and sparse denoising. arXiv preprint arXiv:1104.0262 (2011)Google Scholar
  31. 31.
    Raskar, R., Agrawal, A., Tumblin, J.: Coded exposure photography: motion deblurring using fluttered shutter. ACM Trans. Graph. 25, 795–804 (2006)CrossRefGoogle Scholar
  32. 32.
    Reddy, D., Veeraraghavan, A., Chellappa, R.: P2C2: Programmable pixel compressive camera for high speed imaging. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011, pp. 329–336, 2011Google Scholar
  33. 33.
    Salehi, J.A.: Code division multiple-access techniques in optical fiber networks. i. fundamental principles. IEEE Trans. Commun. 37(8), 824–833 (1989)Google Scholar
  34. 34.
    Sankaranarayanan, A.C., Studer, C., Baraniuk, R.G.: CS-MUVI: Video compressive sensing for spatial-multiplexing cameras. In: IEEE International Conference on Computational Photography (ICCP), pp. 1–10, 2012Google Scholar
  35. 35.
    Shankar, M., Pitsianis, N.P., Brady, D.J.: Compressive video sensors using multichannel imagers. Appl. Opt. 49(10), B9–B17 (2010)CrossRefGoogle Scholar
  36. 36.
    Son, H., Marks, D.L, Tremblaym, E.J., Ford, J., Hahn, J., Stack, R., Johnson, A., McLaughlin, P., Shaw, J., Kim, J.: A multiscale, wide field, gigapixel camera. In: Computational Optical Sensing and Imaging, p. JTuE2, Optical Society of America, 2011Google Scholar
  37. 37.
    Teich, M.C., Saleh, B.: Fundamentals of Photonics, p. 3. Wiley Interscience, Canada (1991)Google Scholar
  38. 38.
    Tendero, Y., Rougé, B., Morel, J.-M.: A formalization of the flutter shutter. J. Phys. Conf. Ser. 386(1), 012001 IOP Publishing (2012)Google Scholar
  39. 39.
    Tropp, J., Gilbert, A.C.: Signal recovery from partial information via orthogonal matching pursuit (2005)Google Scholar
  40. 40.
    Tsai, T.-H., Brady, D.J.: Coded aperture snapshot spectral polarization imaging. Appl. Opt. 52(10), 2153–2161 (2013)CrossRefGoogle Scholar
  41. 41.
    Tsaig, Y., Donoho, D.L.: Extensions of compressed sensing. Signal Process. 86(3), 549–571 (2006)CrossRefGoogle Scholar
  42. 42.
    Veeraraghavan, A., Reddy, D., Raskar, R.: Coded strobing photography: compressive sensing of high speed periodic videos. IEEE Trans. Pattern Anal. Mach. Intell. 33(4), 671–686 (2011)CrossRefGoogle Scholar
  43. 43.
    Wagadarikar, A., John, R., Willett, R., Brady, D.: Single disperser design for coded aperture snapshot spectral imaging. Appl. Opt. 47(10), B44–B51 (2008)CrossRefGoogle Scholar
  44. 44.
    Wagadarikar, A.A., Pitsianis, N.P., Sun, X., Brady, D.J.: Video rate spectral imaging using a coded aperture snapshot spectral imager. Opt. Express 17(8), 6368–6388 (2009)CrossRefGoogle Scholar
  45. 45.
    Watts, C.M., Shrekenhamer, D., Montoya, J., Lipworth, G., Hunt, J., Sleasman, T., Krishna, S., Smith, D.R., Padilla,W.J.: Coded and compressive thz imaging with metamaterials. In: International Society for Optics and Photonics SPIE OPTO, pp. 89851N–89851N, 2014Google Scholar
  46. 46.
    Yang, J., Liao, X., Yuan, X., Llull, P., Brady, D.J., Sapiro, G., Carin, L.: Compressive Sensing by learning a gaussian mixture model from measurements. IEEE Trans. Image Process 24(1), 106–119 (2015). doi:  10.1109/TIP.2014.2365720 MathSciNetCrossRefGoogle Scholar
  47. 47.
    Yang, J., Yuan, X., Liao, X., Llull, P., Brady, D.J., Sapiro, G., Carin, L.: Video compressive sensing using Gaussian mixture models. IEEE Trans. Image Process 23(11), 4863–4878 (2014). doi:  10.1109/TIP.2014.2344294 MathSciNetCrossRefGoogle Scholar
  48. 48.
    Yang, J., Yuan, X., Liao, X., Llull, P., Sapiro, G., Brady, D.J., Carin, L.: Gaussian mixture model for video compressive sensing. In: International Conference on Image Processing, 2013CrossRefGoogle Scholar
  49. 49.
    Yu, G., Sapiro, G.: Statistical compressed sensing of Gaussian mixture models. IEEE Trans. Signal Process. 59(12), 5842–5858 (2011)MathSciNetCrossRefGoogle Scholar
  50. 50.
    Yu, G., Sapiro, G., Mallat, S.: Solving inverse problems with piecewise linear estimators: from Gaussian mixture models to structured sparsity. IEEE Trans. Image Process. 21(5), 2481–2499 (2012)MathSciNetCrossRefGoogle Scholar
  51. 51.
    Yuan, X., Yang, J., Llull, P., Liao, X., Sapiro, G., Brady, D.J., Carin, L.: Adaptive temporal compressive sensing for video. In: IEEE International Conference on Image Processing, 2013CrossRefGoogle Scholar
  52. 52.
    Yuan, X., Llull, P., Liao, X., Yang, J., Sapiro, G., Brady, D.J., Carin, L.: Low-cost compressive sensing for color video and depth. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014Google Scholar
  53. 53.
    Zhou, M., Chen, H., Paisley, J., Ren, L., Li, L., Xing, Z., Dunson, D., Sapiro, G., Carin, L.: Nonparametric Bayesian dictionary learning for analysis of noisy and incomplete images. IEEE Trans. Image Process. 21(1), 130–144 (2012)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Patrick Llull
    • 1
    Email author
  • Xin Yuan
    • 1
  • Xuejun Liao
    • 1
  • Jianbo Yang
    • 1
  • David Kittle
    • 1
  • Lawrence Carin
    • 1
  • Guillermo Sapiro
    • 1
  • David J. Brady
    • 1
  1. 1.Department of Electrical and Computer EngineeringDuke UniversityDurhamUSA

Personalised recommendations