Skip to main content
SpringerLink
Log in

A Nuclear-Norm Model for Multi-Frame Super-Resolution Reconstruction from Video Clips

  • Chapter
  • First Online:
Structured Matrices in Numerical Linear Algebra

Part of the book series: Springer INdAM Series ((SINDAMS,volume 30))

  • 1308 Accesses

Abstract

We propose a variational approach to obtain super-resolution images from multiple low-resolution frames extracted from video clips. First the displacement between the low-resolution frames and the reference frame is computed by an optical flow algorithm. Then a low-rank model is used to construct the reference frame in high resolution by incorporating the information of the low-resolution frames. The model has two terms: a 2-norm data fidelity term and a nuclear-norm regularization term. Alternating direction method of multipliers is used to solve the model. Comparison of our methods with other models on synthetic and real video clips shows that our resulting images are more accurate with less artifacts. It also provides much finer and discernable details.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 59.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Altunbasak, Y., Patti, A., Mersereau, R.: Super-resolution still and video reconstruction from mpeg-coded video. IEEE Trans. Circuits Syst. Video Technol. 12(4), 217–226 (2002)

    Article  Google Scholar 

  2. Bishop, C.M., Blake, A., Marthi, B.: Super-resolution enhancement of video. In: Proc. Artificial Intelligence and Statistics, vol. 2. Key West, FL, USA (2003)

    Google Scholar 

  3. Bose, N., Boo, K.: High-resolution image reconstruction with multisensors. Int. J. Imaging Syst. Technol. 9(4), 294–304 (1998)

    Article  Google Scholar 

  4. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)

    Article  Google Scholar 

  5. Candès, E.J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM 58(3), 11 (2011)

    Article  MathSciNet  Google Scholar 

  6. Chan, R.H., Chan, T.F., Shen, L., Shen, Z.: Wavelet algorithms for high-resolution image reconstruction. SIAM J. Sci. Comput. 24(4), 1408–1432 (2003)

    Article  MathSciNet  Google Scholar 

  7. Chan, R.H., Riemenschneider, S.D., Shen, L., Shen, Z.: Tight frame: an efficient way for high-resolution image reconstruction. Appl. Comput. Harmon. Anal. 17(1), 91–115 (2004)

    Article  MathSciNet  Google Scholar 

  8. Chan, R.H., Shen, Z., Xia, T.: A framelet algorithm for enhancing video stills. Appl. Comput. Harmon. Anal. 23(2), 153–170 (2007)

    Article  MathSciNet  Google Scholar 

  9. Chen, X., Qi, C.: A single-image super-resolution method via low-rank matrix recovery and nonlinear mappings. In: 20th IEEE International Conference on Image Processing (ICIP), 2013, pp. 635–639 (2013). https://doi.org/10.1109/ICIP.2013.6738131

  10. Duponchel, L., Milanfar, P., Ruckebusch, C., Huvenne, J.P.: Super-resolution and Raman chemical imaging: from multiple low resolution images to a high resolution image. Anal. Chim. Acta 607(2), 168–175 (2008)

    Article  Google Scholar 

  11. Farsiu, S., Robinson, M.D., Elad, M., Milanfar, P.: Fast and robust multiframe super resolution. IEEE Trans. Image Process. 13(10), 1327–1344 (2004)

    Article  Google Scholar 

  12. Farsiu, S., Elad, M., Milanfar, P.: Multiframe demosaicing and super-resolution of color images. IEEE Trans. Image Process. 15(1), 141–159 (2006)

    Article  Google Scholar 

  13. Gilliam, C., Blu, T.: Local all-pass filters for optical flow estimation. In: IEEE International Conference on Acoustics, Speech and Signal Processing, vol. 1, IEEE (2015)

    Google Scholar 

  14. Jin, C., Nunez-Yanez, J., Achim, A.: Video super-resolution using low rank matrix completion. In: 20th IEEE International Conference on Image Processing (ICIP), 2013, Melbourne, Australia, pp. 1376–1380. (2013)

    Google Scholar 

  15. Levenberg, K.: A method for the solution of certain non-linear problems in least squares. Q. Appl. Math. 2, 164–168 (1944)

    Article  MathSciNet  Google Scholar 

  16. Li, Y.R., Dai, D.Q., Shen, L.: Multiframe super-resolution reconstruction using sparse directional regularization. IEEE Trans. Circuits Syst. Video Technol. 20(7), 945–956 (2010)

    Article  Google Scholar 

  17. Liu, C.: Beyond pixels: exploring new representations and applications for motion analysis. Ph.D. thesis, Citeseer (2009)

    Google Scholar 

  18. Liu, C., Sun, D.: On Bayesian adaptive video super resolution. IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 346–360 (2014)

    Article  Google Scholar 

  19. Lu, Y., Shen, L., Xu, Y.: Multi-parameter regularization methods for high-resolution image reconstruction with displacement errors. IEEE Trans. Circuits Syst. Regul. Pap. 54(8), 1788–1799 (2007)

    Article  MathSciNet  Google Scholar 

  20. Ma, Z., Liao, R., Tao, X., Xu, L., Jia, J., Wu, E.: Handling motion blur in multi-frame super-resolution. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5224–5232 (2015). http://dx.doi.org/10.1109/CVPR.2015.7299159

    Google Scholar 

  21. Marquardt, D.W.: An algorithm for least-squares estimation of nonlinear parameters. J. Soc. Ind. Appl. Math. 11(2), 431–441 (1962)

    Article  MathSciNet  Google Scholar 

  22. Narayanan, B., Hardie, R.C., Barner, K.E., Shao, M.: A computationally efficient super-resolution algorithm for video processing using partition filters. IEEE Trans. Circuits Syst. Video Technol. 17(5), 621–634 (2007)

    Article  Google Scholar 

  23. Ng, M.K., Chan, R.H., Tang, W.C.: A fast algorithm for deblurring models with Neumann boundary conditions. SIAM J. Sci. Comput. 21(3), 851–866 (1999)

    Article  MathSciNet  Google Scholar 

  24. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60(1), 259–268 (1992)

    Article  MathSciNet  Google Scholar 

  25. Shankar, P.M., Neifeld, M.A.: Sparsity constrained regularization for multiframe image restoration. JOSA A 25(5), 1199–1214 (2008)

    Article  Google Scholar 

  26. Shen, L., Sun, Q.: Biorthogonal wavelet system for high-resolution image reconstruction. IEEE Trans. Signal Process. 52(7), 1997–2011 (2004)

    Article  MathSciNet  Google Scholar 

  27. Takeda, H., Milanfar, P., Protter, M., Elad, M.: Super-resolution without explicit subpixel motion estimation. IEEE Trans. Image Process. 18(9), 1958–1975 (2009)

    Article  MathSciNet  Google Scholar 

  28. Tsai, R., Huang, T.: Multiframe image restoration and registration. Adv. Comput. Vis. Image Process. 1(2), 317–339 (1984)

    Google Scholar 

  29. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). http://dx.doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  30. Wang, C., Xue, P., Lin, W.: Improved super-resolution reconstruction from video. IEEE Trans. Circuits Syst. Video Technol. 16(11), 1411–1422 (2006)

    Article  Google Scholar 

  31. Zibetti, M.V.W., Mayer, J.: A robust and computationally efficient simultaneous super-resolution scheme for image sequences. IEEE Trans. Circuits Syst. Video Technol. 17(10), 1288–1300 (2007)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by HKRGC Grants Nos. CUHK14306316, HKRGC CRF Grant C1007-15G, and HKRGC AoE Grant AoE/M-05/12.

Author information

Authors and Affiliations

  1. Department of Mathematics, The Chinese University of Hong Kong, Shatin, NT, Hong Kong

    Rui Zhao

  2. Department of Mathematics, City University of Hong Kong, KLN, Hong Kong

    Raymond HF Chan

Authors
  1. Rui Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Raymond HF Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Raymond HF Chan .

Editor information

Editors and Affiliations

  1. Department of Mathematics, University of Pisa, Pisa, Italy

    Dario Andrea Bini

  2. Department of Mathematics, University of Genoa, Genoa, Italy

    Fabio Di Benedetto

  3. Institute of Numerical Mathematics, Russian Academy of Sciences, Moscow, Russia

    Eugene Tyrtyshnikov

  4. Department of Computer Science, KU Leuven, Heverlee, Belgium

    Marc Van Barel

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Zhao, R., Chan, R.H. (2019). A Nuclear-Norm Model for Multi-Frame Super-Resolution Reconstruction from Video Clips. In: Bini, D., Di Benedetto, F., Tyrtyshnikov, E., Van Barel, M. (eds) Structured Matrices in Numerical Linear Algebra. Springer INdAM Series, vol 30. Springer, Cham. https://doi.org/10.1007/978-3-030-04088-8_16

Download citation

Publish with us

Policies and ethics

Search

Navigation