A Nuclear-Norm Model for Multi-Frame Super-Resolution Reconstruction from Video Clips

  • Rui Zhao
  • Raymond HF ChanEmail author
Part of the Springer INdAM Series book series (SINDAMS, volume 30)


We propose a variational approach to obtain super-resolution images from multiple low-resolution frames extracted from video clips. First the displacement between the low-resolution frames and the reference frame is computed by an optical flow algorithm. Then a low-rank model is used to construct the reference frame in high resolution by incorporating the information of the low-resolution frames. The model has two terms: a 2-norm data fidelity term and a nuclear-norm regularization term. Alternating direction method of multipliers is used to solve the model. Comparison of our methods with other models on synthetic and real video clips shows that our resulting images are more accurate with less artifacts. It also provides much finer and discernable details.


Image processing Super-resolution Low-rank approximation 



This work was supported by HKRGC Grants Nos. CUHK14306316, HKRGC CRF Grant C1007-15G, and HKRGC AoE Grant AoE/M-05/12.


  1. 1.
    Altunbasak, Y., Patti, A., Mersereau, R.: Super-resolution still and video reconstruction from mpeg-coded video. IEEE Trans. Circuits Syst. Video Technol. 12(4), 217–226 (2002)CrossRefGoogle Scholar
  2. 2.
    Bishop, C.M., Blake, A., Marthi, B.: Super-resolution enhancement of video. In: Proc. Artificial Intelligence and Statistics, vol. 2. Key West, FL, USA (2003)Google Scholar
  3. 3.
    Bose, N., Boo, K.: High-resolution image reconstruction with multisensors. Int. J. Imaging Syst. Technol. 9(4), 294–304 (1998)CrossRefGoogle Scholar
  4. 4.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)CrossRefGoogle Scholar
  5. 5.
    Candès, E.J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM 58(3), 11 (2011)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Chan, R.H., Chan, T.F., Shen, L., Shen, Z.: Wavelet algorithms for high-resolution image reconstruction. SIAM J. Sci. Comput. 24(4), 1408–1432 (2003)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Chan, R.H., Riemenschneider, S.D., Shen, L., Shen, Z.: Tight frame: an efficient way for high-resolution image reconstruction. Appl. Comput. Harmon. Anal. 17(1), 91–115 (2004)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Chan, R.H., Shen, Z., Xia, T.: A framelet algorithm for enhancing video stills. Appl. Comput. Harmon. Anal. 23(2), 153–170 (2007)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Chen, X., Qi, C.: A single-image super-resolution method via low-rank matrix recovery and nonlinear mappings. In: 20th IEEE International Conference on Image Processing (ICIP), 2013, pp. 635–639 (2013).
  10. 10.
    Duponchel, L., Milanfar, P., Ruckebusch, C., Huvenne, J.P.: Super-resolution and Raman chemical imaging: from multiple low resolution images to a high resolution image. Anal. Chim. Acta 607(2), 168–175 (2008)CrossRefGoogle Scholar
  11. 11.
    Farsiu, S., Robinson, M.D., Elad, M., Milanfar, P.: Fast and robust multiframe super resolution. IEEE Trans. Image Process. 13(10), 1327–1344 (2004)CrossRefGoogle Scholar
  12. 12.
    Farsiu, S., Elad, M., Milanfar, P.: Multiframe demosaicing and super-resolution of color images. IEEE Trans. Image Process. 15(1), 141–159 (2006)CrossRefGoogle Scholar
  13. 13.
    Gilliam, C., Blu, T.: Local all-pass filters for optical flow estimation. In: IEEE International Conference on Acoustics, Speech and Signal Processing, vol. 1, IEEE (2015)Google Scholar
  14. 14.
    Jin, C., Nunez-Yanez, J., Achim, A.: Video super-resolution using low rank matrix completion. In: 20th IEEE International Conference on Image Processing (ICIP), 2013, Melbourne, Australia, pp. 1376–1380. (2013)Google Scholar
  15. 15.
    Levenberg, K.: A method for the solution of certain non-linear problems in least squares. Q. Appl. Math. 2, 164–168 (1944)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Li, Y.R., Dai, D.Q., Shen, L.: Multiframe super-resolution reconstruction using sparse directional regularization. IEEE Trans. Circuits Syst. Video Technol. 20(7), 945–956 (2010)CrossRefGoogle Scholar
  17. 17.
    Liu, C.: Beyond pixels: exploring new representations and applications for motion analysis. Ph.D. thesis, Citeseer (2009)Google Scholar
  18. 18.
    Liu, C., Sun, D.: On Bayesian adaptive video super resolution. IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 346–360 (2014)CrossRefGoogle Scholar
  19. 19.
    Lu, Y., Shen, L., Xu, Y.: Multi-parameter regularization methods for high-resolution image reconstruction with displacement errors. IEEE Trans. Circuits Syst. Regul. Pap. 54(8), 1788–1799 (2007)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Ma, Z., Liao, R., Tao, X., Xu, L., Jia, J., Wu, E.: Handling motion blur in multi-frame super-resolution. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5224–5232 (2015). Scholar
  21. 21.
    Marquardt, D.W.: An algorithm for least-squares estimation of nonlinear parameters. J. Soc. Ind. Appl. Math. 11(2), 431–441 (1962)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Narayanan, B., Hardie, R.C., Barner, K.E., Shao, M.: A computationally efficient super-resolution algorithm for video processing using partition filters. IEEE Trans. Circuits Syst. Video Technol. 17(5), 621–634 (2007)CrossRefGoogle Scholar
  23. 23.
    Ng, M.K., Chan, R.H., Tang, W.C.: A fast algorithm for deblurring models with Neumann boundary conditions. SIAM J. Sci. Comput. 21(3), 851–866 (1999)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60(1), 259–268 (1992)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Shankar, P.M., Neifeld, M.A.: Sparsity constrained regularization for multiframe image restoration. JOSA A 25(5), 1199–1214 (2008)CrossRefGoogle Scholar
  26. 26.
    Shen, L., Sun, Q.: Biorthogonal wavelet system for high-resolution image reconstruction. IEEE Trans. Signal Process. 52(7), 1997–2011 (2004)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Takeda, H., Milanfar, P., Protter, M., Elad, M.: Super-resolution without explicit subpixel motion estimation. IEEE Trans. Image Process. 18(9), 1958–1975 (2009)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Tsai, R., Huang, T.: Multiframe image restoration and registration. Adv. Comput. Vis. Image Process. 1(2), 317–339 (1984)Google Scholar
  29. 29.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). Scholar
  30. 30.
    Wang, C., Xue, P., Lin, W.: Improved super-resolution reconstruction from video. IEEE Trans. Circuits Syst. Video Technol. 16(11), 1411–1422 (2006)CrossRefGoogle Scholar
  31. 31.
    Zibetti, M.V.W., Mayer, J.: A robust and computationally efficient simultaneous super-resolution scheme for image sequences. IEEE Trans. Circuits Syst. Video Technol. 17(10), 1288–1300 (2007)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of MathematicsThe Chinese University of Hong KongShatin, NTHong Kong
  2. 2.Department of MathematicsCity University of Hong KongKLNHong Kong

Personalised recommendations