Skip to main content
SpringerLink
Log in

An Optimal Fluid Optical Flow Registration for Super-resolution with Lamé Parameters Learning

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

The main idea of multi-frame super-resolution (SR) algorithms is to recover a single high-resolution image through a series of low-resolution ones of a captured scene. The success of the SR approaches is often related to well registration and restoration steps. In this work, we propose a new approach based on fluid optical flow image registration and a second-order regularization term to treat both the registration and restoration steps. The fluid registration is introduced to avoid misregistration errors, while the second-order regularization resolved by the Bregman iteration is employed to reduce the image artifacts. Moreover, we propose a bilevel supervised learning framework to compute the Lamé coefficients \(\lambda \) and \(\mu \), which perform the nonparametric registration of the super-resolution result. The numerical part demonstrated that the proposed method copes with some competitive SR methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Ardila, J.P., Tolpekin, V.A., Bijker, W., Stein, A.: Markov-random-field-based super-resolution mapping for identification of urban trees in vhr images. ISPRS J. Photogramm. Remote Sens. 66(6), 762–775 (2011)

    Google Scholar 

  2. Baker, S., Kanade, T.: Super-resolution optical flow. Carnegie Mellon University, The Robotics Institute, Pittsburgh (1999)

    Google Scholar 

  3. Borzi, A., Ito, K., Kunisch, K.: Optimal control formulation for determining optical flow. SIAM J. Sci. Comput. 24(3), 818–847 (2003)

    MathSciNet  MATH  Google Scholar 

  4. Buades, A., Coll, B., Morel, J.-M.: The staircasing effect in neighborhood filters and its solution. IEEE Trans. Image Process. 15(6), 1499–1505 (2006)

    Google Scholar 

  5. Castro, P., Reyes, J.C.D.L.: A bilevel learning approach for optimal observation placement in variational data assimulation. Inverse Probl. 36(3), 1–31 (2020)

    Google Scholar 

  6. Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)

    MathSciNet  MATH  Google Scholar 

  7. Chen, J., Nunez-Yanez, J., Achim, A.: Video super-resolution using generalized gaussian Markov random fields. IEEE Signal Process. Lett. 19(2), 63–66 (2012)

    Google Scholar 

  8. Cheong, J.Y., Park, I.K.: Deep CNN-based super-resolution using external and internal examples. IEEE Signal Process. Lett. 24(8), 1252–1256 (2017)

    Google Scholar 

  9. Clason, C., Valkonen, T.: Primal-dual extragradient methods for nonlinear nonsmooth PDE-constrained optimization. SIAM J. Optim. 27(3), 1314–1339 (2017)

    MathSciNet  MATH  Google Scholar 

  10. Dempe, S., Harder, F., Mehlitz, P., Wachsmuth, G.: Solving inverse optimal control problems via value functions to global optimality. J. Global Optim. 74(2), 297–325 (2019)

    MathSciNet  MATH  Google Scholar 

  11. El Mourabit, I., El Rhabi, M., Hakim, A., Laghrib, A., Moreau, E.: A new denoising model for multi-frame super-resolution image reconstruction. Signal Process. 132, 51–65 (2017)

    Google Scholar 

  12. Engl, H.W., Hanke, M., Neubauer, A.: Regularization of inverse problems (mathematics and its applications). Springer, Netherlands (2000)

    MATH  Google Scholar 

  13. Farsiu, S., Robinson, M.D., Elad, M., Milanfar, P.: Fast and robust multiframe super resolution. IEEE Trans. Image Process. 13(10), 1327–1344 (2004)

    Google Scholar 

  14. Fransens, R., Strecha, C., Van Gool, L.: Optical flow based super-resolution: a probabilistic approach. Computer Vis. Underst. 106(1), 106–115 (2007)

    Google Scholar 

  15. Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Graph. Appl. 22(2), 56–65 (2002)

    Google Scholar 

  16. Gao, X., Zhang, K., Tao, D., Li, X.: Joint learning for single-image super-resolution via a coupled constraint. IEEE Trans. Image Process. 21(2), 469–480 (2012)

    MathSciNet  MATH  Google Scholar 

  17. Gernot Holler, K.K., Barnard, R.C.: A bilevel approach for parameter learning in inverse problems. Inverse Probl. 34(11), 115012 (2018)

    MathSciNet  MATH  Google Scholar 

  18. Ghosh, D., Kaabouch, N., Hu, W.-C.: A robust iterative super-resolution mosaicking algorithm using an adaptive and directional Huber-Markov regularization. J. Vis. Commun. Image Represent. 40, 98–110 (2016)

    Google Scholar 

  19. Goldstein, T., Osher, S.: The split Bregman method for l1-regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009)

    MathSciNet  MATH  Google Scholar 

  20. Golub, G.H., Heath, M., Wahba, G.: Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21(2), 215–223 (1979)

    MathSciNet  MATH  Google Scholar 

  21. Hansen, P.C.: Analysis of discrete ill-posed problems by means of the l-curve. SIAM Rev. 34(4), 561–580 (1992)

    MathSciNet  MATH  Google Scholar 

  22. Hansen, P.C., O’Leary, D.P.: The use of the l-curve in the regularization of discrete ill-posed problems. SIAM J. Sci. Comput. 14(6), 1487–1503 (1993)

    MathSciNet  MATH  Google Scholar 

  23. He, Y., Yap, K.-H., Chen, L., Chau, L.-P.: A nonlinear least square technique for simultaneous image registration and super-resolution. IEEE Trans. Image Process. 16(11), 2830–2841 (2007)

    MathSciNet  Google Scholar 

  24. Jia, K., Gong, S.: Hallucinating multiple occluded face images of different resolutions. Pattern Recognit. Lett. 27(15), 1768–1775 (2006)

    Google Scholar 

  25. Kasetkasem, T., Arora, M.K., Varshney, P.K.: Super-resolution land cover mapping using a Markov random field based approach. Remote Sens. Environ. 96(3–4), 302–314 (2005)

    Google Scholar 

  26. Khattab, M.M., Zeki, A.M., Alwan, A.A., Badawy, A.S.: Regularization-based multi-frame super-resolution: a systematic review. J. King Saud Univ. Comput. Inf. Sci. 32(7), 755–762 (2018)

    Google Scholar 

  27. Kim, K.I., Kwon, Y.: Single-image super-resolution using sparse regression and natural image prior. IEEE Trans. Pattern Anal. Mach. Intell. 32(6), 1127–1133 (2010)

    Google Scholar 

  28. Konstantin, P., Mattias, G.: Necessary conditions for a class of bilevel optimal control problems exploiting the value function. Pure Appl. Funct. Anal. 1(4), 505–524 (2016)

    MathSciNet  MATH  Google Scholar 

  29. Kumar, N., Verma, R., Sethi, A.: Convolutional neural networks for wavelet domain super resolution. Pattern Recognit. Lett. 90, 65–71 (2017)

    Google Scholar 

  30. Laghrib, A., Chakib, A., Hadri, A., Hakim, A.: A nonlinear fourth-order PDE for multi-frame image super-resolution enhancement. Discret. Contin. Dyn. Syst. B 25(1), 415 (2020)

    MathSciNet  MATH  Google Scholar 

  31. Laghrib, A., Ezzaki, M., El Rhabi, M., Hakim, A., Monasse, P., Raghay, S.: Simultaneous deconvolution and denoising using a second order variational approach applied to image super resolution. Comput. Vis. Image Underst. 168, 50–63 (2018)

    Google Scholar 

  32. Laghrib, A., Ghazdali, A., Hakim, A., Raghay, S.: A multi-frame super-resolution using diffusion registration and a nonlocal variational image restoration. Comput. Math. Appl. 72(9), 2535–2548 (2016)

    MathSciNet  MATH  Google Scholar 

  33. Laghrib, A., Hadri, A., Hakim, A.: An edge preserving high-order PDE for multiframe image super-resolution. J. Franklin Inst. 356(11), 5834–5857 (2019)

    MathSciNet  MATH  Google Scholar 

  34. Laghrib, A., Hadri, A., Hakim, A., Raghay, S.: A new multiframe super-resolution based on nonlinear registration and a spatially weighted regularization. Inf. Sci. 493, 34–56 (2019)

    MathSciNet  MATH  Google Scholar 

  35. Laghrib, A., Hakim, A., Raghay, S.: A combined total variation and bilateral filter approach for image robust super resolution. EURASIP J. Image Video Process. 2015(1), 1–10 (2015)

    Google Scholar 

  36. Laghrib, A., Hakim, A., Raghay, S.: An iterative image super-resolution approach based on Bregman distance. Signal Process. Image Commun. 58, 24–34 (2017)

    Google Scholar 

  37. Lang, L.F., Neumayer, S., Öktem, O., Schönlieb, C.-B.: Template-based image reconstruction from sparse tomographic data. Appl. Math. Optim. 82, 1081–1109 (2020)

    MathSciNet  MATH  Google Scholar 

  38. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4681–4690

  39. Lucas, A., Lopez-Tapia, S., Molina, R., Katsaggelos, A.K.: Generative adversarial networks and perceptual losses for video super-resolution. IEEE Trans. Image Process. 28(7), 3312–3327 (2019)

    MathSciNet  MATH  Google Scholar 

  40. Maiseli, B.J., Ally, N., Gao, H.: A noise-suppressing and edge-preserving multiframe super-resolution image reconstruction method. Signal Process. Image Communi. 34, 1–13 (2015)

    Google Scholar 

  41. Papafitsoros, K., Schönlieb, C.-B.: A combined first and second order variational approach for image reconstruction. J. Math. Imaging Vis. 48(2), 308–338 (2014)

    MathSciNet  MATH  Google Scholar 

  42. Peyré, G., Bougleux, S., Cohen, L.D.: Non-local regularization of inverse problems. Inverse Probl. Imaging 5(2), 511–530 (2011)

    MathSciNet  MATH  Google Scholar 

  43. Rasti, P., Demirel, H., and Anbarjafari, G. (2014). Improved iterative back projection for video super-resolution. In 2014 22nd Signal Processing and Communications Applications Conference (SIU), pages 552–555. IEEE

  44. Robinson, M. D., Chiu, S. J., Toth, C. A., Izatt, J. A., Lo, J. Y., and Farsiu, S. (2017). New applications of super-resolution in medical imaging. In Super-Resolution Imaging, pages 401–430. CRC Press

  45. Su, H., Jiang, N., Wu, Y., Zhou, J.: Single image super-resolution based on space structure learning. Pattern Recognit. Lett. 34(16), 2094–2101 (2013)

    Google Scholar 

  46. Tsai, R.: Multiframe image restoration and registration. Adv. Comput. Vis. Image Process. 1, 317–339 (1984)

    Google Scholar 

  47. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Google Scholar 

  48. Wheeler, F. W., Hoctor, R. T., and Barrett, E. B. (2005). Super-resolution image synthesis using projections onto convex sets in the frequency domain. In Computational Imaging III, volume 5674, pages 479–490. International Society for Optics and Photonics

  49. Yang, S., Wang, M., Sun, Y., Sun, F., Jiao, L.: Compressive sampling based single-image super-resolution reconstruction by dual-sparsity and non-local similarity regularizer. Pattern Recognit. Lett. 33(9), 1049–1059 (2012)

    Google Scholar 

  50. Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., Zhang, L.: Image super-resolution: the techniques, applications, and future. Signal Process. 128, 389–408 (2016)

    Google Scholar 

  51. Zeng, X., Yang, L.: A robust multiframe super-resolution algorithm based on half-quadratic estimation with modified BTV regularization. Digital Signal Process. 23(1), 98–109 (2013)

    MathSciNet  Google Scholar 

  52. Zhang, K., Gao, X., Tao, D., Li, X.: Single image super-resolution with non-local means and steering kernel regression. IEEE Trans. Image Process. 21(11), 4544–4556 (2012)

    MathSciNet  MATH  Google Scholar 

  53. Zhang, Y., Liu, J., Yang, W., Guo, Z.: Image super-resolution based on structure-modulated sparse representation. IEEE Trans. Image Process. 24(9), 2797–2810 (2015)

    MathSciNet  MATH  Google Scholar 

  54. Zhao, L., Bai, H., Liang, J., Zeng, B., Wang, A., Zhao, Y.: Simultaneous color-depth super-resolution with conditional generative adversarial networks. Pattern Recognit. 88, 356–369 (2019)

    Google Scholar 

  55. Zhao, W., Sawhney, H., Hansen, M., and Samarasekera, S. (2002). Super-fusion: a super-resolution method based on fusion. In Object Recognition Supported by User Interaction for Service Robots, volume 2, pages 269–272. IEEE

Download references

Acknowledgements

The authors are grateful to the anonymous reviewers for their helpful comments.

Author information

Authors and Affiliations

  1. Mathematics and Interactions Teams (EMI), Sultan Moulay Slimane University, Beni Mellal, Morocco

    Abdelmajid El Hakoume, Amine Laghrib & Lekbir Afraites

  2. Computer Systems and Vision Laboratory (Lab SIV), Ibn Zohr University, Agadir, Morocco

    Aissam Hadri

Authors
  1. Abdelmajid El Hakoume
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amine Laghrib
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aissam Hadri
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lekbir Afraites
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amine Laghrib.

Ethics declarations

Conflict of interest

Not available.

Additional information

Communicated by Gabriel Peyré.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

El Hakoume, A., Laghrib, A., Hadri, A. et al. An Optimal Fluid Optical Flow Registration for Super-resolution with Lamé Parameters Learning. J Optim Theory Appl 197, 508–538 (2023). https://doi.org/10.1007/s10957-023-02186-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-023-02186-4

Keywords

Search

Navigation