Skip to main content
Log in

Super-resolution network-based fractional-pixel motion compensation

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

In video coding, the efficiency of inter-prediction in highly efficient video coding (HEVC) is improved by fractional-pixel motion compensation and is widely used. The traditional fractional-pixel motion compensation method usually interpolates fractional pixels from the integer pixels. However, the existing fractional interpolation methods based on deep learning either only generates half pixels or needs to train corresponding models for corresponding sub-pixel positions. In this paper, we propose to use the super-resolution reconstruction model as the basic structure of the network to solve the problem of fractional-pixel motion compensation in HEVC. This is constructed from a detail component and a super-resolution network to make feature extraction accurate and generate fractional pixels. Firstly, we extract the detail components of adjacent frames as complementary information to achieve accurate feature extraction. Secondly, the data expansion methods are utilized to make full use of the dataset when generating training data. Finally, we set the upscaling factor to four in the deconvolution layer to generate fractional pixels. Experimental results show that this method can save more bits compared with HEVC.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Bevilacqua, M., Roumy, A., Guillemot, C., Alberimorel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: British Machine Vision Conference, pp. 1–10 (2012)

  2. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision, pp. 184–199 (2014)

  3. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: European Conference on Computer Vision, pp. 391–407 (2016)

  4. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: International Conference on Computer Vision, pp. 1026–1034 (2015)

  5. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

  6. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations, pp. 1–15 (2015)

  7. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Neural Information Processing Systems, pp. 1097–1105 (2012)

  8. Liu, G., Reda, F.A., Shih, K.J., Wang, T., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: European Conference on Computer Vision, pp. 89–105 (2018)

  9. Liu, J., Xia, S., Yang, W., Li, M., Liu, D.: One-for-all: Grouped variation network-based fractional interpolation in video coding. IEEE Transactions on Image Processing 28(5), 2140–2151 (2019)

    Article  MathSciNet  Google Scholar 

  10. Lv, H., Wang, R., Xie, X., Jia, H., Gao, W.: A comparison of fractional-pel interpolation filters in hevc and h.264/avc. In: Visual Communications and Image Processing, pp. 1–6 (2012)

  11. Mahmood, A., Bennamoun, M., An, S., Sohel, F., Boussaid, F.: Resfeats: Residual network based features for underwater image classification. Image and Vision Computing 93, 103811 (2020)

    Article  Google Scholar 

  12. Martin, D., Fowlkes, C.C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: International Conference on Computer Vision, pp. 416–423 (2001)

  13. Murn, L., Blasi, S., Smeaton, A.F., O’Connor, N.E., Mrak, M.: Interpreting cnn for low complexity learned sub-pixel motion compensation in video coding. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 798–802 (2020). https://doi.org/10.1109/ICIP40778.2020.9191193

  14. Remez, T., Litany, O., Giryes, R., Bronstein, A.M.: Class-aware fully convolutional gaussian and poisson denoising. IEEE Transactions on Image Processing 27(11), 5707–5722 (2018)

    Article  MathSciNet  Google Scholar 

  15. Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: Computer Vision and Pattern Recognition, pp. 3791–3799 (2015)

  16. Sullivan, G.J., Ohm, J., Han, W., Wiegand, T.: Overview of the high efficiency video coding (hevc) standard. IEEE Transactions on Circuits and Systems for Video Technology 22(12), 1649–1668 (2012)

    Article  Google Scholar 

  17. Timofte, R., De Smet, V., Van Gool, L.: A+: Adjusted anchored neighborhood regression for fast super-resolution. In: Asian Conference on Computer Vision, pp. 111–126 (2014)

  18. Wei, J., Lu, G., Liu, H., Yan, J.: Facial image inpainting with deep generative model and patch search using region weight. IEEE Access 7, 67456–67468 (2019)

    Article  Google Scholar 

  19. Wiegand, T.., Sullivan, G..J.., Bjontegaard, G.., Luthra, A..: Overview of the h.264/avc video coding standard. IEEE Transactions on Circuits and Systems for Video Technology 13(7), 560–560 (2003)

    Article  Google Scholar 

  20. Yan, N., Liu, D., Li, H., Li, B., Li, L., Wu, F.: Convolutional neural network-based fractional-pixel motion compensation. IEEE Transactions on Circuits and Systems for Video Technology 29(3), 840–853 (2019)

    Article  Google Scholar 

  21. Yan, N., Liu, D., Li, H., Wu, F.: A convolutional neural network approach for half-pel interpolation in video coding. In: International Symposium on Circuits and Systems, pp. 1–4 (2017)

  22. Yang, C., Ma, C., Yang, M.: Single-image super-resolution: A benchmark. In: European Conference on Computer Vision, pp. 372–386 (2014)

  23. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Transactions on Image Processing 19(11), 2861–2873 (2010)

    Article  MathSciNet  Google Scholar 

  24. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: Computer Vision and Pattern Recognition, pp. 5505–5514 (2018)

  25. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: European Conference on Computer Vision, pp. 818–833 (2014)

  26. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: International Conference on Curves and Surfaces, pp. 711–730 (2010)

  27. Zhang, H., Li, L., Song, L., Yang, X., Li, Z.: Advanced cnn based motion compensation fractional interpolation. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 709–713 (2019). https://doi.org/10.1109/ICIP.2019.8804199

  28. Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep cnn denoiser prior for image restoration. In: Computer Vision and Pattern Recognition, pp. 2808–2817 (2017)

  29. Zhao, H., Shao, W., Bao, B., Li, H.: A simple and robust deep convolutional approach to blind image denoising. In: International Conference on Computer Vision, pp. 3943–3951 (2019)

  30. Zhou, F., Hang, R., Liu, Q., Yuan, X.: Hyperspectral image classification using spectral-spatial lstms. Neurocomputing 328(FEB.7), 39–47 (2019)

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under (Grant no. 61703196), the Key Science Foundation of Zhangzhou City under (Grant no. ZZ2019ZD11) and the Fujian Province Nature Science Foundation under (Grant No.2020J01813).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Wenyuan Yang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Z., Liu, J., Yang, J. et al. Super-resolution network-based fractional-pixel motion compensation. SIViP 15, 1547–1554 (2021). https://doi.org/10.1007/s11760-021-01887-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-021-01887-5

Keywords

Navigation