Abstract
The majority of conventional video super-resolution algorithms aim at reconstructing low-resolution videos after down-sampling. However, numerous low-resolution videos will be further compressed to adapt to the limited storage size and transmission bandwidth, leading to further video quality degradation. Significantly, the noise brought by compression often has a strong correlation with the content of the video frame itself. If we super-resolve compressed video frames directly, the noise may be amplified, leading to loss of important information or lower super-resolution performance. To ease those problems, we present an end-to-end deep feature fusion network with ordinary differential equation and dual attention mechanism for joint video compression artifacts reduction and super-resolution. The proposed network commendably enhances the spatial-temporal features fusion of different depths, improves the acquisition of meaningful information ability, and perfects reconstruction quality. In addition, we leverage several skip connections to fuse the captured in-depth feature information and the shallow to prevent information loss. The experimental results show that our proposed method is competent to reduce bit-rate and have excellent quality improvement effectively.
Similar content being viewed by others
References
Anoop V, Bipin PR (2020) Super-resolution based automatic diagnosis of retinal disease detection for clinical applications. Neural Process Lett 52(2):1155–1170
Brandi F, de Queiroz R, Mukherjee D (2008) Super-resolution of video using key frames and motion estimation. In: 2008 15th IEEE international conference on image processing. IEEE, pp 321–324
Cao H, Liu X, Wang Y, Li Y, Lei W (2020) Enhanced down/up-sampling-based video coding using the residual compensation. In: 2020 5th international conference on computer and communication systems (ICCCS). IEEE, pp 286–290
Chang B, Meng L, Haber E, Ruthotto L, Begert D, Holtham E (2018) Reversible architectures for arbitrarily deep residual neural networks. In: Proceedings of the AAAI conference on artificial intelligence, vol 32
Dai Y, Liu D, Wu F (2017) A convolutional neural network approach for post-processing in hevc intra coding. In: International conference on multimedia modeling. Springer, pp 28–39
Feng L, Zhang X, Zhang X, Wang S, Wang R, Ma S (2018) A dual-network based super-resolution for compressed high definition video. In: Pacific Rim conference on multimedia. Springer, pp 600–610
Fuoli D, Gu S, Timofte R (2019) Efficient video super-resolution through recurrent latent space propagation. In: 2019 IEEE/CVF international conference on computer vision workshop (ICCVW). IEEE, pp 3476–3485
Georgis G, Lentaris G, Reisis D (2015) Reduced complexity superresolution for low-bitrate video compression. IEEE Trans Circuits Syst Video Technol 26(2):332–345
Guan Z, Xing Q, Xu M, Yang R, Liu T, Wang, Z (2019) Mfqe 2.0: a new approach for multiframe quality enhancement on compressed video. IEEE Trans Pattern Anal Mach Intell 43(3): 949–963
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: European conference on computer vision. Springer, pp 630–645
Ho MM, He G, Wang Z, Zhou J (2020) Down-sampling based video coding with degradation-aware restoration-reconstruction deep neural network. In: International conference on multimedia modeling. Springer, pp 99–110
Ho MM, Zhou J, He G (2021) Rr-dncnn v2. 0: enhanced restoration–reconstruction deep neural network for down-sampling-based video coding. IEEE Trans Image Process 30:1702–1715
Ho MM, Zhou J, He G, Li M, Li L (2020) Sr-cl-dmc: P-frame coding with super-resolution, color learning, and deep motion compensation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 124–125
Hoang TM, Zhou J (2019) B-drrn: a block information constrained deep recursive residual network for video compression artifacts reduction. In: 2019 picture coding symposium (PCS). IEEE, pp 1–5
Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141
Isobe T, Jia X, Gu S, Li S, Wang S, Tian Q (2020) Video super-resolution with recurrent structure-detail network. In: European conference on computer vision. Springer, pp 645–660
Isobe T, Li S, Jia X, Yuan S, Slabaugh G, Xu C, Li YL, Wang S, Tian Q (2020) Video super-resolution with temporal group attention. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8008–8017
Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980
Köhler T, Huang X, Schebesch F, Aichert A, Maier A, Hornegger J (2016) Robust multiframe super-resolution employing iteratively re-weighted minimization. IEEE Trans Comput Imaging 2(1):42–58
Larsson G, Maire M, Shakhnarovich G (2016) Fractalnet: ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648
Lee JK, Kim N, Cho S, Kang JW (2020) Deep video prediction network-based inter-frame coding in hevc. IEEE Access 8:95906–95917
Li C, Zhang B, Hu H, Dai J (2019) Enhanced bird detection from low-resolution aerial image using deep neural networks. Neural Process Lett 49(3):1021–1039
Li F, Bai H, Zhao Y (2020) Learning a deep dual attention network for video super-resolution. IEEE Trans Image Process 29:4474–4488
Li X, Hu Y, Gao X, Tao D, Ning B (2010) A multi-frame image super-resolution method. Signal Process 90(2):405–414
Li Y, Liu D, Li H, Li L, Li Z, Wu F (2018) Learning a convolutional neural network for image compact-resolution. IEEE Trans Image Process 28(3):1092–1107
Li Y, Liu D, Li H, Li L, Wu F, Zhang H, Yang H (2017) Convolutional neural network-based block up-sampling for intra frame coding. IEEE Trans Circuits Syst Video Technol 28(9):2316–2330
Liao R, Tao X, Li R, Ma Z, Jia J (2015) Video super-resolution via deep draft-ensemble learning. In: Proceedings of the IEEE international conference on computer vision, pp 531–539
Lin J, Liu D, Yang H, Li H, Wu F (2018) Convolutional neural network-based block up-sampling for hevc. IEEE Trans Circuits Syst Video Technol 29(12):3701–3715
Liu C, Sun D (2013) On Bayesian adaptive video super resolution. IEEE Trans Pattern Anal Mach Intell 36(2):346–360
Lu Y, Zhong A, Li Q, Dong B (2018) Beyond finite layer neural networks: bridging deep architectures and numerical differential equations. In: International conference on machine learning. PMLR, pp 3276–3285
Ma Z, Liao R, Tao X, Xu L, Jia J, Wu E (2015) Handling motion blur in multi-frame super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5224–5232
Mnih V, Heess N, Graves A et al (2014) Recurrent models of visual attention. In: Advances in neural information processing systems, pp 2204–2212
Molina R, Katsaggelos AK, Alvarez L, Mateos J (2006) Toward a new video compression scheme using super-resolution. In: Visual communications and image processing 2006, vol 6077. International Society for Optics and Photonics, p 607706
Peng H, Chen X, Zhao J (2020) Residual pixel attention network for spectral reconstruction from RGB images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 486–487
Pourazad MT, Doutre C, Azimi M, Nasiopoulos P (2012) Hevc: the new gold standard for video compression: how does hevc compare with h.264/avc? IEEE Consum Electron Mag 1(3):36–46
Shen M, Xue P, Wang C (2011) Down-sampling based video coding using super-resolution technique. IEEE Trans Circuits Syst Video Technol 21(6):755–765
Soh JW, Park J, Kim Y, Ahn B, Lee HS, Moon YS, Cho NI (2018) Reduction of video compression artifacts based on deep temporal networks. IEEE Access 6:63094–63106
Tang T, Li L (2016) Adaptive deblocking method for low bitrate coded hevc video. J Vis Commun Image Represent 38:721–734
Tian Y, Zhang Y, Fu Y, Xu C (2020) Tdan: temporally-deformable alignment network for video super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3360–3369
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008
Wang L, Guo Y, Lin Z, Deng X, An W (2018) Learning for video super-resolution through HR optical flow estimation. In: Asian conference on computer vision. Springer, pp 514–529
Wang L, Guo Y, Liu L, Lin Z, Deng X, An W (2020) Deep video super-resolution using HR optical flow estimation. IEEE Trans Image Process 29:4323–4336
Weinan E (2017) A proposal on machine learning via dynamical systems. Commun Math Stat 5(1):1–11
Yang R, Xu M, Wang Z, Li T (2018) Multi-frame quality enhancement for compressed video. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6664–6673
Ying X, Wang L, Wang Y, Sheng W, An W, Guo Y (2020) Deformable 3d convolution for video super-resolution. IEEE Signal Process Lett 27:1500–1504
Zhang X, Dong H, Hu Z, Lai WS, Wang F, Yang MH (2018) Gated fusion network for joint image deblurring and super-resolution. arXiv preprint arXiv:1807.10806
Zhang X, Li Z, Change Loy C, Lin D (2017) Polynet: a pursuit of structural diversity in very deep networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 718–726
Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European conference on computer vision (ECCV), pp 286–301
Zhao H, Kong X, He J, Qiao Y, Dong C (2020) Efficient image super-resolution using pixel attention. In: European conference on computer vision. Springer, pp 56–72
Zhou Y, Sun X, Zha ZJ, Zeng W (2018) Mict: mixed 3d/2d convolutional tube for human action recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 449–458
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grants 61871279 & 62081330105 and in part by the Fundamental Research Funds for the Central Universities under Grant 2021SCU12061.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Wang, Y., Wu, X., He, X. et al. Deep Feature Fusion Network for Compressed Video Super-Resolution. Neural Process Lett 54, 4427–4441 (2022). https://doi.org/10.1007/s11063-022-10816-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-022-10816-7