Abstract
Taking advantage of spatial and temporal correlations, deep learning-based video compressive sensing reconstruction (VCSR) technologies have tremendously improved reconstructed video quality. Existing VCSR works mainly focus on improving deep learning-based motion compensation without optimizing local and global information, leaving much space for further improvements. This paper proposes a video compressive sensing reconstruction method with joint in-loop reference enhancement and out-loop super-resolution (JVCSR), focusing on removing reconstruction artifacts and increasing the resolution simultaneously. As an in-loop part, the enhanced frame is utilized as a reference to improve the recovery performance of the current frame. Furthermore, it is the first time to propose out-loop super-resolution for VCSR to obtain high-quality images at low bitrates. As a result, JVCSR obtains an average improvement of 1.37 dB PSNR compared with state-of-the-art compressive sensing methods at the same bitrate.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Virat description. http://www.viratdata.org/
Chen, H., He, X., Ren, C., Qing, L., Teng, Q.: CISRDCNN: super-resolution of compressed images using deep convolutional neural networks. Neurocomputing 285, 204–219 (2018)
Cheng, Z., Sun, H., Takeuchi, M., Katto, J.: Performance comparison of convolutional autoencoders, generative adversarial networks and super-resolution for image compression. In: CVPR Workshops, pp. 2613–2616 (2018)
Dong, C., Deng, Y., Loy, C.C., Tang, X.: Compression artifacts reduction by a deep convolutional network. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 576–584 (2015)
Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_13
Duarte, M.F., et al.: Single-pixel imaging via compressive sampling. IEEE Sign. Process. Mag. 25(2), 83–91 (2008)
Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1664–1673 (2018)
Ho, M.M., Zhou, J., He, G., Li, M., Li, L.: SR-CL-DMC: P-frame coding with super-resolution, color learning, and deep motion compensation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 124–125 (2020)
Ho, M.-M., He, G., Wang, Z., Zhou, J.: Down-sampling based video coding with degradation-aware restoration-reconstruction deep neural network. In: Ro, Y.M., et al. (eds.) MMM 2020. LNCS, vol. 11961, pp. 99–110. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-37731-1_9
Huang, B., Zhou, J., Yan, X., Jing, M., Wan, R., Fan, Y.: CS-MCNet: a video compressive sensing reconstruction network with interpretable motion compensation. In: Proceedings of the Asian Conference on Computer Vision (2020)
Li, Z., Yang, J., Liu, Z., Yang, X., Jeon, G., Wu, W.: Feedback network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3867–3876 (2019)
Mercat, A., Viitanen, M., Vanne, J.: UVG dataset: 50/120fps 4K sequences for video codec analysis and development. In: Proceedings of the 11th ACM Multimedia Systems Conference, pp. 297–302 (2020)
Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3147–3155 (2017)
Wang, H., et al.: MCL-JCV: a JND-based h. 264/AVC video quality assessment dataset. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 1509–1513. IEEE (2016)
Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: CBAM: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)
Xu, K., Ren, F.: CSVideoNet: a real-time end-to-end learning framework for high-frame-rate video compressive sensing. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1680–1688. IEEE (2018)
Yao, S., Guan, Q., Wang, S., Xie, X.: Fast sparsity adaptive matching pursuit algorithm for large-scale image reconstruction. EURASIP J. Wirel. Commun. Netw. 2018(1), 1–8 (2018). https://doi.org/10.1186/s13638-018-1085-6
Zhang, J., Ghanem, B.: ISTA-NET: interpretable optimization-inspired deep network for image compressive sensing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1828–1837 (2018)
Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)
Zhang, Z., Liu, Y., Liu, J., Wen, F., Zhu, C.: AMP-NET: denoising-based deep unfolding for compressive image sensing. IEEE Trans. Image Process. 30, 1487–1500 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, J., Pham, C.DK., Zhou, J. (2022). JVCSR: Video Compressive Sensing Reconstruction with Joint In-Loop Reference Enhancement and Out-Loop Super-Resolution. In: Þór Jónsson, B., et al. MultiMedia Modeling. MMM 2022. Lecture Notes in Computer Science, vol 13141. Springer, Cham. https://doi.org/10.1007/978-3-030-98358-1_36
Download citation
DOI: https://doi.org/10.1007/978-3-030-98358-1_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-98357-4
Online ISBN: 978-3-030-98358-1
eBook Packages: Computer ScienceComputer Science (R0)