Abstract
As a special kind of video coding, screen content coding (SCC) has received widespread attention because of the popularity of online classes and conferences. However, few people use neural networks to improve the compression efficiency of SCC. Intra block copy (IBC) is one of the most important coding tools in SCC, which can save half of the bitrate. Due to the need to copy the content of the reference block, the performance of IBC mode largely depends on the quality of the reference block. In the standard encoding process of Versatile Video Coding (VVC), the IBC reference block is not filtered, and there are still serious compression artifacts. This will result in a decrease in IBC search accuracy and SCC compression efficiency. Inspired by in-loop filtering, we propose an IBC reference blocks enhancement network based on GAN (IREGAN) to filter the reference blocks before IBC estimation, which can improve the quality of IBC reference block and the accuracy of IBC matching. In addition to the generator used for image enhancement, our model also includes a variance-based classifier and a discriminator obtained from adversarial training. The classifier can effectively improve the efficiency of the model and the discriminator can improve the robustness of the entire system. Experimental results demonstrate the performance gains of IREGAN with VTM10.0, offering about 6.98% BDBR reduction, 0.71dB BDPSNR gains in average (luminance). SSIM increased by 0.0113 and the number of blocks using IBC mode is increased by 1.42%.
This work was supported by Key-Area R&D Program of Guangdong Province under Grant 2019B010135002, and Innovative & Enterprising Team of Zhuhai under Grant 2019ZHCDGY07.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Zhao, X., Liu, S., Zhao, L., Xu, X., Zhu, B., Li, X.: A comparative study of HEVC, VVC, VP9, AV1 and AVS3 video codecs. In: Applications of Digital Image Processing XLIII, vol. 11510, p. 1151011. International Society for Optics and Photonics (2020)
Xu, X., et al.: Intra block copy in HEVC screen content coding extensions. IEEE J. Emerg. Sel. Top. Circuits Syst. 6(4), 409–419 (2016)
Hu, Y., Li, Y., Chen, Z., Xu, X., Liu, S.: Performance analysis of intra block copy for screen content coding in AVS3. In: 2020 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pp. 123–126. IEEE (2020)
Xu, X., Liu, S.: Screen content coding in recently developed video coding standards. In: 2020 IEEE International Conference on Visual Communications and Image Processing (VCIP), pp. 1–2. IEEE (2020)
Cao, J., Li, Z., Liang, F., Wang, J.: An intra-affine current picture referencing mode for screen content coding in VVC. In: 2019 Picture Coding Symposium (PCS), pp. 1–5. IEEE (2019)
Tsang, S.H., Kwong, N.W., Chan, Y.L.: Fastsccnet: fast mode decision in VVC screen content coding via fully convolutional network. In: 2020 IEEE International Conference on Visual Communications and Image Processing (VCIP), pp. 177–180. IEEE (2020)
Xu, X., Li, X., Liu, S.: Intra block copy in versatile video coding with reference sample memory reuse. In: 2019 Picture Coding Symposium (PCS), pp. 1–5. IEEE (2019)
Pan, Z., Yi, X., Zhang, Y., Jeon, B., Kwong, S.: Efficient in-loop filtering based on enhanced deep convolutional neural networks for HEVC. IEEE Trans. Image Process. 29, 5352–5366 (2020)
Zhang, Y., Shen, T., Ji, X., Zhang, Y., Xiong, R., Dai, Q.: Residual highway convolutional neural networks for in-loop filtering in HEVC. IEEE Trans. Image Process. 27(8), 3827–3841 (2018)
Dai, Y., Liu, D., Wu, F.: A convolutional neural network approach for post-processing in HEVC intra coding. In: Amsaleg, L., Guðmundsson, G.Þ, Gurrin, C., Jónsson, B.Þ, Satoh, S. (eds.) MMM 2017. LNCS, vol. 10132, pp. 28–39. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-51811-4_3
Lu, M., Chen, T., Liu, H., Ma, Z.: Learned image restoration for VVC intra coding. In: CVPR Workshops (2019)
Xue, Y., Su, J.: Attention based image compression post-processing convlutional neural network. In: CVPR Workshops (2019)
Cho, S., et al.: Low bit-rate image compression based on post-processing with grouped residual dense network. In: CVPR Workshops (2019)
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544 (2016)
Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017)
Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., Matas, J.: Deblurgan: blind motion deblurring using conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8183–8192 (2018)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)
Galteri, L., Bertini, M., Seidenari, L., Uricchio, T., Del Bimbo, A.: Increasing video perceptual quality with GANs and semantic coding. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 862–870 (2020)
Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)
Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2794–2802 (2017)
Bjontegaard, G.: Calculation of average PSNR differences between RD-curves. VCEG-M33 (2001)
Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition, pp. 2366–2369. IEEE (2010)
The VTM reference software for VVC development, version 10.0. https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/tree/VTM-10.0
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, P. et al. (2022). An IBC Reference Block Enhancement Model Based on GAN for Screen Content Video Coding. In: Þór Jónsson, B., et al. MultiMedia Modeling. MMM 2022. Lecture Notes in Computer Science, vol 13142. Springer, Cham. https://doi.org/10.1007/978-3-030-98355-0_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-98355-0_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-98354-3
Online ISBN: 978-3-030-98355-0
eBook Packages: Computer ScienceComputer Science (R0)