Skip to main content
SpringerLink
Log in

Fast QTMT decision tree for Versatile Video Coding based on deep neural network

  • 1221: Deep Learning for Image/Video Compression and Visual Quality Assessment
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Versatile Video Coding (VVC), the emerging video coding standard, outperforms the coding efficiency of the previous standard named High Efficiency Video Coding (HEVC) at the cost of an encoding complexity increase. In fact, VVC proposes a new partitioning block structure called quadtree with nested multi-type tree (QTMT) that introduces a more flexible partition shape compared to the previous splitting algorithms namely quadtree plus binary tree (QTBT) and quadtree (QT) structures adopted in HEVC. However, QTMT increases the encoding time due to the rate-distortion cost (RDcost) process. In order to overcome this issue, this paper proposes a fast intra partitioning algorithm based on a Deep Learning (DL) approach using a Convolution Neural Network (CNN). First, a fast QTMT partition algorithm based on a CNN-binary tree horizontal (CNN-BTH) network is developed to predict the BTH mode decision at 32×32 Coding Units (CUs). The BTV decision tree algorithm is also predicted at this level by a CNN-binary tree vertical (CNN-BTV). Then, two algorithms are combined to suggest a new fast intra QTMT decision tree algorithm. Compared to the VVC reference software VTM-3.0, the proposed overall intra QTMT partition approach reaches a significant complexity reduction down to 37% compared to the original software VTM-3.0, and an average of 31% in terms of encoding time saving with a slight loss in coding performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Abdallah B, Belghith F, Ayed MAB, Masmoudi N (2021) Low-complexity qtmt partition based on deep neural network for versatile video coding. SIViP:1–8

  2. Amestoy T, Mercat A, Hamidouche W, Menard D, Bergeron C (2019) Tunable vvc frame partitioning based on lightweight machine learning. IEEE Trans Image Process 29:1313–1328

    Article  MathSciNet  MATH  Google Scholar 

  3. Bjøntegaard G (2001) Calculation of average psnr differences between rd-curves (vceg-m33). In: VCEG meeting (ITU-T SG 16 Q. 6), pp 2–4

  4. Bossen F, Boyce J, Li X, Seregin V, Sühring K (2018) Jvet common test conditions and software reference configurations for sdr video. Joint Video Experts Team (JVET) of ITU-T SG, vol 16

  5. Cao J, Tang N, Wang J, Liang F (2020) Texture-based fast cu size decision and intra mode decision algorithm for vvc. In: International conference on multimedia modeling. Springer, pp 739–751

  6. Chang CY, Srinivasan K, Wang WC, Ganapathy GP, Vincent DR, Deepa N (2020) Quality assessment of tire shearography images via ensemble hybrid faster region-based convnets. Electronics 9(1):45

    Article  Google Scholar 

  7. Fan Y, Chen J, Sun H, Katto J, Jing M (2020) A fast qtmt partition decision strategy for vvc intra prediction. IEEE Access 8:107900–107911. https://doi.org/10.1109/ACCESS.2020.3000565

    Article  Google Scholar 

  8. Fu T, Zhang H, Mu F, Chen H (2019) Fast cu partitioning algorithm for h. 266/vvc intra-frame coding. In: 2019 IEEE International conference on multimedia and expo (ICME). IEEE, pp 55–60

  9. Fu T, Zhang H, Mu F, Chen H (2019) Fast cu partitioning algorithm for h.266/vvc intra-frame coding. In: 2019 IEEE International conference on multimedia and expo (ICME), pp 55–60. https://doi.org/10.1109/ICME.2019.00018

  10. Jin Z, An P, Yang C, Shen L (2018) Fast qtbt partition algorithm for intra frame coding through convolutional neural network. IEEE Access 6:54660–54673

    Article  Google Scholar 

  11. Kibeya H, Belghith F, Ayed MAB, Masmoudi N (2016) Fast coding unit selection and motion estimation algorithm based on early detection of zero block quantified transform coefficients for high-efficiency video coding standard. IET Image Process 10(5):371–380

    Article  Google Scholar 

  12. Kibeya H, Belghith F, Ben Ayed MA, Masmoudi N (2016) Fast intra-prediction algorithms for high efficiency video coding standard. J Electr Imaging vol 25(1)

  13. Kim S, Jun D, Kim BG, Beack S, Lee M, Lee T (2021) Two-dimensional audio compression method using video coding schemes. Electronics 10 (9):1094

    Article  Google Scholar 

  14. Li T, Xu M, Tang R (2020) Deepqtmt: a deep learning approach for fast qtmt-based cu partition of intra-mode vvc. arXiv:2006.13125

  15. Liu X, Li Y, Liu D, Wang P, Yang LT (2017) An adaptive cu size decision algorithm for hevc intra prediction based on complexity classification using machine learning. IEEE Trans Circuits Syst Video Technol 29(1):144–155

    Article  Google Scholar 

  16. Liu Z, Yu X, Gao Y, Chen S, Ji X, Wang D (2016) Cu partition mode decision for hevc hardwired intra encoder using convolution neural network. IEEE Trans Image Process 25(11):5088–5103

    Article  MathSciNet  MATH  Google Scholar 

  17. Park SH, Kang JW (2019) Context-based ternary tree decision method in versatile video coding for fast intra coding. IEEE Access 7:172597–172605

    Article  Google Scholar 

  18. Sidaty N, Hamidouche W, Deforges O, Philippe P (2017) Emerging video coding performance: 4k quality monitoring. In: 2017 Ninth international conference on quality of multimedia experience (qoMEX). IEEE, pp 1–3

  19. Tang G, Jing M, Zeng X, Fan Y (2019) Adaptive cu split decision with pooling-variable cnn for vvc intra encoding. In: 2019 IEEE visual communications and image processing (VCIP), pp 1–4, DOI 10.1109/VCIP47243.2019.8965679, (to appear in print)

  20. VVC Test Model (VTM) vesion (2018) VVC Test Model (VTM) vesion 3.0: [online] available. https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftwareVTM/tree/VTM-3.0 (December 2018)

  21. Wang Z, Wang S, Zhang J, Wang S, Ma S (2017) Effective quadtree plus binary tree block partition decision for future video coding. In: 2017 Data compression conference (DCC), pp 23–32. https://doi.org/10.1109/DCC.2017.70

  22. Wang Z, Wang S, Zhang X, Wang S, Ma S (2018) Fast qtbt partitioning decision for interframe coding with convolution neural network. In: 2018 25th IEEE international conference on image processing (ICIP). IEEE, pp 2550–2554

  23. Yang H, Shen L, Dong X, Ding Q, An P, Jiang G (2019) Low complexity ctu partition structure decision and fast intra mode decision for versatile video coding. IEEE Trans Circuits Syst Video Technol

  24. Yang H, Shen L, Dong X, Ding Q, An P, Jiang G (2020) Low-complexity ctu partition structure decision and fast intra mode decision for versatile video coding. IEEE Trans Circuits Syst Video Technol 30:1668–1682

    Article  Google Scholar 

  25. Zhong G, Wang J, Hu J, Liang F (2021) A gan-based video intra coding. Electronics 10(2):132

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Electronics and Information Technology Laboratory, National Engineering School of Sfax, University of Sfax, Sfax, 3035, Tunisie

    Bouthaina Abdallah, Fatma Belghith & Nouri Masmoudi

  2. New Technologies and Telecom Systems Laboratory (NTS’COM), Sfax National School of Electronics and Communication (ENET’COM), University of Sfax, Sfax, Tunisia

    Mohamed Ali Ben Ayed

Authors
  1. Bouthaina Abdallah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fatma Belghith
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohamed Ali Ben Ayed
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nouri Masmoudi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bouthaina Abdallah.

Ethics declarations

Competing interests

The authors have no relevant financial or non-financial interests to disclose..

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abdallah, B., Belghith, F., Ben Ayed, M.A. et al. Fast QTMT decision tree for Versatile Video Coding based on deep neural network. Multimed Tools Appl 81, 42731–42747 (2022). https://doi.org/10.1007/s11042-022-13479-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13479-7

Keywords

Search

Navigation