Skip to main content

BSQ-rate: a New Approach for Video-codec Performance Comparison and Drawbacks of Current Solutions

Abstract

This paper is dedicated to the analysis of the existing approaches to video codecs comparisons. It includes the revealed drawbacks of popular comparison methods and proposes new techniques. The performed analysis of user-generated videos collection showed that two of the most popular open video collections from media.xiph.org which are widely used for video-codecs analysis and development do not cover real-life videos complexity distribution. A method for creating representative video sets covering all segments of user videos the spatial and temporal complexity is also proposed. One of the sections discusses video quality estimation algorithms used for video codec comparisons and shows the disadvantages of popular methods VMAF and NIQE. Also, the paper describes the drawbacks of the BD-rate – generally used method for video codecs final ranking during comparisons. A new ranking method called BSQ-rate which considers the identified issues is proposed. The results of this investigation were obtained during the series of research conducted as part of the annual video-codecs comparisons organized by video group of computer graphics and multimedia laboratory at Moscow State University.

This is a preview of subscription content, access via your institution.

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.
Fig. 9.
Fig. 10.
Fig. 11.
Fig. 12.
Fig. 13.
Fig. 14.

REFERENCES

  1. 1

    Cisco VNI Report 2017-2022, 2018 update, 2019. https://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white-paper-c11-741490.html

  2. 2

    Xiph.org Test Media, 2019. https://media.xiph.org/

  3. 3

    Video Quality Experts Group Test Sequences, 2019. ftp://ftp.crc.ca/crc/vqeg/TestSequences/

  4. 4

    MPEG-2 Transport Stream Test Patterns and Tools, 2019. http://www.w6rz.net/

  5. 5

    Sveriges Television: The SVT High Definition Multi-Format Test Set, 2019. ftp://vqeg.its.bldrdoc.gov/HDTV/SVT_MultiFormat/

  6. 6

    Columbia Consumer Video (CCV) Database, 2019. http://www.ee.columbia.edu/ln/dvmm/CCV/

  7. 7

    CDVL The Consumer Digital Video Library, 2019. https://www.cdvl.org/

  8. 8

    LIVE Public-Domain Subjective Video Quality Database, 2019. http://live.ece.utexas.edu/research/quality/live_video.html

  9. 9

    Video samples from KODI Wiki, 2019. https://kodi.wiki/view/Samples

  10. 10

    Ultra Video Group test sequences, 2019. http://ultravideo.cs.tut.fi/#testsequences

  11. 11

    Chen, C., Inguva, S., Rankin, A., and Kokaram, A., A subjective study for the design of multi-resolution ABR video streams with the vp9 codec, Electron. Imag., 2016, vol. 2016, pp. 1–5.

    Google Scholar 

  12. 12

    Vatolin, D., Kulikov, D., Erofeev, M., Antsiferova, A., Zvezdakov, S., Kondranin, D., and Grokholsky, S., MSU FullHD Video Codec Comparison 2019. http://compression.ru/video/codec_comparison/hevc_2019/#main_report.

  13. 13

    MSU Video Codecs Comparisons, 2019. http://compression.ru/video/codec_comparison/index_en.html

  14. 14

    Chikkerur, S., Sundaram, V., Reisslein, M., and Karam, L.J., Objective video quality assessment methods: a classification, review, and performance comparison, IEEE Trans. Broadcast., 2011, vol. 57, no. 2, pp. 165–182.

    Article  Google Scholar 

  15. 15

    Wang, Z., Bovik, A.C., Sheikh, H.R., and Simoncelli, E.P., Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Processing, 2004, vol. 13, no. 4, pp. 600–612.

    Article  Google Scholar 

  16. 16

    Sheikh, H.R. and Bovik, A.C., Image information and visual quality, IEEE Trans. Image Process., 2006, vol. 15, no. 2, pp. 430–444.

    Article  Google Scholar 

  17. 17

    VMAF: Perceptual video quality assessment based on multi-method fusion, 2019. https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652

  18. 18

    MSU Video Codec Comparison 2017. Part III: Full HD Content, Subjective Evaluation. http://www.compression.ru/video/codec_comparison/hevc_2017/MSU_HEVC_comparison_2017_P3_subjective.pdf

  19. 19

    Deb, K., Pratap, A., Agarwal, S., and Meyarivan, T.A.M.T., A fast and elitist multiobjective genetic algorithm: NSGA-II, IEEE Trans. Evol. Comput., 2002, vol. 6, no. 2, pp. 182–197.

    Article  Google Scholar 

  20. 20

    Vatolin, D.S. and Grishin, S.V., Double up-conversion of video frame rate based on bidirectional motion compensation, Progr. Comput. Software, 2009, vol. 35, no. 6, pp. 351–364.

    Article  Google Scholar 

  21. 21

    Mittal, A., Soundararajan, R., and Bovik, A.C., Making a “completely blind” image quality analyzer, IEEE Signal Process. Lett., 2012, vol. 20, no. 3, pp. 209–212.

    Article  Google Scholar 

  22. 22

    MSU Video Codec Comparison 2018. Subjective Report. http://compression.ru/video/codec_comparison/hevc_2018/#subjective_report

  23. 23

    Bjontegaard, G., Calculation of average PSNR differences between RD-curves, Proc. 13th ITU-T VCEG Meeting, Austin, TX, 2001, document VCEG-M33.

Download references

ACKNOWLEDGMENTS

This work was partially supported by the Russian Foundation for Basic Research under Grant 19-01-00785a. Special thanks to Georgiy Osipov and Denis Kondranin who helped to analyze all detected issues and improved NIQE implementation in MSU VQMT, Mikhail Erofeev who helped with MSU comparison methodology improvement and conducted subjective comparisons, and Moscow State University Graphics and Media Lab team for valuable advice and support in our projects.

Author information

Affiliations

Authors

Corresponding authors

Correspondence to A. V. Zvezdakova, D. L. Kulikov, S. V. Zvezdakov or D. S. Vatolin.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zvezdakova, A.V., Kulikov, D.L., Zvezdakov, S.V. et al. BSQ-rate: a New Approach for Video-codec Performance Comparison and Drawbacks of Current Solutions. Program Comput Soft 46, 183–194 (2020). https://doi.org/10.1134/S0361768820030111

Download citation

Keywords:

  • video quality
  • no-reference metric
  • quality measuring
  • video-codec comparison
  • comparison methodology
  • bsq-rate