This paper is dedicated to the analysis of the existing approaches to video codecs comparisons. It includes the revealed drawbacks of popular comparison methods and proposes new techniques. The performed analysis of user-generated videos collection showed that two of the most popular open video collections from media.xiph.org which are widely used for video-codecs analysis and development do not cover real-life videos complexity distribution. A method for creating representative video sets covering all segments of user videos the spatial and temporal complexity is also proposed. One of the sections discusses video quality estimation algorithms used for video codec comparisons and shows the disadvantages of popular methods VMAF and NIQE. Also, the paper describes the drawbacks of the BD-rate – generally used method for video codecs final ranking during comparisons. A new ranking method called BSQ-rate which considers the identified issues is proposed. The results of this investigation were obtained during the series of research conducted as part of the annual video-codecs comparisons organized by video group of computer graphics and multimedia laboratory at Moscow State University.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price includes VAT (USA)
Tax calculation will be finalised during checkout.
Cisco VNI Report 2017-2022, 2018 update, 2019. https://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white-paper-c11-741490.html
Xiph.org Test Media, 2019. https://media.xiph.org/
Video Quality Experts Group Test Sequences, 2019. ftp://ftp.crc.ca/crc/vqeg/TestSequences/
MPEG-2 Transport Stream Test Patterns and Tools, 2019. http://www.w6rz.net/
Sveriges Television: The SVT High Definition Multi-Format Test Set, 2019. ftp://vqeg.its.bldrdoc.gov/HDTV/SVT_MultiFormat/
Columbia Consumer Video (CCV) Database, 2019. http://www.ee.columbia.edu/ln/dvmm/CCV/
CDVL The Consumer Digital Video Library, 2019. https://www.cdvl.org/
LIVE Public-Domain Subjective Video Quality Database, 2019. http://live.ece.utexas.edu/research/quality/live_video.html
Video samples from KODI Wiki, 2019. https://kodi.wiki/view/Samples
Ultra Video Group test sequences, 2019. http://ultravideo.cs.tut.fi/#testsequences
Chen, C., Inguva, S., Rankin, A., and Kokaram, A., A subjective study for the design of multi-resolution ABR video streams with the vp9 codec, Electron. Imag., 2016, vol. 2016, pp. 1–5.
Vatolin, D., Kulikov, D., Erofeev, M., Antsiferova, A., Zvezdakov, S., Kondranin, D., and Grokholsky, S., MSU FullHD Video Codec Comparison 2019. http://compression.ru/video/codec_comparison/hevc_2019/#main_report.
MSU Video Codecs Comparisons, 2019. http://compression.ru/video/codec_comparison/index_en.html
Chikkerur, S., Sundaram, V., Reisslein, M., and Karam, L.J., Objective video quality assessment methods: a classification, review, and performance comparison, IEEE Trans. Broadcast., 2011, vol. 57, no. 2, pp. 165–182.
Wang, Z., Bovik, A.C., Sheikh, H.R., and Simoncelli, E.P., Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Processing, 2004, vol. 13, no. 4, pp. 600–612.
Sheikh, H.R. and Bovik, A.C., Image information and visual quality, IEEE Trans. Image Process., 2006, vol. 15, no. 2, pp. 430–444.
VMAF: Perceptual video quality assessment based on multi-method fusion, 2019. https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652
MSU Video Codec Comparison 2017. Part III: Full HD Content, Subjective Evaluation. http://www.compression.ru/video/codec_comparison/hevc_2017/MSU_HEVC_comparison_2017_P3_subjective.pdf
Deb, K., Pratap, A., Agarwal, S., and Meyarivan, T.A.M.T., A fast and elitist multiobjective genetic algorithm: NSGA-II, IEEE Trans. Evol. Comput., 2002, vol. 6, no. 2, pp. 182–197.
Vatolin, D.S. and Grishin, S.V., Double up-conversion of video frame rate based on bidirectional motion compensation, Progr. Comput. Software, 2009, vol. 35, no. 6, pp. 351–364.
Mittal, A., Soundararajan, R., and Bovik, A.C., Making a “completely blind” image quality analyzer, IEEE Signal Process. Lett., 2012, vol. 20, no. 3, pp. 209–212.
MSU Video Codec Comparison 2018. Subjective Report. http://compression.ru/video/codec_comparison/hevc_2018/#subjective_report
Bjontegaard, G., Calculation of average PSNR differences between RD-curves, Proc. 13th ITU-T VCEG Meeting, Austin, TX, 2001, document VCEG-M33.
This work was partially supported by the Russian Foundation for Basic Research under Grant 19-01-00785a. Special thanks to Georgiy Osipov and Denis Kondranin who helped to analyze all detected issues and improved NIQE implementation in MSU VQMT, Mikhail Erofeev who helped with MSU comparison methodology improvement and conducted subjective comparisons, and Moscow State University Graphics and Media Lab team for valuable advice and support in our projects.
About this article
Cite this article
Zvezdakova, A.V., Kulikov, D.L., Zvezdakov, S.V. et al. BSQ-rate: a New Approach for Video-codec Performance Comparison and Drawbacks of Current Solutions. Program Comput Soft 46, 183–194 (2020). https://doi.org/10.1134/S0361768820030111
- video quality
- no-reference metric
- quality measuring
- video-codec comparison
- comparison methodology