Content adaptive pre-filtering for video compression

  • Mehdi SaeediEmail author
  • Boris Ivanovic
  • Tomasz Stolarczyk
  • Ihab Amer
  • Gabor Sines
Original Paper


Bitrate reduction with little to no degradation in visual perception is a long-standing challenge in video coding. This paper targets this challenge by adaptively filtering the content prior to video compression and in the preprocessing stage. This is done by applying a bilateral filter where the filter parameters are selected according to regional content complexity and estimated visual importance besides bitrate and quality requirements. A multi-scale metric based on 2D gradient is employed to determine bandwidth requirements of different regions. A random forest regression model is trained to predict distortion and bit requirements for a block, if it is filtered and encoded at a given quality. The predicted distortion and bit requirements are used to select filter parameters considering a cost function. The proposed approach is applied to both H.264 and HEVC encoders, with different GOP structures. The results show up to 60% bitrate reduction in terms of BD-Rate (about 20% on average) for the attempted test cases with little to no noticeable quality degradation.


Adaptive Pre-filtering Machine learning Video encoding Visual perception 



  1. 1.
    Bjontegaard, G.: Calculation of average PSNR differences between RD-curves. In: Proceedings of the ITU-T Video Coding Experts Group, Thirteenth Meeting (2001)Google Scholar
  2. 2.
    Bossen, F.: Common test conditions and software reference configurations. JCT-VC, Technical report I1100 (2012)Google Scholar
  3. 3.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefGoogle Scholar
  4. 4.
    Chen, Z., Liu, H.: JND modeling: approaches and applications. In: International Conference on Digital Signal Processing, pp. 827–830 (2014)Google Scholar
  5. 5.
    Duvar, R., Akbulut, O., Urhan, O.: Fast inter mode decision exploiting intra-block similarity in HEVC. Signal Process. Image Commun. 78, 503–510 (2019)CrossRefGoogle Scholar
  6. 6.
    Grois, D., Nguyen, T., Marpe, D.: Performance comparison of AV1, JEM, VP9, and HEVC encoders. In: Proceedings of SPIE, 10396, (2018)Google Scholar
  7. 7.
    Guo, L., Cock, J.D., Aaron, A.: Compression performance comparison of x264, x265, libvpx and aomenc for on-demand adaptive streaming applications. In: Picture Coding Symposium, pp. 26–30 (2018)Google Scholar
  8. 8.
    Heidari, B., Ramezanpour, M.: Reduction of intra-coding time for HEVC based on temporary direction map. J. Real-Time Image Process. (2018). CrossRefGoogle Scholar
  9. 9.
    Huang, X., Zhang, Q., Zhao, X., Zhang, W., Zhang, Y., Gan, Y.: Fast inter-prediction mode decision algorithm for HEVC. Signal Image Video Process. 11(1), 33–40 (2017)CrossRefGoogle Scholar
  10. 10.
    Jenab, M., Amer, I., Ivanovic, B., Saeedi, M., Liu, Y., Sines, G., Shirani, S.: Content-adaptive resolution control to improve video coding efficiency. In: IEEE International Conference on Multimedia Expo Workshops, pp. 1–4 (2018)Google Scholar
  11. 11.
    Karunaratne, P.V., Segall, C.A., Katsaggelos, A.K.: A rate-distortion optimal video pre-processing algorithm. In: International Conference on Image Processing, 1, 481–484 (2001)Google Scholar
  12. 12.
    Kerofsky, L.J., Vanam, R., Reznik, Y.A.: Improved adaptive video delivery system using a perceptual pre-processing filter. In: Signal and Information Processing, pp. 1058–1062 (2014)Google Scholar
  13. 13.
    Ki, S., Bae, S., Kim, M., Ko, H.: Learning-based just-noticeable-quantization-distortion modeling for perceptual video coding. IEEE Trans. Image Process. 27(7), 3178–3193 (2018)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Kim, W.J., Yi, J.W., Kim, S.D.: A bit allocation method based on picture activity for still image coding. IEEE Trans. Image Process. 8(7), 974–977 (1999)CrossRefGoogle Scholar
  15. 15.
    Lee, J., Ebrahimi, T.: Perceptual video compression: a survey. IEEE J.Sel. Top. Signal Process. 6(6), 684–697 (2012)CrossRefGoogle Scholar
  16. 16.
    Li, S., Zhang, F., Ma, L., Ngan, K.N.: Image quality assessment by separately evaluating detail losses and additive impairments. IEEE Trans. Multimed. 13(5), 935–949 (2011)CrossRefGoogle Scholar
  17. 17.
    Li, Z., Aaron, A., Katsavounidis, I., Moorthy, A., Manohara, M.: Toward a practical perceptual video quality metric. Netflix Technology Blog (2016)Google Scholar
  18. 18.
    Li, Z., Bampis, C., Novak, J., Aaron, A., Swanson, K., Moorthy, A., Cock, J.D.: VMAF: The journey continues. Netflix Technology Blog (2018)Google Scholar
  19. 19.
    Lu, S.P., Zhang, S.R.: Saliency-based fidelity adaptation preprocessing for video coding. J. Comput. Sci. Technol. 26(1), 195–202 (2011)CrossRefGoogle Scholar
  20. 20.
    Najafabadi, N., Ramezanpour, M.: Mass center direction-based decision method for intraprediction in HEVC standard. J. Real-Time Image Process. (2019). CrossRefGoogle Scholar
  21. 21.
    Nguyen, T., Marpe, D.: Future video coding technologies: a performance evaluation of AV1, JEM, VP9, and HM. In: Picture Coding Symposium, pp. 31–35 (2018)Google Scholar
  22. 22.
    Oh, H., Kim, W.: Video processing for human perceptual visual quality-oriented video coding. IEEE Trans. Image Process. 22(4), 1526–1535 (2013)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Ramezanpour, M., Zargari, F.: Fast CU size and prediction mode decision method for HEVC encoder based on spatial features. Signal Image Video Process. 10(7), 1233–1240 (2016)CrossRefGoogle Scholar
  24. 24.
    Rassool, R.: VMAF reproducibility: validating a perceptual practical video quality metric. In: IEEE International Symposium on Broadband Multimedia Systems and Broadcasting, pp. 1–2 (2017)Google Scholar
  25. 25.
    Shaw, M.Q., Allebach, J.P., Delp, E.J.: Color difference weighted adaptive residual preprocessing using perceptual modeling for video compression. Image Commun. 39, 355–368 (2015)Google Scholar
  26. 26.
    Sheikh, H.R., Bovik, A.C.: Image information and visual quality. IEEE Trans. Image Process. 15(2), 430–444 (2006)CrossRefGoogle Scholar
  27. 27.
    Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: International Conference on Computer Vision, pp. 839–846 (1998)Google Scholar
  28. 28.
    Vanam, R., Kerofsky, L.J., Reznik, Y.A.: Perceptual pre-processing filter for adaptive video on demand content delivery. In: IEEE International Conference on Image Processing, pp. 2537–2541 (2014)Google Scholar
  29. 29.
    Vidal, E., Sturmel, N., Guillemot, C., Corlay, P., Coudoux, F.X.: New adaptive filters as perceptual preprocessing for rate-quality performance optimization of video coding. Signal Process. Image Commun. 52, 124–137 (2017)CrossRefGoogle Scholar
  30. 30.
    Yang, X., Ling, W., Lu, Z., Ong, E.P., Yao, S.: Just noticeable distortion model and its applications in video coding. Signal Process. Image Commun. 20(7), 662–680 (2005)CrossRefGoogle Scholar
  31. 31.
    Zhang, X., Lin, W., Wang, S., Liu, J., Ma, S., Gao, W.: Fine-grained quality assessment for compressed images. IEEE Trans Image Process. 28(3), 1163–1175 (2019)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2020

Authors and Affiliations

  1. 1.Advanced Micro Devices, Inc. (AMD)MarkhamCanada

Personalised recommendations