Skip to main content
SpringerLink
Log in

SUR-FeatNet: Predicting the satisfied user ratio curve for image compression with deep feature learning

  • Research Article
  • Published:
Quality and User Experience Aims and scope Submit manuscript

Abstract

The satisfied user ratio (SUR) curve for a lossy image compression scheme, e.g., JPEG, characterizes the complementary cumulative distribution function of the just noticeable difference (JND), the smallest distortion level that can be perceived by a subject when a reference image is compared to a distorted one. A sequence of JNDs can be defined with a suitable successive choice of reference images. We propose the first deep learning approach to predict SUR curves. We show how to apply maximum likelihood estimation and the Anderson–Darling test to select a suitable parametric model for the distribution function. We then use deep feature learning to predict samples of the SUR curve and apply the method of least squares to fit the parametric model to the predicted samples. Our deep learning approach relies on a siamese convolutional neural network, transfer learning, and deep feature learning, using pairs consisting of a reference image and a compressed image for training. Experiments on the MCL-JCI dataset showed state-of-the-art performance. For example, the mean Bhattacharyya distances between the predicted and ground truth first, second, and third JND distributions were 0.0810, 0.0702, and 0.0522, respectively, and the corresponding average absolute differences of the peak signal-to-noise ratio at a median of the first JND distribution were 0.58, 0.69, and 0.58 dB. Further experiments on the JND-Pano dataset showed that the method transfers well to high resolution panoramic images viewed on head-mounted displays.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Bhattacharyya A (1943) On a measure of divergence between two statistical populations defined by their probability distributions. Bull Calcutta Math Soc 35:99–109

    MathSciNet  MATH  Google Scholar 

  2. Bosse S, Maniry D, Müller KR, Wiegand T, Samek W (2018) Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans Image Process 27(1):206–219

    Article  MathSciNet  Google Scholar 

  3. Bromley J, Guyon I, LeCun Y, Säckinger E, Shah R (1993) Signature verification using a siamese time delay neural network. Adv neural inf process sys 7(4):737–744

    Google Scholar 

  4. Chollet F et al (2015) Keras. https://keras.io

  5. Chopra S, Hadsell R, LeCun Y (2005) Learning a similarity metric discriminatively, with application to face verification. In: Computer vision and pattern recognition (CVPR), vol 1. IEEE, pp 539–546

  6. Fan C, Lin H, Hosu V, Zhang Y, Jiang Q, Hamzaoui R, Saupe D (2019) SUR-Net: predicting the satisfied user ratio curve for image compression with deep learning. In: Eleventh international conference on quality of multimedia experience (QoMEX). IEEE, pp 1–6

  7. Fan C, Zhang Y, Zhang H, Hamzaoui R, Jiang Q (2019) Picture-level just noticeable difference for symmetrically and asymmetrically compressed stereoscopic images: subjective quality assessment study and datasets. J Vis Commun Image Represent 62:140–151

    Article  Google Scholar 

  8. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT press, Cambridge

    MATH  Google Scholar 

  9. Hadizadeh H, Reza Heravi A, Bajic IV, Karami P (2018) A perceptual distinguishability predictor for JND-noise-contaminated images. IEEE Trans Image Process 28:2242–2256

    Article  MathSciNet  Google Scholar 

  10. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  11. Hosu V, Goldlucke B, Saupe D (2019) Effective aesthetics prediction with multi-level spatially pooled features. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9375–9383

  12. Hosu V, Lin H, Sziranyi T, Saupe D (2020) KonIQ-10k: an ecologically valid database for deep learning of blind image quality assessment. IEEE Trans Image Process 29:4041–4056. https://doi.org/10.1109/TIP.2020.2967829

    Article  Google Scholar 

  13. Huang Q, Wang H, Lim SC, Kim HY, Jeong SY, Kuo CCJ (2017) Measure and prediction of HEVC perceptually lossy/lossless boundary QP values. In: Data compression conference (DCC), pp 42–51

  14. Jin L, Lin JY, Hu S, Wang H, Wang P, Katsavounidis I, Aaron A, Kuo CCJ (2016) Statistical study on perceived JPEG image quality via MCL-JCI dataset construction and analysis. In: IS&T international symposium on electronic imaging. Image quality and system performance, vol XIII, pp 1–9

  15. Johnson NL, Kotz S, Balakrishnan N (1993) Continuous univariate distributions, vol 1. Wiley, Hoboken

    MATH  Google Scholar 

  16. Johnson NL, Kotz S, Balakrishnan N (1994) Continuous univariate distributions, vol 2. Wiley, Hoboken

    MATH  Google Scholar 

  17. Keelan BW, Urabe H (2003) Iso 20462: a psychophysical image quality measurement standard. In: The proceeding of SPIE-IS&T electronic imaging, image quality and system performance, vol 5294. International Society for Optics and Photonics, pp 181–189

  18. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980

  19. Li Z, Aaron A, Katsavounidis I, Moorthy A, Manohara M (2016) Toward a practical perceptual video quality metric. In: The Netflix Tech Blog, vol 29

  20. Lin H, Hosu V, Saupe D (2019) KADID-10k: a large-scale artificially distorted IQA database. In: Eleventh international conference on quality of multimedia experience (QoMEX). IEEE, pp 1–3

  21. Lin JY, Jin L, Hu S, Katsavounidis I, Li Z, Aaron A, Kuo CCJ (2015) Experimental design and analysis of jnd test on coded image/video. In: Applications of digital image processing XXXVIII, vol 9599. International Society for Optics and Photonics, p 95990Z

  22. Lin TY, Goyal P, Girshick R, He K, Dollar P (2017) Focal loss for dense object detection. In: IEEE International conference on computer vision (ICCV), pp 2999–3007

  23. Liu H, Zhang Y, Zhang H, Fan C, Kwong S, Kuo CJ, Fan X (2020) Deep learning-based picture-wise just noticeable distortion prediction model for image compression. IEEE Trans Image Process 29:641–656

    Article  MathSciNet  Google Scholar 

  24. Liu X, Chen Z, Wang X, Jiang J, Kowng S (2018) JND-Pano: database for just noticeable difference of JPEG compressed panoramic images. In: Pacific rim conference on multimedia (PCM). Springer, New York, pp 458–468

  25. Nafchi HZ, Shahkolaei A, Hedjam R, Cheriet M (2016) Mean deviation similarity index: efficient and reliable full-reference image quality evaluator. IEEE Access 4:5579–5590

    Article  Google Scholar 

  26. Nist/Sematech e-Handbook of Statistical Methods (2020). http://www.itl.nist.gov/div898/handbook

  27. Recommendation ITU-R BT.500-11 (2002) Methodology for the subjective assessment of the quality of television pictures

  28. Redi J, Liu H, Alers H, Zunino R, Heynderickx I (2010) Comparing subjective image quality measurement methods for the creation of public databases. In: Proceeding of SPIE-IS&T electronic imaging, image quality and system performance VII, vol 7529. International Society for Optics and Photonics, p 752903

  29. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 779–788

  30. SUR-FeatNet Source Code (2019). https://github.com/Linhanhe/SUR-FeatNet

  31. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2818–2826

  32. Wang H, Gan W, Hu S, Lin JY, Jin L, Song L, Wang P, Katsavounidis I, Aaron A, Kuo CCJ (2016) MCL-JCV: a JND-based H. 264/AVC video quality assessment dataset. In: IEEE international conference on image processing (ICIP), pp 1509–1513

  33. Wang H, Katsavounidis I, Huang Q, Zhou X, Kuo CCJ (2018) Prediction of satisfied user ratio for compressed video. In: IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 6747–6751

  34. Wang H, Katsavounidis I, Zhou J, Park J, Lei S, Zhou X, Pun MO, Jin X, Wang R, Wang X et al (2017) VideoSet: a large-scale compressed video quality dataset based on JND measurement. J Vis Commun Image Represent 46:292–302

    Article  Google Scholar 

  35. Wiedemann O, Hosu V, Lin H, Saupe D (2018) Disregarding the big picture: towards local image quality assessment. In: International conference on quality of multimedia experience (QoMEX)

  36. Zhang X, Yang S, Wang H, Xu W, Kuo CCJ (2020) Satisfied-user-ratio modeling for compressed video. IEEE Trans Image Process 29:3777–3789

    Article  Google Scholar 

Download references

Funding

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)—Project-ID 251654672–TRR 161 (Project A05).

Author information

Authors and Affiliations

  1. Department of Computer and Information Science, University of Konstanz, Konstanz, Germany

    Hanhe Lin, Vlad Hosu & Dietmar Saupe

  2. Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China

    Chunling Fan & Yun Zhang

  3. School of Engineering, University of Edinburgh, Edinburgh, UK

    Yuchen Mu

  4. School of Engineering and Sustainable Development, De Montfort University, Leicester, UK

    Raouf Hamzaoui

Authors
  1. Hanhe Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vlad Hosu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chunling Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuchen Mu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Raouf Hamzaoui
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dietmar Saupe
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hanhe Lin.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, H., Hosu, V., Fan, C. et al. SUR-FeatNet: Predicting the satisfied user ratio curve for image compression with deep feature learning. Qual User Exp 5, 5 (2020). https://doi.org/10.1007/s41233-020-00034-1

Download citation

  • Received:

  • Published:

  • DOI: https://doi.org/10.1007/s41233-020-00034-1

Keywords

Search

Navigation