Abstract
These days, social media holds a large portion of our daily lives. Millions of people post their images using a social media platform. The enormous amount of images shared on social network presents serious challenges and requires massive computing resources to ensure efficient data processing. However, images are subject to a wide range of distortions in real application scenarios during the processing, transmission, sharing, or combination of many factors. So, there is a need to guarantee acceptable delivery content, even though some distorted images do not have access to their original version. In this paper, we present a framework developed to process a large amount of images in real-time while estimating and assisting in the enhancement of the No-Reference and Full-Reference image quality. Our quality evaluation is measured using a Convolutional Neural Network, which is tuned by the objective quality methods, in addition to the face alignment metric and enhanced with the help of a Super-Resolution Model. A set of experiments is conducted to evaluate our proposed approach.
Similar content being viewed by others
References
(2015) Apache storm—concepts. [Online]. Available: http://storm.apache.org/releases/current/Concepts.html
(2015) Setting up a development environment [Online]. Available: http://storm.apache.org/releases/1.0.6/Setting-up-development-environment.html
(2018) Apache storm cluster architecture. [Online]. Available: http://storm.apache.org/releases/1.0.6/Setting-up-development-environment.html
Abaza A (2014) Design and evaluation of photometric image quality measures for effective face recognition. IET Biometrics 3(10):314–324. [Online]. Available: https://digital-library.theiet.org/content/journals/10.1049/iet-bmt.2014.0022
Agrawal P, Narayanan PJ (2011) Person de-identification in videos. IEEE Trans Circ Syst Video Technol 21(3):299–310
Aggarwal CC, Hinneburg A, Keim DA (2001) On the surprising behavior of distance metrics in high dimensional space. In: Van den Bussche J, Vianu V (eds) Database theory—ICDT 2001. Springer, Berlin, pp 420–434
al Bouna B, Chbeir R, Gabillon A (2011) The image protector—a flexible security rule specification toolkit. In: SECRYPT, pp 345–350
Bai X, Wang J, Simons D, Sapiro G (2009) Video snapcut: robust video object cutout using localized classifiers. In: ACM SIGGRAPH 2009 papers, ser. SIGGRAPH ’09. ACM, New York, pp 70:1–70:11
Bianco S, Cadène R, Celona L, Napoletano P (2018) Benchmark analysis of representative deep neural network architectures. IEEE Access 6:64270–64277
Bosse S, Maniry D, Müller K, Wiegand T, Samek W (2018) Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans Image Process 27(1):206–219
Caviedes J, Gurbuz S (2002) No-reference sharpness metric based on local edge kurtosis. In: Proceedings. International conference on image processing, vol 3, pp III–III
Chami Z, Bouna BAL, Abou Jaoude C, Chbeir R (2019) A real-time multimedia data quality assessment framework, 270–276
Chuang Y-Y, Agarwala A, Curless B, Salesin DH, Szeliski R (2002) Video matting of complex scenes. In: Proceedings of the 29th annual conference on computer graphics and interactive techniques, ser. SIGGRAPH ’02. ACM, New York, pp 243–248
Ciregan D, Meier U, Schmidhuber J (2012) Multi-column deep neural networks for image classification. In: 2012 IEEE conference on computer vision and pattern recognition, pp 3642–3649
De Bruyne S, De Schrijver D, De Neve W, Van Deursen D, Van de Walle R (2007) Enhanced shot-based video adaptation using mpeg-21 generic bitstream syntax schema. In: IEEE symposium on computational intelligence in image and signal processing, 2007. CIISP 2007, pp 380–385
Dong C, Loy CC, He K, Tang X (2014) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38:12
Dutta A, Veldhuis RNJ, Spreeuwers LJ (2015) Predicting face recognition performance using image quality. CoRR arXiv:1510.07119. [Online]. Available: 1510.07119
El-Khoury V, Bennani N, Coquil D (2010) Utility function for semantic video content adaptation. In: Proceedings of the 12th international conference on information integration and web-based applications & services, ser. iiWAS ’10. [Online]. Available: https://doi.org/10.1145/1967486.1967649. Association for Computing Machinery, New York, pp 921–924
Fan J, Luo H, Hacid M-S, Bertino E (2005) A novel approach for privacy-preserving video sharing. In: Proceedings of the 14th ACM international conference on Information and knowledge management, ser. CIKM ’05. ACM, New York, pp 609–616
Gang Z, Chia L-T, Zongkai Y (2004) Mpeg-21 digital item adaptation by applying perceived motion energy to h.264 video. In: 2004 international conference on image processing, 2004. ICIP ’04, vol 4, pp 2777–2780
Ghadiyaram D, Bovik AC (2016) Massive online crowdsourced study of subjective and objective picture quality. IEEE Trans Image Process 25(1):372–387
Gudivada VN, Raghavan VV (1995) Content based image retrieval systems. Computer 28(9):18–22
He L, Tao D, Li X, Gao X (2012) Sparse representation for blind image quality assessment. In: 2012 IEEE conference on computer vision and pattern recognition, pp 1146–1153
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: The IEEE conference on computer vision and pattern recognition (CVPR)
Herranz L (2007) Integrating semantic analysis and scalable video coding for efficient content-based adaptation. Multimed Syst 13:103–118. https://doi.org/10.1007/s00530-007-0090-0
Hou L, Yu C, Samaras D Squared earth mover’s distance-based loss for training deep neural networks. CoRR arXiv:1611.05916, 2016. [Online]. Available: 1611.05916
Huynh-Thu Q (2008) Scope of validity of psnr in image/video quality assessment. Electron Lett 44(1):800–801. [Online]. Available: https://digital-library.theiet.org/content/journals/10.1049/el_20080522
Kang L, Ye P, Li Y, Doermann D (2014) Convolutional neural networks for no-reference image quality assessment. In: The IEEE conference on computer vision and pattern recognition (CVPR)
Kephart JO, Das R (2007) Achieving self-management via utility functions. IEEE Internet Comput 11(1):40–48
Kim J, Lee S (2017) Fully deep blind image quality predictor. IEEE J Sel Top Signal Process 11(1):206–220
Kim CS, Sohn H, Neve WD, Ro YM (2009) An objective perceptual quality-based adte for adapting mobile svc video content. IEICE Trans Inf Syst E92.D(1):93–96
Li Q, Lin W, Fang Y (2016) No-reference quality assessment for multiply-distorted images in gradient domain. IEEE Signal Process Lett 23(4):541–545
Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, Shi W (2017) Photo-realistic single image super-resolution using a generative adversarial network, pp 105–114
Lian Z, Li Y, Tao J, Huang J, Niu M -Y (2019) Expression analysis based on face regions in read-world conditions. Int J Autom Comput 17
Lin K-Y, Wang G (2018) Hallucinated-iqa: No-reference image quality assessment via adversarial learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)
Liu Z, Luo P, Wang X, Tang X (2015) Deep learning face attributes in the wild. In: Proceedings of international conference on computer vision (ICCV)
Liu X, van de Weijer J, Bagdanov AD (2017) Rankiqa: learning from rankings for no-reference image quality assessment. In: Proceedings of the IEEE international conference on computer vision (ICCV)
Majumdar S (2016) Image super resolution. https://github.com/titu1994/Image-Super-Resolution
Marziliano P, Dufaux F, Winkler S, Ebrahimi T (2002) A no-reference perceptual blur metric. In: Proceedings. International conference on image processing, vol 3, pp III–III
Meesters L, Martens J-B (2002) A single-ended blockiness measure for jpeg-coded images. Signal Process 82(3):369–387. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0165168401001773
Mittal A, Moorthy AK, Bovik AC (2011) Blind/referenceless image spatial quality evaluator. In: 2011 Conference record of the forty fifth asilomar conference on signals, systems and computers (ASILOMAR), pp 723–727
Moorthy AK, Bovik AC (2010) A two-step framework for constructing blind image quality indices. IEEE Signal Process Lett 17(5):513–516
Moorthy AK, Bovik AC (2011) Blind image quality assessment: from natural scene statistics to perceptual quality. IEEE Trans Image Process 20 (12):3350–3364
Mu M, Cerqueira E, Boavida F, Mauthe A (2009) Quality of experience management framework for real-time multimedia applications. IJIPT 4:54–64
Newton EM, Sweeney L, Malin B (2005) Preserving privacy by de-identifying face images. IEEE Trans Knowl Data Eng 17(2):232–243
Nguyen SM, Ogino M, Asada M Real-time face swapping as a tool for understanding infant self-recognition. CoRR arXiv:1112.2095, 2011. [Online]. Available: 1112.2095
Ponomarenko N, Jin L, Ieremeiev O, Lukin V, Egiazarian K, Astola J, Vozel B, Chehdi K, Carli M, Battisti F, Jay Kuo C-C (2015) Image database tid2013: peculiarities, results and perspectives. Signal Process: Image Commun 30:57–77. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0923596514001490
Prangl M, Szkaliczki T, Hellwagner H (2007) A framework for utility-based multimedia adaptation. IEEE Trans Circ Syst Video Technol 17(6):719–728
Rippel O, Bourdev L (2017) Real-time adaptive image compression
Rueangprathum A, Limsiroratana S, Witosurapot S (2016) User-driven multimedia adaptation framework for context-aware learning content service. J Adv Inf Technol 7:182–185
Saad MA, Bovik AC, Charrier C (2012) Blind image quality assessment: a natural scene statistics approach in the dct domain. IEEE Trans Image Process 21(8):3339–3352
Sheikh HR, Sabir MF, Bovik AC (2006) A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans Image Process 15(11):3440–3451
Social media statistics in 2020. https://dustinstout.com/social-media-statistics/#instagram-stats. Accessed: 24 Jan 2020
Vega MT, Mocanu DC, Famaey J, Stavrou S, Liotta A (2017) Deep learning for quality assessment in live video streaming. IEEE Signal Process Lett 24(6):736–740
Vijay Venkatesh M, Cheung S-CS, Zhao J (2009) Efficient object-based video inpainting. Pattern Recogn Lett 30(2):168–179
Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612
Wickramasuriya J, Datt M, Mehrotra S, Venkatasubramanian N (2004) Privacy protecting data collection in media spaces. In: Proceedings of the 12th annual ACM international conference on multimedia, ser. Multimedia ’04. ACM, New York, pp 48–55
Yang W, Zhang X, Tian Y, Wang W, Xue J-H, Liao Q (2019) Deep learning for single image super-resolution: a brief review. IEEE Trans Multimed 21:3106–3121
Ye P, Kumar J, Kang L, Doermann D (2012) Unsupervised feature learning framework for no-reference image quality assessment. In: 2012 IEEE conference on computer vision and pattern recognition, pp 1098–1105
Zhang L, Zhang L, Bovik AC (2015) A feature-enriched completely blind image quality evaluator. IEEE Trans Image Process 24(8):2579–2591
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This work is jointly funded from the National Council for Scientific Research in Lebanon (CNRS-L), the Antonine University, and the Agence universitaire de la Francophonie (AUF).
Rights and permissions
About this article
Cite this article
Chami, Z.A., Jaoude, C.A., Chbeir, R. et al. A No-Reference and Full-Reference image quality assessment and enhancement framework in real-time. Multimed Tools Appl 81, 32491–32517 (2022). https://doi.org/10.1007/s11042-022-12334-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-022-12334-z