Advertisement

Image Aesthetics Assessment Using Fully Convolutional Neural Networks

  • Konstantinos Apostolidis
  • Vasileios MezarisEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11295)

Abstract

This paper presents a new method for assessing the aesthetic quality of images. Based on the findings of previous works on this topic, we propose a method that addresses the shortcomings of existing ones, by: (a) Making possible to feed higher-resolution images in the network, by introducing a fully convolutional neural network as the classifier. (b) Maintaining the original aspect ratio of images in the input of the network, to avoid distortions caused by re-scaling. And (c) combining local and global features from the image for making the assessment of its aesthetic quality. The proposed method is shown to achieve state of the art results on a standard large-scale benchmark dataset.

Keywords

Image aesthetics Deep learning Fully convolutional neural networks 

Notes

Acknowledgments

This work was supported by the EU’s Horizon 2020 research and innovation programme under contracts H2020-687786 InVID and H2020-732665 EMMA.

References

  1. 1.
    Bhattacharya, S., Sukthankar, R., Shah, M.: A framework for photo-quality assessment and enhancement based on visual aesthetics. In: Proceedings of 18th ACM International Conference on Multimedia (MM), pp. 271–280. ACM (2010)Google Scholar
  2. 2.
    Bianco, S., Celona, L., Napoletano, P., Schettini, R.: Predicting image aesthetics with deep learning. In: Blanc-Talon, J., Distante, C., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2016. LNCS, vol. 10016, pp. 117–125. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-48680-2_11CrossRefGoogle Scholar
  3. 3.
    Cui, C., Fang, H., Deng, X., Nie, X., Dai, H., Yin, Y.: Distribution-oriented aesthetics assessment for image search. In: Proceedings of 40th International SIGIR Conference on Research and Development in Information Retrieval, pp. 1013–1016. ACM (2017)Google Scholar
  4. 4.
    Deng, X., Cui, C., Fang, H., Nie, X., Yin, Y.: Personalized image aesthetics assessment. In: Proceedings of Conference on Information and Knowledge Management, pp. 2043–2046. ACM (2017)Google Scholar
  5. 5.
    Donahue, J., et al.: DeCAF: a deep convolutional activation feature for generic visual recognition. In: Proceedings of International Conference on Machine Learning (ICML), pp. 647–655 (2014)Google Scholar
  6. 6.
    Dong, Z., Shen, X., Li, H., Tian, X.: Photo quality assessment with DCNN that understands image well. In: He, X., Luo, S., Tao, D., Xu, C., Yang, J., Hasan, M.A. (eds.) MMM 2015. LNCS, vol. 8936, pp. 524–535. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-14442-9_57CrossRefGoogle Scholar
  7. 7.
    Dong, Z., Tian, X.: Multi-level photo quality assessment with multi-view features. Neurocomputing 168, 308–319 (2015)CrossRefGoogle Scholar
  8. 8.
    Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury, S., Pal, C.: The importance of skip connections in biomedical image segmentation. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 179–187. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46976-8_19CrossRefGoogle Scholar
  9. 9.
    Guo, Y., Liu, M., Gu, T., Wang, W.: Improving photo composition elegantly: considering image similarity during composition optimization. In: Computer Graphics Forum, vol. 31, pp. 2193–2202. Wiley Online Library (2012)Google Scholar
  10. 10.
    Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, p. 3 (2017)Google Scholar
  11. 11.
    Jin, X., Chi, J., Peng, S., Tian, Y., Ye, C., Li, X.: Deep image aesthetics classification using inception modules and fine-tuning connected layer. In: Proceedings of IEEE 8th International Conference on Wireless Communications & Signal Processing (WCSP), pp. 1–6. IEEE (2016)Google Scholar
  12. 12.
    Kong, S., Shen, X., Lin, Z., Mech, R., Fowlkes, C.: Photo aesthetics ranking network with attributes and content adaptation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 662–679. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46448-0_40CrossRefGoogle Scholar
  13. 13.
    Krizhevsky, A., Ilya, S., Hinton, G.: ImageNet classification with deep convolutional neural networks. In: Proceedings of Conference on Advances in Neural Information Processing Systems (NIPS), pp. 1097–1105. Curran Associates, Inc. (2012)Google Scholar
  14. 14.
    Lemarchand, F.: From computational aesthetic prediction for images to films and online videos. Avant 8, 69–78 (2017)CrossRefGoogle Scholar
  15. 15.
    Li, Y., Zhang, T., Liu, Z., Hu, H.: A concatenating framework of shortcut convolutional neural networks. arXiv preprint arXiv:1710.00974 (2017)
  16. 16.
    Liang, M., Hu, X., Zhang, B.: Convolutional neural networks with intra-layer recurrent connections for scene labeling. In: Proceedings of Conference on Advances in Neural Information Processing Systems (NIPS), Red Hook, NY Curran, pp. 937–945 (2015)Google Scholar
  17. 17.
    Lo, K.Y., Liu, K.H., Chen, C.S.: Assessment of photo aesthetics with efficiency. In: Proceedings of 21st International Conference on Pattern Recognition (ICPR), pp. 2186–2189. IEEE (2012)Google Scholar
  18. 18.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431–3440. IEEE (2015)Google Scholar
  19. 19.
    Lu, X., Lin, Z., Jin, H., Yang, J., Wang, J.Z.: Rapid: rating pictorial aesthetics using deep learning. In: Proceedings of 22nd ACM Internatioanl Conference on Multimedia (MM), pp. 457–466. ACM (2014)Google Scholar
  20. 20.
    Lu, X., Lin, Z., Jin, H., Yang, J., Wang, J.Z.: Rating image aesthetics using deep learning. IEEE Trans. Multimedia 17(11), 2021–2034 (2015)CrossRefGoogle Scholar
  21. 21.
    Mai, L., Jin, H., Liu, F.: Composition-preserving deep photo aesthetics assessment. In: Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 497–506. IEEE (2016)Google Scholar
  22. 22.
    Marchesotti, L., Perronnin, F., Larlus, D., Csurka, G.: Assessing the aesthetic quality of photographs using generic image descriptors. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 1784–1791. IEEE (2011)Google Scholar
  23. 23.
    Matan, O., Burges, C.J., LeCun, Y., Denker, J.S.: Multi-digit recognition using a space displacement neural network. In: Proceedings of Conference on Advances in Neural Information Processing Systems (NIPS), pp. 488–495 (1992)Google Scholar
  24. 24.
    Mavridaki, E., Mezaris, V.: A comprehensive aesthetic quality assessment method for natural images using basic rules of photography. In: Proceedings of IEEE International Conference on Image Processing (ICIP), pp. 887–891. IEEE (2015)Google Scholar
  25. 25.
    Murray, N., Marchesotti, L., Perronnin, F.: AVA: a large-scale database for aesthetic visual analysis. In: Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2408–2415. IEEE (2012)Google Scholar
  26. 26.
    Nejdl, W., Niederee, C.: Photos to remember, photos to forget. IEEE Trans. MultiMedia (TMM) 22(1), 6–11 (2015)CrossRefGoogle Scholar
  27. 27.
    Obrador, P., Anguera, X., de Oliveira, R., Oliver, N.: The role of tags and image aesthetics in social image search. In: Proceedings of 1st SIGMM Workshop on Social Media, pp. 65–72. ACM (2009)Google Scholar
  28. 28.
    Pittaras, N., Markatopoulou, F., Mezaris, V., Patras, I.: Comparison of fine-tuning and extension strategies for deep convolutional neural networks. In: Amsaleg, L., Guðmundsson, G.Þ., Gurrin, C., Jónsson, B.Þ., Satoh, S. (eds.) MMM 2017. LNCS, vol. 10132, pp. 102–114. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-51811-4_9CrossRefGoogle Scholar
  29. 29.
    Ren, J., Shen, X., Lin, Z.L., Mech, R., Foran, D.J.: Personalized image aesthetics. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 638–647. IEEE (2017)Google Scholar
  30. 30.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Proceedings of International Conference on Learning Representations (ICLR) (2015)Google Scholar
  31. 31.
    Su, H.H., Chen, T.W., Kao, C.C., Hsu, W.H., Chien, S.Y.: Preference-aware view recommendation system for scenic photos based on bag-of-aesthetics-preserving features. IEEE Trans. Multimedia (TMM) 14(3), 833–843 (2012)CrossRefGoogle Scholar
  32. 32.
    Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9. IEEE (2015)Google Scholar
  33. 33.
    Wang, W., Zhao, M., Wang, L., Huang, J., Cai, C., Xu, X.: A multi-scene deep learning model for image aesthetic evaluation. Sig. Process. Image Commun. 47, 511–518 (2016)CrossRefGoogle Scholar
  34. 34.
    Wang, Z., Liu, D., Chang, S., Dolcos, F., Beck, D., Huang, T.: Image aesthetics assessment using deep Chatterjee’s machine. In: Proceedings of IEEE International Joint Conference on Neural Networks (IJCNN), pp. 941–948. IEEE (2017)Google Scholar
  35. 35.
    Yeh, C.H., Ho, Y.C., Barsky, B.A., Ouhyoung, M.: Personalized photograph ranking and selection system. In: Proceedings of 18th ACM International Conference on Multimedia (MM), pp. 211–220. ACM (2010)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Information Technologies Institute/CERTHThermi, ThessalonikiGreece

Personalised recommendations