Skip to main content

Personalised Aesthetics with Residual Adapters

Part of the Lecture Notes in Computer Science book series (LNIP,volume 11867)

Abstract

The use of computational methods to evaluate aesthetics in photography has gained interest in recent years due to the popularization of convolutional neural networks and the availability of new annotated datasets. Most studies in this area have focused on designing models that do not take into account individual preferences for the prediction of the aesthetic value of pictures. We propose a model based on residual learning that is capable of learning subjective, user-specific preferences over aesthetics in photography, while surpassing the state-of-the-art methods and keeping a limited number of user-specific parameters in the model. Our model can also be used for picture enhancement, and it is suitable for content-based or hybrid recommender systems in which the amount of computational resources is limited.

C. Rodríguez-Pardo—Work performed at the University of Edinburgh.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-31332-6_44
  • Chapter length: 13 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   84.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-31332-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   109.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.

Notes

  1. 1.

    Please visit https://github.com/crp94/Personalised-aesthetic-assessment-using-residual-adapters for our implementation in PyTorch.

References

  1. Alif, M.A.R., Ahmed, S., Hasan, M.A.: Isolated Bangla handwritten character recognition with convolutional neural network. In: 2017 20th International Conference of Computer and Information Technology (ICCIT), pp. 1–6. IEEE, December 2017. http://ieeexplore.ieee.org/document/8281823/

  2. Shaji, A.: Understanding aesthetics with deep learning (2016). https://devblogs.nvidia.com/understanding-aesthetics-deep-learning/

  3. Bhattacharya, S., Sukthankar, R., Shah, M.: A framework for photo-quality assessment and enhancement based on visual aesthetics. In: Proceedings of the International Conference on Multimedia - MM 2010, p. 271. ACM Press, New York (2010). http://dl.acm.org/citation.cfm?doid=1873951.1873990

  4. Bianco, S., Celona, L., Napoletano, P., Schettini, R.: Predicting image aesthetics with deep learning. In: Blanc-Talon, J., Distante, C., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2016. LNCS, vol. 10016, pp. 117–125. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48680-2_11

    CrossRef  Google Scholar 

  5. Browniee, J.: A gentle introduction to transfer learning for deep learning (2017). https://machinelearningmastery.com/transfer-learning-for-deep-learning/

  6. Chen, Y.L., Huang, T.W., Chang, K.H., Tsai, Y.C., Chen, H.T., Chen, B.Y.: Quantitative analysis of automatic image cropping algorithms: a dataset and comparative study. In: Proceedings - 2017 IEEE Winter Conference on Applications of Computer Vision, WACV, pp. 226–234 (2017). https://arxiv.org/pdf/1701.01480.pdf

  7. Datta, R., Joshi, D., Li, J., Wang, J.Z.: Studying aesthetics in photographic images using a computational approach. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3953, pp. 288–301. Springer, Heidelberg (2006). https://doi.org/10.1007/11744078_23

    CrossRef  Google Scholar 

  8. Datta, R., Li, J., Wang, J.Z.: Algorithmic inferencing of aesthetics and emotion in natural images: an exposition (2008). http://riemann.ist.psu.edu/

  9. Deng, Y., Loy, C.C., Tang, X.: Image aesthetic assessment: an experimental survey. IEEE Signal Process. Mag. 34(4), 80–106 (2017). https://arxiv.org/pdf/1610.00838.pdf

    CrossRef  Google Scholar 

  10. Denzler, J., Rodner, E., Simon, M.: Convolutional neural networks as a computational model for the underlying processes of aesthetics perception. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9913, pp. 871–887. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46604-0_60

    CrossRef  Google Scholar 

  11. Hayn-Leichsenring, G.U., Lehmann, T., Redies, C.: Subjective ratings of beauty and aesthetics: correlations with statistical image properties in Western oil paintings (2017)

    Google Scholar 

  12. Hong, L., Doumith, A.S., Davison, B.D.: Co-factorization machines. In: Proceedings of the Sixth ACM International Conference on Web Search and Data Mining - WSDM 2013, p. 557. ACM Press, New York (2013). http://dl.acm.org/citation.cfm?doid=2433396.2433467

  13. Isinkaye, F., Folajimi, Y., Ojokoh, B.: Recommendation systems: principles, methods and evaluation. Egypt. Inform. J. 16(3), 261–273 (2015). https://www.sciencedirect.com/science/article/pii/S1110866515000341

    CrossRef  Google Scholar 

  14. Jiang, W., Loui, A.C., Cerosaletti, C.D.: Automatic aesthetic value assessment in photographic images. In: 2010 International Conference on Multimedia and Expo, pp. 920–925. IEEE, July 2010. http://ieeexplore.ieee.org/document/5582588/

  15. Jin, X., Zhao, M., Chen, X., Zhao, Q., Zhu, S.-C.: Learning artistic lighting template from portrait photographs. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 101–114. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15561-1_8

    CrossRef  Google Scholar 

  16. Joshi, D., et al.: Aesthetics and emotions in images. IEEE Signal Process. Mag. 28(5), 94–115 (2011). http://ieeexplore.ieee.org/document/5999579/

    CrossRef  Google Scholar 

  17. Kong, S., Shen, X., Lin, Z., Mech, R., Fowlkes, C.: Photo aesthetics ranking network with attributes and content adaptation (2016). https://arxiv.org/pdf/1606.01621.pdf

  18. Leder, H., Belke, B., Oeberst, A., Augustin, D.: A model of aesthetic appreciation and aesthetic judgments. Br. J. Psychol. 95(4), 489–508 (2010). https://doi.org/10.1348/0007126042369811

    CrossRef  Google Scholar 

  19. Liu, Y., Zhang, Y.M., Zhang, X.Y., Liu, C.L.: Adaptive spatial pooling for image classification. Pattern Recogn. 55(C), 58–67 (2016). https://linkinghub.elsevier.com/retrieve/pii/S0031320316000510

    Google Scholar 

  20. Lu, X., Lin, Z., Jin, H., Yang, J., Wang, J.Z.: Rapid: rating pictorial aesthetics using deep learning. In: Proceedings of the ACM International Conference on Multimedia - MM 2014, pp. 457–466 (2014). http://dl.acm.org/citation.cfm?doid=2647868.2654927

  21. Luo, Y., Tang, X.: Photo and video quality evaluation: focusing on the subject. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5304, pp. 386–399. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88690-7_29

    CrossRef  Google Scholar 

  22. Murray, N., Marchesotti, L., Perronnin, F.: AVA: a large-scale database for aesthetic visual analysis. http://refbase.cvc.uab.es/files/MMP2012a.pdf

  23. Niu, Y., Liu, F.: What makes a professional video? A computational aesthetics approach. IEEE Trans. Circuits Syst. Video Technol. 22(7), 1037–1049 (2012). http://ieeexplore.ieee.org/document/6162974/

    CrossRef  Google Scholar 

  24. O’Donovan, P., Agarwala, A., Hertzmann, A.: Collaborative filtering of color aesthetics. In: Proceedings of Workshop Computational Aesthetics - CAe 2014, pp. 33–40. ACM Press, New York (2014). http://dl.acm.org/citation.cfm?doid=2630099.2630100

  25. Rebuffi, S.A., Bilen, H., Vedaldi, A.: Learning multiple visual domains with residual adapters (2017). http://arxiv.org/abs/1705.08045

  26. Rebuffi, S.A., Bilen, H., Vedaldi, A.: Efficient parametrization of multi-domain deep neural networks (2018). http://arxiv.org/abs/1803.10082

  27. Ren, J., Shen, X., Lin, Z., Mech, R., Foran, D.J.: Personalized image aesthetics. In: Proceedings of the IEEE International Conference on Computer Vision, October 2017, pp. 638–647 (2017)

    Google Scholar 

  28. Rothe, R., Timofte, R., Van Gool, L.: Some like it hot-visual guidance for preference prediction. Technical report (2016). http://hotornot.com

  29. Schafer, J.B., Frankowski, D., Herlocker, J., Sen, S.: Collaborative filtering recommender systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web. LNCS, vol. 4321, pp. 291–324. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-72079-9_9

    CrossRef  Google Scholar 

  30. de Stoutz, E., Ignatov, A., Kobyshev, N., Timofte, R., Van Gool, L.: Fast perceptual image enhancement, December 2018. http://arxiv.org/abs/1812.11852

  31. Tang, X., Luo, W., Wang, X.: Content-based photo quality assessment. IEEE Trans. Multimed. 15(8), 1930–1943 (2013). http://ieeexplore.ieee.org/document/6544270/

    CrossRef  Google Scholar 

  32. Vogel, D., Khan, S.S.: Evaluating visual aesthetics in photographic portraiture. In: Proceedings of Eighth Annual Symposium on Computational Aesthetics in Graphics, Visualization, and Imaging, p. 128 (2012). https://dl.acm.org/citation.cfm?id=2328898

  33. Wang, W., Shen, J.: Deep cropping via attention box prediction and aesthetics assessment. In: Proceedings of the IEEE International Conference on Computer Vision, October 2017, pp. 2205–2213 (2017)

    Google Scholar 

  34. Ke, Y., Tang, X., Jing, F.: The design of high-level features for photo quality assessment. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), vol. 1, pp. 419–426. IEEE (2006). http://ieeexplore.ieee.org/document/1640788/

  35. Yu, W., Chen, X.: Aesthetic-based clothing recommendation 2 (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Carlos Rodríguez-Pardo .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Rodríguez-Pardo, C., Bilen, H. (2019). Personalised Aesthetics with Residual Adapters. In: Morales, A., Fierrez, J., Sánchez, J., Ribeiro, B. (eds) Pattern Recognition and Image Analysis. IbPRIA 2019. Lecture Notes in Computer Science(), vol 11867. Springer, Cham. https://doi.org/10.1007/978-3-030-31332-6_44

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-31332-6_44

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-31331-9

  • Online ISBN: 978-3-030-31332-6

  • eBook Packages: Computer ScienceComputer Science (R0)