Content-based artwork recommendation: integrating painting metadata with neural and manually-engineered visual features

Abstract

Recommender Systems help us deal with information overload by suggesting relevant items based on our personal preferences. Although there is a large body of research in areas such as movies or music, artwork recommendation has received comparatively little attention, despite the continuous growth of the artwork market. Most previous research has relied on ratings and metadata, and a few recent works have exploited visual features extracted with deep neural networks (DNN) to recommend digital art. In this work, we contribute to the area of content-based artwork recommendation of physical paintings by studying the impact of the aforementioned features (artwork metadata, neural visual features), as well as manually-engineered visual features, such as naturalness, brightness and contrast. We implement and evaluate our method using transactional data from UGallery.com, an online artwork store. Our results show that artwork recommendations based on a hybrid combination of artist preference, curated attributes, deep neural visual features and manually-engineered visual features produce the best performance. Moreover, we discuss the trade-off between automatically obtained DNN features and manually-engineered visual features for the purpose of explainability, as well as the impact of user profile size on predictions. Our research informs the development of next-generation content-based artwork recommenders which rely on different types of data, from text to multimedia.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Notes

  1. 1.

    www.ugallery.com.

  2. 2.

    www.singulart.com.

  3. 3.

    www.artspace.com.

  4. 4.

    http://www.UGallery.com/.

  5. 5.

    Our collaborators at UGallery requested us not to disclose the exact dates when the data was collected.

  6. 6.

    http://caffe.berkeleyvision.org/.

  7. 7.

    To obtain the weights for the different methods, we initialize the coefficients based on the individual performance (concretely, Recall@10) of each method and then we iterate with a grid search, each time narrowing the weight search space in a greedy fashion. The performance tend to converge after two to three iterations.

References

  1. Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17(6), 734–749 (2005)

    Article  Google Scholar 

  2. Aggarwal, C.C.: Content-based recommender systems. In: Recommender Systems, pp. 139–166. Springer, Berlin (2016). https://doi.org/10.1007/978-3-319-29659-3_4

  3. Akay, S., Kundegorski, M.E., Devereux, M., Breckon, T.P.: Transfer learning using convolutional neural networks for object classification within X-ray baggage security imagery. In: Proceedings of the IEEE International Conference on Image Processing (ICIP), pp. 1057–1061 (2016)

  4. Albanese, M., d’Acierno, A., Moscato, V., Persia, F., Picariello, A.: A multimedia semantic recommender system for cultural heritage applications. In: Proceedings of the Fifth IEEE International Conference on Semantic Computing (ICSC), pp. 403–410 (2011)

  5. Amatriain, X.: Mining large streams of user data for personalized recommendations. ACM SIGKDD Explor. Newsl. 14(2), 37–48 (2013)

    Article  Google Scholar 

  6. Aroyo, L., Wang, Y., Brussee, R., Gorgels, P., Rutledge, L., Stash, N.: Personalized museum experience: the rijksmuseum use case. In: Proceedings of Museums and the Web (2007)

  7. Bennett, J., Lanning, S., et al.: The netflix prize. In: Proceedings of KDD Cup and Workshop, vol. 2007, p. 35 (2007)

  8. Benouaret, I., Lenne, D.: Personalizing the museum experience through context-aware recommendations. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 743–748 (2015)

  9. Celma, O.: Music recommendation. In: Music Recommendation and Discovery, pp. 43–85. Springer, Berlin (2010). https://doi.org/10.1007/978-3-642-13287-2_3

  10. Cremonesi, P., Koren, Y., Turrin, R.: Performance of recommender algorithms on top-n recommendation tasks. In: Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys ’10, pp. 39–46 (2010)

  11. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (CVPR) 1, 886–893 (2005)

    Google Scholar 

  12. David, O.E., Netanyahu, N.S.: DeepPainter: Painter Classification Using Deep Convolutional Autoencoders, pp. 20–28. Springer, Berlin (2016)

    Google Scholar 

  13. de Gemmis, M., Lops, P., Musto, C., Narducci, F., Semeraro, G.: Semantics-aware content-based recommender systems. In: Recommender Systems Handbook, pp. 119–159. Springer, Berlin (2015)

  14. Deldjoo, Y., Elahi, M., Cremonesi, P., Garzotto, F., Piazzolla, P., Quadrana, M.: Content-based video recommendation system based on stylistic visual features. J. Data Semant. 5(2), 99–113 (2016)

    Article  Google Scholar 

  15. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255 (2009)

  16. Ekstrand, M.D., Kluver, D., Harper, F.M., Konstan, J.A.: Letting users choose recommender algorithms: an experimental study. In: Proceedings of the 9th ACM Conference on Recommender Systems, RecSys ’15, pp. 11–18 (2015). https://doi.org/10.1145/2792838.2800195

  17. Elahi, M., Deldjoo, Y., Bakhshandegan Moghaddam, F., Cella, L., Cereda, S., Cremonesi, P.: Exploring the semantic gap for movie recommendations. In: Proceedings of the Eleventh ACM Conference on Recommender Systems, RecSys’17, pp. 326–330 (2017)

  18. Esman, A.R.: The World’s Strongest Economy? The Global Art Market. https://www.forbes.com/sites/abigailesman/2012/02/29/the-worlds-strongest-economy-the-global-art-market/ (2012). Accessed 21 March 2017

  19. Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576 (2015)

  20. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)

  21. Gomez-Uribe, C.A., Hunt, N.: The netflix recommender system: algorithms, business value, and innovation. ACM Trans. Manag. Inf. Syst. (TMIS) 6(4), 13 (2016)

    Google Scholar 

  22. Gonzalez, R.C., Eddins, S.L., Woods, R.E.: Digital Image Publishing Using MATLAB. Prentice Hall, Upper Saddle River (2004)

    Google Scholar 

  23. Harper, F.M., Konstan, J.A.: The movielens datasets: history and context. ACM Trans. Interact. Intell. Syst. 5(4), 19:1–19:19 (2015)

    Article  Google Scholar 

  24. He, R., Fang, C., Wang, Z., McAuley, J.: Vista: A visually, socially, and temporally-aware model for artistic recommendation. In: Proceedings of the 10th ACM Conference on Recommender Systems, RecSys ’16, pp. 309–316 (2016)

  25. He, R., McAuley, J.: VBPR: Visual Bayesian personalized ranking from implicit feedback. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp. 144–150 (2016)

  26. Hidasi, B., Quadrana, M., Karatzoglou, A., Tikk, D.: Parallel recurrent neural network architectures for feature-rich session-based recommendations. In: Proceedings of the 10th ACM Conference on Recommender Systems, RecSys’16, pp. 241–248 (2016)

  27. Kannala, J., Rahtu, E.: Bsif: Binarized statistical image features. In: Proceedings of 21st International Conference on Pattern Recognition (ICPR), pp. 1363–1366 (2012)

  28. Karnowski, J.: AlexNet + SVM. https://jeremykarnowski.files.wordpress.com/2015/07/alexnet2.png (2015). Accessed 1 Dec 2017

  29. Knijnenburg, B.P., Bostandjiev, S., O’Donovan, J., Kobsa, A.: Inspectability and control in social recommenders. In: Proceedings of the Sixth ACM Conference on Recommender Systems, pp. 43–50 (2012)

  30. Koch, G., Zemel, R., Salakhutdinov, R.: Siamese neural networks for one-shot image recognition. In: Proceedings of ICML Deep Learning Workshop, vol. 2 (2015)

  31. Konstan, J.A., Riedl, J.: Recommender systems: from algorithms to user experience. User Model. User Adapt. Interact. 22(1–2), 101–123 (2012)

    Article  Google Scholar 

  32. Koren, Y.: Collaborative filtering with temporal dynamics. Commun. ACM 53(4), 89–97 (2010)

    Article  Google Scholar 

  33. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Proceedings of Advances in Neural Information Processing Systems 25(NIPS), pp. 1097–1105 (2012)

  34. La Cascia, M., Sethi, S., Sclaroff, S.: Combining textual and visual cues for content-based image retrieval on the world wide web. In: Proceedings of the IEEE Workshop on Content-Based Access of Image and Video Libraries, pp. 24–28 (1998)

  35. Lacic, E., Kowald, D., Eberhard, L., Trattner, C., Parra, D., Marinho, L.B.: Utilizing online social network and location-based data to recommend products and categories in online marketplaces. In: Atzmueller M., Chin A., Scholz C., Trattner C. (eds) Mining, Modeling, and Recommending ‘Things’ in Social Media. MUSE 2013, MSM 2013. Lecture Notes in Computer Science, vol 8940. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-14723-9_6

  36. Larrain, S., Trattner, C., Parra, D., Graells-Garrido, E., Nørvåg, K.: Good times bad times: a study on recency effects in collaborative filtering for social tagging. In: Proceedings of the 9th ACM Conference on Recommender Systems, RecSys’15, pp. 269–272 (2015)

  37. Lei, C., Liu, D., Li, W., Zha, Z.J., Li, H.: Comparative deep learning of hybrid representations for image recommendations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2545–2553 (2016)

  38. Liao, W.H., Young, T.J.: Texture classification using uniform extended local ternary patterns. In: Proceedings of IEEE International Symposium on Multimedia (ISM), pp. 191–195 (2010)

  39. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  40. Maaten, L.V.D., Hinton, G.: Visualizing data using t-sne. J. Mach. Learn. Res. 9(Nov), 2579–2605 (2008)

    MATH  Google Scholar 

  41. Macedo, A.Q., Marinho, L.B., Santos, R.L.: Context-aware event recommendation in event-based social networks. In: Proceedings of the 9th ACM Conference on Recommender Systems, pp. 123–130 (2015)

  42. Maes, P., et al.: Agents that reduce work and information overload. Commun. ACM 37(7), 30–40 (1994)

    Article  Google Scholar 

  43. Manning, C.D., Raghavan, P., Schütze, H., et al.: Introduction to Information Retrieval, vol. 1. Cambridge University Press Cambridge, Cambridge (2008)

    Book  MATH  Google Scholar 

  44. Mathieu, M.F., Zhao, J.J., Zhao, J., Ramesh, A., Sprechmann, P., LeCun, Y.: Disentangling factors of variation in deep representation using adversarial training. In: Proceedings of Advances in Neural Information Processing Systems, pp. 5040–5048 (2016)

  45. McAuley, J., Targett, C., Shi, Q., Van Den Hengel, A.: Image-based recommendations on styles and substitutes. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 43–52 (2015)

  46. Mensink, T., Van Gemert, J.: The rijksmuseum challenge: Museum-centered visual recognition. In: Proceedings of International Conference on Multimedia Retrieval, p. 451 (2014)

  47. Nguyen, A., Yosinski, J., Clune, J.: Multifaceted feature visualization: uncovering the different types of features learned by each neuron in deep neural networks. arXiv preprint arXiv:1602.03616 (2016)

  48. Nunes, I., Jannach, D.: A systematic review and taxonomy of explanations in decision support and recommender systems. User Model. User Adapt. Interact. 27(3), 393–444 (2017)

    Article  Google Scholar 

  49. Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recognit. 29(1), 51–59 (1996)

    Article  Google Scholar 

  50. Olah, C., Mordvintsev, A., Schubert, L.: Feature visualization. Distill (2017) . https://doi.org/10.23915/distill.00007

  51. Parra, D., Brusilovsky, P.: User-controllable personalization: a case study with setfusion. Int. J. Hum. Comput. Stud. 78, 43–67 (2015)

    Article  Google Scholar 

  52. Parra, D., Sahebi, S.: Recommender systems: sources of knowledge and evaluation metrics. In: Advanced Techniques in Web Intelligence-2, pp. 149–175. Springer, Berlin (2013). https://doi.org/10.1007/978-3-642-33326-2_7

  53. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)

  54. Rendle, S., Freudenthaler, C., Gantner, Z., Schmidt-Thieme, L.: BPR: Bayesian personalized ranking from implicit feedback. In: Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pp. 452–461 (2009)

  55. Rui, Y., Huang, T.S., Ortega, M., Mehrotra, S.: Relevance feedback: a power tool for interactive content-based image retrieval. IEEE Trans. Circuits Syst. Video Technol. 8(5), 644–655 (1998)

    Article  Google Scholar 

  56. Rush, J.C.: Acquiring a concept of painting style. Stud. Art Educ. 20(3), 43–51 (1979)

    Article  Google Scholar 

  57. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)

    MathSciNet  Article  Google Scholar 

  58. San Pedro, J., Siersdorfer, S.: Ranking and classifying attractiveness of photos in folksonomies. In: Proceedings of the 18th International Conference on World Wide Web, WWW’09, pp. 771–780 (2009)

  59. Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823 (2015)

  60. Semeraro, G., Lops, P., De Gemmis, M., Musto, C., Narducci, F.: A folksonomy-based recommender system for personalized access to digital artworks. J. Comput. Cult. Herit. (JOCCH) 5(3), 11 (2012)

    Google Scholar 

  61. Shankar, D., Narumanchi, S., Ananya, H., Kompalli, P., Chaudhury, K.: Deep learning based large scale visual recommendation and search for e-commerce. arXiv preprint arXiv:1703.02344 (2017)

  62. Sharif Razavian, A., Azizpour, H., Sullivan, J., Carlsson, S.: CNN features off-the-shelf: an astounding baseline for recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 806–813 (2014)

  63. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  64. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: Proceedings of AAAI, vol. 4, p. 12 (2017)

  65. Tintarev, N., Masthoff, J.: Explaining recommendations: design and evaluation. In: Ricci, F., Rokach, L., Shapira, B. (eds.) Recommender Systems Handbook. Springer, Boston (2015). https://doi.org/10.1007/978-1-4899-7637-6_10

    Google Scholar 

  66. Trattner, C., Elsweiler, D.: Investigating the healthiness of internet-sourced recipes: implications for meal planning and recommender systems. In: Proceedings of the 26th International Conference on World Wide Web, pp. 489–498 (2017)

  67. Verbert, K., Parra, D., Brusilovsky, P., Duval, E.: Visualizing recommendations to support exploration, transparency and controllability. In: Proceedings of the 2013 International Conference on Intelligent User Interfaces, pp. 351–362 (2013)

  68. Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J., Philbin, J., Chen, B., Wu, Y.: Learning fine-grained image similarity with deep ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1386–1393 (2014)

  69. Weinswig, D.: Art Market Cooling, But Online Sales Booming. https://www.forbes.com/sites/deborahweinswig/2016/05/13/art-market-cooling-but-online-sales-booming/ (2016). Accessed 21 March 2017

  70. Yang, L., Cui, Y., Zhang, F., Pollak, J.P., Belongie, S., Estrin, D.: PlateClick: bootstrapping food preferences through an adaptive visual interface. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM’15, pp. 183–192 (2015)

Download references

Acknowledgements

This research has been supported by the Chilean research agency Conicyt, under Fondecyt Grant 11150783, and partially funded by the Millennium Institute for Foundational Research on Data (IMFD). We also acknowledge the help from Felipe del Río and Domingo Mery, who helped us frame some evaluations and provided us with some interesting ideas for future work.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Pablo Messina.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Messina, P., Dominguez, V., Parra, D. et al. Content-based artwork recommendation: integrating painting metadata with neural and manually-engineered visual features. User Model User-Adap Inter 29, 251–290 (2019). https://doi.org/10.1007/s11257-018-9206-9

Download citation

Keywords

  • Artwork
  • Recommender systems
  • Content-based recommender
  • Hybrid recommendations
  • Metadata
  • Visual features
  • Deep neural networks