Machine Vision and Applications

, Volume 25, Issue 6, pp 1385–1397 | Cite as

Painting-91: a large scale database for computational painting categorization

  • Fahad Shahbaz Khan
  • Shida Beigpour
  • Joost van de Weijer
  • Michael Felsberg
Original Paper

Abstract

Computer analysis of visual art, especially paintings, is an interesting cross-disciplinary research domain. Most of the research in the analysis of paintings involve medium to small range datasets with own specific settings. Interestingly, significant progress has been made in the field of object and scene recognition lately. A key factor in this success is the introduction and availability of benchmark datasets for evaluation. Surprisingly, such a benchmark setup is still missing in the area of computational painting categorization. In this work, we propose a novel large scale dataset of digital paintings. The dataset consists of paintings from 91 different painters. We further show three applications of our dataset namely: artist categorization, style classification and saliency detection. We investigate how local and global features popular in image classification perform for the tasks of artist and style categorization. For both categorization tasks, our experimental results suggest that combining multiple features significantly improves the final performance. We show that state-of-the-art computer vision methods can correctly classify 50 % of unseen paintings to its painter in a large dataset and correctly attribute its artistic style in over 60 % of the cases. Additionally, we explore the task of saliency detection on paintings and show experimental findings using state-of-the-art saliency estimation algorithms.

Keywords

Painting categorization Visual features Image classification 

References

  1. 1.
    Arora, R., Elgammal, A.: Towards automated classification of fine-art painting style: a comparative study. In: Proceedings of the ICPR (2012)Google Scholar
  2. 2.
    Bosch, A., Zisserman, A., Munoz, X.: Representing shape with a spatial pyramid kernel. In: Proceedings of the CIVR (2007)Google Scholar
  3. 3.
    Carneiro, G., da Silva, N.P., Bue, A.D., Costeira, J.P.: Artistic image classification: an analysis on the printart database. In: Proceedings of the ECCV (2012)Google Scholar
  4. 4.
    Cinzia Di Dio, G.R., Macaluso, E.: The golden beauty: brain response to classical and renaissance sculptures. Plos One 2(11), e1201 (2007)Google Scholar
  5. 5.
    Condorovici, R.G., Vranceanu, R., Vertan, C.: Saliency map retrieval for artistic paintings inspired from human understanding. In: Proceedings of the SPAMEC (2011)Google Scholar
  6. 6.
    Csurka, G., Bray, C., Dance, C., Fan, L.: Visual categorization with bags of keypoints. In: Proceedings of the Workshop on Statistical Learning in Computer Vision, ECCV (2004)Google Scholar
  7. 7.
    Danelljan, M., Khan, F.S., Felsberg, M., van de Weijer, J.: Adaptive color attributes for real-time visual tracking. In: Proceedings of the CVPR (2014)Google Scholar
  8. 8.
    Deng, J., Dong, W., Socher, R., Jia, L., Kai, L., Li, F.F.: Imagenet: a large-scale hierarchical image database. In: Proceedings of the CVPR (2009)Google Scholar
  9. 9.
    Elfiky, N., Khan, F.S., van de Weijer, J., Gonzalez, J.: Discriminative compact pyramids for object and scene recognition. Pattern Recognit. 45(4), 1627–1636 (2012)CrossRefMATHGoogle Scholar
  10. 10.
    Everingham, M., Gool, L.J.V., Williams, C.K.I., Winn, J.M., Zisserman, A.: The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)CrossRefGoogle Scholar
  11. 11.
    Gehler, P.V., Nowozin, S.: On feature combination for multiclass object classification. In: Proceedings of the ICCV (2009)Google Scholar
  12. 12.
    Goguen, J.: Art and the Brain. Imprint Academic, UK (1999)Google Scholar
  13. 13.
    Guo, Z., Zhang, L., Zhang, D.: A completed modeling of local binary pattern operator for texture classification. Trans. Image Process. 19(6), 1657–1663 (2010)CrossRefMathSciNetGoogle Scholar
  14. 14.
    Harel, J., Koch, C., Perona, P.: Graph-based visual saliency. In: Proceedings of the NIPS (2006)Google Scholar
  15. 15.
    Hou, X., Zhang, L.: Saliency detection: a spectral residual approach. In: Proceedings of the CVPR (2007)Google Scholar
  16. 16.
    Hou, X., Harel, J., Koch, C.: Image signature: highlighting sparse salient regions. Pattern Anal. Mach. Intell. 34(1), 194–201 (2012)CrossRefGoogle Scholar
  17. 17.
    Itti, L., Cristof, K., Ernst, N.: A model of saliency-based visual attention for rapid scene analysis. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)CrossRefGoogle Scholar
  18. 18.
    Jiang, S., Huang, Q., Ye, Q., Gao, W.: An effective method to detect and categorize digitized traditional chinese paintings. Pattern Recognit. Lett. 27(7), 734–746 (2006)CrossRefGoogle Scholar
  19. 19.
    Johnson, R., Hendriks, E., Berezhnoy, I.J., Brevdo, E., Hughes, S.M., Daubechies, I., Li, J., Postma, E., Wang, J.Z.: Image processing for artist identification. IEEE Signal Process. Mag. 25(4), 37–48 (2008)CrossRefGoogle Scholar
  20. 20.
    Judd, T., Ehinger, K.A., Durand, F., Torralba, A.: Learning to predict where humans look. In: Proceedings of the ICCV (2009)Google Scholar
  21. 21.
    Khan, F.S., Anwer, R.M., van de Weijer, J., Bagdanov, A.D., Vanrell, M., Lopez, A.M.: Color attributes for object detection. In: Proceedings of the CVPR (2012a)Google Scholar
  22. 22.
    Khan, F.S., van de Weijer, J., Ali, S., Felsberg, M.: Evaluating the impact of color on texture recognition. In: Proceedings of the CAIP (2013b)Google Scholar
  23. 23.
    Khan, F.S., van de Weijer, J., Bagdanov, A.D., Vanrell, M.: Portmanteau vocabularies for multi-cue image representations. In: Proceedings of the NIPS (2011)Google Scholar
  24. 24.
    Khan, F.S., van de Weijer, J., Vanrell, M.: Top-down color attention for object recognition. In: Proceedings of the ICCV (2009)Google Scholar
  25. 25.
    Khan, F.S., van de Weijer, J., Vanrell, M.: Modulating shape features by color attention for object recognition. Int. J. Comput. Vis. 98(1), 49–64 (2012b)CrossRefGoogle Scholar
  26. 26.
    Khan, F.S., Anwer, R.M., van de Weijer, J., Bagdanov, A., Lopez, A., Felsberg, M.: Coloring action recognition in still images. Int. J. Comput. Vis. 105(3), 205–221 (2013a)CrossRefGoogle Scholar
  27. 27.
    Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: Proceedings of the CVPR (2006)Google Scholar
  28. 28.
    Lazebnik, S., Schmid, C., Ponce, J.: A sparse texture representation using local affine regions. Pattern Anal. Mach. Intell. 27(8), 1265–1278 (2005)CrossRefGoogle Scholar
  29. 29.
    Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., Shum, H.Y.: Learning to detect a salient object. Pattern Anal. Mach. Intell. 32(2), 353–367 (2011)Google Scholar
  30. 30.
    Lowe, D.G.: Distinctive image features from scale-invariant points. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRefGoogle Scholar
  31. 31.
    Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. Pattern Anal. Mach. Intell. 27(10), 1615–1630 (2005)CrossRefGoogle Scholar
  32. 32.
    Murray, N., Marchesotti, L., Perronnin, F.: Ava: a large-scale database for aesthetic visual analysis. In: Proceedings of the CVPR (2012)Google Scholar
  33. 33.
    Mutch, J., Lowe, D.G.: Object class recognition and localization using sparse features with limited receptive fields. Int. J. Comput. Vis. 80(1), 45–57 (2008)CrossRefGoogle Scholar
  34. 34.
    Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)CrossRefGoogle Scholar
  35. 35.
    Oliva, A., Torralba, A.B.: Modeling the shape of the scene: a holistic representation of the spatial envelope. Int. J. Comput. Vis. 42(3), 145–175 (2001)CrossRefMATHGoogle Scholar
  36. 36.
    Quiroga, R.Q., Pedreira, C.: How do we see art: an eye-tracker study. Front. Hum. Neurosci. 5, 98 (2011)Google Scholar
  37. 37.
    Ramachandran, V., Hirstein, W.: The science of art: a neurological theory of aesthetic experience. J. Conscious. Stud. 6(7), 15–51 (1999)Google Scholar
  38. 38.
    Rojas, D., Khan, F.S., van de Weijer, J., Gevers, T.: The impact of color on bag-of-words based object recognition. In: Proceedings of the ICPR (2010)Google Scholar
  39. 39.
    Sablatnig, R., Kammerer, P., Zolda, E.: Hierarchical classification of paintings using face- and brush-stroke models. In: Proceedings of the ICPR (1998)Google Scholar
  40. 40.
    Shamir, L., Macura, T., Orlov, N., Eckley, D.M., Goldberg, I.G.: Impressionism, expressionism, surrealism: automated recognition of painters and schools of art. ACM Trans. Appl. Percept. 7(2), 8–16 (2010)CrossRefGoogle Scholar
  41. 41.
    Shamir, L.: Computer analysis reveals similarities between the artistic styles of Van Gogh and Pollock. Leonardo 45(2), 149–154 (2012)CrossRefGoogle Scholar
  42. 42.
    Shamir, L., Tarakhovsky, J.A.: Computer analysis of art. J. Comput. Cult. Herit. 5(2), 1–7 (2012)CrossRefGoogle Scholar
  43. 43.
    Shechtman, E., Irani, M.: Matching local self-similarities across images and videos. In: Proceedings of the CVPR (2007)Google Scholar
  44. 44.
    Shen, J.: Stochastic modeling western paintings for effective classification. Pattern Recognit. 42(2), 293–301 (2009)CrossRefMATHGoogle Scholar
  45. 45.
    Siddiquie, B., Vitaladevuni, S.N.P., Davis, L.S.: Combining multiple kernels for efficient image classification. In: Proceedings of the WACV (2009)Google Scholar
  46. 46.
    Vikram, T., Tscherepanow, M., Wrede, B.: A saliency map based on sampling an image into random rectangular regions of interest. Pattern Recognit. 45(9), 3114–3124 (2012)Google Scholar
  47. 47.
    van de Sande, K.E.A., Gevers, T., Snoek, C.G.M.: Evaluating color descriptors for object and scene recognition. Pattern Anal. Mach. Intell. 32(9), 1582–1596 (2010)CrossRefGoogle Scholar
  48. 48.
    van de Weijer, J., Schmid, C., Verbeek, J.J., Larlus, D.: Learning color names for real-world applications. Trans. Image Process. 18(7), 1512–1524 (2009)CrossRefMathSciNetGoogle Scholar
  49. 49.
    Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: large-scale scene recognition from abbey to zoo. In: Proceedings of the CVPR (2010)Google Scholar
  50. 50.
    Zhang, J., Marszalek, M., Lazebnik, S., Schmid, C.: Local features and kernels for classification of texture and object catergories: a comprehensive study. Int. J. Comput. Vis. 73(2), 213–218 (2007)CrossRefGoogle Scholar
  51. 51.
    Zujovic, J., Gandy, L., Friedman, S., Pardo, B., Pappas, T.: Classifying paintings by artistic genre: an analysis of features and classifiers. In: Proceedings of the MMSP (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Fahad Shahbaz Khan
    • 1
  • Shida Beigpour
    • 2
  • Joost van de Weijer
    • 3
  • Michael Felsberg
    • 1
  1. 1.Computer Vision LaboratoryLinköping UniversityLinköpingSweden
  2. 2.Norwegian Colour and Visual Computing LaboratoryGjovik University CollegeGjøvikNorway
  3. 3.Computer Vision Centre BarcelonaUniversitat Autonoma de BarcelonaCataloniaSpain

Personalised recommendations