Large Image Collection Visualization Using Perception-Based Similarity with Color Features

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10072)


This paper introduces the basic steps to build a similarity-based visualization tool for large image collections. We build the similarity metrics based on human perception. Psychophysical experiments have shown that human observers can recognize the gist of scenes within 100 milliseconds (ms) by comprehending the global properties of an image. Color also plays an important role in human rapid scene recognition. However, previous works often neglect color features. We propose new scene descriptors that preserve the information from coherent color regions, as well as the spatial layouts of scenes. Experiments show that our descriptors outperform existing state-of-the-art approaches. Given the similarity metrics, a hierarchical structure of an image collection can be built in a top-down manner. Representative images are chosen for image clusters and visualized using a force-directed graph.


Visual Word Spatial Layout Similarity Metrics Image Collection Scene Classification 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Michel, F.: How many public photos are uploaded to flickr every day, month, year? (2015). Accessed 5 August 2016
  2. 2. (2016). Accessed 5 August 2016
  3. 3.
    Nguyen, G.P., Worring, M.: Interactive access to large image collections using similarity-based visualization. J. Vis. Lang. Comput. 19, 203–224 (2008)CrossRefGoogle Scholar
  4. 4.
    Yang, J., Fan, J., Hubball, D., Gao, Y., Luo, H., Ribarsky, W., Ward, M.: Semantic image browser: bridging information visualization with automated intelligent image analysis. In: 2006 IEEE Symposium on Visual Analytics Science and Technology, pp. 191–198. IEEE (2006)Google Scholar
  5. 5.
    Jing, Y., Rowley, H., Wang, J., Tsai, D., Rosenberg, C., Covell, M.: Google image swirl: a large-scale content-based image visualization system. In: Proceedings of the 21st International Conference Companion on World Wide Web, pp. 539–540. ACM (2012)Google Scholar
  6. 6.
    Barnard, K., Forsyth, D.: Learning the semantics of words and pictures. In: Proceedings of the Eighth IEEE International Conference on Computer Vision, ICCV 2001, vol. 2, pp. 408–415. IEEE (2001)Google Scholar
  7. 7.
    Carson, C., Belongie, S., Greenspan, H., Malik, J.: Blobworld: image segmentation using expectation-maximization and its application to image querying. IEEE Trans. Pattern Anal. Mach. Intell. 24, 1026–1038 (2002)CrossRefGoogle Scholar
  8. 8.
    Schyns, P.G., Oliva, A.: From blobs to boundary edges: evidence for time-and spatial-scale-dependent scene recognition. Psychol. Sci. 5, 195–200 (1994)CrossRefGoogle Scholar
  9. 9.
    Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. Int. J. Comput. Vis. 42, 145–175 (2001)CrossRefMATHGoogle Scholar
  10. 10.
    Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 2169–2178. IEEE (2006)Google Scholar
  11. 11.
    Oliva, A.: Gist of the scene. Neurobiol. Atten. 696, 251–258 (2005)CrossRefGoogle Scholar
  12. 12.
    Greene, M.R., Oliva, A.: The briefest of glances the time course of natural scene understanding. Psychol. Sci. 20, 464–472 (2009)CrossRefGoogle Scholar
  13. 13.
    Greene, M.R., Oliva, A.: Recognition of natural scenes from global properties: seeing the forest without representing the trees. Cogn. Psychol. 58, 137–176 (2009)CrossRefGoogle Scholar
  14. 14.
    Castelhano, M.S., Henderson, J.M.: The influence of color on the perception of scene gist. J. Exp. Psychol. Hum. Percept. Perform. 34, 660 (2008)CrossRefGoogle Scholar
  15. 15.
    Torralba, A., Oliva, A.: Statistics of natural image categories. Netw. Comput. Neural Syst. 14, 391–412 (2003)CrossRefGoogle Scholar
  16. 16.
    Oliva, A., Torralba, A.: Building the gist of a scene: the role of global image features in recognition. Prog. Brain Re. 155, 23–36 (2006)CrossRefGoogle Scholar
  17. 17.
    Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: large-scale scene recognition from abbey to zoo. In: 2010 IEEE conference on Computer vision and pattern recognition (CVPR), pp. 3485–3492. IEEE (2010)Google Scholar
  18. 18.
    Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., Oliva, A.: Learning deep features for scene recognition using places database. In: Advances in Neural Information Processing Systems, pp. 487–495 (2014)Google Scholar
  19. 19.
    Chang, P., Krumm, J.: Object recognition with color cooccurrence histograms. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2. IEEE (1999)Google Scholar
  20. 20.
    Pass, G., Zabih, R.: Histogram refinement for content-based image retrieval. In: Proceedings 3rd IEEE Workshop on Applications of Computer Vision, WACV 1996, pp. 96–102. IEEE (1996)Google Scholar
  21. 21.
    Huang, J., Kumar, S.R., Mitra, M., Zhu, W.J., Zabih, R.: Image indexing using color correlograms. In: Proceedings of the 1997 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 762–768. IEEE (1997)Google Scholar
  22. 22.
    Van De Sande, K.E., Gevers, T., Snoek, C.G.: Evaluating color descriptors for object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 32, 1582–1596 (2010)CrossRefGoogle Scholar
  23. 23.
    Bosch, A., Zisserman, A., Muoz, X.: Scene classification using a hybrid generative/discriminative approach. IEEE Trans. Pattern Anal. Mach. Intell. 30, 712–727 (2008)CrossRefGoogle Scholar
  24. 24.
    Oliva, A., Schyns, P.G.: Diagnostic colors mediate scene recognition. Cogn. Psychol. 41, 176–210 (2000)CrossRefGoogle Scholar
  25. 25.
    Yang, J., Jiang, Y.G., Hauptmann, A.G., Ngo, C.W.: Evaluating bag-of-visual-words representations in scene classification. In: Proceedings of the International Workshop on Workshop on Multimedia Information Retrieval, pp. 197–206. ACM (2007)Google Scholar
  26. 26.
    Hou, J., Gao, H., Xia, Q., Qi, N.: Feature combination and the knn framework in object classification. IEEE Trans. Neural Netw. Learn. Syst. 27, 1368–1378 (2016)CrossRefGoogle Scholar
  27. 27.
    Swain, M.J., Ballard, D.H.: Color indexing. Int. J. Comput. Vis. 7, 11–32 (1991)CrossRefGoogle Scholar
  28. 28.
    Tibshirani, R., Walther, G., Hastie, T.: Estimating the number of clusters in a data set via the gap statistic. J. Roy. Stat. Soc. Ser. B (Stat. Methodol.) 63, 411–423 (2001)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Department of Computer ScienceNorth Carolina State UniversityRaleighUSA

Personalised recommendations