Visualisation and Browsing of Image Databases

  • William Plant
  • Gerald Schaefer
Part of the Studies in Computational Intelligence book series (SCI, volume 346)

Abstract

In this chapter we provide a comprehensive overview of the emerging field of visualising and browsing image databases. We start with a brief introduction to content-based image retrieval and the traditional query-by-example search paradigm that many retrieval systems employ. We specify the problems associated with this type of interface, such as users not being able to formulate a query due to not having a target image or concept in mind. The idea of browsing systems is then introduced as a means to combat these issues, harnessing the cognitive power of the human mind in order to speed up image retrieval.We detail common methods in which the often high-dimensional feature data extracted from images can be used to visualise image databases in an intuitive way. Systems using dimensionality reduction techniques, such as multi-dimensional scaling, are reviewed along with those that cluster images using either divisive or agglomerative techniques as well as graph-based visualisations. While visualisation of an image collection is useful for providing an overview of the contained images, it forms only part of an image database navigation system. We therefore also present various methods provided by these systems to allow for interactive browsing of these datasets. A further area we explore are user studies of systems and visualisations where we look at the different evaluations undertaken in order to test usability and compare systems, and highlight the key findings from these studies. We conclude the chapter with several recommendations for future work in this area.

Keywords

Image Retrieval User Study Image Database Representative Image Relevance Feedback 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Abdel-Mottaleb, M., Krischnamachari, S., Mankovich, N.J.: Performance Evaluation of Clustering Algorithms for Scalable Image Retrieval. In: IEEE Computer Society Workshop on Empirical Evaluation of Computer Vision Algorithms (1998)Google Scholar
  2. 2.
    Assfalg, J., Del-Bimbo, A., Pala, P.: Virtual Reality for Image Retrieval. Journal of Visual Languages and Computing 11(2), 105–124 (2000)CrossRefGoogle Scholar
  3. 3.
    Atkinson, K.E.: An Introduction to Numerical Analysis. John Wiley and Sons, Chichester (1989)MATHGoogle Scholar
  4. 4.
    Bederson, B.: Quantum Treemaps and Bubblemaps for a Zoomable Image Browser. In: ACM Symposium on User Interface Software and Technology, pp. 71–80 (2001)Google Scholar
  5. 5.
    Borth, D., Schulze, C., Ulges, A., Breuel, T.: Navidgator - Similarity Based Browsing for Image and Video Databases. In: German Conference on Advances in Artificial Intelligence, pp. 22–29 (2008)Google Scholar
  6. 6.
    Chen, C.: Information Visualization. Springer, Heidelberg (2004)Google Scholar
  7. 7.
    Chen, C., Gagaudakis, G., Rosin, P.: Similarity-Based Image Browsing. In: International Conference on Intelligent Information Processing, pp. 206–213 (2000)Google Scholar
  8. 8.
    Chen, Y., Butz, A.: Photosim: Tightly Integrating Image Analysis into a Photo Browsing UI. In: International Symposium on Smart Graphics (2008)Google Scholar
  9. 9.
    Combs, T., Bederson, B.: Does Zooming Improve Image Browsing? In: ACM Conference on Digital Libraries, pp. 130–137 (1999)Google Scholar
  10. 10.
    Cruz-Neira, C., Sandin, D., DeFanti, T.: Surround-screen Projection-based Virtual Reality: The Design and Implementation of the CAVE. In: 20th Annual Conference on Computer Graphics and Interactive Techniques, pp. 135–142 (1993)Google Scholar
  11. 11.
    Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image Retrieval: Ideas, Influences, and Trends of the New Age. ACM Computing Surveys 40(2), 1–60 (2008)CrossRefGoogle Scholar
  12. 12.
    Deng, D., Zhang, J., Purvis, M.: Visualisation and Comparison of Image Collections based on Self-organised Maps. In: Workshop on Australasian Information Security, Data Mining and Web Intelligence, and Software Internationalisation, pp. 97–102 (2004)Google Scholar
  13. 13.
    Dontcheva, M., Agrawala, M., Cohen, M.: Metadata Visualization for Image Browsing. In: ACM Symposium on User Interface Software and Technology (2005)Google Scholar
  14. 14.
    Eidenberger, H.: A Video Browsing Application Based on Visual MPEG-7 Descriptors and Self-organising Maps. International Journal of Fuzzy Systems 6(3) (2004)Google Scholar
  15. 15.
    Faloutsos, C., Equitz, W., Flickner, M., Niblack, W., Petkovic, D., Barber, R.: Efficient and Effective Querying by Image Content. Journal of Intelligent Information Systems 3, 231–262 (1994)CrossRefGoogle Scholar
  16. 16.
    Faloutsos, C., Lin, K.: FastMap: A Fast Algorithm for Indexing, Datamining and Visualization of Traditional and Multimedia Datasets. In: ACM SIGMOD International Conference on Management of Data, pp. 163–174 (1995)Google Scholar
  17. 17.
    Flickr (2009), http://www.flickr.com/
  18. 18.
    Gomi, A., Miyazaki, R., Itoh, T., Li, J.: CAT: A Hierarchical Image Browser Using a Rectangle Packing Technique. In: International Conference on Information Visualization, pp. 82–87 (2008)Google Scholar
  19. 19.
    Graham, A., Garcia-Molina, H., Paepcke, A., Winograd, T.: Time as Essence for Photo Browsing through Personal Digital Libraries. In: ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 326–335 (2002)Google Scholar
  20. 20.
    Gupta, A., Jain, R.: Visual Information Retrieval. Communications of the ACM 40(5), 70–79 (1997)CrossRefGoogle Scholar
  21. 21.
    Hare, J.S., Lewis, P.H.: Content-based Image Retrieval Using a Mobile Device as a Novel Interface. In: SPIE Storage and Retrieval Methods and Applications for Multimedia, pp. 64–75 (2005)Google Scholar
  22. 22.
    Heesch, D., Rüger, S.M.: NNk Networks for Content-Based Image Retrieval. In: McDonald, S., Tait, J.I. (eds.) ECIR 2004. LNCS, vol. 2997, pp. 253–266. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  23. 23.
    Heesch, D., Rüger, S.: Three Interfaces for Content-Based Access to Image Collections. In: International Conference on Image and Video Retrieval, pp. 491–499 (2004)Google Scholar
  24. 24.
    Hilliges, O., Baur, D., Butz, A.: Photohelix: Browsing, Sorting and Sharing Digital Photo Collections. In: IEEE Tabletop Workshop on Horizontal Interactive Human-Computer Systems, pp. 87–94 (2007)Google Scholar
  25. 25.
    Hilliges, O., Kunath, P., Pryakhin, A., Butz, A., Kriegel, H.P.: Browsing and Sorting Digital Pictures using Automatic Image Classification and Quality Analysis. In: International Conference on Human-Computer Interaction, pp. 882–891 (2007)Google Scholar
  26. 26.
    Hinton, G., Roweis, S.: Stochastic Neighbor Embedding. In: Advances in Neural Information Processing Systems, vol. 15, pp. 833–840 (2002)Google Scholar
  27. 27.
  28. 28.
    Jacobs, C.E., Finkelstein, A., Salesin, D.H.: Fast Multiresolution Image Querying. In: Conference on Computer Graphics and Interactive Techniques, pp. 277–286 (1995)Google Scholar
  29. 29.
    Jain, A.K., Dubes, R.C.: Algorithms for Clustering Data. Prentice-Hall, Englewood Cliffs (1988)MATHGoogle Scholar
  30. 30.
    Jambu, M.: Exploratory and Multivariate Data Analysis. Academic Press, London (1991)MATHGoogle Scholar
  31. 31.
    Keller, I., Meiers, T., Ellerbrock, T., Sikora, T.: Image Browsing with PCA-Assisted User-Interaction. In: IEEE Workshop on Content-Based Access of Image and Video Libraries, pp. 102–108 (2001)Google Scholar
  32. 32.
    Khella, A., Bederson, B.: Pocket PhotoMesa: A Zooming Image Browser for PDA’s. In: International Conference on Mobile and Ubiquitous Multimedia, pp. 19–24 (2004)Google Scholar
  33. 33.
    Kohonen, T.: Self-organizing Maps. Springer, Heidelberg (1997)MATHGoogle Scholar
  34. 34.
    Koikkalainen, P., Oja, E.: Self-organizing Hierarchical Feature Maps. In: International Joint Conference on Neural Networks, vol. 2, pp. 279–285 (1990)Google Scholar
  35. 35.
    Krischnamachari, S., Abdel-Mottaleb, M.: Image Browsing using Hierarchical Clustering. In: IEEE Symposium Computers and Communications, pp. 301–307 (1999)Google Scholar
  36. 36.
    Kruskal, J.B., Wish, M.: Multidimensional Scaling. Sage, Thousand Oaks (1978)Google Scholar
  37. 37.
    Laaksonen, J., Koskela, M., Oja, E.: PicSOM – Self-organizing Image Retrieval with MPEG-7 Content Descriptors. IEEE Transactions on Neural Networks: Special Issue on Multimedia Processing 13(4), 841–853 (2002)Google Scholar
  38. 38.
    Lew, M.S., Sebe, N.: Visual Websearching Using Iconic Queries. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 788–789 (2000)Google Scholar
  39. 39.
    Li, J., Wang, J.Z.: Real-Time Computerized Annotation of Pictures. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(6), 985–1002 (2008)CrossRefGoogle Scholar
  40. 40.
    Lim, S., Chen, L., Lu, G., Smith, R.: Browsing Texture Image Databases. In: International Conference on Multimedia Modelling, pp. 328–333 (2005)Google Scholar
  41. 41.
    Linde, Y., Buzo, A., Gray, R.: An Algorithm for Vector Quantizer Design. IEEE Transactions on Communications 28, 84–94 (1980)CrossRefGoogle Scholar
  42. 42.
    Liu, H., Xie, X., Tang, X., Li, Z.W., Ma, W.Y.: Effective Browsing of Web Image Search Results. In: ACM International Workshop on Multimedia Information Retrieval, pp. 84–90 (2004)Google Scholar
  43. 43.
    Ma, W.Y., Manjunath, B.S.: NeTra: A Toolbox for Navigating Large Image Databases. Multimedia Systems 7(3), 184–198 (1999)CrossRefGoogle Scholar
  44. 44.
    Milanese, R., Squire, D., Pun, T.: Correspondence Analysis and Hierarchical Indexing for Content-Based Image Retrieval. In: IEEE International Conference on Image Processing, pp. 859–862 (1996)Google Scholar
  45. 45.
    Moghaddam, B., Tian, Q., Lesh, N., Shen, C., Huang, T.: Visualization and User-Modeling for Browsing Personal Photo Libraries. International Journal of Computer Vision 56(1/2), 109–130 (2004)CrossRefGoogle Scholar
  46. 46.
    Müller, H., Müller, W., Squire, D.M., Marchand-Maillet, S., Pun, T.: Performance Evaluation in Content-Based Image Retrieval: Overview and Proposals. Pattern Recognition Letters 22(5), 593–601 (2001)CrossRefMATHGoogle Scholar
  47. 47.
    Nakazato, M., Huang, T.: 3D MARS: Immersive Virtual Reality for Content-Based Image Retrieval. In: IEEE International Conference on Multimedia and Expo., pp. 44–47 (2001)Google Scholar
  48. 48.
    Nakazato, M., Manola, L., Huang, T.: ImageGrouper: A Group-Oriented User Interface for Content-Based Image Retrieval and Digital Image Arrangement. Journal of Visual Language and Computing 14(4), 363–386 (2003)CrossRefGoogle Scholar
  49. 49.
    Nguyen, G.P., Worring, M.: Interactive Access to Large Image Collections using Similarity Based Visualization. Journal of Visual Languages and Computing 19, 203–224 (2008)CrossRefGoogle Scholar
  50. 50.
    Osman, T., Thakker, D., Schaefer, G., Lakin, P.: An Integrative Semantic Framework for Image Annotation and Retrieval. In: IEEE/WIC/ACM International Conference on Web Intelligence, pp. 366–373 (2007)Google Scholar
  51. 51.
    Pecenovic, Z., Do, M.N., Vetterli, M., Pu, P.: Integrated Browsing and Searching of Large Image Collections. In: Laurini, R. (ed.) VISUAL 2000. LNCS, vol. 1929, pp. 279–289. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  52. 52.
    Pentland, A., Picard, W.R., Sclaroff, S.: Photobook: Content-Based Manipulation of Image Databases. International Journal of Computer Vision 18(3), 233–254 (1996)CrossRefGoogle Scholar
  53. 53.
    Google Picasa (2009), http://picasa.google.com/
  54. 54.
    Plaisant, C., Carr, D., Shneiderman, B.: Image Browsers: Taxonomy, Guidelines, and Informal Specifications. IEEE Software 12, 21–32 (1995)CrossRefGoogle Scholar
  55. 55.
    Platt, J., Czerwinski, M., Field, B.: PhotoTOC: Automatic Clustering for Browsing Personal Photographs. Technical report, Microsoft Research (2002)Google Scholar
  56. 56.
    Platt, J.C.: AutoAlbum: Clustering Digital Photographs using Probalistic Model Merging. In: IEEE Workshop on Content-Based Access of Image and Video Libraries, pp. 96–100 (2000)Google Scholar
  57. 57.
    Porta, M.: New Visualization Modes for Effective Image Presentation. International Journal of Image and Graphics 9(1), 27–49 (2009)CrossRefGoogle Scholar
  58. 58.
    Rodden, K.: Evaluating Similarity-Based Visualisations as Interfaces for Image Browsing. PhD thesis, University of Cambridge Computer Laboratory (2001)Google Scholar
  59. 59.
    Rodden, K., Basalaj, W., Sinclair, D., Wood, K.: Evaluating a Visualisation of Image Similarity as a Tool for Image Browsing. In: IEEE Symposium on Information Visualisation, pp. 36–43 (1999)Google Scholar
  60. 60.
    Rodden, K., Basalaj, W., Sinclair, D., Wood, K.: A Comparison of Measures for Visualising Image Similarity. In: The Challenge of Image Retrieval (2000)Google Scholar
  61. 61.
    Rodden, K., Basalaj, W., Sinclair, D., Wood, K.: Does Organisation by Similarity Assist Image Browsing? In: SIGCHI Conference on Human Factors in Computing Systems, pp. 190–197 (2001)Google Scholar
  62. 62.
    Rodden, K., Wood, K.: How Do People Manage Their Digital Photographs? In: SIGCHI Conference on Human Factors in Computing Systems, pp. 409–416 (2003)Google Scholar
  63. 63.
    Roweis, S., Saul, L.: Nonlinear Dimensionality Reduction by Locally Linear Embedding. Science 290(5500), 2323–2326 (2000)CrossRefGoogle Scholar
  64. 64.
    Rubner, Y., Guibas, L.J., Tomasi, C.: The Earth Movers Distance, Multi-dimensional Scaling, and Color-based Image Retrieval. In: APRA Image Understanding Workshop, pp. 661–668 (1997)Google Scholar
  65. 65.
    Rui, Y., Huang, T.S., Ortega, M., Mehrotra, M.: Relevance Feedback: A Power Tool for Interactive Content-based Image Retrieval. IEEE Transaction on Circuits and Systems for Video Technology 8(5), 644–655 (1998)CrossRefGoogle Scholar
  66. 66.
    Ruszala, S., Schaefer, G.: Visualisation Models for Image Databases: A Comparison of Six Approaches. In: Irish Machine Vision and Image Processing Conference, pp. 186–191 (2004)Google Scholar
  67. 67.
    Sammon, J.W.: A Nonlinear Mapping for Data Structure Analysis. IEEE Transactions on Computers 18(5), 401–409 (1969)CrossRefGoogle Scholar
  68. 68.
    Santini, S., Jain, R.: Integrated Browsing and Querying for Image Databases. IEEE Multimedia 7, 26–39 (2000)CrossRefGoogle Scholar
  69. 69.
    Sarkar, M., Brown, M.: Graphical Fisheye Views. Communications of the ACM 37(12), 73–83 (1994)CrossRefGoogle Scholar
  70. 70.
    Sarvas, R., Herrarte, E., Wilhelm, A., Davis, M.: Metadata Creation System for Mobile Images. In: International Conference on Mobile Systems, Applications, and Services, pp. 36–48 (2004)Google Scholar
  71. 71.
    Schaefer, G., Ruszala, S.: Image database navigation: A globe-al approach. In: Bebis, G., Boyle, R., Koracin, D., Parvin, B. (eds.) ISVC 2005. LNCS, vol. 3804, pp. 279–286. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  72. 72.
    Schaefer, G., Ruszala, S.: Image Database Navigation on a Hierarchical Hue Sphere. In: International Symposium on Visual Computing, pp. 814–823 (2006)Google Scholar
  73. 73.
    Schaefer, G., Ruszala, S.: Image Database Navigation on a Hierarchical MDS Grid. In: 28th Pattern Recognition Symposium, pp. 304–313 (2006)Google Scholar
  74. 74.
    Schaefer, G., Stich, M.: UCID – An Uncompressed Colour Image Database. In: Storage and Retrieval Methods and Applications for Multimedia, pp. 472–480 (2004)Google Scholar
  75. 75.
    Smeulders, A., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-Based Image Retrieval at the End of the Early Years. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12), 1349–1380 (2000)CrossRefGoogle Scholar
  76. 76.
    Swain, M., Ballard, D.: Color Indexing. International Journal of Computer Vision 7(1), 11–32 (1991)CrossRefGoogle Scholar
  77. 77.
    Tao, D., Tang, X., Li, X., Rui, Y.: Direct Kernal Biased Discriminant Analysis: A New Content-Based Image Retrieval Relevance Feedback Algorithm. IEEE Transactions on Multimedia 8(4), 716–727 (2006)CrossRefGoogle Scholar
  78. 78.
    Tao, D., Tang, X., Li, X., Wu, X.: Asymmetric Bagging and Random Subspace for Support Vector Machines-Based Relevance Feedback in Image Retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(7),1088–1099 (2006)CrossRefGoogle Scholar
  79. 79.
    Tenenbaum, J., Silva, V., Langford, J.: A Global Geometric Framework for Nonlinear Dimensionality Reduction. Science 290(5500), 2319–2322 (2000)CrossRefGoogle Scholar
  80. 80.
    Tian, G.Y., Taylor, D.: Colour Image Retrieval Using Virtual Reality. In: IEEE International Conference on Information Visualization, pp. 221–225 (2000)Google Scholar
  81. 81.
    Urban, J., Jose, J.M.: EGO: A Personalized Multimedia Management and Retrieval Tool. International Journal of Intelligent Systems 21(7), 725–745 (2006)CrossRefMATHGoogle Scholar
  82. 82.
    van Liere, R., de Leeuw, W.: Exploration of Large Image Collections Using Virtual Reality Devices. In: Workshop on New Paradigms in Information Visualization and Manipulation, held in conjunction with the 8th ACM International Conference on Information and Knowledge Management, pp. 83–86 (1999)Google Scholar
  83. 83.
    Worring, M., de Rooij, O., van Rijn, T.: Browsing Visual Collections Using Graphs. In: International Workshop on Multimedia Information Retrieval, pp. 307–312 (2007)Google Scholar
  84. 84.
    Yeh, T., Tollmar, K., Darrell, T.: Searching the Web with Mobile Images for Location Recognition. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 76–81 (2004)Google Scholar
  85. 85.
    Zhang, H., Zhong, D.: A Scheme for Visual Feature Based Image Indexing. In: SPIE/IS&T Conference on Storage and Retrieval for Image and Video Databases, pp. 36–46 (1995)Google Scholar
  86. 86.
    Zhou, X., Huang, T.: Relevance Feedback in Image Retrieval: A Comprehensive Review. Multimedia Systems 8(6), 536–544 (2003)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • William Plant
    • 1
  • Gerald Schaefer
    • 2
  1. 1.School of Engineering and Applied ScienceAston UniversityBirminghamU.K.
  2. 2.Department of Computer ScienceLoughborough UniversityLoughboroughU.K.

Personalised recommendations