Multimedia Systems

, Volume 12, Issue 4–5, pp 325–338 | Cite as

Building a user-centered semantic hierarchy in image databases

  • Manjeet Rege
  • Ming Dong
  • Farshad Fotouhi
Regular Paper


Recent development in the field of digital media technology has resulted in the generation of a huge number of images. Consequently, content-based image retrieval has emerged as an important area in multimedia computing. Research in human perception of image content suggests that the semantic cues play an important role in image retrieval. In this paper, we present a new paradigm to establish the semantics in image databases based on multi-user relevance feedback. Relevance feedback mechanism is one way to incorporate the users’ perception during image retrieval. By treating each feedback as a weak classifier and combining them together, we are able to capture the categories in the users’ mind and build a user-centered semantic hierarchy in the database to support semantic browsing and searching. We present an image retrieval system based on a city-landscape image database comprising of 3,009 images. We also compare our approach with other typical methods to organize an image database. Superior results have been achieved by the proposed framework.


Image Retrieval Image Database Relevance Feedback Image Retrieval System Semantic Graph 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Smeulders A.W.M., Worring M., Santini S., Gupta A., Jain R. (2000). Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12):1349–1380CrossRefGoogle Scholar
  2. 2.
    Mehrotra, S., Rui, R., Ortega, M., Huang, T.S.: Supporting content-based queries over images in mars. In: Proceedings of IEEE International Conference on Multimedia Computing Systems, pp. 632–633 (1997)Google Scholar
  3. 3.
    Ma, W.Y., Manjunath, B.: Netra: A toolbox for navigating large image databases. In: Proceedings of IEEE ICIP, pp. 568–571 (1997)Google Scholar
  4. 4.
    Gupta A., Jain R. (1997). Visual information retrieval. Commun. ACM 40(5):70–79CrossRefGoogle Scholar
  5. 5.
    Rogowitz B., Frese T., Smith J., Bouman C., Kalin E. (1998). Perceptual image smilarity experiments. Proc. SPIE 8:576–590CrossRefGoogle Scholar
  6. 6.
    Wang J., Li J., Wiederhold G. (2002). Simplicity: semantics-sensitive integrated matching for picture libraries. IEEE Trans. PAMI 23(9):947–963Google Scholar
  7. 7.
    Vailaya A., Figueiredo M.A.T., Jain A.K., Zhang H.J. (2001). Image classification for content-based image retrieval. IEEE Trans. Image Process. 10(1):117–129CrossRefzbMATHGoogle Scholar
  8. 8.
    Chen J., Bouman C.A., Dalton J.C. (2000). Hierarchical browsing and search of large image databases. IEEE Trans. Image Process. 9:442–455CrossRefGoogle Scholar
  9. 9.
    Ji C., Ma S. (1997). Combinations of weak classifiers. IEEE Trans. Neural Netw. 8:32–42CrossRefGoogle Scholar
  10. 10.
    Bishop C.M. (1995). Neural Networks for Pattern Recognition. University Press, New YorkGoogle Scholar
  11. 11.
    Kuncheva L. (2002). A theoretical study on six classifier fusion strategies. IEEE Trans. Pattern Anal. Mach. Intell. 24(2):281–286CrossRefGoogle Scholar
  12. 12.
    Ponnusamy, A., Breen, C., Khan, L., Wang, L.: Ontology-based image classification using neural networks. In: SPIE: The International Society for Optical Engineering, Boston (2002)Google Scholar
  13. 13.
    Guo G.-D., Jain A.K., Ma W.-Y., Zhang J.-J. (2002). Learning similarity measure for natural image retrieval with relevance feedback. IEEE Trans. Neural Netw. 13(4):811–820CrossRefGoogle Scholar
  14. 14.
    Schapire, R.: Theoretical views of boosting and applications. In: 10th International Conference on Algorithmic Learning Theory, pp. 13–25 (1999)Google Scholar
  15. 15.
    Li J., Wang J. (2003). Automatic linguistic indexing of pictures by a statistical modeling approach. IEEE Trans. Pattern Anal. Mach. Intell. 25(9):1075–1088CrossRefGoogle Scholar
  16. 16.
    Mori, Y., Takahashia, H., Oka, R.: Image-to-word transformation based on dividing and vector quantizing images with words. In: Proceedings of MISRM (1999)Google Scholar
  17. 17.
    Duygulu, P., Barnard, K., de Freitas N., Forsyth, D.: Object recognition as machine translation: learning a lexicon for a fixed image vocabulary. In: Proceedings of ECCV (2002)Google Scholar
  18. 18.
    Jeon, J., Lavrenko, V., Manmatha, R.: Automatic image annotation and retrieval using cross-media relevance models. In: Proceedings of ACM SIGIR (2003)Google Scholar
  19. 19.
    Barnard K., Duygulu P., de Fretias N., Forsyth D., Blei D., Jordan M.I. (2003). Matching words and pictures. J. Mach. Learn. Res. 3:1107–1135CrossRefzbMATHGoogle Scholar
  20. 20.
    Lu, Y., Hu, C., Zhu, X., Zhang, H.J., Yang, Q.: A unified framework for semantics and feature based relevance feedback in image retrieval systems. In: Proceedings of ACM Multimedia (2000)Google Scholar
  21. 21.
    Lee, C.S., Ma, W.-Y., Zhang, H.J.: Information embedding based on users relevance feedback for image retrieval. In: Proceedings of SPIE International Conference on Multimedia Storage and Archiving Systems (1999)Google Scholar
  22. 22.
    Fournier, J., Cord, M.: Long-term similarity learning in content-based image retrieval. In: Proceedings of IEEE International Conference on Image Processing (2002)Google Scholar
  23. 23.
    Yang, J., Li, Q., Zhuang, Y.: Image retrieval and relevance feedback using peer indexing. In: Proceedings of IEEE International Conference on Multimedia and Expo (2002)Google Scholar
  24. 24.
    Li M., Chen Z., Zhang H. (2002). Statistical correlation analysis in image retrieval. Pattern Recogn. 35:2687–2693CrossRefMathSciNetzbMATHGoogle Scholar
  25. 25.
    Zhuang, Y., Yang, J., Li, Q., Pan, Y.: A graphic-theoretic model for incremental relevance feedback in image retrieval. In: Proceedings of IEEE International Conference on Image Processing (2002)Google Scholar
  26. 26.
    Heisterkamp, D.R.: Building a latent semantic index of an image database from patterns of relevance feedback. In: Proceedings of 16th International Conference on Pattern Recognition (2002)Google Scholar
  27. 27.
    He X., King O., Ma W.-Y., Li M., Zhang H.-J. (2003). Learning a semantic space from user’s relevance feedback for image retrieval. IEEE Trans. Circuits Syst. Video Technol. 13(1):39–48CrossRefGoogle Scholar
  28. 28.
    Hoi S.C.H., Lyu M.R., Jin R. (2006). A unified log-based relevance feedback scheme for image retrieval. IEEE Trans. Knowl. Data Eng. 18(4):509–524CrossRefGoogle Scholar
  29. 29.
    Kohrs, A., Merialdo, B.: Improving collaborative filtering with multimedia indexing techniques to create user-adapting web sites. In: Proceedings of ACM International Conference on Multimedia (1999)Google Scholar
  30. 30.
    Zhang, H.J., Zhong, D.: A scheme for visual feature based image indexing. In: Proceedings of SPIE Conference on Storage Retrieval Image Video Databases, San Jose, pp. 36–46 (1995)Google Scholar
  31. 31.
    Zhong, D., Zhang, H.J., Chang, S.F.: Clustering methods for video browsing and annotation. In: Proceedings of SPIE Conference on Storage Retrieval Image Video Databases IV, San Jose, pp. 239–246 (1996)Google Scholar
  32. 32.
    Tieu K., Viola P. (2000). Boosting image retrieval. Proc. IEEE CVPR 1:228–235Google Scholar
  33. 33.
    Santini S., Gupta A., Jain R. (2001). Emergent semantics through interaction in image databases. IEEE Trans. Knowl. Data Eng. 13(3):337–351CrossRefGoogle Scholar
  34. 34.
    Picard R., Minka T., Szummer M. (1996). Modeling subjectivity in image libraries. Proc. IEEE ICIP. 1:777–780CrossRefGoogle Scholar
  35. 35.
    Rui Y., Huang T., Ortega M., Mehrotra S. (2002). Relevance feedback: A power tool in interactive content-based image . IEEE Trans. Circuits Syst. Video Tech. 13(4):811–820Google Scholar
  36. 36.
    Chawla N., Moore T., Hall L., Bowyer K., Kegelmeyer W., Springer C. (2003). Distributed learning with bagging-like performance. Pattern Recogn. Lett. 24:455–471CrossRefGoogle Scholar
  37. 37.
    Vailaya A., Jain A.K., Zhang H.J. (1998). On image classification: City vs. landscape. Pattern Recogn. 31:1921–1936CrossRefGoogle Scholar
  38. 38.
    Vailaya A., Zhang H., Liu F., Jain A.K. (2002). Automatic image orientation detection. IEEE Trans. Image Process. 11:746–755CrossRefGoogle Scholar
  39. 39.
    Canny J. (1986). A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8:679–698CrossRefGoogle Scholar
  40. 40.
    Jain A., Vailaya A. (1996). Image retrieval using color and shape. Pattern Recogn. 29:1233–1244CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2006

Authors and Affiliations

  1. 1.Machine Vision and Pattern Recognition Lab at Department of Computer ScienceWayne State UniversityDetroitUSA
  2. 2.Database and Multimedia Systems Group at Department of Computer ScienceWayne State UniversityDetroitUSA

Personalised recommendations