Improving Text-Based Image Search with Textual and Visual Features Combination

  • Xuan-Son Vu
  • Thanh Vu
  • Huong Nguyen
  • Quang-Thuy Ha
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 326)


With the huge number of available images on the web, an effective image retrieval system has been more and more needed. Improving the performance is one of crucial tasks in modern text-based image retrieval systems such as Google Image Search, Frickr, etc. In this paper, we propose a unified framework to cluster and re-rank returned images with respect to an input query. However, owning to a difference to previous methods of using only either textual or visual features of an image, we combine the textual and visual features to improve search performance. The experimental results show that our proposed model can significantly improve the performance of a text-based image search system (i.e. Flickr). Moreover, the performance of the system with the combination of textual and visual features outperforms the performance of both the textual-based system and the visual-based system.


Image search system Meta-search engine Textual features Visual features Re-ranking Clustering 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Banerjee, A., Merugu, S., Dhillon, I., Ghosh, J.: Clustering with bregman divergences. Journal of Machine Learning Reserach, 1705–1749 (2005)Google Scholar
  2. 2.
    Cao, L., Pozo, A.D., Jin, X., Luo, J., Han, J., Huang, T.S.: Rankcompete: Simultaneous ranking and clustering of web photos. In: Proceedings of the 19th International Conference on World Wide Web, WWW 2010, pp. 1071–1072. ACM, New York (2010)CrossRefGoogle Scholar
  3. 3.
    Chang, R., Lin, S., Ho, J., Fann, C., Wang, Y.: A novel content based image retrieval system using k-means/knn with feature extraction. Computer Science and Information Systems 9, 1645–1662 (2012)CrossRefGoogle Scholar
  4. 4.
    Chang, S.K., Hsu, A.: Image information systems: Where do we go from here? IEEE Trans. on Knowl. and Data Eng. 4(5), 431–442 (1992)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Deselaers, T., Keysers, D., Ney, H.: Features for image retrieval: An experimental comparison. Inf. Retr. 11(2), 77–107 (2008)CrossRefGoogle Scholar
  6. 6.
    Ding, H., Liu, J., Lu, H.: Hierarchical clustering-based navigation of image search results. In: Proceedings of the 16th ACM International Conference on Multimedia, MM 2008, pp. 741–744. ACM, New York (2008)Google Scholar
  7. 7.
    Fischer, I.: An algorithm for vector quantization design. Report No.IDSIA-12-04 (2004)Google Scholar
  8. 8.
    Gupta, A., Jain, R.: Visual information retrieval. Commun. ACM 40(5), 70–79 (1997)CrossRefGoogle Scholar
  9. 9.
    Kanungo, T., Mount, D.M., Netanyahu, N.S., Piatko, C.D., Silverman, R., Wu, A.Y.: An efficient k-means clustering algorithm: Analysis and implementation. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 881–892 (2002)CrossRefGoogle Scholar
  10. 10.
    Liu, G., Lee, B.: A color-based clustering approach for web image search results. In: Proceedings of the 2009 International Conference on Hybrid Information Technology, ICHIT 2009, pp. 481–484. ACM, New York (2009)CrossRefGoogle Scholar
  11. 11.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  12. 12.
    Luo, B., Wang, X., Tang, X.: A world wide web based image search engine using text and image content features. In: Proc. of IS-T/SPIE Electronic Imaging 2003, Internet Imaging IV (2003)Google Scholar
  13. 13.
    Mitra, S., Acharya, T.: Data mining - multimedia, soft computing, and bioinformatics. Wiley (2003)Google Scholar
  14. 14.
    Mukherjea, S., Hirata, K., Hara, Y.: Using clustering and visualization for refining the results of a www image search engine. In: Proceedings of the 1998 Workshop on New Paradigms in Information Visualization and Manipulation, NPIV 1998, pp. 29–35. ACM, New York (1998)CrossRefGoogle Scholar
  15. 15.
    Nguyen, C.-T.: Flickrsearcher: A java implementation for image retrieval with flickr API,
  16. 16.
    Niblack, C.W., Barber, R., Equitz, W., Flickner, M.D., Glasman, E.H., Petkovic, D., Yanker, P., Faloutsos, C., Taubin, G.: Qbic project: querying images by content, using color, texture, and shape. In: Proc. SPIE, vol. 1908, pp. 173–187 (1993)Google Scholar
  17. 17.
  18. 18.
    Pentland, A., Picard, R.W., Sclaroff, S.: Photobook: Content-based manipulation of image databases. Int. J. Comput. Vision 18(3), 233–254 (1996)CrossRefGoogle Scholar
  19. 19.
    Sevil, S., Zitouni, H., Ikizler, N., Ozkan, D., Duygulu, P.: Re-ranking of image search results using a graph algorithm. In: IEEE 16th Signal Processing, Communication and Applications Conference, SIU 2008, pp. 1–4 (April 2008)Google Scholar
  20. 20.
    Tamura, H., Mori, S., Yamawaki, T.: Texture features corresponding to visual perception. IEEE Transactions on Systems, Man and Cybernetics 8(6) (1978)Google Scholar
  21. 21.
    Tamura, H., Yokoya, N.: Image database systems: A survey. Pattern Recognition 17(1), 29–43 (1984), Knowledge Based Image AnalysisGoogle Scholar
  22. 22.
    Thomas, D., Daniel, K., Hermann, N.: Clustering visually similar images to improve image search engines. Informatiktage der Gesellschaft für InformatikGoogle Scholar
  23. 23.
    Thomas, O.: Facebook: Users upload 300 million images a day,
  24. 24.
    Upstill, T., Nagappan, R., Craswell, N.: Visual clustering of image search results. In: SPIE Visual Data Exploration and Analysis VIII, San Jose, California USA (2001),
  25. 25.
    Wang, X.-J., Ma, W.-Y., He, Q.-C., Li, X.: Grouping web image search result. In: Proceedings of the 12th Annual ACM International Conference on Multimedia, MULTIMEDIA 2004, pp. 436–439. ACM, New York (2004)Google Scholar
  26. 26.
    Ding, Z.Y.: An image retrieval algorithm based on hue entropy. Taiyuan Normal Uni: Nat. Sci. Ed., 10–11Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Xuan-Son Vu
    • 1
  • Thanh Vu
    • 2
  • Huong Nguyen
    • 3
  • Quang-Thuy Ha
    • 3
  1. 1.School of Computer Science and EngineeringKyungpook National UniversityDaeguKorea
  2. 2.The Open UniversityMilton KeynesUK
  3. 3.KTLab, University of Engineering and Technology (VNU-UET)Vietnam National University, Hanoi (VNUH)Ha NoiViet Nam

Personalised recommendations