The Browsing Issue in Multimodal Information Retrieval: A Navigation Tool Over a Multiple Media Search Result Space

  • Umer Rashid
  • Marco Viviani
  • Gabriella Pasi
  • Muhammad Afzal Bhatti
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 400)


In the field of Multimodal Information Retrieval, one of the issue to tackle is how to effectively browse the search result space. In addressing this issue, it is particularly important to take into consideration that, especially nowadays, data is highly semantically interlinked. In this scenario, we present a tool to navigate and visualize the results produced by the evaluation of a query over a set of multiple media objects. The search result space can be represented via a graph-based data model where (i) multiple media objects are represented as nodes with multiple modalities of information associated with them, and (ii) media objects can be connected via different kinds of relationships. Our idea is to give to the user the possibility to navigate the space of the results of a query, constituted by multiple media objects, as s/he was exploring a graph of connected entities. As a preliminary work, in this paper we only deal with textual information for building similarity relationships among media objects and part-of relationships in the case of media objects belonging to a same (multimedia) document. This way, we show how a user can navigate and visualize the result space following different links connecting media objects. We illustrate our navigation and visualization tool with different examples.


Query Term Video Object Media Object Multimedia Document Content Object 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Lauer, C.: Contending with terms: Multimodal and Multimedia in the Academic and Public Spheres. J. Computers and Composition 26, 225–239 (2009)CrossRefGoogle Scholar
  2. 2.
    Rafailidis, D., Manolopoulou, S., Daras, P.: A unified framework for multimodal retrieval. J. Pattern Recognition 4, 358–3370 (2013)Google Scholar
  3. 3.
    Tjondronegoro, D., Spink, A., Jansen, B.J.: A study and comparison of multimedia Web searching: 1997–2006. J. American Society for Information Science and Technology 60, 1756–1768 (2009)CrossRefGoogle Scholar
  4. 4.
    Bozzon, A., Fraternali, P.: Chapter 8: Multimedia and multimodal information retrieval. In: Ceri, S., Brambilla, M. (eds.) Search Computing. LNCS, vol. 5950, pp. 135–155. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  5. 5.
    Sushmita, S., Joho, H., Lalmas, M., Villa, R.: Factors affecting click-through behavior in aggregated search interfaces. In: 19th ACM International Conference on Information and Knowledge Management, pp. 519–528. ACM (2010)Google Scholar
  6. 6.
    Bron, M., Van Gorp, J., Nack, F., Baltussen, L.B., de Rijke, M.: Aggregated search interface preferences in multi-session search tasks. In: Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 15–20. ACM, Japan (2013)Google Scholar
  7. 7.
    Mei, T., Rui, Y., Li, S., Tian, Q.: Multimedia search Re-ranking: A literature survey. ACM Computing Surveys 46(38) (2014)Google Scholar
  8. 8.
    Kalamaras, I., Malassiotis, S., Tzovaras, D., Mademlis, S.: Novel framework for retrieval and interactive visualization of multimodal data. J. Electronic Letters on Computer Vision and Image Analysis 12, 28–29 (2013)Google Scholar
  9. 9.
    Lauer, C.: Precision-recall is wrong for multimedia. J. IEEE MultiMedia 18, 04–07 (2009)Google Scholar
  10. 10.
    Kopliku, A., Pinel-Sauvagnat, K., Boughanem, M.: Aggregated search: A new information retrieval paradigm. ACM Computing Surveys 46(41) (2014)Google Scholar
  11. 11.
    Zavesky, E., Chang, S.F., Yang, C.C.: Visual islands: intuitive browsing of visual search results. In: Proceedings of International Conference on Content-Based Image and Video Retrieval, pp. 617–626. ACM (2008)Google Scholar
  12. 12.
    Wiesener, S., Kowarschick, W., Bayer, R.: Semalink: an approach for semantic browsing through large distributed document spaces. In: Proceedings of the Third Forum on Research and Technology Advances in Digital Libraries, pp. 86–94. IEEE (1996)Google Scholar
  13. 13.
    Szegõ, D.: A logical framework for analyzing properties of multimedia web documents. In: Workshop on Multimedia Discovery and Mining, ECML/PKDD, pp. 19–30. (2003)Google Scholar
  14. 14.
    Rigamonti, M., Lalanne, D., Ingold, R.: Faericworld: browsing multimedia events through static documents and links. In: Baranauskas, C., Abascal, J., Barbosa, S.D.J. (eds.) INTERACT 2007. LNCS, vol. 4662, pp. 102–115. Springer, Heidelberg (2007) CrossRefGoogle Scholar
  15. 15.
    Lazaridis, M., Axenopoulos, A., Rafailidis, D., Daras, P.: Multimedia search and retrieval using multimodal annotation propagation and indexing techniques. Signal Processing: Image Communication 28, 351–367 (2013)Google Scholar
  16. 16.
    Sabetghadam, S., Lupu, M., Bierig, R., Rauber, A.: Reachability analysis of graph modelled collections. In: Hanbury, A., Kazai, G., Rauber, A., Fuhr, N. (eds.) ECIR 2015. LNCS, vol. 9022, pp. 370–381. Springer, Heidelberg (2015) Google Scholar
  17. 17.
    Rizzo, G., Steiner, T., Troncy, R., Verborgh, R., Redondo Garcia, J.L.: What fresh media are you looking for?: retrieving media items from multiple social networks. In: Proceedings of International Workshop on Socially-Aware Multimedia, pp. 15–20. ACM, Japan (2012)Google Scholar
  18. 18.
    Hearst, M.: Search User Interfaces. Cambridge University Press, UK (2009)CrossRefGoogle Scholar
  19. 19.
    Huang, A.: Similarity measures for text document clustering. In: Proceedings of the Sixth New Zealand Computer Science Research Student Conference, pp. 49–56 (2008)Google Scholar
  20. 20.
    Chim, H., Deng, X.: Efficient phrase-based document similarity for clustering & Retrieval. IEEE Transactions on Knowledge and Data Engineering 29, 1217–1229 (2009). New ZealandGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Umer Rashid
    • 1
  • Marco Viviani
    • 2
  • Gabriella Pasi
    • 2
  • Muhammad Afzal Bhatti
    • 1
  1. 1.Quaid-i-Azam UniversityIslamabadPakistan
  2. 2.University of Milano-BicoccaMilanItaly

Personalised recommendations