Visual Analytics and IR Experimental Evaluation

  • Nicola FerroEmail author
  • Giuseppe Santucci
Part of the The Information Retrieval Series book series (INRE, volume 41)


We investigate the application of Visual Analytics (VA) techniques to the exploration and interpretation of Information Retrieval (IR) experimental data. We first briefly introduce the main concepts about VA and then we present some relevant examples of VA prototypes developed for better investigating IR evaluation data. Finally, we conclude with an discussion of the current trends and future challenges on this topic.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Andrienko G, Andrienko N, Jankowski P, Keim DA, Kraak MJ, MacEachren A, Wrobel S (2007) Geovisual analytics for spatial decision support: setting the research agenda. Int J Geogr Inf Sci 21(8):839–858CrossRefGoogle Scholar
  2. Angelini M, Ferro N, Santucci G, Silvello G (2012) Visual interactive failure analysis: supporting users in information retrieval evaluation. In: Kamps J, Kraaij W, Fuhr N (eds) Proceedings of 4th symposium on information interaction in context (IIIX 2012). ACM Press, New York, pp 195–203Google Scholar
  3. Angelini M, Ferro N, Santucci G, Silvello G (2014) VIRTUE: a visual tool for information retrieval performance evaluation and failure analysis. J Vis Lang Comput 25(4):394–413CrossRefGoogle Scholar
  4. Angelini M, Ferro N, Santucci G, Silvello G (2016a) A visual analytics approach for what-if analysis of information retrieval systems. In: Perego R, Sebastiani F, Aslam J, Ruthven I, Zobel J (eds) Proceedings of 39th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2016). ACM Press, New York, pp 1081–1084Google Scholar
  5. Angelini M, Ferro N, Santucci G, Silvello G (2016b) What-if analysis: a visual analytics approach to information retrieval evaluation. In: Di Nunzio GM, Nardini FM, Orlando S (eds) Proceedings of 7th Italian information retrieval workshop (IIR 2016). CEUR workshop proceedings ( ISSN 1613-0073.
  6. Angelini M, Ferro N, Santucci G, Silvello G (2017) Visual analytics for information retrieval evaluation campaigns. In: Sedlmair M, Tominski C (eds) Proceedings of 8th international workshop on visual analytics (EuroVA 2017). Eurographics Association, Goslar, pp 25–29Google Scholar
  7. Angelini M, Fazzini V, Ferro N, Santucci G, Silvello G (2018) CLAIRE: a combinatorial visual analytics system for information retrieval evaluation. Inf Process Manag 54(6), 1077–1100CrossRefGoogle Scholar
  8. Banks D, Over P, Zhang NF (1999) Blind men and elephants: six approaches to TREC data. Inf Retriev 1(1–2):7–34CrossRefGoogle Scholar
  9. Behrisch M, Davey J, Simon S, Schreck T, Keim D, Kohlhammer J (2013) Visual comparison of orderings and rankings. In: Pohl M, Schumann H (eds) Proceedings of 4th international workshop on visual analytics (EuroVA 2013). Eurographics Association, GoslarGoogle Scholar
  10. Buckley C, Voorhees EM (2005) Retrieval system evaluation. In: Harman DK, Voorhees EM (eds) TREC. experiment and evaluation in information retrieval. MIT Press, Cambridge, pp 53–78Google Scholar
  11. Card SK, Mackinlay JD, Shneiderman B (1999) Readings in information visualization: using vision to think. Morgan Kaufmann Publishers, San Francisco, CAGoogle Scholar
  12. Chen C (2004) Information visualization - beyond the horizon. Springer, LondonGoogle Scholar
  13. Cleverdon CW (1967) The cranfield tests on index languages devices. Aslib Proc 19(6):173–194CrossRefGoogle Scholar
  14. Crestani F, Vegas J, de la Fuente P (2004) A graphical user interface for the retrieval of hierarchically structured documents. Inf Process Manag 40(2):269–289CrossRefGoogle Scholar
  15. Derthick M, Christel MG, Hauptmann AG, Wactlar HD (2003) Constant density displays using diversity sampling. In: Munzner T, North S (eds) Proceedings of 9th IEEE symposium on information visualization (INFOVIS 2003). IEEE Computer Society, Los Alamitos, pp 137–144Google Scholar
  16. Ferro N, Harman D (2010) CLEF 2009: Grid@CLEF pilot track overview. In: Peters C, Di Nunzio GM, Kurimo M, Mandl T, Mostefa D, Peñas A, Roda G (eds) Multilingual information access evaluation vol. I text retrieval experiments – tenth workshop of the cross–language evaluation forum (CLEF 2009). Revised selected papers. Lecture notes in computer science (LNCS), vol 6241. Springer, Heidelberg, pp 552–565Google Scholar
  17. Ferro N, Silvello G (2016) A general linear mixed models approach to study system component effects. In: Perego R, Sebastiani F, Aslam J, Ruthven I, Zobel J (eds) Proceedings of 39th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2016). ACM Press, New York, pp 25–34Google Scholar
  18. Fowler RH, Lawrence-Fowler WA, Wilson BA (1991) Integrating query, thesaurus, and documents through a common visual representation. In: Fox EA (ed) Proceedings of 14th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 1991). ACM Press, New York, pp 142–151Google Scholar
  19. Hearst MA (2009) Search user interfaces, 1st edn. Cambridge University Press, New YorkCrossRefGoogle Scholar
  20. Hearst MA (2011) “Natural” search user interfaces. Commun ACM 54(11):60–67CrossRefGoogle Scholar
  21. Inselberg A (2009) Parallel coordinates. Visual multidimensional geometry and its applications. Springer, New YorkzbMATHGoogle Scholar
  22. Ioannakis G, Koutsoudis A, Pratikakis I, Chamzas C (2018) Retrieval–an online performance evaluation tool for information retrieval methods. IEEE Trans Multimedia 20(1):119–127CrossRefGoogle Scholar
  23. Järvelin K, Kekäläinen J (2002) Cumulated gain-based evaluation of IR techniques. ACM Trans Inf Syst 20(4):422–446CrossRefGoogle Scholar
  24. Keim DA (2001) Visual exploration of large data sets. Commun ACM 44(8):38–44CrossRefGoogle Scholar
  25. Keim DA, Mansmann F, Schneidewind J, Ziegler H (2006) Challenges in visual data analysis. In: Banissi E (ed) Proceedings of the 10th international conference on information visualization (IV 2006). IEEE Computer Society, Los Alamitos, pp 9–16Google Scholar
  26. Keim DA, Kohlhammer J, Ellis G, Mansmann F (eds) (2010) Mastering the information age – solving problems with visual analytics. Eurographics Association, GoslarGoogle Scholar
  27. Koshman S (2005) Testing user interaction with a prototype visualization-based information retrieval system. J Am Soc Inf Sci Technol 56(8):824–833CrossRefGoogle Scholar
  28. Lipani A, Lupu M, Hanbury A (2017) Visual pool: a tool to visualize and interact with the pooling method. In: Kando N, Sakai T, Joho H, Li H, de Vries AP, White RW (eds) Proceedings of 40th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2017). ACM Press, New York, pp 1321–1324Google Scholar
  29. McGill R, Tukey JW, Larsen WA (1978) Variations of box plots. Am Stat 32(1):12–16Google Scholar
  30. Moffat A, Zobel J (2008) Rank-biased precision for measurement of retrieval effectiveness. ACM Trans Inf Syst (TOIS) 27(1):2:1–2:27CrossRefGoogle Scholar
  31. Morse EL, Lewis M, Olsen KA (2002) Testing visual information retrieval methodologies case study: comparative analysis of textual, icon, graphical, and spring displays. J Am Soc Inf Sci Technol 53(1):28–40CrossRefGoogle Scholar
  32. Rocco G, Silvello G (2019) An InfoVis tool for interactive component-based evaluation. arXivorg, information retrieval (csIR) arXiv:1901.11372Google Scholar
  33. Sankey HR (1898) Introductory note on the thermal efficiency of steam-engines. Report of the committee appointed on the 31st March, 1896, to consider and report to the council upon the subject of the definition of a standard or standards of thermal efficiency for steam-engines: with an introductory note. Minutes of proceedings of the institution of civil engineers, vol 134, pp 278–283 including Plate 5Google Scholar
  34. Schmidt M (2008) The sankey diagram in energy and material flow management. J Ind Ecol 12(1):82–94CrossRefGoogle Scholar
  35. Seo J, Shneiderman B (2005) A rank-by-feature framework for interactive exploration of multidimensional data. Inf Vis 4(2):96–113CrossRefGoogle Scholar
  36. Sormunen E, Hokkanen S, Kangaslampi P, Pyy P, Sepponen B (2002) Query performance analyser – a web-based tool for IR research and instruction. In: Järvelin K, Beaulieu M, Baeza-Yates R, Hyon Myaeng S (eds) Proceedings of 25th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2002). ACM Press, New York, p 450Google Scholar
  37. Spence R (2007) Information visualization: design for interaction, 2nd edn. Pearson Education Limited, LondonGoogle Scholar
  38. Ware C (2012) Information visualization - perception for design, 3rd edn. Morgan Kaufmann Publishers, San FranciscoGoogle Scholar
  39. Wong PC, Thomas JJ (2004) Visual analytics - guest editors’ introduction. IEEE Comput Graph Appl 24(5):20–21CrossRefGoogle Scholar
  40. Zhang J (2001) TOFIR: a tool of facilitating information retrieval - introduce a visual retrieval model. Inf Process Manag 37(4):639–657CrossRefGoogle Scholar
  41. Zhang J (2008) Visualization for information retrieval. Springer, HeidelbergCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Information EngineeringUniversity of PaduaPadovaItaly
  2. 2.Department of Computer, Control, and Management Engineering “Antonio Ruberti”Sapienza University of RomeRomeItaly

Personalised recommendations