To Re-rank or to Re-query: Can Visual Analytics Solve This Dilemma?

  • Emanuele Di Buccio
  • Marco Dussin
  • Nicola Ferro
  • Ivano Masiero
  • Giuseppe Santucci
  • Giuseppe Tino
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6941)

Abstract

Evaluation has a crucial role in (IR) since it allows for identifying possible points of failure of an IR approach, thus addressing them to improve its effectiveness. Developing tools to support researchers and analysts when analyzing results and investigating strategies to improve IR system performance can help make the analysis easier and more effective. In this paper we discuss a VA-based approach to support the analyst when deciding whether or not to investigate re-ranking to improve the system effectiveness measured after a retrieval run. Our approach is based on effectiveness measures that exploit graded relevance judgements and it provides both a principled and intuitive way to support analysis. A prototype is described and exploited to discuss some case studies based on TREC data.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Harman, D., Buckley, C.: Overview of the Reliable Information Access Workshop. Information Retrieval 12, 615–641 (2009)CrossRefGoogle Scholar
  2. 2.
    Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information System 20, 422–446 (2002)CrossRefGoogle Scholar
  3. 3.
    Keskustalo, H., Järvelin, K., Pirkola, A., Kekäläinen, J.: Intuition-supporting visualization of user’s performance based on explicit negative higher-order relevance. In: Proceedings of SIGIR 2008, pp. 675–682. ACM, New York (2008)Google Scholar
  4. 4.
    Teevan, J., Dumais, S.T., Horvitz, E.: Potential for personalization. ACM Transactions on Computer-Human Interaction (TOCHI) 17, 1–31 (2010)CrossRefGoogle Scholar
  5. 5.
    Keim, D., Andrienko, G., Fekete, J.D., Görg, C., Kohlhammer, J., Melançon, G.: Information visualization, pp. 154–175. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  6. 6.
    Card, S.K., Mackinlay, J.: The structure of the information visualization design space. In: Proceedings of InfoVis 1997, pp. 92–99. IEEE Computer Society, Washington, DC, USA (1997)Google Scholar
  7. 7.
    Shneiderman, B.: The eyes have it: A task by data type taxonomy for information visualizations. In: Proceedings of the 1996 IEEE Symposium on Visual Languages, pp. 336–343. IEEE Computer Society, Washington, DC, USA (1996)CrossRefGoogle Scholar
  8. 8.
    Keim, D., Kohlhammer, J., Santucci, G., Mansmann, F., Wanner, F., Schäfer, M.: Visual Analytics Challenges. In: Proceedings of the eChallenges 2009 (2009)Google Scholar
  9. 9.
    Seo, J., Shneiderman, B.: A rank-by-feature framework for interactive exploration of multidimensional data. Information Visualization 4, 96–113 (2005)CrossRefGoogle Scholar
  10. 10.
    Derthick, M., Christel, M.G., Hauptmann, A.G., Wactlar, H.D.: Constant density displays using diversity sampling. In: Proceedings of InfoVis 2003, pp. 137–144. IEEE Computer Society, Washington, DC, USA (2003)Google Scholar
  11. 11.
    Banks, D., Over, P., Zhang, N.-F.: Blind Men and Elephants: Six Approaches to TREC data. Information Retrieval 1, 7–34 (1999)CrossRefGoogle Scholar
  12. 12.
    Sormunen, E., Hokkanen, S., Kangaslampi, P., Pyy, P., Sepponen, B.: Query performance analyser -: a web-based tool for ir research and instruction. In: Proceedings of SIGIR 2002, p. 450. ACM, New York (2002)Google Scholar
  13. 13.
    Ferro, N., Sabetta, A., Santucci, G., Tino, G., Veltri, F.: Visual comparison of Ranked Result Cumulated Gains. In: Proceedings of EuroVA 2011 (2011)Google Scholar
  14. 14.
    Di Buccio, E., Dussin, M., Ferro, N., Masiero, I., Santucci, G., Tino, G.: Interactive analysis and exploration of experimental evaluation results. In: Proceedings of EuroHCIR 2011 (to appear, 2011)Google Scholar
  15. 15.
    Melucci, M.: Weighted rank correlation in information retrieval evaluation. In: Lee, G.G., Song, D., Lin, C.-Y., Aizawa, A., Kuriyama, K., Yoshioka, M., Sakai, T. (eds.) AIRS 2009. LNCS, vol. 5839, pp. 75–86. Springer, Heidelberg (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Emanuele Di Buccio
    • 1
  • Marco Dussin
    • 1
  • Nicola Ferro
    • 1
  • Ivano Masiero
    • 1
  • Giuseppe Santucci
    • 2
  • Giuseppe Tino
    • 2
  1. 1.University of PaduaItaly
  2. 2.Sapienza University of RomeItaly

Personalised recommendations