A Statistical and Graphical Methodology for Comparing Bilingual to Monolingual Cross-Language Information Retrieval

  • Franco Crivellari
  • Giorgio Maria Di Nunzio
  • Nicola Ferro

Abstract

A new methodology for the evaluation of MultiLingual Information Access (MLIA) systems is proposed. This two-fold methodology exploits both statistical analyses and graphical tools in order to provide MLIA researchers guidelines, hints, and directions to drive the design and development of the next generation systems, and to provide a means to interpret and compare experimental results and to present these results to other research communities. An example of the application of this methodology is applied in the real-case study of the monolingual and bilingual tasks of the CLEF 2005 and 2006.

Keywords

multilingual information access experimental evaluation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Braschler, M., Di Nunzio, G.M., Ferro, N., Peters, C.: CLEF 2004: Ad Hoc Track Overview and Results Analysis. In: C. Peters, P. Clough, J. Gonzalo, G.J.F. Jones, M. Kluck, B. Magnini (eds.) Multilingual Information Access for Text, Speech and Images: Fifth Workshop of the Cross–Language Evaluation Forum (CLEF 2004) Revised Selected Papers, pp. 10–26. Lecture Notes in Computer Science (LNCS) 3491, Springer, Heidelberg, Germany (2005)Google Scholar
  2. 2.
    Conover, W.J.: Practical Nonparametric Statistics, 1st edn. John Wiley and Sons, New York, USA (1971)Google Scholar
  3. 3.
    Di Nunzio, G.M., Ferro, N.: Appendix A. Results of the Core Tracks and Domain-Specific Tracks. In: C. Peters, V. Quochi (eds.) Working Notes for the CLEF 2005 Workshop. http://www.clef-campaign.org/2005/working_notes/workingnotes2005/appendix_a.pdf [last visited 2007, March 23] (2005)Google Scholar
  4. 4.
    Di Nunzio, G.M., Ferro, N.: Appendix A: Results of the Ad-hoc Bilingual and Monolingual Tasks. In: A. Nardi, C. Peters, J.L. Vicedo (eds.) Working Notes for the CLEF 2006 Workshop. http://www.clef-campaign.org/2006/working_notes/workingnotes2006/Appendix_Ad-Hoc.pdf [last visited 2007, March 23] (2006)Google Scholar
  5. 5.
    Di Nunzio, G.M., Ferro, N., Jones, G.J.F., Peters, C.: CLEF 2005: Ad Hoc Track Overview. In: C. Peters, F.C. Gey, J. Gonzalo, G.J.F. Jones, M. Kluck, B. Magnini, H. Müller, M. de Rijke (eds.) Accessing Multilingual Information Repositories: Sixth Workshop of the Cross–Language Evaluation Forum (CLEF 2005). Revised Selected Papers, pp. 11–36. Lecture Notes in Computer Science (LNCS) 4022, Springer, Heidelberg, Germany (2006)Google Scholar
  6. 6.
    Di Nunzio, G.M., Ferro, N., Mandl, T., Peters, C.: CLEF 2006: Ad Hoc Track Overview. In: A. Nardi, C. Peters, J.L. Vicedo (eds.) Working Notes for the CLEF 2006 Workshop. http://www.clef-campaign.org/2006/working_notes/workingnotes2006/dinunzioOCLEF2006.pdf [last visited 2007, March 23] (2006)Google Scholar
  7. 7.
    European Commission: Commission Recommendation of 24 August 2006 on the digitisation and online accessibility of cultural material and digital preservation. Official Journal of the European Union, OJ L 236, 31.8.2006 49, 28–30 (2006)Google Scholar
  8. 8.
    European Commission Information Society and Media: i2010: Digital Libraries. http://europa.eu.int/information_society/activities/digital_libraries/doc/brochures/dl_brochure_2006.pdf [last visited 2007, March 23] (2006)Google Scholar
  9. 9.
    Gonzalo, J., Peters, C.: The Impact of Evaluation on Multilingual Text Retrieval. In: R. Baeza-Yates, N. Ziviani, G. Marchionini, A. Moffat, J. Tait (eds.) Proc. 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2005), pp. 603–604. ACM Press, New York, USA (2005)Google Scholar
  10. 10.
    Hull, D.: Using Statistical Testing in the Evaluation of Retrieval Experiments. In: R. Korfhage, E. Rasmussen, P. Willett (eds.) Proc. 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1993), pp. 329–338. ACM Press, New York, USA (1993)Google Scholar
  11. 11.
    Judge, G.G., Hill, R.C., Griffiths, W.E., Lütkepohl, H., Lee, T.C.: Introduction to the Theory and Practice of Econometrics, 2nd edn. John Wiley and Sons, New York, USA (1988)MATHGoogle Scholar
  12. 12.
    Peters, C.: Multilingual Information Access for Digital Libraries: The Impact of Evaluation on System Development. In: C. Thanos (ed.) DELOS Research Activities 2006, pp. 105–107. ISTI-CNR at Gruppo ALI, Pisa, Italy (2006)Google Scholar
  13. 13.
    Tague-Sutcliffe, J.: The Pragmatics of Information Retrieval Experimentation, Revisited. In: K. Spack Jones, P. Willett (eds.) Readings in Information Retrieval, pp. 205–216. Morgan Kaufmann Publisher, Inc., San Francisco, California, USA (1997)Google Scholar
  14. 14.
    Wang, J., Oard, D.W.: Combining Bidirectional Translation and Synonymy for Cross-Language Information Retrieval. In: E.N. Efthimiadis, S. Dumais, D. Hawking, K. Järvelin (eds.) Proc. 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006), pp. 202–209. ACM Press, New York, USA (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Franco Crivellari
    • 1
  • Giorgio Maria Di Nunzio
    • 1
  • Nicola Ferro
    • 1
  1. 1.Department of Information EngineeringUniversity of PaduaVia Gradenigo 6/aItaly

Personalised recommendations