Skip to main content
SpringerLink
Log in
Book cover

Collaborative Information Seeking pp 115–137Cite as

Evaluation

  • Chapter
  • First Online:

  • 1065 Accesses

Part of the book series: The Information Retrieval Series ((INRE,volume 34))

Abstract

Evaluating performances of CIS systems and users can be challenging due to the complex interactions that take place among various users and systems processes. While some of the aspects of a CIS system or user could be measured by typical assessment techniques from single-user IR/IS, one often needs to go beyond them to provide a meaningful evaluation. This chapter presents a framework for CIS evaluation with two major parts: system-based and user-based. It outlines several of the relevant assessments employed for IR/IS/CIS works, and provides a comprehensive list of measures, along with suitable methods for deploying them for empirical evaluations.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    Also available from http://oldwww.acm.org/perlman/question.cgi?form=CSUQ.

  2. 2.

    Taken from http://www.cc.gatech.edu/classes/AY2005/cs7470_fall/papers/manual.pdf.

References

  1. Maria Aneiros and Vladimir Estivill-Castro. Usability of real-time unconstrained WWW-co-browsing for educational settings. In Proceedings of the IEEE/WIC International Conference on Web Intelligence, pages 105–111, Compiegne University of Technology, France, September 2005.

    Google Scholar 

  2. Ricardo Baeza-Yates and Jose A. Pino. A first step to formally evaluate collaborative work. In Proceedings of GROUP, pages 56–60, 1997.

    Google Scholar 

  3. Marcia J. Bates. Information search tactics. Journal of the American Society for Information Science, 30(4):205–214, 1979.

    Article  Google Scholar 

  4. N.J. Belkin, P.G. Marchetti and C. Cool. BRAQUE: design of an interface to support user interaction in information retrieval. Information Processing and Management, 29(3):325–344, 1993.

    Article  Google Scholar 

  5. Raya Fidel, Annelise Mark Pejtersen, Bryan Cleal and Harry Bruce. A multidimensional approach to the study of human-information interaction: A case study of collaborative information retrieval. Journal of the American Society for Information Science and Technology, 55(11):939–953, 2004.

    Article  Google Scholar 

  6. S. Fox, K. Karnawat, M. Mydland, S. Dumais and T. White. Evaluating implicit measures to improve web search. ACM Transactions on Information Systems, 23(2):147–168, 2005.

    Article  Google Scholar 

  7. Jawaid A. Ghani, Roberta Supnick and Pamela Rooney. The experience of flow in computer-mediated and in face-to-face groups. In Proceedings of International Conference on Information Systems, pages 229–237, University of Minnesota, Minneapolis, MN, USA, 1991.

    Google Scholar 

  8. John M. Govern and Lisa A. Marsch. Development and validation of the situational self-awareness scale. Consciousness and Cognition, 10(3):366–378, 2001.

    Article  Google Scholar 

  9. D. Johnson, R. Johnson, E. Holubec and P. Roy. Circles of Learning. Cooperation in the Classroom. Association for Supervision and Curriculum Development, Alexandria, 1984.

    Google Scholar 

  10. Yann Laurillau and Laurence Nigay. CoVitesse: a groupware interface for collaborative navigation on the WWW. In Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW), pages 236–240, New Orleans, Louisiana, USA, November 2002.

    Google Scholar 

  11. J.R. Lewis. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1):57–58, 1995.

    Article  Google Scholar 

  12. Meredith Ringel Morris and Eric Horvitz. SearchTogether: an interface for collaborative web search. In ACM Symposium on User Interface Software and Technology (UIST), pages 3–12, Newport, RI, October 2007.

    Chapter  Google Scholar 

  13. Jakob Nielsen. Usability inspection methods. In Conference Companion on Human Factors in Computing Systems—CHI ’94, pages 413–414, New York, USA, April 1994. ACM, New York.

    Chapter  Google Scholar 

  14. R. Olivares. Evaluación del Impacto de un Juego Colaborativo Basado en Computador en el Apoyo al Aprendizaje de la Colaboración en Escolares de Enseñanza Básica. Master’s thesis, Universidad de Santiago de Chile, Santiago, Chile, 2006.

    Google Scholar 

  15. Gary M. Olson, Judith S. Olson, Mark R. Carter and Marianne Storrosten. Small Group Design Meetings: An Analysis of Collaboration. Human-Computer Interaction, 7(4):347–374, 1992.

    Article  Google Scholar 

  16. Jeremy Pickens, Gene Golovchinsky, Chirag Shah, Pernilla Qvarfordt and Maribeth Back. Algorithmic mediation for collaborative exploratory search. In Proceedings of the Annual ACM Conference on Research and Development in Information Retrieval (SIGIR), Singapore, July 2008.

    Google Scholar 

  17. Paul Prekop. A qualitative study of collaborative information seeking. Journal of Documentation, 58(5):538–547, 2002.

    Article  Google Scholar 

  18. Chirag Shah. Working in collaboration—what, why, and how. In Proceedings of Collaborative Information Retrieval workshop at CSCW 2010, Savannah, GA, 2010.

    Google Scholar 

  19. Chirag Shah and Roberto Gonzalez-Ibanez. Exploring information seeking processes in collaborative search tasks. In Annual Meeting of the American Society for Information Science, Pittsburgh, PA, 2010.

    Google Scholar 

  20. Chirag Shah and Roberto Gonzalez-Ibanez. Evaluating the synergic effect of collaboration in information seeking. In Proceedings of the Annual ACM Conference on Research and Development in Information Retrieval (SIGIR), pages 913–922, Beijing, China, 2011.

    Google Scholar 

  21. Chirag Shah, Jeremy Pickens and Gene Golovchinsky. Role-based results redistribution for collaborative information retrieval. Information Processing & Management, 46(6):773–781, 2010.

    Article  Google Scholar 

  22. Barry Smyth, Evelyn Balfe, Oisin Boydell, Keith Bradley, Peter Briggs, Maurice Coyle and Jill Freyne. A live-user evaluation of collaborative web search. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), Edinburgh, Scotland, July 2005.

    Google Scholar 

  23. Barry Smyth, Evelyn Balfe, Peter Briggs, Maurice Coyle and Jill Freyne. Collaborative web search. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 1417–1419, Acapulco, Mexico, August 2003. Morgan Kaufmann, San Mateo.

    Google Scholar 

  24. James Surowiecki. Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. Doubleday, New York, 2004.

    Google Scholar 

  25. Michael B. Twidale, David M. Nichols and Chris D. Paice. Supporting collaborative learning during information searching. In Proceedings of Computer Supported Collaborative Learning (CSCL), pages 367–374, Bloomington, Indiana, 1995.

    Google Scholar 

  26. Robert A. Wagner and Michael J. Fischer. The string-to-string correction problem. Journal of the ACM, 21(1):168–173, 1974.

    Article  MATH  MathSciNet  Google Scholar 

  27. David Watson, Lee A. Clark and Auke Tellegen. Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54(6):1063–1070, 1988.

    Article  Google Scholar 

  28. Ryen W. White and Jeff Huang. Assessing the scenic route: measuring the value of search trails in web logs. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2010), pages 587–594, 2010.

    Chapter  Google Scholar 

  29. Max L. Wilson and m.c. schraefel. Evaluating collaborative search interfaces with information seeking theory. In Workshop on Collaborative Information Retrieval, Pittsburgh, PA, June 2008.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Communication & Information, The State University of New Jersey, New Brunswick, NJ, USA

    Chirag Shah

Authors
  1. Chirag Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Shah, C. (2012). Evaluation. In: Collaborative Information Seeking. The Information Retrieval Series, vol 34. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-28813-5_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-28813-5_7

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-28812-8

  • Online ISBN: 978-3-642-28813-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation