Abstract
Evaluating performances of CIS systems and users can be challenging due to the complex interactions that take place among various users and systems processes. While some of the aspects of a CIS system or user could be measured by typical assessment techniques from single-user IR/IS, one often needs to go beyond them to provide a meaningful evaluation. This chapter presents a framework for CIS evaluation with two major parts: system-based and user-based. It outlines several of the relevant assessments employed for IR/IS/CIS works, and provides a comprehensive list of measures, along with suitable methods for deploying them for empirical evaluations.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Also available from http://oldwww.acm.org/perlman/question.cgi?form=CSUQ.
- 2.
References
Maria Aneiros and Vladimir Estivill-Castro. Usability of real-time unconstrained WWW-co-browsing for educational settings. In Proceedings of the IEEE/WIC International Conference on Web Intelligence, pages 105–111, Compiegne University of Technology, France, September 2005.
Ricardo Baeza-Yates and Jose A. Pino. A first step to formally evaluate collaborative work. In Proceedings of GROUP, pages 56–60, 1997.
Marcia J. Bates. Information search tactics. Journal of the American Society for Information Science, 30(4):205–214, 1979.
N.J. Belkin, P.G. Marchetti and C. Cool. BRAQUE: design of an interface to support user interaction in information retrieval. Information Processing and Management, 29(3):325–344, 1993.
Raya Fidel, Annelise Mark Pejtersen, Bryan Cleal and Harry Bruce. A multidimensional approach to the study of human-information interaction: A case study of collaborative information retrieval. Journal of the American Society for Information Science and Technology, 55(11):939–953, 2004.
S. Fox, K. Karnawat, M. Mydland, S. Dumais and T. White. Evaluating implicit measures to improve web search. ACM Transactions on Information Systems, 23(2):147–168, 2005.
Jawaid A. Ghani, Roberta Supnick and Pamela Rooney. The experience of flow in computer-mediated and in face-to-face groups. In Proceedings of International Conference on Information Systems, pages 229–237, University of Minnesota, Minneapolis, MN, USA, 1991.
John M. Govern and Lisa A. Marsch. Development and validation of the situational self-awareness scale. Consciousness and Cognition, 10(3):366–378, 2001.
D. Johnson, R. Johnson, E. Holubec and P. Roy. Circles of Learning. Cooperation in the Classroom. Association for Supervision and Curriculum Development, Alexandria, 1984.
Yann Laurillau and Laurence Nigay. CoVitesse: a groupware interface for collaborative navigation on the WWW. In Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW), pages 236–240, New Orleans, Louisiana, USA, November 2002.
J.R. Lewis. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1):57–58, 1995.
Meredith Ringel Morris and Eric Horvitz. SearchTogether: an interface for collaborative web search. In ACM Symposium on User Interface Software and Technology (UIST), pages 3–12, Newport, RI, October 2007.
Jakob Nielsen. Usability inspection methods. In Conference Companion on Human Factors in Computing Systems—CHI ’94, pages 413–414, New York, USA, April 1994. ACM, New York.
R. Olivares. Evaluación del Impacto de un Juego Colaborativo Basado en Computador en el Apoyo al Aprendizaje de la Colaboración en Escolares de Enseñanza Básica. Master’s thesis, Universidad de Santiago de Chile, Santiago, Chile, 2006.
Gary M. Olson, Judith S. Olson, Mark R. Carter and Marianne Storrosten. Small Group Design Meetings: An Analysis of Collaboration. Human-Computer Interaction, 7(4):347–374, 1992.
Jeremy Pickens, Gene Golovchinsky, Chirag Shah, Pernilla Qvarfordt and Maribeth Back. Algorithmic mediation for collaborative exploratory search. In Proceedings of the Annual ACM Conference on Research and Development in Information Retrieval (SIGIR), Singapore, July 2008.
Paul Prekop. A qualitative study of collaborative information seeking. Journal of Documentation, 58(5):538–547, 2002.
Chirag Shah. Working in collaboration—what, why, and how. In Proceedings of Collaborative Information Retrieval workshop at CSCW 2010, Savannah, GA, 2010.
Chirag Shah and Roberto Gonzalez-Ibanez. Exploring information seeking processes in collaborative search tasks. In Annual Meeting of the American Society for Information Science, Pittsburgh, PA, 2010.
Chirag Shah and Roberto Gonzalez-Ibanez. Evaluating the synergic effect of collaboration in information seeking. In Proceedings of the Annual ACM Conference on Research and Development in Information Retrieval (SIGIR), pages 913–922, Beijing, China, 2011.
Chirag Shah, Jeremy Pickens and Gene Golovchinsky. Role-based results redistribution for collaborative information retrieval. Information Processing & Management, 46(6):773–781, 2010.
Barry Smyth, Evelyn Balfe, Oisin Boydell, Keith Bradley, Peter Briggs, Maurice Coyle and Jill Freyne. A live-user evaluation of collaborative web search. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), Edinburgh, Scotland, July 2005.
Barry Smyth, Evelyn Balfe, Peter Briggs, Maurice Coyle and Jill Freyne. Collaborative web search. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 1417–1419, Acapulco, Mexico, August 2003. Morgan Kaufmann, San Mateo.
James Surowiecki. Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. Doubleday, New York, 2004.
Michael B. Twidale, David M. Nichols and Chris D. Paice. Supporting collaborative learning during information searching. In Proceedings of Computer Supported Collaborative Learning (CSCL), pages 367–374, Bloomington, Indiana, 1995.
Robert A. Wagner and Michael J. Fischer. The string-to-string correction problem. Journal of the ACM, 21(1):168–173, 1974.
David Watson, Lee A. Clark and Auke Tellegen. Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54(6):1063–1070, 1988.
Ryen W. White and Jeff Huang. Assessing the scenic route: measuring the value of search trails in web logs. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2010), pages 587–594, 2010.
Max L. Wilson and m.c. schraefel. Evaluating collaborative search interfaces with information seeking theory. In Workshop on Collaborative Information Retrieval, Pittsburgh, PA, June 2008.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Shah, C. (2012). Evaluation. In: Collaborative Information Seeking. The Information Retrieval Series, vol 34. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-28813-5_7
Download citation
DOI: https://doi.org/10.1007/978-3-642-28813-5_7
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-28812-8
Online ISBN: 978-3-642-28813-5
eBook Packages: Computer ScienceComputer Science (R0)