Evaluation Methods for Groupware Systems

  • Valeria Herskovic
  • José A. Pino
  • Sergio F. Ochoa
  • Pedro Antunes
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4715)


Evaluation of collaborative systems is necessary in several situations. However, evaluation is frequently done in an ad-hoc manner or not at all. This paper presents a survey of evaluation methods for groupware systems. The analysis, comparison and classification of these methods will help developers choose the appropriate methods for their situation. Furthermore, the survey allows identification of strengths and weaknesses of existing methods, opening opportunities for research in this area. The proposed comparison criteria represent a framework to evaluate and classify new evaluation methods.


Evaluation Method Computer Support Cooperative Work Collaborative System Shared Workspace Cooperation Scenario 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Antunes, P., Costa, C.: Perceived value: A low-cost approach to evaluate meetingware. In: Favela, J., Decouchant, D. (eds.) CRIWG 2003. LNCS, vol. 2806, pp. 109–125. Springer, Heidelberg (2003)Google Scholar
  2. 2.
    Antunes, P., Ferreira, A., Pino, J.: Analyzing shared workspaces design with human-performance models. In: Dimitriadis, Y.A., Zigurs, I., Gómez-Sánchez, E. (eds.) CRIWG 2006. LNCS, vol. 4154, pp. 62–77. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  3. 3.
    Baeza-Yates, R., Pino, J.: Towards formal evaluation of collaborative work and its application to information retrieval. Information Research 11(4), 271 (2006)Google Scholar
  4. 4.
    Baker, K., Greenberg, S., Gutwin, C.: Empirical development of a heuristic evaluation methodology for shared workspace groupware. In: CSCW 2002, pp. 96–105 (2002)Google Scholar
  5. 5.
    Grudin, J.: Why CSCW applications fail: problems in the design and evaluation of organization of organizational interfaces. In: CSCW 1988, pp. 85–93. ACM Press, New York (1988)CrossRefGoogle Scholar
  6. 6.
    Gutwin, C., Greenberg, S.: The mechanics of collaboration: Developing low cost usability evaluation methods for shared workspaces. In: WETICE 2000, pp. 98–103. IEEE Comp. Soc., Los Alamitos (2000)Google Scholar
  7. 7.
    Haynes, S., Purao, S., Skattebo, A.: Situating evaluation in scenarios of use. In: CSCW 2004, pp. 92–101. ACM Press, New York (2004)CrossRefGoogle Scholar
  8. 8.
    Huang, J.: A conceptual framework for understanding collab. systems evaluation. In: WETICE 2005. 14th I. Workshop on Enabling Tech., pp. 215–220. IEEE Comp. Soc., Los Alamitos (2005)Google Scholar
  9. 9.
    Hughes, J., King, V., Rodden, T., Andersen, H.: Moving out from the control room: Ethnography in system design. In: CSCW 1994, pp. 429–439. ACM Press, New York (1994)CrossRefGoogle Scholar
  10. 10.
    Huis in’t Veld, M., Andriessen, J., Verburg, R.: E-magine: The development of an evaluation method to assess groupware applications. In: WETICE 2003, vol. 153, IEEE, Los Alamitos (2003)Google Scholar
  11. 11.
    Inkpen, K., Mandryk, R., DiMicco, J., Scott, S.: Methodology for evaluating collaboration in co-located environments. Interactions 11(6) (2004)Google Scholar
  12. 12.
    Pinelle, D., Gutwin, C.: A review of groupware evaluations. In: WETICE 2000. 9th International Workshop on Enabling Technologies, pp. 86–91. IEEE Computer Society, Los Alamitos (2000)Google Scholar
  13. 13.
    Pinelle, D., Gutwin, C.: Groupware walkthrough: adding context to groupware usability evaluation. In: CHI 2002, pp. 455–462. ACM Press, New York (2002)CrossRefGoogle Scholar
  14. 14.
    Pinelle, D., Gutwin, C., Greenberg, S.: Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with the mechanics of collaboration. ACM Transactions on Computer-Human Interaction 10(4), 281–311 (2003)CrossRefGoogle Scholar
  15. 15.
    Plowman, L., Rogers, Y., Ramage, M.: What are workplace studies for? In: ECSCW 1995. Fourth European Conference on Computer-Supported Cooperative Work, pp. 309–324 (1995)Google Scholar
  16. 16.
    Ross, S., Ramage, M., Rogers, Y.: PETRA: Participatory evaluation through redesign and analysis. Interacting with Computers 7(4), 335–360 (1995)CrossRefGoogle Scholar
  17. 17.
    Stiemerling, O., Cremers, A.: The use of cooperation scenarios in the design and evaluation of a CSCW system. IEEE Transact. on Software Engineering 25, 140 (1999)Google Scholar
  18. 18.
    Twidale, M., Randall, D., Bentley, R.: Situated evaluation for cooperative systems. In: CSCW 1994, pp. 441–452. ACM Press, New York (1994)CrossRefGoogle Scholar
  19. 19.
    Vizcaíno, A., Martinez, M., Aranda, G., Piattini, M.: Evaluating collaborative applications from a knowledge management approach. In: WETICE 2005, pp. 221–225. IEEE Comp. Soc., Los Alamitos (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Valeria Herskovic
    • 1
  • José A. Pino
    • 1
  • Sergio F. Ochoa
    • 1
  • Pedro Antunes
    • 2
  1. 1.Department of Computer Science, Universidad de ChileChile
  2. 2.Department of Informatics, University of LisbonPortugal

Personalised recommendations