Computer Supported Cooperative Work (CSCW)

, Volume 17, Issue 4, pp 353–393 | Cite as

The CACHE Study: Group Effects in Computer-supported Collaborative Analysis

  • Gregorio Convertino
  • Dorrit Billman
  • Peter Pirolli
  • J. P. Massar
  • Jeff Shrager
Article

Abstract

The present experiment investigates effects of group composition in computer-supported collaborative intelligence analysis. Human cognition, though highly adaptive, is also quite limited, leading to systematic errors and limitations in performance – that is, biases. We experimentally investigated the impact of group composition on an individual’s bias, by composing groups that differ in whether their members initial beliefs are diverse (heterogeneous group) or similar (homogeneous group). We study three-member, distributed, computer-supported teams in heterogeneous, homogeneous, and solo (or nominal) groups. We measured bias in final judgment, and also in the selection and evaluation of the evidence that contributed to the final beliefs. The distributed teams collaborated via CACHE-A, a web-based software environment that supports a collaborative version of Analysis of Competing Hypotheses (or ACH, a method used by intelligence analysts). Individuals in Heterogeneous Groups showed no net process cost, relative to noninteracting individuals. Both heterogeneous and solo (noninteracting) groups debiased strongly, given a stream of balanced evidence. In contrast, individuals in Homogenous Groups did worst, accentuating their initial bias rather than debiasing. We offer suggestions about how CACHE-A supports collaborative analysis, and how experimental investigation in this research area can contribute to design of CSCW systems.

Key words

CACHE collaboration intelligence analysis CSCW group bias group decision-making 

References

  1. Adelman, L., Tolcott, M.A. and T.A. Bresnick (1993): Examining the effect of information order on expert judgment, Organizational Behavior and Human Decision Processes, vol. 56, pp. 348–369.CrossRefGoogle Scholar
  2. Arnott, D. (2006): Cognitive Biases and Decision Support Systems Development: A Design Science Approach. Information Systems Journal, vol. 16(1), pp. 55–78 doi:10.1111/j.1365-2575.2006.00208.x.CrossRefGoogle Scholar
  3. Benbasat, I. and J. Lim (2000): Information Technology Support For Debiasing Group Judgments: An Empirical Evaluation. Organizational Behavior and Human Decision Processes, vol. 83, pp. 167–183 doi:10.1006/obhd.2000.2905.CrossRefGoogle Scholar
  4. Benjamin, K. (1990): Why We Still Use Our Heads Instead of Formulas: Toward an Integrative Approach. Psychological Bulletin, vol. 107(3), pp. 296–310 doi:10.1037/0033-2909.107.3.296.CrossRefGoogle Scholar
  5. Bornstein, B.H. and A.C. Emler (2001): Rationality in Medical Decision Making: A Review of the Literature On Doctors’ Decision-making Biases. Journal of Evaluation in Clinical Practice, vol. 7(2), pp. 97–107 doi:10.1046/j.1365-2753.2001.00284.x.CrossRefGoogle Scholar
  6. Camerer, C.F. and E.J. Johnson (1991): The Process–Performance Paradox in Expert Judgment: How Can Experts Know so Much and Predict so Badly? In K.A. Ericsson and J. Smith (eds): Towards a General Theory of Expertise: Prospects and LimitsNew York: Cambridge University Press, pp. 195–217.Google Scholar
  7. Card, S.K. (2005): The Science of Analytical Reasoning. In J.J. Thomas and K.A. Cook (eds): Illuminating the Path: the Research and Development Agenda for Visual AnalyticsIEEE CS: Los Alamitos, CA.Google Scholar
  8. Chapman, G.B., G.R. Bergus and A.S. Elstein (1996): Order of Information Affects Clinical Judgment. Journal of Behavioral Decision Making, vol. 9(3), pp. 201–211 doi:10.1002/(SICI)1099-0771(199609)9:3<201::AID-BDM229>3.0.CO;2-J.CrossRefGoogle Scholar
  9. Cheikes, B.A., M.J. Brown, P.E. Lehner and L. Alderman (2004): Confirmation Bias in Complex Analyses. Technical Report No. MTR 04B0000017. Bedford, MA: MITRE.Google Scholar
  10. Cho, H.-K. (2004): The effect of Delphi Structure on Small and Medium-sized Asynchronous Groups. Ph. D. Dissertation Thesis, New Jersey Institute of Technology, Information Systems Department.Google Scholar
  11. Cho, H.-K. and M. Turoff (2001): Debiasing Group Judgments through Computerized Delphi Systems. In Proceedings of AMCIS 2001, August, Boston, MA.Google Scholar
  12. Convertino, G., H.M. Mentis, P. Bhambare, C. Ferro, J.M. Carroll and M.B. Rosson (2008): Comparing Media in Emergency Planning. In Proceedings of the 5th International ISCRAM. Conference. Washington, DC, USA, May 4–7, 2008.Google Scholar
  13. Cook, M.B. and H.S. Smallman (2007): Collaborative intelligence analysis: Debiasing through graphical evidence layout. In: Proceedings of the 51st Annual Meeting of the Human Factors and Ergonomics Society, Baltimore, MD, pp. 16–20.Google Scholar
  14. Cummings, J.N. (2004): Work Groups, Structural Diversity, and Knowledge Sharing in a Global Organization. Management Science, vol. 50(3), pp. 352–364.CrossRefMathSciNetGoogle Scholar
  15. Davis, E.B. and R.H. Ashton (2002): Threshold Adjustment in Response to Asymmetric Loss Functions: The Case of Auditors’ “Substantial Doubt” Thresholds. Organizational Behavior and Human Decision Processes, vol. 89, pp. 1082–1099 doi:10.1016/S0749-5978(02)00009-2.CrossRefGoogle Scholar
  16. Dennis, A.R. and J.S. Valacich (1993): Computer Brainstorms: More Heads are Better Than One. The Journal of Applied Psychology, vol. 78(4), pp. 531–537 doi:10.1037/0021-9010.78.4.531.CrossRefGoogle Scholar
  17. Dennis, A.R., K.M. Hilmer and N.J. Taylor (1997): Information Exchange and Use in GSS and Verbal Group Decision Making: Effects of Minority Influence. Journal of Management Information Systems Archive, vol. 14(3), pp. 61–88.CrossRefGoogle Scholar
  18. Fjermestad, J. and S.R. Hiltz (2001): An Assessment of Group Support Systems Research: Methodology. Journal of Management Information Systems, vol. 15(3), pp. 7–149.Google Scholar
  19. Fugelsang, J.A., C.B. Stein, A.E. Green and K.N. Dunbar (2004): Theory and Data Interactions of the Scientific Mind: Evidence From the Molecular and the Cognitive Laboratory. Canadian Journal of Experimental Psychology, vol. 58(2), pp. 86–95 doi:10.1037/h0085799.Google Scholar
  20. Gallupe, B.R., L.M. Bastianutti and W.H. Cooper (1991): Unblocking Brainstorms. The Journal of Applied Psychology, vol. 76, pp. 137–142 doi:10.1037/0021-9010.76.1.137.CrossRefGoogle Scholar
  21. George, J.F., K. Duffy and M. Ahuja (2000): Countering the Anchoring and Adjustment Bias with Decision Support Systems. Decision Support Systems, vol. 29(2), pp. 195–206 doi:10.1016/S0167-9236(00)00074-9.CrossRefGoogle Scholar
  22. Gettys, C.F., C. Kelly III and C.R. Peterson (1982): The Best-guess Hypothesis in Multistage Inference. In D. Kahneman, P. Slovic and A. Tversky (eds): Judgment Under Undertainty: Heuristics and BiasesNew York: Cambridge University Press, pp. 370–377.Google Scholar
  23. Gigone, D. and R. Hastie (1996): The Impact of Information on Small Group Choice. Journal of Personality and Social Psychology, vol. 72, pp. 132–140 doi:10.1037/0022-3514.72.1.132.CrossRefGoogle Scholar
  24. Heuer, Richards J. Jr. (1999): The Psychology of Intelligence Analysis. Washington, DC: Center for the Study of Intelligence, Central Intelligence Agency.Google Scholar
  25. Hightower, R. and L. Sayeed (1995): The Impact of Computer-mediated Communication Systems on Biased Group Discussion. Computers in Human Behavior, vol. 11(1), pp. 33–44 doi:10.1016/0747-5632(94)00019-E.CrossRefGoogle Scholar
  26. Hogarth, R.M. and Hillel J. Einhorn (1992): Order effects in belief updating: The belief-adjustment model. Cognitive Psychology, vol. 24(1), pp. 1–55 doi:10.1016/0010-0285(92)90002-J.CrossRefGoogle Scholar
  27. Johnston, R. (2003): Reducing Analytic Error: Integrating Methodologists into Teams of Substantive Experts. Studies in Intelligence, vol. 47(1), pp. 57–65.Google Scholar
  28. Johnston, R. (2005): Analytic Culture in the U.S. Intelligence Community: An Ethnographic Study. Washingon, DC: Central Intelligence Agency, Center for the Study of Intelligence.Google Scholar
  29. Johnson, E.M. and S.M. Halpin (1974): Multistage Inference Models For Intelligence Analysis. Report AD 785-639. Arlington, VA: Army Research Institute for the Behavioral and Social Sciences(Distributed by US Department of Commerce, Springfield, VA).Google Scholar
  30. Kahneman, D., P. Slovic and A. Tversky (eds) (1982): Judgment under Uncertainty: Heuristics and Biases. New York: Cambridge University Press.Google Scholar
  31. Keefer, Donald L., Craig W. Kirkwood and James L. Corner (2004): Perspective on Decision Analysis Applications, 1990–2001. Decision Analysis, vol. 1(1), pp. 4–22 doi:10.1287/deca.1030.0004.CrossRefGoogle Scholar
  32. Kerr, N.L. and S.R. Tindale (2004): R.S. Group Performance and Decision Making. Annual Review of Psychology, vol. 55, pp. 623–655 doi:10.1146/annurev.psych.55.090902.142009.CrossRefGoogle Scholar
  33. Kerr, Norbert L., R.J. MacCoun and G.P. Kramer (1996): Bias in Judgment: Comparing Individuals and Groups. Psychological Review, vol. 103(4), pp. 687–719 doi:10.1037/0033-295X.103.4.687.CrossRefGoogle Scholar
  34. Kerstholt, J.H. and J.L. Jackson (1998): Judicial Decision Making: Order of Evidence Presentation and Availability of Background Information. Applied Cognitive Psychology, vol. 12, pp. 445–454.CrossRefGoogle Scholar
  35. Kiesler, S. and L. Sproull (1992): Group Decision Making and Communication Technology. Organizational Behavior and Human Decision Processes, vol. 52, pp. 96–123 doi:10.1016/0749-5978(92)90047-B.CrossRefGoogle Scholar
  36. Klayman, J. and Y.-W. Ha (1987): Confirmation, Disconfirmation, and Information in Hypothesis Testing. Psychological Review, vol. 94(2), pp. 211–228 doi:10.1037/0033-295X.94.2.211.CrossRefGoogle Scholar
  37. Kraut, R.E. (2003): Applying Social Psychological Theory to the Problems of Group Work. In John M. Carroll (ed): HCI Models, Theories, and Frameworks: Toward a Multidisciplinary ScienceNew York: Morgan Kaufmann, pp. 325–356.CrossRefGoogle Scholar
  38. Larson, J.R. Jr., P.G. Foster-Fishman and C.B. Keys (1994): Discussion of Shared and Unshared Information in Decision-making Groups. Journal of Personality and Social Psychology, vol. 67, pp. 446–461 doi:10.1037/0022-3514.67.3.446.CrossRefGoogle Scholar
  39. Lim, L.-H. and I. Benbasat (1997): The Debiasing Role of Group Support Systems: An Experimental Investigation of the Representativeness Bias. Int. J. Human–Computer Studies, vol. 47, pp. 453–471 doi:10.1006/ijhc.1997.0137.CrossRefGoogle Scholar
  40. Lipshitz, R., G. Klein, J. Orasanu and E. Salas (2001): Taking stock of naturalistic decision making. Journal of Behavioral Decision Making, vol. 14(5), pp. 331–352 doi:10.1002/bdm.381.CrossRefGoogle Scholar
  41. Neale, D.C., J.M. Carroll and R.M. Beth (2004): Evaluating Computer-supported Cooperative WOrk: Models And Frameworks. In Proceedings of the ACM conference on Computer Supported Cooperative Work, Chicago, IL.Google Scholar
  42. Nunamaker, J.F. Jr., A.R. Dennis, J.S. Valacich, D.R. Vogel and J.F. George (1991): Electronic Meeting Systems to Support Group Work. Communications of the ACM, vol. 34(7), pp. 40–61.CrossRefGoogle Scholar
  43. Perrin, B.M., B.J. Barnett, L. Walrath and J.D. Grossman (2001): Information Order and Outcome Framing: An Assessment of Judgment Bias, In a Naturalistic Decision-Making Context. Human Factors, vol. 43(2), pp. 227–238 doi:10.1518/001872001775900968.CrossRefGoogle Scholar
  44. Pirolli, P., T. Lee and Stuart K. Card (2004): Leverage Points for Analyst Technology Identified through Cognitive Task Analysis Technical Report. Palo Alto, CA: PARC.Google Scholar
  45. Pirolli, P., L. Good, J. Heiser, J. Shrager and S. Hutchins (2005): UIR Technical Report. Palo Alto, CA: PARC.Google Scholar
  46. Reagan-Cirincione, P. (1994): Improving the accuracy of group judgment: a process intervention combining group facilitation, social judgment analysis, and information technology. Organizational Behavior and Human Decision Processes, vol. 58, pp. 246–270 doi:10.1006/obhd.1994.1036.CrossRefGoogle Scholar
  47. Russell, D.M., M.J. Stefik, P. Pirolli, and S.K. Card (1993): The Cost Structure of Sensemaking. In: Proceedings of the SIGCHI conference on Human factors in computing systems, April 24–29, 1993, Amsterdam, The Netherlands, pp. 269–276.Google Scholar
  48. Scholtz, J., E. Morse and P.S. Michelle (2006): Evaluation metrics and methodologies for user-centered evaluation of intelligent systems. Interacting with Computers, vol. 18, pp. 1186–1214 doi:10.1016/j.intcom.2006.08.014.CrossRefGoogle Scholar
  49. Schultz-Hart, S., D. Frey, C. Lüthgens and S. Moscovici (2000): Biased Information Search in Group Decision Making. Journal of Personality and Social Psychology, vol. 78(4), pp. 655–669 doi:10.1037/0022-3514.78.4.655.CrossRefGoogle Scholar
  50. Shanteau, J. (1992): Competence in Experts: The Role of Task Characteristics. Organizational Behavior and Human Decision Processes, vol. 53, pp. 252–266 doi:10.1016/0749-5978(92)90064-E.CrossRefGoogle Scholar
  51. Shrager, J. (2005): CACHE; The Collaborative Analysis of Competing Hypotheses Environment [computer software]. Palo Alto, CA: Xerox Palo Alto Research Center.Google Scholar
  52. Shrager, J., D. Billman, G. Convertino, J.P. Massar and P. Pirolli (in review): CACHE: Web-based support for distributed multi-stage inference.Google Scholar
  53. Smallman, H.S. (2008): JIGSAW – Joint Intelligence Graphical Situation Awareness Web for Collaborative Intelligence Analysis. In M.P. Letsky, N.W. Warner, S.M. Fiore and C.A.P. Smith (eds.): Macrocognition in Teams: Theories and Methodologies. Ashgate Publishing Limited, Hampshire, England, pp. 321–337 (in press).Google Scholar
  54. Stasser, G. and W. Titus (2003): Hidden Profiles: A Brief History. Psychological Inquiry, vol. 14(3–4), pp. 302–311.Google Scholar
  55. Straus, S.G. and J.E. McGrath (1994): Does the Medium Matter? The Interaction of Task Type and Technology on Group Performance and Member Reactions. The Journal of Applied Psychology, vol. 79(1), pp. 87–89 doi:10.1037/0021-9010.79.1.87.CrossRefGoogle Scholar
  56. Tolcott, M.A., F.F. Marvin and P.E. Lehner (1989): Expert Decisionmaking in Evolving Situations. IEEE Transact. on Systems. Man, and Cybernetics, vol. 19(3), pp. 606–615 doi:10.1109/21.31066.CrossRefGoogle Scholar
  57. Trent, S.A., E.S. Patterson and D.D. Woods (2007): Challenges for Cognition. J. of Cognitive Engineering and Decision Making, vol. 1(1), pp. 75–97.Google Scholar
  58. Tversky, A. and D. Kahneman (1974): Judgments under Uncertainty. Heuristics and Biases. Science, vol. 185, pp. 1124–1131 doi:10.1126/science.185.4157.1124.CrossRefGoogle Scholar
  59. van Knippenberg, D. and M.C. Schippers (2007): Work Group Diversity. Annual Review of Psychology, vol. 58, pp. 515–541 doi:10.1146/annurev.psych.58.110405.085546.CrossRefGoogle Scholar
  60. Wason, P.C. (1960): On the Failure to Eliminate Hypotheses in a Conceptual Task. The Quarterly Journal of Experimental Psychology, vol. 12, pp. 129–140 doi:10.1080/17470216008416717.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2008

Authors and Affiliations

  • Gregorio Convertino
    • 1
  • Dorrit Billman
    • 2
    • 5
  • Peter Pirolli
    • 2
  • J. P. Massar
    • 3
  • Jeff Shrager
    • 4
  1. 1.College of Information Sciences and TechnologyPenn State UniversityUniversity ParkUSA
  2. 2.Palo Alto Research CenterPalo AltoUSA
  3. 3.BerkeleyUSA
  4. 4.Symbolic Systems ProgramStanford UniversityStanfordUSA
  5. 5.Center for Study of Language and InformationStanford UniversityStanfordUSA

Personalised recommendations