Computer Supported Cooperative Work (CSCW)

, Volume 21, Issue 4–5, pp 449–473 | Cite as

Enabling Large-Scale Deliberation Using Attention-Mediation Metrics

  • Mark KleinEmail author


Humanity now finds itself faced with a range of highly complex and controversial challenges—such as climate change, the spread of disease, international security, scientific collaborations, product development, and so on—that call upon us to bring together large numbers of experts and stakeholders to deliberate collectively on a global scale. Collocated meetings can however be impractically expensive, severely limit the concurrency and thus breadth of interaction, and are prone to serious dysfunctions such as polarization and hidden profiles. Social media such as email, blogs, wikis, chat rooms, and web forums provide unprecedented opportunities for interacting on a massive scale, but have yet to realize their potential for helping people deliberate effectively, typically generating poorly-organized, unsystematic and highly redundant contributions of widely varying quality. Large-scale argumentation systems represent a promising approach for addressing these challenges, by virtue of providing a simple systematic structure that radically reduces redundancy and encourages clarity. They do, however, raise an important challenge. How can we ensure that the attention of the deliberation participants is drawn, especially in large complex argument maps, to where it can best serve the goals of the deliberation? How can users, for example, find the issues they can best contribute to, assess whether some intervention is needed, or identify the results that are mature and ready to “harvest”? Can we enable, for large-scale distributed discussions, the ready understanding that participants typically have about the progress and needs of small-scale, collocated discussions?. This paper will address these important questions, discussing (1) the strengths and limitations of current deliberation technologies, (2) how argumentation technology can help address these limitations, and (3) how we can use attention-mediation metrics to enhance the effectiveness of large-scale argumentation-based deliberations.

Key words

Deliberation Metrics Argumentation 



The author would like to gratefully acknowledge the many useful conversations he has had on the topic of deliberation metrics with Prof Ali Gurkan (Ecole Centrale Paris), Prof. Luca Iandoli (University of Naples), and Prof. Haji Reijers (Eindhoven University of Technology).

The work has been supported by the National Science Foundation.


  1. Adomavicius, G., & Tuzhilin, A. (2005). Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions. IEEE Transactions on Knowledge and Data Engineering, 17(6), 734–749.CrossRefGoogle Scholar
  2. Benbasat, I., & Lim, J. (2000). Information technology support for debiasing group judgments: an empirical evaluation. Organizational Behavior and Human Decision Processes, 83(1), 167–183.CrossRefGoogle Scholar
  3. Benkler, Y. (2006). The Wealth of Networks: How Social Production Transforms Markets and Freedom. Yale University Press.Google Scholar
  4. Boland, R. J., Maheshwari, A. K., Te’eni, D., Schwartz, D., & Tenkasi, R. V. (1992). Sharing Perspectives in Distributed Decision Making. Computer-Supported Cooperative Work.Google Scholar
  5. Bolstad, W. M. (2010). Understanding Computational Bayesian Statistics. John Wiley.Google Scholar
  6. Bonaccorsi, A., & Rossi, C. (2004). Altruistic individuals, selfish firms? The structure of motivation in Open Source software.Google Scholar
  7. Cappella, J. N., Price, V., & Nir, L. (2002). Argument Repertoire as a Reliable and Valid Measure of Opinion Quality: Electronic Dialogue During Campaign 2000. Political Communication, 19(1), 73–93.CrossRefGoogle Scholar
  8. Carr, C. S. (2003). Using computer supported argument visualization to teach legal argumentation. In P. A. Kirschner, S. J. B. Shum, & C. S. Carr (Eds.), Visualizing argumentation: software tools for collaborative and educational sense-making (pp. 75–96). Springer-Verlag.Google Scholar
  9. Chklovski, T., Ratnakar, V., & Gil, Y. (2005). User interfaces with semi-formal representations: a study of designing argumentation structures. Proceedings of the 10th international conference on Intelligent user interfaces, 130–136.Google Scholar
  10. Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on socially shared cognition (pp. 127–149). Washington, DC, US: American Psychological Association.CrossRefGoogle Scholar
  11. Conklin, J. (2005). Dialogue Mapping: Building Shared Understanding of Wicked Problems. John Wiley and Sons, Ltd.Google Scholar
  12. Convertino, G., Billman, D., Shrager, J., Pirolli, P., & Massar, J. P. (2008). The CACHE Study: Group Effects in Computer-Supported Collaborative Analysis. Journal of Computer Supported Cooperative Work, 17(4), 353–393.CrossRefGoogle Scholar
  13. Cook, M. B., & Smallman, H. S. (2007). Visual Evidence Landscapes: Reducing Bias in Collaborative Intelligence Analysis. Human Factors and Ergonomics Society Annual Meeting Proceedings.Google Scholar
  14. Dennis, A. R., & Valacich, J. S. (1993). Computer brainstorms: More heads are better than one. Journal of Applied Psychology, 78(4), 531–537.CrossRefGoogle Scholar
  15. Eemeren, F. H. v., & Grootendorst, R. (2003). A Systematic Theory of Argumentation: The Pragma-dialectical Approach. Cambridge University Press.Google Scholar
  16. Erickson, T., Halverson, C., Kellogg, W. A., Laff, M., & Wolf, T. (2002). Social Translucence: Designing Social Infrastructures that Make Collective Activity Visible. Communications of the ACM, 45(4), 40–44.CrossRefGoogle Scholar
  17. Farnham, S., Chesley, H. R., McGhee, D. E., Kawal, R., & Landau, J. (2000). Structured online interactions: improving the decision-making of small discussion groups. Computer Supported Cooperative Work.Google Scholar
  18. Finholt, T. A. (2002). Collaboratories. Annual Review of Information Science and Technology, 36(1), 73–107.CrossRefGoogle Scholar
  19. Gladwell, M. (2002). The Tipping Point: How Little Things Can Make a Big Difference. Back Bay Books.Google Scholar
  20. Gopal, A., & Prasad, P. (2000). Understanding GDSS in Symbolic Context: Shifting the Focus from Technology to Interaction. MIS Quarterly, 24(3), 509–546.CrossRefGoogle Scholar
  21. Group, R. D. F. D. A. W. (2008). SPARQL Query Language for RDF.
  22. Hars, A., & Ou, S. (2002). Working for Free? Motivations for Participating in Open-Source Projects. International Journal of Electronic Commerce, 6(3), 25–39.Google Scholar
  23. Heng, M. S. H., & de Moor, A. (2003). From Habermas' s communicative theory to practice on the internet. Information Systems Journal, 13(4), 331–352.CrossRefGoogle Scholar
  24. Householder, A. S. (1958). Unitary Triangularization of a Nonsymmetric Matrix. Journal of the ACM, 5(4), 339–342.MathSciNetzbMATHCrossRefGoogle Scholar
  25. Iandoli, L., Klein, M., & Zollo, G. (2009). Enabling on-line deliberation and collective decision-making through large-scale argumentation: a new approach to the design of an Internet-based mass collaboration platform. International Journal of Decision Support System Technology, 1(1), 69–91.CrossRefGoogle Scholar
  26. Johnson, E. M., & Halpin, S. M. (1974). Multistage inference models for intelligence analysis.Google Scholar
  27. Jonassen, D., & Jr, H. R. (2005). Mapping alternative discourse structures onto computer conferences. International Journal of Knowledge and Learning, 1(1/2), 113–129.CrossRefGoogle Scholar
  28. Karacapilidis, N., Loukis, E., & Dimopoulos, S. (2004). A Web-Based System for Supporting Structured Collaboration in the Public Sector. LECTURE NOTES IN COMPUTER SCIENCE, 218–225.Google Scholar
  29. Kirschner, P. A., Shum, S. J. B., & Carr, C. S. (2003). Visualizing Argumentation: Software tools for collaborative and educational sense-making. Springer.Google Scholar
  30. Kittur, A., Suh, B., Pendleton, B. A., & Chi, E. H. (2007). He says, she says: conflict and coordination in Wikipedia. SIGCHI Conference on Human Factors in Computing Systems.Google Scholar
  31. Klein, M. (2003). A Knowledge-Based Methodology for Designing Reliable Multi-Agent Systems. In P. Giorgini, J. P. Mueller, & J. Odell (Eds.), Agent-Oriented Software Engineering IV (Vol. 2935, pp. 85–95). Springer-Verlag.Google Scholar
  32. Klein, M. (2007). The MIT Collaboratorium: Enabling Effective Large-Scale Deliberation for Complex Problems.Google Scholar
  33. Klein, M., & Bernstein, A. (2004). Towards High-Precision Service Retrieval. IEEE Internet Computing Journal, 8(1), 30–36.CrossRefGoogle Scholar
  34. Klein, M., & Iandoli, L. (2008). Supporting Collaborative Deliberation Using a Large-Scale Argumentation System: The MIT Collaboratorium. Directions and Implications of Advanced Computing; Conference on Online Deliberation (DIAC-2008/OD2008).Google Scholar
  35. Klein, M., Sayama, H., Faratin, P., & Bar-Yam, Y. (2003). The Dynamics of Collaborative Design: Insights From Complex Systems and Negotiation Research. Concurrent Engineering Research & Applications, 11(3), 201–210.CrossRefGoogle Scholar
  36. Kramer, K., & King, J. (1988). Computer-Based Systems for Coperative Work and Group Decision Making. Computing Surveys, 20(June), 115–146.CrossRefGoogle Scholar
  37. Lakhani, K. R., & Wolf, R. G. (2005). Why Hackers Do What They Do: Understanding Motivation and Effort in Free/Open Source Software Projects. In J. Feller, B. Fitzgerald, S. Hissam, & K. R. Lakhani (Eds.), Perspectives on Free and Open Source Software. MIT Press.Google Scholar
  38. Lowrance, J. D., Harrison, I. W., & Rodriguez, A. C. (2001). Capturing Analytic Thought. First International Conference on Knowledge Capture, 84–91.Google Scholar
  39. Luppicini, R. (2007). Review of computer mediated communication research for education. Instructional Science, 35(2), 141–185.CrossRefGoogle Scholar
  40. Macaulay, L. A., & Alabdulkarim, A. (2005). Facilitation of e-Meetings: State-of-the-Art Review. e-Technology, e-Commerce and e-Service (EEE’05).Google Scholar
  41. Moor, A. d., & Aakhus, M. (2006). Argumentation Support: From Technologies to Tools. Communications of the ACM, 49(3), 93.CrossRefGoogle Scholar
  42. Nisbet, D. (2004). Measuring the Quantity and Quality of Online Discussion Group Interaction. Journal of eLiteracy, 1, 122–139.Google Scholar
  43. Pervan, G. P., & Atkinson, D. J. (1995). GDSS research: An overview and historical analysis. Group Decision and Negotiation, 4(6), 475–483.CrossRefGoogle Scholar
  44. Poole, M. S., Holmes, M., & DeSanctis, G. (1988). Conflict Management and Group Decision Support Systems. Proceedings from Proceedings of Computer Supported Cooperative Work.Google Scholar
  45. Powell, A., Piccoli, G., & Ives, B. (2004). Virtual teams: a review of current literature and directions for future research. ACM SIGMIS Database, 35(1), 6–36.CrossRefGoogle Scholar
  46. Rahwan, I. (2008). Mass argumentation and the semantic web. Journal of Web Semantics, 6(1), 29–37.CrossRefGoogle Scholar
  47. Reagan-Cirincione, P. (1994). Improving the accuracy of group judgment: a process intervention combining group facilitation, social judgment analysis, and information technology. Organizational Behavior and Human Decision Processes, 58(2), 246–270.CrossRefGoogle Scholar
  48. Roberts, J. E. F. F. R. E. Y., Hann, I. L.-H. O. R. N., & Slaughter, S. A. N. D. R. A. (2006). Understanding the Motivations, Participation and Performance of Open Source Software Developers: A Longitudinal Study of the Apache Projects. Management Science, 52(7), 984–999.CrossRefGoogle Scholar
  49. Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006). Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market. Science, 311(5762), 854–856.CrossRefGoogle Scholar
  50. Schulz-Hardt, S., Frey, D., Lüthgens, C., & Moscovici, S. (2000). Biased information search in group decision making. Journal of Personality and Social Psychology, 78(4), 655–669.CrossRefGoogle Scholar
  51. Shrager, J., Billman, D., Convertino, G., Massar, J. P., & Pirolli, P. (2010). Soccer Science and the Bayes Community: Exploring the Cognitive Implications of Modern Scientific Communication. Topics in Cognitive Science, 2(1), 53–72.CrossRefGoogle Scholar
  52. Shum, S. B., Liddo, A. D., Iandoli, L., & Quinto, I. (2012). A Debate Dashboard to Support the Adoption of Online Knowledge Mapping Tools. VINE Journal of information and Knowledge Management Systems.Google Scholar
  53. Shum, S. J. B., Selvin, A. M., Sierhuis, M., Conklin, J., & Haley, C. B. (2006). Hypermedia Support for Argumentation-Based Rationale: 15 Years on from gIBIS and QOC. In A. H. Dutoit, R. McCall, I. Mistrik, & B. Paech (Eds.), Rationale Management in Software Engineering. Springer-Verlag.Google Scholar
  54. Smallman, H. S. (2008). JIGSAW – Joint Intelligence Graphical Situation Awareness Web for collaborative intelligence analysis. In M. P. Letsky, N. Warner, S. Fiore, & C. A. P. Smith (Eds.), Macrocognition in Teams: Theories and Methodologies (pp. 321–337). Aldershot, UK.: Ashgate.Google Scholar
  55. Spatariu, A., Hartley, K., & Bendixen, L. D. (2004). Defining and Measuring Quality in Online Discussions. The Journal of Interactive Online Learning, 2(4).Google Scholar
  56. Steenbergen, M. R., Bachtiger, A., Sporndli, M., & Steiner, J. (2003). Measuring political deliberation: a discourse quality index. Comparative European Politics, 1(1), 21–48.CrossRefGoogle Scholar
  57. Stromer-Galley, J. (2007). Measuring deliberation’s content: A coding scheme. Journal of Public Deliberation, 3(1), 12.Google Scholar
  58. Sunstein, C. R. (2006). Infotopia: How Many Minds Produce Knowledge. Oxford University Press.Google Scholar
  59. Surowiecki, J. (2005). The Wisdom of Crowds. Anchor.Google Scholar
  60. Tapscott, D., & Williams, A. D. (2006). Wikinomics: How Mass Collaboration Changes Everything. Portfolio Hardcover.Google Scholar
  61. Trénel, M. (2004). Measuring the quality of online deliberation. Coding scheme 2.4. Social Science Research Center Berlin, Germany. Available at: www. wz-berlin. de/online-mediation/files/publications/quod_2_4. pdf.Google Scholar
  62. Tversky, A., & Kahneman, D. (1974). Judgments under uncertainty. Heuristics and biases. Science, 185(4157), 1124–1131.CrossRefGoogle Scholar
  63. Viegas, F. B., Wattenberg, M., & Dave, K. (2004). Studying cooperation and conflict between authors with history flow visualizations. SIGCHI conference on Human factors in computing systems.Google Scholar
  64. Walton, D. N. (2005). Fundamentals of Critical Argumentation (Critical Reasoning and Argumentation). Cambridge (MA): Cambridge University Press.CrossRefGoogle Scholar
  65. Walton, D. N., & Krabbe, E. C. W. (1995). Commitment in dialogue: Basic concepts of interpersonal reasoning. Albany, NY: State University of New York Press.Google Scholar

Copyright information

© Springer 2012

Authors and Affiliations

  1. 1.MIT Center for Collective IntelligenceCambridgeUSA

Personalised recommendations