Social Indicators Research

, Volume 139, Issue 2, pp 473–490 | Cite as

Using Quali-Quantitative Indicators for Assessing the Quality of Citizen Participation: A Study on Three Citizen Juries

  • Terri Mannarini
  • Angela Fedi


Over the last 3 decades, citizen involvement has become rather common in policymaking processes. Its rationale, as well as its potential benefits and limitations, are manifold. The literature on the evaluation of public participation is copious and it is crucial both to implement effective processes, and to achieve high-quality outcomes. Inspired by deliberative democracy theory, dialogue/fairness and knowledge/competence have been considered the two main criteria to assess the quality of deliberative processes. Based on the analysis of three citizen juries, the paper focuses on the process through which citizen deliberation occurs. Specifically, three properties related to dialogue, i.e., equity, cooperation, and cognitive openness, were treated as quality indicators of the deliberative process. Both quantitative and qualitative methods were used, and three sources of data utilized: (a) semi-structured interviews to jurors; (b) post-jury questionnaires; and (c) jurors’ conversational turns. Altogether, the analyses showed that despite the imbalance in participation, the deliberation process was perceived as fair. However, findings also suggested that the participatory setting did not promote the ability of participants to generate new collective knowledge.


Deliberation Citizen participation Evaluation Quality indicators 



This work benefits from a research project “The quality of deliberation” (PRIN) funded by the Ministry of Education, University and Research of the Italian Government.


  1. Abdel-Monem, T., Bingham, S., Marincic, J., & Tomkins, A. (2010). Deliberation and diversity: Perceptions of small group discussions by race and ethnicity. Small Group Research, 41, 746–776.CrossRefGoogle Scholar
  2. Abelson, J., Forest, P. G., Eyles, J., Smith, P., Martin, E., & Gauvin, F. P. (2003). Deliberations about deliberative methods: issues in the design and evaluation of public participation processes. Social Science and Medicine, 57(2), 239–251.CrossRefGoogle Scholar
  3. Asch, S. E. (1956). Studies of independence and conformity: I. A minority of one against a unanimous majority. Psychological Monographs General and Applied, 70(9), 1–70.CrossRefGoogle Scholar
  4. Beauvais, E., & Baechtiger, A. (2016). Taking the goals of deliberation seriously: a differentiated view on equality and equity in deliberative designs and processes. Journal of Public Deliberation, 12(2), 1–18.
  5. Beierle, T. C., & Cayford, J. (2002). Democracy in practice: Public participation in environmental decisions. Washington, D.C.: Resources for the Future.Google Scholar
  6. Beste, S. (2013). Contemporary trends of deliberative research: Synthesizing a new study agenda. Journal of Public Deliberation, 9(2).
  7. Bobbio, L. (2013). La qualità della deliberazione [The quality of deliberation]. Roma: Carocci.Google Scholar
  8. Botes, L., & van Rensburg, D. (2000). Community participation in development: Nine plagues and twelve commandments. Community Development Journal, 1, 41–58.CrossRefGoogle Scholar
  9. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77–101.CrossRefGoogle Scholar
  10. Buchy, M., & Hoverman, S. (2000). Understanding public participation: A review. Forest Policy and Economics, 1, 15–25.CrossRefGoogle Scholar
  11. Burton, P. (2003). Community involvement in neighbourhood regeneration: Stairway to heaven or road to nowhere?. Paper n. 13, ESRC Centre for Neighbourhood Research.Google Scholar
  12. Canary, D. J., & Spitzberg, B. H. (1987). Appropriateness and effectiveness perceptions of conflict strategies. Human Communication Research, 14, 93–120.CrossRefGoogle Scholar
  13. Carnes, S. A., Schweitzer, M., Peelle, E. B., Wolfe, A. K., & Munro, J. F. (1998). Measuring the success of public participation on environmental restoration and waste management activities in the US Department of Energy. Technology in Society, 20(4), 385–406.CrossRefGoogle Scholar
  14. Chambers, S. (2003). Deliberative democratic theory. Annual Review of Political Science, 6, 307–326.CrossRefGoogle Scholar
  15. Charnley, S., & Engelbert, B. (2005). Evaluating public participation in environmental decision-making: EPA’s superfund community involvement program. Journal of Environmental Management, 77, 165–182.CrossRefGoogle Scholar
  16. Chavis, D. M., & Wandersman, A. (1990). Sense of community in the urban environment: A catalyst for participation and community development. American Journal of Community Psychology, 18, 55–81.CrossRefGoogle Scholar
  17. Chess, C., & Purcell, K. (1999). Public participation and the environment: Do we know what works? Environmental Science and Technology, 33(16), 2685–2692.CrossRefGoogle Scholar
  18. Coelho, V. S. R. P., & Waisbich, L. (2016). Participatory mechanisms and inequality reduction: searching for plausible relations. Journal of Public Deliberation, 12(2).
  19. Cohen, J. (1996). Procedure and substance in deliberative democracy. In S. Benhabib (Ed.), Democracy and difference: Contesting the boundaries of the political (pp. 95–119). Princeton, NJ: Princeton University Press.Google Scholar
  20. Cohen, J. (1997). Deliberation and democratic legitimacy. In J. F. Bohman & W. Rehg (Eds.), Deliberative democracy: Essays on reason and politics (pp. 67–91). Cambridge, MA: MIT Press.Google Scholar
  21. Crosby, N. (1995). Citizens juries: One solution for difficult environmental questions. In O. Renn, T. Webler, & P. Wiedemann (Eds.), Fairness and competence in citizen participation (pp. 157–174). London: Kluwer Academic Publishers.CrossRefGoogle Scholar
  22. de Castro-Silva, C. R., & Cavichioli, S. (2013). La participación politíca en una ONG/SIDA y las posibilidades de emancipación en un contexto de exclusión social. Revista Interamericana de Psicología/Interamerican Journal of Psychology, 47, 9–16.Google Scholar
  23. De Vries, R., Stanczyk, A., Wall, I. F., Uhlmann, R., Damschroder, L. J., & Kim, S. Y. (2010). Assessing the quality of democratic deliberation: A case study of public deliberation on the ethics of surrogate consent for research. Social Science and Medicine, 70(12), 1896–1903.CrossRefGoogle Scholar
  24. Delli Carpini, M. X., Cook, F. L., & Jacobs, L. R. (2004). Public deliberation, discursive participation, and citizen engagement: A review of the empirical literature. Annual Review of Political Science, 7, 315–344.CrossRefGoogle Scholar
  25. Dunkerley, D., & Glasner, P. (1998). Empowering the public? Citizens juries and the new genetic technologies. Critical Public Health, 8, 181–192.CrossRefGoogle Scholar
  26. Edwards, P., Hindmarsh, R., Merer, H., Bond, M., & Rowland, A. (2008). A three-stage evaluation of a deliberative event on climate change and transforming energy. Journal of Public Deliberation, 4(1).
  27. Font, J., & Blanco, I. (2007). Procedural legitimacy and political trust: The case of citizen juries in Spain. European Journal of Political Research, 46, 557–589.CrossRefGoogle Scholar
  28. Gastil, J., & Black, L. (2008). Public deliberation as the organizing principle of political communication research. Journal of Public Deliberation, 4(1).
  29. Goodin, R. E., & Dryzek, J. S. (2006). Deliberative impacts: The macro-political uptake of mini-publics. Politics and Society, 34, 219–244.CrossRefGoogle Scholar
  30. Grice, P. (1975). Logic and conversation. In P. Cole & J. Morgan (Eds.), Syntax and semantics: Speech acts (Vol. 3, pp. 41–58). New York: Academic Press.Google Scholar
  31. Habermas, J. (1984). The theory of communicative action. Reason and the rationalization of society. Boston: Beacon Press.Google Scholar
  32. Habermas, J. (1989). The structural transformation of the public sphere. Cambridge, MA: MIT Press.Google Scholar
  33. Habermas, J. (1996). Between facts and norms: Contributions to a discourse theory of law and democracy. Cambridge, MA: MIT Press.Google Scholar
  34. Huntington, S. P. (1991). Democracy’s third wave. The Journal of Democracy, 2, 12–34.CrossRefGoogle Scholar
  35. Jackson, D. N. (1984). Personality research form. Manual. Port Huron, MI: Research Psychologists Press.Google Scholar
  36. Janis, I. L. (1982). Groupthink: Psychological studies of policy decisions and fiascos (2nd ed.). New York, NY: Houghton Mifflin.Google Scholar
  37. Kadlec, A., & Friedman, W. (2007). Deliberative democracy and the problem of power. Journal of Public Deliberation, 3(1).
  38. Karpowitz, C. F., & Mansbridge, J. (2005). Disagreement and consensus: The need for dynamic updating in public deliberation. Journal of Public deliberation, 1(1), 348–364.
  39. Keyes, C. L. M. (1998). Social well-being. Social Psychological Quarterly, 61(2), 121–140.CrossRefGoogle Scholar
  40. King, R. C., Hartzel, K. S., Schilhavy, R. A., Melone, N. P., & McGuire, T. W. (2010). Social responsibility and stakeholder influence: Does technology matter during stakeholder deliberation with high-impact decisions? Decision Support Systems, 48(4), 536–547.CrossRefGoogle Scholar
  41. Klijn, E. H., & Koppenjan, J. F. M. (2000). Politicians and interactive decision making: Institutional spoilsports or playmakers. Public Administration, 2, 365–387.CrossRefGoogle Scholar
  42. Korfmacher, K. S. (2001). The politics of participation in watershed modeling. Environment Management, 27, 161–176.CrossRefGoogle Scholar
  43. Kroon, M., van Kreveld, D., & Rabbie, J. (1992). Group versus individual decision making. Effects of accountability and gender on groupthink. Small Group Research, 4, 427–458.CrossRefGoogle Scholar
  44. Kruglanski, A., Raviv, A., Bar-Tal, D., Raviv, A., Sharvit, K., Ellis, S., et al. (2005). Says who?: Epistemic authority effects in social judgment. In M. P. Zanna (Ed.), Advances in experimental social psychology (Vol. 37, pp. 345–392). New York: Academic Press.Google Scholar
  45. Kruglanski, A., & Webster, D. (1996). Motivated closing of mind: “Seizing” and “freezing”. Psychological Review, 2, 263–283.CrossRefGoogle Scholar
  46. Ledwith, M., & Springett, J. (2010). Participatory practice. Bristol, UK: Policy Press.Google Scholar
  47. Linnel, P., & Luckman, T. (1991). Asymmetries in dialogue: Some conceptual preliminaries. In I. Markova & K. Foppa (Eds.), Asymmetries in dialogue (pp. 1–20). Hemel Hempstead: Harvester Wheatsheaf.Google Scholar
  48. Linnell, P., Gustavsson, L., & Juvonen, P. (1988). Interactional dominance in dyadic communication: A presentation of initiative-response analysis. Linguistics, 26, 415–442.Google Scholar
  49. Mannarini, T. (2011). Public involvement and competent communities: Towards a social psychology of public participation. International Journal of Humanities and Social Sciences, 1(7), 66–72.Google Scholar
  50. Mannarini, T. (2014). Riding paradox: Lessons learned from Italian participatory policy-making experiences. Interamerican Journal of Psychology, 48, 71–81.Google Scholar
  51. Mendelberg, T. (2002). The deliberative citizen: Theory and evidence. In M. X. Delli Carpini, L. Huddy, & R. Y. Shapiro (Eds.), Political decision-making, deliberation and participation. Research in Micropolitics (Vol. 6, pp. 151–194). Amsterdam Boston: Emerald Group Publishing.Google Scholar
  52. Mendelberg, T. (2006). Small group deliberation. Paper presented at the Annual Meeting of the American Political Science Association, Philadelphia, PA.Google Scholar
  53. Montero, M. (2004). Introducción a la Psicología comunitaria. Buenos Aires: Paidos.Google Scholar
  54. Moscovici, S., & Doise, W. (1991). Dissensus et consensus. Une theéorie générale des decisions collectives. Paris: Presses Universitaires de France.Google Scholar
  55. Moscovici, S., & Zavalloni, M. (1969). The group as a polarizer of attitudes. Journal of Personality and Social Psychology, 12(2), 125–135.CrossRefGoogle Scholar
  56. Moscrop, D. R., & Warren, M. E. (2016). When is Deliberation democratic?. Journal of Public Deliberation, 12(2).
  57. Nemeth, C. J. (1986). The differential contributions of majority and minority influence. Psychological Review, 93, 23–32.CrossRefGoogle Scholar
  58. Noelle-Neuman, E. (1984). The spiral of silence. A theory of public opinion. Chicago: University of Chicago Press.Google Scholar
  59. Nussbaum, M. (1999). Sex & social justice. New York: Oxford University Press.Google Scholar
  60. Papadopulos, Y., & Warin, P. (2007). Are innovative, participatory and deliberative procedures in policy-making democratic and effective? European Journal of Political Research, 46, 445–472.CrossRefGoogle Scholar
  61. Prestby, J., Wandersman, A., Florin, P., Rich, R., & Chavis, D. M. (1990). Benefits, costs, incentives management and participation in voluntary associations. American Journal of Community Psychology, 18, 117–149.CrossRefGoogle Scholar
  62. Putnam, R. D. (2000). Bowling alone: The collapse and revival of american community. New York: Simon & Schuster.CrossRefGoogle Scholar
  63. Radcliff, B., & Shufeldt, G. (2016). Direct democracy and subjective well-being: The initiative and life satisfaction in the American States. Social Indicators Research, 128(3), 1405–1423.CrossRefGoogle Scholar
  64. Rondinella, T., Segre, E., & Zola, D. (2017). Participative processes for measuring progress: Deliberation, consultation and the role of civil society. Social Indicators Research, 130(3), 959–982.CrossRefGoogle Scholar
  65. Rowe, G., & Frewer, L. J. (2000). Public participation methods: A framework for evaluation. Science, Technology and Human Values, 25(1), 3–29.CrossRefGoogle Scholar
  66. Rowe, G., & Frewer, L. J. (2004). Evaluating public-participation exercises: A research agenda. Science, Technology and Human Values, 29(4), 512–557.CrossRefGoogle Scholar
  67. Rowe, G., Marsh, R., & Frewer, L. (2004). Evaluation of a deliberative conference in science. Technology and Human Values, 29(1), 88–121.CrossRefGoogle Scholar
  68. Ryfe, D. M. (2005). Does deliberative democracy work? Annual Review of Political Science, 8(1), 49–71.CrossRefGoogle Scholar
  69. Sanders, L. (1997). Against deliberation. Political Theory, 25(3), 347–376.CrossRefGoogle Scholar
  70. Steiner, J., Steenbergen, M. R., Bachtiger, A., & Spörndli, M. (2003). Measuring political deliberation: A discourse quality index. Comparative European Politics, 1, 21–48.CrossRefGoogle Scholar
  71. Stephens, J. B., & Berner, M. (2011). Learning from your neighbor: The value of public participation evaluation for public policy dispute resolution. Journal of Public Deliberation, 7(1), art. 10,
  72. Stoner, J. A. F. (1968). Risky and cautious shifts in group decisions: The influence of widely held values. Journal of Experimental Social Psychology, 4(4), 442–459.CrossRefGoogle Scholar
  73. Street, J., Duszynski, K., Krawczyk, S., & Braunack-Mayer, A. (2013). Citizens’ juries in health policy decision-making: A systematic review. Social Science and Medicine, 109, 1–9.CrossRefGoogle Scholar
  74. Stromer-Galley, J. (2007). Measuring deliberation’s content: A coding scheme. Journal of Public Deliberation, 3(1).
  75. Sunstein, C. (2000). Deliberative Trouble? Why groups go to extremes. The Yale Law Journal, 1, 71–119.CrossRefGoogle Scholar
  76. Sunstein, C. (2002). The law of group polarization. The Journal of Political Philosophy, 2, 175–195.CrossRefGoogle Scholar
  77. Sunstein, C. (2005). Group judgments: Statistical means, deliberation and information markets. New York University Law Review, 80, 962–1049.Google Scholar
  78. Sunstein, C. & Hastie, R. (2008). Fur failures of deliberating groups. Public law and legal theory working paper series. Retrieved April 28, 2017 from
  79. The Jefferson Center (2004). Citizens jury handbook. Washington, DC. Accessed July 5, 2016.
  80. Thomson, A. M., & Perry, J. L. (2006). Collaboration processes: Inside the black box. Public Administration Review, 66(s1), 20–32.CrossRefGoogle Scholar
  81. Wandersman, A., & Florin, P. (2000). Citizen participation and community organizations. In J. Rappaport & E. Seidman (Eds.), Handbook of community psychology (pp. 247–272). New York, NY: Kluwer Academic/Plenum Press.CrossRefGoogle Scholar
  82. Webler, T. (1995). ‘Right’ discourse in citizen participation: An evaluative yardstick. In O. Renn, T. Webler, & P. Wiedemann (Eds.), Fairness and competence in citizen participation: Evaluating models for environmental discourse (pp. 35–86). Boston; MA: Kluwer Academic.CrossRefGoogle Scholar
  83. Webler, T., & Tuler, S. (2001). Public participation in watershed management planning: Views on process from people in the field. Human Ecology Review, 8(2), 29–39.Google Scholar
  84. Wollabæk, D., & Selle, P. (2003). Participation and social capital formation: Norway in a comparative perspective. Scandinavian Political Studies, 26(1), 67–91.CrossRefGoogle Scholar
  85. Young, I. M. (2000). Democracy and inclusion. New York, NY: Oxford University Press.Google Scholar
  86. Zimmerman, M. A., & Rappaport, J. (1988). Citizen participation, perceived control, and psychological empowerment. American Journal of Community Psychology, 5, 725–750.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2017

Authors and Affiliations

  1. 1.Department of History, Society and Human StudiesUniversity of SalentoLecceItaly
  2. 2.Department of PsychologyUniversity of TorinoTurinItaly

Personalised recommendations