Abstract
This article investigates how certainty and timing of evidence introduction impact the uptake of evidence by policy-makers in collective deliberations. Little is known about how experts or researchers should time the introduction of uncertain evidence for policy-makers. With a computational model based on the Hegselmann–Krause opinion dynamics model, we simulate how policy-makers update their opinions in light of new evidence. We illustrate the use of our model with two examples in which timing and certainty matter for policy-making: intelligence analysts scouting potential terrorist activity and food safety inspections of chicken meat. Our computations indicate that evidence should come early to convince policy-makers, regardless of how certain it is. Even if the evidence is quite certain, it will not convince all policy-makers. Next to its substantive contribution, the article also showcases the methodological innovation that agent-based models can bring for a better understanding of the science–policy nexus. The model can be endlessly adapted to generate hypotheses and simulate interactions that cannot be empirically tested.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ascough, J., II., Maier, H., Ravalico, J., & Strudley, M. (2008). Future research challenges for incorporation of uncertainty in environmental and ecological decision-making. Ecological Modelling, 219(3–4), 383–399.
Bammer, G., & Smithson, M. (2012). Uncertainty and Risk: Multidisciplinary Perspectives. Routledge.
Bandola-Gill, J., Arthur, M., & Ivor Leng, R. (2022). What is co-production? Conceptualising and understanding co-production of knowledge and policy across different theoretical perspectives. Evidence & Policy, pp. 1–24.
Botterill, L. C., & Hindmoor, A. (2012). Turtles all the way down: Bounded rationality in an evidence-based age. Policy Studies, 33(5), 367–379.
Boyer, J. F., & Langbein, L. I. (1991). Factors influencing the use of health evaluation research in congress. Evaluation Review, 15(5), 507–532.
Brase, G. L., & Hill, W. T. (2015). Good fences make for good neighbors but bad science: A review of what improves Bayesian reasoning and why. Frontiers in Psychology, 6, 340.
Burns, K. (2015). Bayesian HELP: Assisting inferences in all-source intelligence. 2015 AAAI Fall Symposium Series.
Cairney, P., Oliver, K., & Wellstead, A. (2016). To bridge the divide between evidence and policy: Reduce ambiguity as much as uncertainty. Public Administration Review, 76(3), 399–402.
Capano, G., & Malandrino, A. (2022). Mapping the use of knowledge in policymaking: Barriers and facilitators from a subjectivist perspective (1990–2020). Policy Sciences, 55(3), 399–428.
Caplan, N. (1979). The two-communities theory and knowledge utilization. American Behavioral Scientist, 22(3), 459–470.
Choi, T., & Park, S. (2021). Theory building via agent-based modeling in public administration research: Vindications and limitations. International Journal of Public Sector Management, 34(6), 614–629.
Claver, A., & van de Meeberg, H. M. (2021). Devil’s advocacy within Dutch military intelligence (2008–2020): An effective instrument for quality assurance? Intelligence and National Security, 36(6), 849–862.
Cousins, J. B., & Leithwood, K. A. (1986). Current empirical research on evaluation utilization. Review of Educational Research, 56(3), 331–364.
Dobbins, M., Cockerill, R., Barnsley, J., & Ciliska, D. (2001). Factors of the innovation, organization, environment, and individual that predict the influence five systematic reviews had on public health decisions. International Journal of Technology Assessment in Health Care, 17(4), 467–478.
Douven, I. (2019). Computational models in social epistemology. The Routledge Handbook of Social Epistemology (pp. 457–465). Routledge.
Douven, I., & Wenmackers, S. (2017). Inference to the best explanation versus Bayes’s rule in a social setting. The British Journal for the Philosophy of Science, 68(2), 535–570.
Doyle, E. E., McClure, J., Paton, D., & Johnston, D. M. (2014). Uncertainty and decision making: Volcanic crisis scenarios. International Journal of Disaster Risk Reduction, 10, 75–101.
Ebel, E. D., Williams, M. S., Golden, N. J., & Tankson, J. (2016). Bayesian techniques for comparison of the test performance of PCR and culture for the identification of Campylobacter in enriched comminuted chicken samples. Journal of Applied Microbiology, 120(5), 1418–1426.
Elias, N. (2006). An Essay on Time. University College Dublin Press.
Foley, R. (2009). Beliefs, degrees of belief, and the Lockean thesis. Degrees of Belief (pp. 37–47). Springer.
Friedman, J. A., & Zeckhauser, R. (2012). Assessing uncertainty in intelligence. Intelligence and National Security, 27(6), 824–847.
Gabriel, N., & O’Connor, C. (2022). Can confirmation bias improve group learning? (MetaArXiv preprint https://doi.org/10.31222/osf.io/dzych)
Gitomer, D., & Crouse, K. (2019). Studying the use of research evidence: A review of methods. New York, NY: William T. Grant Foundation. Retrieved March, 17 , 2019.
Gustafson, A., & Rice, R. E. (2020). A review of the effects of uncertainty in public science communication. Public Understanding of Science, 29(6), 614–633.
Hartmann, K., & Wenzelburger, G. (2021). Uncertainty, risk and the use of algorithms in policy decisions: A case study on criminal justice in the USA. Policy Sciences, 54(2), 269–287.
Haukkala, H. (2013). Timing is everything: The time, space, and strategies for scholarly analysis in the making of foreign policy. International Studies Perspectives, 14(2), 221–233.
Hedström, P., Ylikoski, P., et al. (2010). Causal mechanisms in the social sciences. Annual Review of Sociology, 36(1), 49–67.
Hegselmann, R., & Krause, U. (2002). Opinion dynamics and bounded confidence models, analysis, and simulation. Journal of Artificial Societies and Social Simulation, 5 (3).
Heidelberg, R. L. (2020). The becoming of the policy maker. Administration & Society, 52(8), 1239–1267.
Ho, E. H., Budescu, D. V., Dhami, M. K., & Mandel, D. R. (2015). Improving the communication of uncertainty in climate science and intelligence analysis. Behavioral Science & Policy, 1(2), 43–55.
Hom, A. (2018). Timing is everything: Toward a better understanding of time and international politics. International Studies Quarterly, 62(1), 69–79.
Hom, A., & Beasley, R. (2021). Constructing time in foreign policymaking: Brexit’s timing entrepreneurs, malcontemps and apparatchiks. International Affairs, 97(2), 267–285.
Innvaer, S., Vist, G., Trommald, M., & Oxman, A. (2002). Health policy-makers’ perceptions of their use of evidence: A systematic review. Journal of Health Services Research & Policy, 7(4), 239–244.
Johnson, K., Greenseid, L. O., Toal, S. A., King, J. A., Lawrenz, F., & Volkov, B. (2009). Research on evaluation use: A review of the empirical literature from 1986 to 2005. American Journal of Evaluation, 30(3), 377–410.
Kahneman, D., & Tversky, A. (1972). Subjective probability: A judgment of representativeness. Cognitive Psychology, 3(3), 430–454.
Karvetski, C.W., & Mandel, D.R. (2020). Coherence of probability judgments from uncertain evidence: Does ACH help? Judgment & Decision Making, 15(6), 939–958.
Karvetski, C. W., Olson, K. C., Gantz, D. T., & Cross, G. A. (2013). Structuring and analyzing competing hypotheses with Bayesian networks for intelligence analysis. EURO Journal on Decision Processes, 1(3–4), 205–231.
Klein, G. (2011). Streetlights and Shadows: Searching for the Keys to Adaptive Decision Making. MIT Press.
Klein, G., Phillips, J.K., Rall, E.L., & Peluso, D.A. (2007). A data-frame theory of sensemaking. Expertise Out of Context (pp. 118–160). Psychology Press.
Lammers, W., Ferrari, S., Wenmackers, S., Pattyn, V.E., & Van de Walle, S. (2024) [Forthcoming]. Theories of uncertainty communication: an interdisciplinary literature review. Science Communication.
Leitgeb, H. (2015). I—The Humean thesis on belief. Aristotelian Society Supplementary Volume (Vol. 89, pp. 143–185).
Leman, C. K., & Nelson, R. H. (1981). Ten commandments for policy economists. Journal of Policy Analysis and Management, 1(1), 97–117.
Leviton, L. C., & Hughes, E. F. (1981). Research on the utilization of evaluations: A review and synthesis. Evaluation Review, 5(4), 525–548.
Lorenz, J. (2007). Continuous opinion dynamics under bounded confidence: A survey. International Journal of Modern Physics C, 18(12), 1819–1838.
Mulgan, G. (2005). Government, knowledge and the business of policy making: The potential and limits of evidence-based policy. Evidence & Policy: A Journal of Research, Debate and Practice, 1(2), 215–226.
Newman, J. (2019). The role of uncertainty in regulating e-cigarettes: The emergence of a regulatory regime, 2005–15. Politics & Policy, 47(2), 407–429.
Norton, J. D. (2011). Challenges to Bayesian confirmation theory. Philosophy of Statistics (pp. 391–439). Elsevier.
Nowotny, H. (2018). Time: The Modern and Postmodern Experience. John Wiley & Sons.
Nutley, S.M., Walter, I., & Davies, H.T. (2007). Using Evidence: How Research can Inform Public Services. Policy Press.
O’Connor, C., & Weatherall, J. O. (2021). Modeling how false beliefs spread. The Routledge Handbook of Political Epistemology (pp. 203–213). Routledge.
Oliver, K., Innvar, S., Lorenc, T., Woodman, J., & Thomas, J. (2014). A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Services Research, 14(1), 1–12.
Pattyn, V., & Bouterse, M. (2020). Explaining use and non-use of policy evaluations in a mature evaluation setting. Humanities and Social Sciences Communications, 7(1), 1–9.
Pollitt, C. (2008). Time, Policy, Management: Governing with the Past. Oxford: Oxford University Press.
Renn, O., Baghramian, M., Capaccioli, M., et al. (2019). Making sense of science for policy under conditions of complexity and uncertainty (Tech. Rep.). SAPEA.
Spiegelhalter, D. J., & Riesch, H. (2011). Don’t know, can’t know: Embracing deeper uncertainties when analysing risks. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 369(1956), 4730–4750.
Stevens, A. (2011). Telling policy stories: An ethnographic study of the use of evidence in policy-making in the UK. Journal of Social Policy, 40(2), 237–255.
Stringer, L. C., & Dougill, A. J. (2013). Channelling science into policy: Enabling best practices from research on land degradation and sustainable land management in dryland Africa. Journal of Environmental Management, 114, 328–335.
Van den Heuvel, C., Alison, L., & Power, N. (2014). Coping with uncertainty: Police strategies for resilient decision-making and action implementation. Cognition, Technology & Work, 16(1), 25–45.
Van der Bles, A. M., Van Der Linden, S., Freeman, A. L., Mitchell, J., Galvao, A. B., Zaval, L., & Spiegelhalter, D. J. (2019). Communicating uncertainty about facts, numbers and science. Royal Society Open Science, 6(5), 181870.
Van der Sluijs, J. (2005). Uncertainty as a monster in the science-policy interface: Four coping strategies. Water Science and Technology, 52(6), 87–92.
Van Dorsser, C., Walker, W. E., Taneja, P., & Marchau, V. A. (2018). Improving the link between the futures field and policymaking. Futures, 104, 75–84.
Van Dyck, M., & Simons, M. (2021). De onoplosbare spanning in expertise-gebaseerd beleid. Filosofie-Tijdschrift, 31(6), 18–21.
Van Oldenborgh, G. J., van der Wiel, K., Kew, S., Philip, S., Otto, F., Vautard, R., & van Aalst, M. (2021). Pathways and pitfalls in extreme event attribution. Climatic Change, 166(1), 1–27.
Walker, W. E., & Marchau, V. A. (2003). Dealing with uncertainty in policy analysis and policymaking. Integrated Assessment, 4(1), 1–4.
Weiss, C. H., & Bucuvalas, M. J. (1977). The challenge of social research to decision making. Using Social Research in Public Policy Making, 11, 213.
Wheaton, K. J., Lee, J., & Deshmukh, H. (2009). Teaching Bayesian statistics to intelligence analysts: Lessons learned. Journal of Strategic Security, 2(1), 39–58.
Wolfson, L. J., Kadane, J. B., & Small, M. J. (1996). Bayesian environmental policy decisions: Two case studies. Ecological Applications, 6(4), 1056–1066.
Young, C.J., & Comptois, J. (1979). Increasing congressional utilization of evaluation. Zweig, F.M. (ed.), Evaluation in Legislation (pp. 57–79). SAGE Publications.
Zollman, K. J. (2012). Social network structure and the achievement of consensus. Politics, Philosophy & Economics, 11(1), 16–44.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare not having competing interests
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Lammers, W., Pattyn, V., Ferrari, S. et al. Evidence for policy-makers: A matter of timing and certainty?. Policy Sci 57, 171–191 (2024). https://doi.org/10.1007/s11077-024-09526-9
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11077-024-09526-9