Assessing the quality of evidence from epidemiological agent-based models for the COVID-19 pandemic

Abstract

Agent-based models (ABMs) are one of the main sources of evidence for decisions regarding mitigation and suppression measures against the spread of SARS-CoV-2. These models have not been previously included in the hierarchy of evidence put forth by the evidence-based medicine movement, which prioritizes those research methods that deliver results less susceptible to the risk of confounding. We point out the need to assess the quality of evidence delivered by ABMs and ask the question of what is the risk that assumptions entertained in ABMs do not include all the key factors and make model predictions susceptible to the problem of confounding.

In this short paper we raise the issue of assessing the quality of evidence from epidemiological agent-based models with respect to the problem of confounding. The unprecedented spread of the novel coronavirus requires governments worldwide to make decisions regarding mitigation and suppression measures. Some of these decisions have been based on agent-based models (ABMs) (Adam 2020), which are an emerging group of epidemiological models that supplement the traditional compartmental models.

With the increase in computational power, ABMs present a relatively novel way to simulate the behavior of complex systems. ABMs consist of entities—namely agents—that interact both with each other and their environment according to a defined set of rules. In this context, an agent represents a human being and the rules describe behavior pattern followed by the particular agent (e.g., agents-children going to schools, agents-adults to work). The simulation represents the way in which people interact with one another and thereby transmit the infectious disease in time. ABMs allow us to compare the baseline scenario (simulation of the epidemic with no changes to agents’ behavior) with effects produced by alternative suppression measures. Thus, ABMs allow for assessing counterfactual causal claims (Marshall and Galea 2015). The standard hierarchies of evidence remain silent on the quality of evidence delivered by ABMs. Neither the manual published by the Oxford Centre for Evidence-Based Medicine (OCEBM) in 2009, nor the National Institute for Health and Care Excellence guidelines (NICE guidelines) from 2014 include ABMs and this can undermine the trust in policy guidance delivered by such models (Maziarz and Zach 2020). Evidence-based medicine (EBM) movement assesses the quality of evidence on the basis of considering the risk of bias or confounding, i.e., the situation where a variable left out from the model accounts for estimated correlations and is the actual causal factor (La Caze 2009). The methods that are less likely to lead to spurious correlations are considered more trustworthy. This approach prioritizes randomized controlled trials over observational studies and cohort over case–control design. Considering the crucial role of epidemiological ABMs in planning mitigation and suppression measures during the SARS-CoV-2 pandemic, the question of the quality of evidence delivered by such models needs to be assessed.

Epidemiological ABMs have been criticized for simplifying social interactions and not accounting for changes of behavior in response to the pandemic and measures themselves (Squazzoni et al. 2020). Much like other scientific practices, theoretical modeling consists of a careful selection of factors that are considered relevant for the task at hand. Thus, by extracting certain features of the studied phenomenon, which, in many cases, are then further modified in various ways (Portides 2018), modelers can indirectly investigate the behavior of a system by first investigating their model. In the process, however, it might be the case that some difference-making factors have been overlooked or otherwise misrepresented in a way that puts the utility of the model in question. Introducing such simplifications can result in findings that may be affected by a confounding factor that is not accounted for in the model (Strevens 2008, p. 288).

Consider, for instance, the attempt to assess the relative accuracy of assumptions in the AceMod model of the spread of the SARS-CoV-2, which is one of the most advanced epidemiological ABMs (Maziarz and Zach 2020). The question remains whether some of the extracted factors, presumed to be key difference-makers, do, in fact, make a difference and whether they do so in accord with the assumptions. The authors of the AceMod SARS-CoV-2 model (Chang et al. 2020) accounted for, among other things, “age-dependent attack rates, a range of reproductive numbers, age-stratified and social context-dependent transmission rates, household clusters and other social mixing contexts, symptomatic-asymptomatic distinction, long and varying incubation periods” (Chang et al. 2020, p. 11). However, the question if all the key difference-making factors have been included in the model remains to be addressed.

For example, the modelers assumed temporal homogeneity of interactions among agents populating AceMod. AceMod predicts that SARS-CoV-2 spreads more quickly in urban areas because social contacts are more frequent in such settings (Chang et al. 2020). However, as shown elsewhere (Nation et al. 2010) these social interactions are less intensive in terms of duration than social interactions in rural areas, which suggests that the spread may be equally rapid or even faster in rural areas, depending on the strength of the confounder’s influence. This is because the duration of exposure is a plausible moderator for the risk of contracting SARS-CoV-2, as stated by, e.g., ECDC (Adlhoch et al. 2020). The failure to properly account for the duration of exposure can have detrimental effects on the accuracy of predictions.

There are many potential confounding factors like the one just discussed that are not included in the model. Given the utmost importance of providing reliable predictions and accurate assessment of public health (non-pharmaceutical) interventions, further research is needed to assess the risk of confounding and quality of evidence delivered by epidemiological ABMs. This task can be divided into two intertwined questions. First, it needs to be established that all the key difference-making factors are accurately included in the model (i.e., no confounding factor is left out from the model). Second, all factors included in the model should be shown to make a difference (approximately) in accord with the rules and assumptions of the model so that the ABM does not include spurious determinants and does not misrepresent the actual factors in a way that would lead to false predictions. Such analysis allows for addressing the question if the causal structure represented by an ABM resembles the actual causal structure and, in effect, assessing the quality and reliability of evidence. However, considering that epidemiological ABMs account for not only biological determinants such as infectivity but also social interactions that differ across the globe, the quality of evidence from ABMs must be assessed on a case-by-case basis. In reaching policy decisions, ABMs should be understood as merely one piece of the puzzle subject to further re-evaluation with respect to value judgments. This is because alternative mitigation measures may disproportionately affect certain social groups. Therefore, the quality assessment aimed at identifying possible confounders that have been left out of a particular ABM should delineate the conflict of interest and vested values related to the ABM and the mitigation measures that it supports.

References

  1. Adam, D. (2020). Special report: The simulations driving the world’s response to COVID-19. Nature, 580(7803), 316–318. https://doi.org/10.1038/d41586-020-01003-6.

    Article  Google Scholar 

  2. Adlhoch, C., Baka, A., Cenciarelli, O., Penttinen, P., Palm, D., Plachouras, D., et al. (2020). Contact tracing: Public health management of persons, including healthcare workers, having had contact with COVID-19 cases in the European Union. https://www.ecdc.europa.eu/sites/default/files/documents/covid-19-public-health-management-contact-novel-coronavirus-cases-EU.pdf.

  3. Chang, S. L., Harding, N., Zachreson, C., Cliff, O. M., & Prokopenko, M. (2020). Modelling transmission and control of the COVID-19 pandemic in Australia. Retrieved May 16, 2020, from http://arxiv.org/abs/2003.10218.

  4. La Caze, A. (2009). Evidence-based medicine must be…. Journal of Medicine and Philosophy, 34(5), 509–527. https://doi.org/10.1093/jmp/jhp034.

    Article  Google Scholar 

  5. Marshall, B. D. L., & Galea, S. (2015). Formalizing the role of agent-based modeling in causal inference and epidemiology. American Journal of Epidemiology, 181(2), 92–99. https://doi.org/10.1093/aje/kwu274.

    Article  Google Scholar 

  6. Maziarz, M., & Zach, M. (2020). Agent-based modelling for SARS-CoV -2 epidemic prediction and intervention assessment: A methodological appraisal. Journal of Evaluation in Clinical Practice, 26(5), 1352–1360. https://doi.org/10.1111/jep.13459.

    Article  Google Scholar 

  7. Nation, M., Fortney, T., & Wandersman, A. (2010). Race, place, and neighboring: Social ties among neighbors in urban, suburban, and rural contexts. Environment and Behavior, 42(5), 581–596. https://doi.org/10.1177/0013916508328599.

    Article  Google Scholar 

  8. Portides, D. (2018). Idealization and abstraction in scientific modeling. Synthese. https://doi.org/10.1007/s11229-018-01919-7.

    Article  Google Scholar 

  9. Squazzoni, F., Polhill, J. G., Edmonds, B., Ahrweiler, P., Antosz, P., Scholz, G., et al. (2020). Computational models that matter during a global pandemic outbreak: A call to action. Journal of Artificial Societies and Social Simulation, 23(2), 10. https://doi.org/10.18564/jasss.4298.

    Article  Google Scholar 

  10. Strevens, M. (2008). Depth: An account of scientific explanation. Harvard: Harvard University Press.

    Google Scholar 

Download references

Acknowledgements

We would like to thank the two anonymous reviewers for thoughtful suggestions which have significantly helped to improve the manuscript.

Funding

The work of Mariusz Maziarz has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant Agreement No. 805498). Mariusz Maziarz received scholarship from the Foundation for Polish Science (FNP).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Mariusz Maziarz.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This note belongs to the Topical Collection “Seeing Clearly Through COVID-19: Current and future questions for the history and philosophy of the life sciences”, edited by G. Boniolo and L. Onaga.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Maziarz, M., Zach, M. Assessing the quality of evidence from epidemiological agent-based models for the COVID-19 pandemic. HPLS 43, 10 (2021). https://doi.org/10.1007/s40656-020-00357-4

Download citation

Keywords

  • SARS-CoV-2
  • EBM
  • Agent-based models