In this short paper we raise the issue of assessing the quality of evidence from epidemiological agent-based models with respect to the problem of confounding. The unprecedented spread of the novel coronavirus requires governments worldwide to make decisions regarding mitigation and suppression measures. Some of these decisions have been based on agent-based models (ABMs) (Adam 2020), which are an emerging group of epidemiological models that supplement the traditional compartmental models.

With the increase in computational power, ABMs present a relatively novel way to simulate the behavior of complex systems. ABMs consist of entities—namely agents—that interact both with each other and their environment according to a defined set of rules. In this context, an agent represents a human being and the rules describe behavior pattern followed by the particular agent (e.g., agents-children going to schools, agents-adults to work). The simulation represents the way in which people interact with one another and thereby transmit the infectious disease in time. ABMs allow us to compare the baseline scenario (simulation of the epidemic with no changes to agents’ behavior) with effects produced by alternative suppression measures. Thus, ABMs allow for assessing counterfactual causal claims (Marshall and Galea 2015). The standard hierarchies of evidence remain silent on the quality of evidence delivered by ABMs. Neither the manual published by the Oxford Centre for Evidence-Based Medicine (OCEBM) in 2009, nor the National Institute for Health and Care Excellence guidelines (NICE guidelines) from 2014 include ABMs and this can undermine the trust in policy guidance delivered by such models (Maziarz and Zach 2020). Evidence-based medicine (EBM) movement assesses the quality of evidence on the basis of considering the risk of bias or confounding, i.e., the situation where a variable left out from the model accounts for estimated correlations and is the actual causal factor (La Caze 2009). The methods that are less likely to lead to spurious correlations are considered more trustworthy. This approach prioritizes randomized controlled trials over observational studies and cohort over case–control design. Considering the crucial role of epidemiological ABMs in planning mitigation and suppression measures during the SARS-CoV-2 pandemic, the question of the quality of evidence delivered by such models needs to be assessed.

Epidemiological ABMs have been criticized for simplifying social interactions and not accounting for changes of behavior in response to the pandemic and measures themselves (Squazzoni et al. 2020). Much like other scientific practices, theoretical modeling consists of a careful selection of factors that are considered relevant for the task at hand. Thus, by extracting certain features of the studied phenomenon, which, in many cases, are then further modified in various ways (Portides 2018), modelers can indirectly investigate the behavior of a system by first investigating their model. In the process, however, it might be the case that some difference-making factors have been overlooked or otherwise misrepresented in a way that puts the utility of the model in question. Introducing such simplifications can result in findings that may be affected by a confounding factor that is not accounted for in the model (Strevens 2008, p. 288).

Consider, for instance, the attempt to assess the relative accuracy of assumptions in the AceMod model of the spread of the SARS-CoV-2, which is one of the most advanced epidemiological ABMs (Maziarz and Zach 2020). The question remains whether some of the extracted factors, presumed to be key difference-makers, do, in fact, make a difference and whether they do so in accord with the assumptions. The authors of the AceMod SARS-CoV-2 model (Chang et al. 2020) accounted for, among other things, “age-dependent attack rates, a range of reproductive numbers, age-stratified and social context-dependent transmission rates, household clusters and other social mixing contexts, symptomatic-asymptomatic distinction, long and varying incubation periods” (Chang et al. 2020, p. 11). However, the question if all the key difference-making factors have been included in the model remains to be addressed.

For example, the modelers assumed temporal homogeneity of interactions among agents populating AceMod. AceMod predicts that SARS-CoV-2 spreads more quickly in urban areas because social contacts are more frequent in such settings (Chang et al. 2020). However, as shown elsewhere (Nation et al. 2010) these social interactions are less intensive in terms of duration than social interactions in rural areas, which suggests that the spread may be equally rapid or even faster in rural areas, depending on the strength of the confounder’s influence. This is because the duration of exposure is a plausible moderator for the risk of contracting SARS-CoV-2, as stated by, e.g., ECDC (Adlhoch et al. 2020). The failure to properly account for the duration of exposure can have detrimental effects on the accuracy of predictions.

There are many potential confounding factors like the one just discussed that are not included in the model. Given the utmost importance of providing reliable predictions and accurate assessment of public health (non-pharmaceutical) interventions, further research is needed to assess the risk of confounding and quality of evidence delivered by epidemiological ABMs. This task can be divided into two intertwined questions. First, it needs to be established that all the key difference-making factors are accurately included in the model (i.e., no confounding factor is left out from the model). Second, all factors included in the model should be shown to make a difference (approximately) in accord with the rules and assumptions of the model so that the ABM does not include spurious determinants and does not misrepresent the actual factors in a way that would lead to false predictions. Such analysis allows for addressing the question if the causal structure represented by an ABM resembles the actual causal structure and, in effect, assessing the quality and reliability of evidence. However, considering that epidemiological ABMs account for not only biological determinants such as infectivity but also social interactions that differ across the globe, the quality of evidence from ABMs must be assessed on a case-by-case basis. In reaching policy decisions, ABMs should be understood as merely one piece of the puzzle subject to further re-evaluation with respect to value judgments. This is because alternative mitigation measures may disproportionately affect certain social groups. Therefore, the quality assessment aimed at identifying possible confounders that have been left out of a particular ABM should delineate the conflict of interest and vested values related to the ABM and the mitigation measures that it supports.