The Boundaries of Cognition and Decision Making

Abstract

This chapter outlines the role that individual-level empirical evidence gathered from psychological experiments and surveys can play in informing agent-based models, and the model-based approach more broadly. To begin with, we provide an overview of the way that this empirical evidence can be used to inform agent-based models. Additionally, we provide three detailed exemplars that outline the development and implementation of experiments conducted to inform an agent-based model of asylum migration, as well as how such data can be used. There is also an extended discussion of important considerations and potential limitations when conducting laboratory or online experiments and surveys, followed by a brief introduction to exciting new developments in experimental methodology, such as gamification and virtual reality, that have the potential to address some of these limitations and open the door to promising and potentially very fruitful new avenues of research.

This chapter outlines the role that individual-level empirical evidence gathered from psychological experiments and surveys can play in informing agent-based models, and the model-based approach more broadly. To begin with, we provide an overview of the way that this empirical evidence can be used to inform agent-based models. Additionally, we provide three detailed exemplars that outline the development and implementation of experiments conducted to inform an agent-based model of asylum migration, as well as how such data can be used. There is also an extended discussion of important considerations and potential limitations when conducting laboratory or online experiments and surveys, followed by a brief introduction to exciting new developments in experimental methodology, such as gamification and virtual reality, that have the potential to address some of these limitations and open the door to promising and potentially very fruitful new avenues of research.

1 The Role of Individual-Level Empirical Evidence in Agent-Based Models

Agents are the key feature that distinguish agent-based models from other forms of micro-simulation. Specifically, within agent-based models, agents can interact with one another in dynamic and non-deterministic ways, allowing macro-level patterns and properties to emerge from the micro-level characteristics and interactions within the model. This key feature of agent-based models means that insights into individual behaviour from psychology and behavioural economics, such as behaviours, personalities, judgements, and decisions, are even more crucial than for other modelling efforts. Within this chapter, we provide an outline as to why it is important to incorporate insights from the study of human behaviour within agent-based models, and give examples of the processes that can be used to do this. As in other chapters within this book, agent-based models of migration are used as an exemplar, however, the information and processes described are applicable to a wide swathe of agent-based models.

Traditionally, many modelling efforts, including agent-based models of demographic processes, have relied on normative models of behaviour, such as expected utility theory, and have assumed that agents behave rationally. However, descriptive models of behaviour, commonly used within psychology and behavioural economics, provide an alternative approach with a focus on behaviour, judgements, and decisions observed using experimental and observational methods. There are many important trade-offs to consider when deciding which approaches to use for an agent-based model and which level of specificity or detail to use. For example, normative models may be more likely to be tractable and already formalised, which gives some key advantages (Jager, 2017). In contrast, many social scientific theories based on observations from areas such as psychology, sociology, and political science may provide much more detailed and nuanced descriptions of how people behave, but are also more likely to be specified using verbal language that is not easily formalised. Therefore, to convert these social science theories from verbal descriptions of empirical results into a form that can be formalised within an agent-based model requires the modeller to make assumptions (Sawyer, 2004). For example, there may be a clear empirical relationship between two variables but the specific causal mechanism that underlies this relationship may not be well established or formalised (Jager, 2017). Similarly, there may be additional variables within an agent-based model that were not incorporated in the initial theory or included in the empirical data. In situations such as these, it often falls to the individual modeller(s) to make assumptions about how to formalise the theory, provide formalised causal mechanisms, and extend the theory to incorporate any additional variables and their potential interactions and impacts.

When it comes to agent-based models of migration, the extent to which empirical insights from the social sciences are used to add complexity and depth to the agents varies greatly (e.g., see Klabunde & Willekens, 2016 for a review of decision making in agent-based models of migration). Additionally, because migration is a complex process that has wide-ranging impacts, there are many options and areas in which additional psychological realism can be added to agent-based models. For example, the personality of the agent is likely to play a role and may be incorporated through giving each agent a propensity for risk taking. Previous research has shown that increased tolerance to risk is associated with a greater propensity to migrate (Akgüç et al., 2016; Dustmann et al., 2017; Gibson & McKenzie, 2011; Jaeger et al., 2010; Williams & Baláž, 2014), and therefore incorporating this psychological aspect within an agent-based model may allow for unique insights to be drawn (e.g., how different levels of heterogeneity in risk tolerance influence the patterns formed, or whether risk tolerance matters more in some migration contexts than others). Additionally, the influence of social networks on migration has been well established (Haug, 2008) so this is also a key area where there may be benefits to adding realism to an agent-based model (Klabunde & Willekens, 2016; Gray et al., 2017). A review of existing models and empirical studies of decision making in the context of migration is offered by Czaika et al. (2021).

When it is believed that an agent-based model can be improved through incorporating additional realism or descriptive insights, designing and implementing an experiment or survey can be a very useful way to gain data, information, and insights. However, there are several different approaches that can be used to derive insights from the social sciences and other empirical literature to inform agent-based models before taking the step of engaging in primary data collection. The first, and most straightforward approach, is to examine the existing literature to see which insights can be gleaned and how people have previously attempted to address the same or similar issues (e.g., if the modeller wants to incorporate emotion or personality into an agent-based model, there are existing formalisms that may be appropriate for use in such instances; Bourgais et al., 2020).

Even if there are no agent-based or other models that have previously addressed the specific research issues or concerns in terms of formalising and incorporating the same descriptive aspect, there may still be pre-existing data that can be used to answer any specific questions that may arise or additional realism that could be incorporated. However, in this situation the modeller will still have to take the additional difficult steps of extracting the information from the existing data or theory (likely a verbal theory) and formalising it for inclusion within an agent-based model. Finally, if it emerges that there are neither pre-existing implementations within a model nor an existing formalism, and there are no verbal theories or relevant data that can be used to build formalisms for inclusion, then it may be time to engage in dedicated primary data collection, and design an experiment and/or survey of the modeller’s own design (see also Gray et al., 2017).

When designing a survey or experiment, it is important to keep in mind the specific goal of the data collection. For example, in terms of agent-based modelling, the goal may be to use the data to inform parameters within the model, or it may be to compare and contrast several different decision rules to decide which has the strongest empirical grounding to include within the model. In the following sections, we outline several experiments that were conducted to better inform agent-based models of asylum migration. The descriptions we provide serve as exemplars, and include an outline of the development of key questions for each experiment, a brief overview of how each experiment was implemented and the methodologies used for the experiments, and finally a discussion of how the data collected in each experiment can be used to inform an agent-based model of migration.

2 Prospect Theory and Discrete Choice

The first set of psychological experiments conducted to better inform agent-based models of migration focused on discrete choice within a migration context. Traditionally, most agent-based models of migration have used expected utility and/or made other assumptions of rationality when building their models (see also the description of neoclassical theories of migration, summarised in Massey et al., 1993). That is, they make assumptions that agents within the models will behave in the way that they ‘should’ behave based on normative models of optimal behaviour. However, research within psychology and behavioural economics has called many of these assumptions into question. The most famous example of this is prospect theory, developed by Kahneman and Tversky (1979) and subsequently updated to become cumulative prospect theory (Tversky & Kahneman, 1992). Based on empirical data, prospect theory proposes that people deviate from the optimal or rational approaches because of biases in the way that they translate information from the objective real-world situation to their subjective internal representations of the world. This has clear implications for how people subsequently make judgements and decisions. Some of the specific empirical findings related to judgement and decision making that are incorporated within prospect theory include loss aversion, overweighting/underweighting of probabilities, differential responses to risk (risk seeking for losses and risk aversion for gains), and framing effects.

Prospect theory was also a useful first area in which to conduct experiments to inform agent-based models of migration because, unlike many other theories of judgement and decision making based on empirical findings, it is already formalised and can therefore be implemented more easily within models. Indeed, in previous work, de Castro et al. (2016) applied prospect theory to agent-based models of financial markets, contrasting these models with agent-based models in which agents behaved according to expected utility theory. De Castro et al. (2016) found that simulations in which agent behaviour was based on prospect theory were a better match to real historical market data than when agent behaviour was based on expected utility theory. Although the bulk of research on prospect theory has focused on financial contexts (for reviews see Barberis, 2013; Wakker, 2010), there is also growing experimental evidence that prospect theory is applicable to other contexts. For example, support for the theory has been found when outcomes of risky decisions are measured in time (Abdellaoui & Kemel, 2014) or related to health such as the number of lives saved (Kemel & Paraschiv, 2018), life years (Attema et al., 2013), and quality of life (Attema et al., 2016).

Czaika (2014) applied prospect theory to migration patterns at a macro-level, finding that the patterns of intra-European migration into Germany were consistent with several aspects of prospect theory, such as reference dependence, loss aversion, and diminished sensitivity. However, because this analysis did not collect micro-level data from individual migrants, it is necessary to assume that the macro-level patterns observed occur (at least partially) due to individual migrants behaving in a way that is consistent with prospect theory. This is a very strong assumption, which risks falling into the trap of the ecological fallacy. At the same time, however, there are also a variety of studies that have examined risk preferences of both economic migrants (Akgüç et al., 2016; Jaeger et al., 2010) and migrants seeking asylum (Ceriani & Verme, 2018; Mironova et al., 2019), and can therefore provide data about some individual level behaviour, judgments and decisions to inform agent-based models of migration. Bocquého et al. (2018) extended this line of research further, using the parametric method of Tanaka et al. (2010) to elicit utility functions from asylum seekers in Luxembourg, finding that the data supported prospect theory over expected utility theory. However, these previous studies examining risk and the application of prospect theory to migration still used standard financial tasks, rather than collecting data within a migration context specifically.

Based on the broad base of existing empirical support, we decided to apply prospect theory to our agent-based models of migration and therefore designed a dedicated experiment to elicit prospect theory parameters within a migration context. There are a variety of potential approaches that can be used to elicit prospect theory parameters (potential issues due to divergent experimental approaches are discussed in Sect. 6.4). To avoid making a priori assumptions about the shape of the utility function, we chose to use a non-parametric methodology adapted from Abdellaoui et al. (2016; methodology presented in Table 6.1). Participants made a series of choices between two gambles within a financial and a migration context. For each choice, both gambles presented a potential gain or loss in monthly income (50% chance of gaining and 50% chance of losing income; see Fig. 6.1 for an example trial). Using this methodology, we elicited six points of the utility function for gains and six points for losses. We then analysed the elicited utility functions for financial and migration decisions to test for loss aversion, whether there was evidence of concavity for gains and/or convexity for losses, and whether there were differences between the migration and financial contexts (see Appendix D for more details on the preregistration of the hypotheses, sample sizes, and ethical issues).

Table 6.1 Procedure for eliciting utility functions

Fig. 6.1

An example of the second gain elicitation (${x}_{\mathbf{2}}^{+}$) within a migration context and with medium stakes. As shown in panel A, ${x}_2^{+}$ is initially set so that both gambles have equivalent utility. The value of ${x}_2^{+}$ is then adjusted in panels B to F depending on the choices made, eliciting the value of ${x}_2^{+}$ that leads to indifference between the two gambles. (Source: own elaboration in Qualtrics)

There are many ways that the results from these experiments can be used to inform agent-based models of migration. The first and perhaps simplest way is to add loss aversion to the model. Because the data collected were within the context of relative changes in gains and losses for potential destination countries, these results can be used within the model to create a distribution of population level loss aversion, from which each agent is assigned an individual level of loss aversion (to allow for variation across agents). Therefore, rather than making assumptions about the extent of loss aversion present within a migration context, instead, each agent within the model would weight potential losses more heavily than potential gains, following the empirical findings from the experiment in a migration context. Similarly, after fitting a function to the elicited points for gains and losses, it is possible to again use this information to inform the shape of the utility functions that are given to agents within the model. That is, the data can be used to inform the extent to which agents place less weight on potential gains and losses as they get further from the reference point (usually implemented as either the current status quo or the currently expected outcome). For example, the empirical data inform us whether people consider a gain of $200 in income to be twice as good as a gain of $100, or only one and a half times as good when they are making a decision.

An additional advantage of including the financial context within the same experiment is that it allows for direct comparisons between that context and a migration context. Therefore, because there is a wide body of existing research on decision making within financial contexts, if the results are similar across conditions then that may provide some supporting evidence that this body of research can be relied on when applied to migration contexts. Conversely, if the results reveal that there are differences between the contexts, then it highlights that modellers should show caution when applying financial insights to other contexts. The presence of differences between contexts would highlight the need to collect additional data within the specific context of interest, rather than relying on assumptions, formalisations, or parameter estimates developed in a different context.

3 Eliciting Subjective Probabilities

The key questions for the second set of psychological experiments emerged from the initial agent-based models presented in Chap. 3 and analysed in Chap. 5. These models highlighted the important role that information sharing and communication between agents can play in influencing the formation and reinforcement of migration routes. Because these aspects played a key role in influencing the results produced by the models, (as indicated by the preliminary sensitivity analysis of the influence of the individual model inputs on a range of outputs, see Chap. 5), it became clear that we needed to gather more information about the processes involved to ensure the model was empirically grounded.

To achieve these aims, we designed a psychological experiment with these specific questions in mind so that the data could be used to inform parameters for the model. Prior to implementing the experiment, we reviewed the relevant literature across domains such as psychology, marketing, and communications to examine what empirical data existed as well as which factors had previously been shown to be relevant. Throughout this process, we kept the specific case study of asylum seeker migration in mind, giving direction and focus to the search and review of the literature. This process led us to focus in on two key factors that were directly relevant to the agent-based model and had also previously been examined within the empirical literature: the source of the information and how people interpret verbal descriptors of likelihood or probability.

Regarding the source of the information, we chose to focus on three specific aspects of source that existing research had shown to be particularly influential: expertise, trust, and social connectedness. Research into the role of source expertise had shown that people are generally more willing to change their views and update their beliefs when the source presenting the information has relevant expertise (Chaiken & Maheswaran, 1994; Hovland & Weiss, 1951; Maddux & Rogers, 1980; Petty et al., 1981; Pilditch et al., 2020; Pornpitakpan, 2004; Tobin & Raymundo, 2009). Trust in a source has also been shown to be a key factor in the interpretation of information and updating of beliefs, with people more strongly influenced by sources in which they place a higher degree of trust (Hahn et al., 2009; Harris et al., 2016; McGinnies & Ward, 1980; Pilditch et al., 2020; Pornpitakpan, 2004). Finally, social connectedness has been found to be an important source characteristic, with people more strongly influenced by sources with whom they have greater social connectedness. For example, people are more influenced by sources that are members of the same racial or religious group and/or sources with whom they have an existing friendship or have worked with collaboratively (Clark & Maass, 1988; Feldman, 1984; Sechrist & Milford-Szafran, 2011; Sechrist & Young, 2011; Suhay, 2015).

The other key aspect was the role of verbal descriptions of likelihood and how people interpret and convert these verbal descriptors into a numerical representation (Budescu et al., 2014; Mauboussin & Mauboussin, 2018; Wintle et al., 2019). This was of particular relevance for the agent-based model of migration because it directly addresses the challenge of converting information from a more fuzzy, verbal description into a numerical response that is easily formalised and can be included within a model. Examining verbal descriptions of likelihood allowed us to address questions such as ‘when someone says that it is likely to be safe to make a migration journey, how should that be numerically quantified’ which is a key step for formalising these processes within the agent-based model.

Having established the areas of focus through an iterative process of generating questions via the agent-based model and reviewing existing literature, it was then possible to design an experiment that provides empirical results to inform the model, and also has the potential to contribute to the scientific literature more broadly by addressing gaps within the literature. We were able to do this by selecting sources that were relevant for asylum seeker migration and also varied on the key source characteristics of expertise, trust, and social connectedness. These choices were also informed by previous research conducted in the Flight 2.0/Flucht 2.0 research project on the media sources used by asylum seekers before, during, and after their journeys from their country of origin to Germany (Emmer et al., 2016; see also Chap. 4 and Appendix B). The specific sources that were chosen for inclusion in the experiment were: a news article, a family member, an official organisation, someone with relevant personal experience, and the travel organiser (i.e., the person organising the boat trip). Additionally, we randomised the verbal likelihood that was communicated by each source to be one of the following: very likely, likely, unlikely, or very unlikely (one verbal likelihood presented per source). For example, a participant may read that a family member says a migration boat journey across the sea is likely to be safe, that an official organisation says the trip is unlikely to be safe, that someone with relevant personal experience says it is very unlikely to be safe, and so on (see Fig. 6.2 for an example).

Fig. 6.2

Vignette for the migration context (panel A), followed by the screening question to ensure participants paid attention (panel B) and an example of the elicitation exercise, in which participants answer questions based on information from a news article (panels C to F). (Source: own elaboration in Qualtrics)

After seeing each piece of information, participants judged the likelihood of travelling safely (0–100) and made a binary decision to travel (yes/no). Additionally, they indicated how confident they were in their likelihood judgement, and whether they would share the information and their likelihood judgement with another traveller. Participants also made overall judgements of the likelihood of travelling safely and hypothetical travel decisions based on all the pieces of information, and indicated their confidence in their overall likelihood judgement, and whether they would share their overall likelihood judgement. At the end of the experiment, participants indicated how much they trusted the five sources in general, as well as whether they had ever seriously considered or made plans to migrate to a new country, and whether they had previously migrated to a new country (again, see Appendix D for details on the preregistration, sample sizes, and ethical issues).

Conducting this experiment provided a rich array of data that can be used to inform an agent-based model of asylum seeker migration. For example, it becomes relatively straightforward to assign numerical judgements about safety to information that agents receive within an agent-based model because data has been collected on how people (experiment participants) interpret phrases such as ‘the boat journey across the sea is likely to be safe’. It is also possible to see whether these interpretations vary depending on the source of the information, such as whether ‘likely to be safe’ should be interpreted differently by an agent within the model depending on whether the information comes from a family member or an official organisation. Additionally, because we collected overall ratings it is possible to examine how people combine and integrate information from multiple sources to form overall judgements. This information can be used within an agent-based model to assign relative weights to different information sources, such as weighting an official organisation as 50% more influential than a news article, a family member as 30% less influential than someone with relevant personal experience, and so on.

To more explicitly illustrate this, the data collected in this experiment were used to inform the model presented in Chap. 8. Specifically, because for each piece of information participants received they provided both a numerical likelihood of safety rating and a binary yes/no decision regarding whether they would travel, it was possible to calculate the decision threshold at which people become willing to travel, as well as how changes in the likelihood of safety ratings influence the probability that someone will decide to travel. We could then use these results to inform parameters within the model that specify how changes in an agent’s internal representation of the safety of travelling translate into changes in the probability of them making specific travel decisions.

4 Conjoint Analysis of Migration Drivers

In the third round of experiments, conjoint analysis is used to elicit the relative weightings of a variety of migration drivers. Specifically, the focus is on characteristics of potential destination countries and analysing which of these characteristics have the strongest influence on people’s choices between destinations. The impetus for this experimental focus again came from some key questions within both the model and the migration literature more broadly. In relation to the model, this line of experimental inquiry arose because the model uses a graphical representation of space that the agents attempt to migrate across towards several potential end cities (end points), with numerous paths and cities present along the way.

In the initial implementations of the Routes and Rumours model, there was no differentiation between the available end points. That is, the agents within the model simply wanted to reach any of the available end cities/points and did not have any preference for some specific end cities over others. This modelling implementation choice was made to get the model operational and to provide results regarding the importance of communication between agents and agent exploration of the paths/cities. However, to enhance the realism of the agent-based model and make it more directly applicable to the real-world scenarios that we would like to model, it became clear that it was important for the end cities to vary in their characteristics and the extent to which agents desire to reach them. Therefore, it was important to gather empirical data about the characteristics of potential end destinations for migration as well as how people weight the different characteristics of these destinations and make trade-offs when choosing to migrate.

Previous research has examined the various factors that influence the desirability of migration destination countries (Carling & Collins, 2018). Recently, a taxonomy of migration drivers has been developed, made up of nine dimensions of drivers and 24 individual driving factors that fit within these nine dimensions (Czaika & Reinprecht, 2020). The nine dimensions identified were: demographic, economic, environmental, human development, individual, politico-institutional, security, socio-cultural, and supra-national. The breadth of areas covered by these dimensions helps to emphasise the large array of characteristics that may influence the choices migrants make about the destination countries of interest.

Research using an experimental approach has also previously been used to examine the importance of a variety of migration drivers, in Baláž et al. (2016) and Baláž and Williams (2018). Both these studies examined how participants searched for information related to wages, living costs, climate, crime rate, life satisfaction, health, freedom and security, and similarity of language (Baláž et al., 2016), as well as the unemployment rate, attitudes towards immigrants, and whether a permit is needed to work in the country (Baláž & Williams, 2018). Additionally, in both studies participants were asked about their previous experience with migration so that results could be compared between migrants and non-migrants. The results of these studies showed that, consistent with many existing neo-classical approaches to migration studies (Borjas, 1989; Harris & Todaro, 1970; Sjaastad, 1962; Todaro, 1969), participants were most likely to request information on economic factors and also weighted these factors the most strongly in their decisions. Specifically, wages and cost of living were the most requested pieces of information and had the highest decision weights. However, they also found that participants with previous migration experience placed more emphasis on non-economic factors, being more likely to request information about life satisfaction and to give more weight to life satisfaction when making their decisions. This suggests that non-economic factors can also play an important role in migration, and that experience of migration may make people more likely to consider and place emphasis on these non-economic factors.

Building on the questions derived from the agent-based model and this previous literature, we decided to conduct an experiment informing the conjoint analysis of the weightings of a variety of migration drivers. Specifically, the approach taken was to examine the existing literature to identify the key characteristics of destination countries that are present and may be relevant for the destination countries within our model. Therefore, we examined the migration drivers included in the previous experimental work (Baláž et al., 2016; Baláž & Williams, 2018) as well as the taxonomy of driver dimensions and individual driver factors (Czaika & Reinprecht, 2020) along with a broader literature review to come up with a long-form list of migration drivers that could potentially be included. Then, through discussions with colleagues and experts within the area of migration studies,¹ we reduced the list down to focus in on the key drivers of interest, while also ensuring the specific drivers chosen provide at least partial coverage across the full breadth of the driver dimensions identified by Czaika and Reinprecht (2020). Specifically, the country-level migration drivers chosen for inclusion were: average wage level, employment level, number of migrants from the country of origin already present, cultural and linguistic links with the country of origin, climate and safety from extreme weather events, openness of migration policies, personal safety and political stability, education and training opportunities, income equality and standard of living, and public infrastructure and services (e.g., health).

Having identified the key drivers for inclusion, the approach used to examine this specific question was an experiment using a conjoint analysis design (Hainmueller et al., 2014, 2015). In a conjoint analysis experiment, participants are presented with a series of trials, each of which presents alternatives that contain information on a number of key attributes (in this case, migration drivers). This approach allows researchers to gain information about the causal role of a number of attributes within a single experiment, rather than conducting multiple experiments or one excessively long experiment that examines the role of each individual attribute one at a time (Hainmueller et al., 2014). Additionally, because all of the attributes are presented together on each trial, it is possible to establish the weightings of each attribute relative to the other presented attributes. That is, a conjoint analysis design allows the analyst to establish not only whether wages have an effect, but how strong that effect is relative to other drivers such as employment level or education and training opportunities. An example of the implementation of the conjoint analysis experiment is presented in Fig. 6.3.

Fig. 6.3

Example of a single trial in the conjoint analysis experiment (panel A) and the questions participants answer for each trial (panel B). (Source: own elaboration in Qualtrics)

Another benefit of the conjoint analysis approach is that because weightings are revealed at least somewhat implicitly (rather than in designs that explicitly ask participants about the weightings or importance they place on specific attributes), and because multiple attributes are presented at the same time, participants may be less influenced by social desirability because they can use any of the attributes present to justify their decision. This is supported by a study by Hainmueller et al. (2015) who found that a paired conjoint analysis design did best at matching the relative weightings of attributes for decisions on applications for citizenship in Switzerland when these weightings were compared to a real-world benchmark (the actual results of referendums on citizenship applications). For these reasons, within the present study we also ask participants to explicitly state how much they weight each variable, allowing for greater understanding of how well people’s stated and revealed preferences align with each other. This comparison between implicit and explicit weightings is also expected to reveal the extent to which people are aware of, and able or willing to communicate the relative value they place on the country attributes that motivate them to choose one destination country over another.

The results from this conjoint analysis experiment can be used to inform the agent-based model by collecting empirical data on the relative weightings of various migration drivers. Additionally, because the experimental data are collected at an individual level, it is also possible to observe to what extent these weightings are heterogenous between individuals (e.g., whether some individuals place more emphasis on safety while others care more about economic opportunities). These relative weightings can then be combined with real-world data on actual migration destination countries or cities to calculate ‘desirability’ scores for potential migration destinations within the model, either at an aggregate level or, if considerable heterogeneity is present, by calculating individual desirability scores for each agent to properly reflect the differences in relative weightings found in the empirical data. The model can then be rerun with migration destinations that vary in terms of desirability to examine what effects this has on aspects such as agent behaviour, route formation, and total number of agents arriving at each destination.

5 Design, Implementation, and Limitations of Psychological Experiments for Agent-Based Models

When designing and implementing psychological experiments, there are several key aspects that must be considered to ensure that valid and reliable conclusions can be drawn from the experiment. Although both reviewing the existing empirical literature and experimental methods have great potential to contribute to the design and implementation of agent-based models, there are also some serious limitations with these approaches. No single experiment or set of experiments is ever perfect, and there are often trade-offs that must be made between various competing interests when designing and implementing a study. In the following section, we discuss several key aspects of designing and implementing psychological experiments using examples from Sects. 6.2, 6.3, and 6.4. The aspects covered include confounding variables, measurement accuracy, participant samples, and external validity of experimental paradigms. In addition to guidance on how these aspects can be addressed we also discuss the limitations of the experimental approaches used (and many psychological experiments more broadly) and suggest ways to overcome these limitations.

When designing a psychological experiment it is important to consider the potential for confounds to influence the outcome (Kovera, 2010). Confounding occurs when there are multiple aspects that vary across experimental conditions, meaning that it is not possible to infer whether the changes seen are due to the intended experimental manipulation, or occur because of another aspect that differs between the conditions. For example, in the experiment discussed in Sect. 6.3, we were interested in the influence of information source on the judgements and decisions that were made. Therefore, we included information from sources such as a news article, an official organisation, and a family member. However, we ensured that the actual information provided to participants was kept consistent regardless of the source (e.g., ‘the migrant sea route is unlikely to be safe’) rather than varying the information across the source formats, such as by presenting a full news article when the source was a news article or a short piece of dialogue when the source was a family member. To examine the role of source, it was crucial that the actual information provided was kept consistent because otherwise it would be impossible to tell whether differences found were due to changes in the source or because of another characteristic such as the length or format of the information provided. However, the drawback in choosing to keep the information presented identical across sources is that the stimuli used are less representative of their real-world counterparts (i.e., the news articles used in the study are less similar to real-world news articles), highlighting that gaining additional experimental control to limit potential confounds can come at the cost of decreasing external validity.

Another key issue to consider is the importance of measurement (for a detailed review see Flake & Fried, 2020). Although a full discussion and evaluation is beyond the scope of the current chapter, some aspects of measurement related issues are made particularly clear through the experiment described in Sect. 6.2. Within this study, we wanted to elicit parameters related to prospect theory. However, previous research by Bauermeister et al. (2018) found that, relevant for prospect theory, the estimates of risk attitudes and probability weightings for the same participants depended on the specific elicitation methodology used. Specifically, Bauermeister et al. compared the methodology from Tanaka et al. (2010) and Wakker and Deneffe (1996), and found that the elicited estimates for participants were more risk averse when the former approach was used, whereas they were more biased in their probability weightings when the latter method was applied (with greater underweighting of high probabilities and overweighting of low probabilities). This raises serious concerns around the robustness of findings, because it suggests that the estimates of prospect theory parameters gathered may be conditional on the experimental methodology used and therefore these estimates are incredibly difficult to generalise and apply to an agent-based model. We attempted to address these issues by using the non-parametric methodology of Abdellaoui et al. (2016), since it requires fewer assumptions than many other elicitation methods. However, the findings of Bauermeister et al. (2018) still highlight the extent to which the results of studies can be highly conditional on the specific methodology and context in which the study takes place, and therefore may be difficult to generalise.

Issues with the typical samples used within psychology and other social sciences have been well documented for many years now (Henrich et al., 2010). Specifically, it has long been pointed out that the populations used for social science research are much more Western, Educated, Industrialised, Rich, and Democratic (WEIRD) than the actual human population of the Earth (Henrich et al., 2010; Rad et al., 2018). This bias means that much of the data within the social sciences literature that can be used to inform agent-based models may not be applicable whenever the social process or system being modelled is not itself comprised solely of WEIRD agents. Even though this issue has been known about for quite some time, there has not yet been much of a shift within the literature to address it. Arnett (2008) found that between 2003 and 2007, 96% of the participants of experiments reported in top psychology journals were from WEIRD samples.

More recently, Rad et al. (2018) found that 95% of the participants of the experiments published in Psychological Science between 2014 and 2017 were from WEIRD samples, suggesting that even though a decade had passed, there had been little change in the extent to which non-WEIRD populations are underrepresented within the psychological literature. Despite their being relatively little research conducted with non-WEIRD samples, that research has produced considerable evidence that there are cultural differences across many areas of human psychology and behaviour, such as visual perception, morality, mating preferences, reasoning, biases, and economic preferences (for reviews see Apicella et al., 2020; Henrich et al., 2010). Of particular relevance for the experiments discussed in the previous sections, Falk et al. (2018) found that economic preferences vary considerably between countries and Rieger et al. (2017) found that, although descriptively, the results from nearly all of the 53 countries they surveyed were consistent with prospect theory, the estimates for the parameters of cumulative prospect theory differed considerably between countries. Therefore, if there is a desire to use results from the broader literature or from a specific study to inform an agent-based model, then it is important for researchers to ensure that the participants included within their studies are representative of the population(s) of interest, rather than continuing to sample almost entirely from WEIRD populations and countries.

The issue of the extent to which findings from experimental contexts can be generalised to the real-world has also received considerable attention across a wide range of fields (Highhouse, 2007; Mintz et al., 2006; Polit & Beck, 2010; Simons et al., 2017). As highlighted by Highhouse (2007), many critiques of experimental methodology place an unnecessarily large emphasis on surface-level ecological validity. That is, the extent to which the materials and experimental setting appear similar to the real-world equivalent (e.g., how much the news articles used as materials within a study look like real-world news articles). However, provided the methodology used allows for proper understanding of “the process by which a result comes about” (Highhouse, 2007, p. 555), then even if the experiment differs considerably from the real world, the information gained is still helpful for developing theoretical understanding that can then be tested and applied more broadly. In the context of asylum migration, additional insights can be gained from some related areas, for example on evacuations during terrorist attacks or natural disasters (Lovreglio et al., 2016), where agent-based models are successfully used to predict and manage the actual human behaviour (e.g. Christensen & Sasaki, 2008; Cimellaro et al., 2019; see also an example of Xie et al., 2014 in Chapter 5). Conceptually, one common factor in such circumstances could be the notion of fear (Kok, 2016).

Nonetheless, migration is an area in which the limitations of lab or online-based experimental methods and the difficulty of truly capturing and understanding the real-world phenomena of interest becomes clear. Deciding to migrate introduces considerable disruption and upheaval to an individual or family’s life, along with potential excitement at new opportunities and discoveries that might await them. How then can a simple experiment or survey conducted in a lab or online via a web browser possibly come close to capturing the real-world stakes or the magnitude of the decisions that are faced by people when they confront these situations in the real world? This problem is likely even more pronounced for migrants seeking asylum, who are likely to be making decisions under considerable stress and where the decisions that they make could have actual life or death consequences. Given the large body of evidence showing that emotion can strongly influence a wide range of human behaviours, judgments, and decisions (Lerner et al., 2015; Schwarz, 2000), it becomes clear that it is incredibly difficult to generalise and apply findings from laboratory and online experimental settings in which the degree of emotional arousal, emotional engagement, and the stakes at play are so greatly reduced from the real-world situations and phenomena of interest.

For the purpose of the modelling work presented in this book, we focus therefore on incorporating the empirical information elicited on the subjective measures (probabilities) related to risky journeys and the related confidence assessment (Sect. 6.3). The process is summarised in Box 6.1.

Box 6.1: Incorporating Psychological Experiment Results Within an Agent-Based Model

Incorporating the results of psychological experiments with an agent-based model may not be a straightforward task, because the specific method of implementation will vary greatly depending on the setup and structure of the model. Therefore, this brief example is designed to outline how results from the experiment in Sect. 6.3 have been incorporated into an agent-based model of migration (see Chap. 8 for more details on the updated version of the model).

In the updated version of the original Routes and Rumours model introduced in Chap. 3, called ‘Risk and Rumours’ (see Chap. 8), agents make safety ratings for the links between cities within the simulation, and these ratings subsequently effect the probability that they will travel along a link. Within the updated Risk and Rumours model, agent beliefs about risk are represented as an estimate v_risk, with a certainty measure t_risk, bounded between 0 and 1.

Within the model, agents form these beliefs based on their experiences travelling through the world as well as by exchanging information with other agents. There is also a scaling parameter for risk, risk_scale which is greater than 1. Based on the above, for risk-related decisions, an agent’s safety estimate for a given link (s) is derived as:

$$ s=t\_ risk\ast {\left(1-v\_ risk\right)}^{risk\_ scale}\ast 100 $$

The logit of the probability to leave for a given link (p) is then calculated as:

$$ p=I+S\ast s $$

The results of the experiment in Sect. 6.3 are incorporated within the model through the values of the intercept I and slope S. These variables take agent-specific values drawn from a bivariate normal distribution, the parameters for which come from the results of a logistic regression conducted on the data collected in the experiment. In this way, the information gained from the psychological experiment about how safety judgments influence people’s willingness to travel is combined with the beliefs that agents within the model have formed, thereby influencing the probability that agents will make the decision to travel along a particular link on their route.

6 Immersive Decision Making in the Experimental Context

The development of more immersive and engaging experimental setups can provide an exciting avenue to address several of the concerns outlined in the previous section. Increasing immersion within experimental studies is particularly helpful for addressing concerns related to realism and emotional engagement of participants. One potentially beneficial approach that can be used to increase emotional engagement, and thereby at least partially close the emotional gap between the experimental and the real-world, is through ‘gamification’. Research has shown that people are motivated by games and that playing games can satisfy several psychological needs such as needs for competence, autonomy, and relatedness (Przybylski et al., 2010; Ryan et al., 2006).

Additionally, Sailer et al. (2017) showed that a variety of aspects of game design can be used to increase feelings of competence, meaningfulness, and social connectedness, feelings that many researchers are likely to want to elicit in participants to increase immersion and emotional engagement while they are completing an experiment. Using gamification to increase participant engagement and motivation does not even require the inclusion of complex or intensive game design elements. Lieberoth (2014) found that when participants were asked to engage in a discussion of environmental issues, simply framing the task as a game through giving participants a game board, cards with discussion items, and pawns increased task engagement and self-reported intrinsic motivation, even though there were no actual game mechanics.

To improve the immersion and emotional engagement of participants in experimental studies of migration, we plan to use gamification aspects in future experiments. Specifically, we aim to design a choose-your-own adventure style of game to explore judgements and decision making within asylum migration context. Inspiration for this approach came from interactive choose-your-own adventure style projects that were developed by the BBC (2015) and Channel 4 (2015) to educate the public about the experiences of asylum seekers on their way to Europe.² We plan to use the agent-based models of migration that have been developed to help generate an experimental setup, and then combine this with aspects of gamification to develop an experiment that can be ‘played’ by participants. For example, by mapping out the experiences, choices, and obstacles that agents within the agent-based models encounter as well as the information that they possess, it is possible to generate sequences of events and choices that occur, and then design a choose-your-own adventure style game in which real-world participants must go through the same sequences of events and choices that the agents within the model face. This allows for the collection of data from real-world participants that can be directly used to calibrate and inform the setup of the agents within the agent-based model, while simultaneously also having the advantage of being more immersive, engaging, and motivating for the participants completing the experiment.

Improvements in technology also allow for the development of even more advanced and immersive experiments in the future, using approaches such as video game modifications (Elson & Quandt, 2016), and virtual reality (Arellana et al., 2020; Farooq et al., 2018; Kozlov & Johansen, 2010; Mol, 2019; Moussaïd et al., 2016; Rossetti & Hurtubia, 2020). Elton and Quandt (2016) highlighted that by using modifications to video games, it is possible for researchers to have control over many aspects of a video game, allowing them to design experiments by operationalising and manipulating variables and creating stimulus materials so that participants in experimental and control groups can play through an experiment in an immersive and engaging virtual environment. At the same time, observational studies based on information from online games allow for studying many aspects of social reality and social dynamics, which may be relevant for agent-based models, such as networks and their structures, collaboration and competition, or inequalities (e.g. Tsvetkova et al., 2018).

The increased availability and decreased costs of virtual reality headsets have also allowed for researchers to test the effectiveness of presenting study materials and experiments within virtual reality. Virtual reality has already been used to examine phenomena such as pedestrian behaviour and traffic management (Arellana et al., 2020; Farooq et al., 2018; Rossetti & Hurtubia, 2020), behaviour during emergency evacuations (Arellana et al., 2020; Moussaïd et al., 2016), and the bystander effect (Kozlov & Johansen, 2010). It has also been applied to a wide range of areas within economics and psychology (for a review see Mol, 2019). In the context of agent-based simulation models, hybrid approaches, with human-computer interactions, have also been the subject of experiments (Collins et al., 2020). These new technological developments allow for the simulation and manipulation of experimental environments in ways that are simply not possible using standard experimental methods, or would be unethical and dangerous to study in the real world. They allow researchers to take several steps towards closing the gap between the laboratory and the real world, and open the door to many exciting new research avenues.