Optimization and Control of AgentBased Models in Biology: A Perspective
 2.4k Downloads
 3 Citations
Abstract
Agentbased models (ABMs) have become an increasingly important mode of inquiry for the life sciences. They are particularly valuable for systems that are not understood well enough to build an equationbased model. These advantages, however, are counterbalanced by the difficulty of analyzing and using ABMs, due to the lack of the type of mathematical tools available for more traditional models, which leaves simulation as the primary approach. As models become large, simulation becomes challenging. This paper proposes a novel approach to two mathematical aspects of ABMs, optimization and control, and it presents a few first steps outlining how one might carry out this approach. Rather than viewing the ABM as a model, it is to be viewed as a surrogate for the actual system. For a given optimization or control problem (which may change over time), the surrogate system is modeled instead, using data from the ABM and a modeling framework for which readymade mathematical tools exist, such as differential equations, or for which control strategies can explored more easily. Once the optimization problem is solved for the model of the surrogate, it is then lifted to the surrogate and tested. The final step is to lift the optimization solution from the surrogate system to the actual system. This program is illustrated with published work, using two relatively simple ABMs as a demonstration, Sugarscape and a consumerresource ABM. Specific techniques discussed include dimension reduction and approximation of an ABM by difference equations as well systems of PDEs, related to certain specific control objectives. This demonstration illustrates the very challenging mathematical problems that need to be solved before this approach can be realistically applied to complex and large ABMs, current and future. The paper outlines a research program to address them.
Keywords
Agentbased modeling Systems theory Optimization Optimal control1 Introduction
 (1)
We consider ABMs “middleware” investigatory objects: selectively abstracted representations of the realworld system that are yet too complex for traditional formal mathematical analysis. To a great degree this complexity arises out of system properties that resist traditional modeling methods, such as variable componentcomponent interactions, spatial heterogeneity and insufficiency of meanfield approximations, and this makes ABMs “sufficiently complex” proxies for the realworld system.
 (2)
The fact that ABMs are computational constructs vastly increases the range of “experimental” conditions able to be applied versus their realworld referent (or realworld physical proxy models); this includes testing putative control strategies.
 (3)However, comprehensive search of modelresponse space using brute force embarrassingly parallel simulation is computationally expensive, and may not be necessary. Therefore, identifying methodological bridges between ABMs and more formally tractable SLMs would be beneficial and serve two purposes:
 (3a)
To reduce the search space for putative controls, which can then be tested via more tractable embarrassingly parallel simulation experiments; and
 (3b)
To facilitate iterative refinement/expansion/reduction of an existing ABM in reference to its intended use with respect to its referent (in this case, the search for practically implementable control strategies).
 (3a)
 (4)
By virtue of being “sufficiently complex” proxy systems, information and knowledge obtained by examining ABMs subject to control may provide insight into how to effectively control the realworld referent. At the very least, this process can provide a first approximation of the set of putative controls.
Main hypothesis If an ABM is treated not as a model of a system of interest but as the system itself, then simpler mathematical models can be derived that capture key features of the ABM for a particular control or optimization objective and, by extension, the biological system of interest.
One might argue that if there is an equationbased model that can be used to solve optimization problems related to the biological system, then one should have constructed such a model in the first place, rather than build an ABM as an intermediary step. This might well be the right approach, if feasible, but there are several reasons why one might nevertheless want to build an ABM first. Firstly, of all model types, an ABM requires arguably the fewest simplifying assumptions to be made, and it can be validated in the most direct way, through, e.g., observation of characteristic patterns rather than surrogate summary statistics. If the biological system is not very well understood, this can be an important reason for an ABM as a first modeling step. Once an ABM is built and validated, it can be used to understand the system better, e.g., the importance of different variables or spatial features. Once a control objective is specified, this understanding can then lead to a possibly much simpler equationbased model that is faithful for the specific control objective, but possibly few or none of the other features. Secondly, the need for optimization and control might be an ongoing process, e.g., for models that are used for policy decisions, and the control and optimization objectives might change over time. In this case, the model is likely intended to capture all possible information about the system and incorporate new information as it becomes available. It is not efficient in this case to build a series of oneoff de novo models for each. Thirdly, the model might be built by domain experts with little expertise in mathematical modeling, who can build an ABM with much greater ease than they can an equationbased model. Thus, the proposed indirect approach to optimization and control of systems is not intended to replace a direct modeling approach in all cases, but is intended to be used in cases where a direct approach is either not feasible or not desirable.
2 AgentBased Models
The life sciences frequently examine systems with interacting components at multiple levels of hierarchy and structure, from cells to connections between cells that lead to tissuelevel properties, to the wholeorganism level, and on to individuals in ecosystems. The reduction of biological systems to physical or chemical phenomena has yielded interesting insights at a fundamental level, but these approaches, to a great degree, fail to sufficiently represent the range and complexity of biological behavior that is often of interest at a level relevant for optimization and control approaches. Biological systems are distinguished by an organizational structure that generates multiscale phenomena arising from the complex interactions between their physical components. They support adaptive behavior of individuals and exhibit great individual variability, whether at the scale of molecules or humans. The nonlinearities associated with the functional transitions between organizational scales challenge the application of many traditional mathematical methods, particularly those oriented toward engineering means of controlling those systems. However, ABMs, which typically simulate interactions between individual components operating in heterogeneous spatial environments to generate populationlevel behaviors, often span two or even more organizational scales (e.g., molecular rules \(<=>\) individual cell behavior \(<=>\) cell population/tissue behavior \(<=>\) multitissue/organism \(<=>\) multiorganism/population). They have become an important technology for the life sciences because of their capacity to account for heterogeneity among components. Additionally, they are able to readily integrate knowledge with data because, in many cases, reductionist experimental data offer better observability of individuals than aggregates. Often, scientists find ABMs simpler to explain to stakeholders such as policy makers, in terms of components for which they have some intuition. It has now become relatively easy for a domain expert to construct an ABM, thanks to easytouse software interfaces for model construction, simulation, and visualization. For these and other reasons, ABMs have been increasingly adopted in the evolving area of computational simulation science. The notion that computation has come to complement experiment and theory as a third pillar of science arises from the use of complex computational simulations not only as a means of integrating and comparing theory and experiment but, more importantly, as tools to aid in theory construction, to illuminate crucial components and uncertainties, to generate and examine new hypotheses, to suggest new experiments and data collection efforts, and to strengthen policy development and decisionmaking.
The basic structure of an ABM consists of individuals/agents with attributes and rules of behavior, rules that govern agent actions and interactions with other agents, as well as the interaction of agents with a potentially complex heterogeneous environment. ABMs often allow for individual variation among agents, challenging the compartmentalization typically used in dynamical systems models. Moreover, agents may have adaptive behavioral rules that lead to unforeseeable interactions and emergence. The time scales of different behavioral rules and environmental pressures can also be quite variable. These features make ABMs difficult to encode in terms of traditional difference or differential equations models. The behaviors and rules of ABMs are typically encoded in software as simple logical rules, coupled with random number generation to model uncertain events and outcomes. As such, ABM construction is more accessible than mathematically involved approaches such as, e.g., systems of differential equations. This simplicity of implementation allows researchers to translate hypotheses into a computational form, so that the ABM plays the role of a digital “sand box,” aiding in the investigation and visualization, advancing theory through conceptual model falsification.
An important and primary feature of ABMs is the populationlevel aggregated behavior that emerges from the rulebased individual interactions, such as the patterns of segregation in the model of (Schelling 1971) or synchronization of breeding in birds (Railsback and Grimm 2012). While differential equations models like the BelousovZhabotinsky reaction model (see, e.g., Murray 2011) can exhibit similar complex pattern evolution, the diversity of pattern and structure formation in ABMs is remarkable (Epstein and Axtell 1996; Gilbert 2008; Railsback and Grimm 2012).
Scientific inquiry into the control points of a system and the key drivers of system behaviors, however, can be difficult with ABMs. For this purpose, computational objects such as cellular automata or modeling methods such as discrete event simulation can be thought of as special cases of ABMs, but to date, there is not currently a rigorous formal description of what constitutes an ABM. A major benefit of using ABMs is their ability to generate, through simulation, nonlinear transitions between multiple scales of organization. However, determining parameter settings that lead to different patterns can be extremely difficult. In contrast, this is relatively straightforward for systems of differential equations, for example. Exhaustive simulation to investigate bifurcations and stability, though cheaper and faster than realworld experimentation, can be prohibitively expensive in terms of compute cycles and the resources needed to execute them. Very complex differential equations models may be similarly expensive to evaluate computationally, but the formal mathematical structure of systems of differential equation often permits analyses in a way that the interactionbased structure of an ABM does not. This difference accounts for the significant appeal of more traditional mathematical models. Reducing the complexity of a differential equations model for design and optimization studies can be challenging, but generally the route is clearer than it is with ABMs, features of which may frustrate attempts at reduction by formal inspection and model reduction methods.
A problem of practical interest is that of policy guidance. In distinction with basic scientific inquiry, ecological management, public health, and medical domains need tools for rational, evidencebased decisionmaking in treatments, interventions, and resource management problems. Models have been used successfully to support such efforts. Social and ecological applications often involve problems for which direct experimentation is, at best, difficult. For example, controlling nonnative species such as the wild hog Sus Scrofa in the Great Smoky Mountains National Park (Peine and Farmer 1990) has created a number of political difficulties, and mathematical models are beginning to suggest management strategies (Salinas et al. 2015; Levy et al. 2016). As another example, college drinking is a major public health problem. Calls for a reduction in the minimum legal drinking age suggest the undertaking of a complex, large scale social and political experiment with potentially major consequences. Computational decision aids can support policy investigation when experimentation and testing must necessarily be limited (McCardell 2008; Fitzpatrick et al. 2012, 2016a). Again, exhaustive simulation of control strategies may not be a desirable or even viable option, and developing mathematically tractable tools for winnowing the vast array of control strategies or policies into a manageable set for simulation can enhance this type of model tremendously.
To illustrate just how widely applicable ABMs are, we point to several additional representative examples. At the population level, EpiSims (Eubank 2005; Stroud et al. 2007) is a very large populationlevel ABM that explicitly represents millions of individuals and their daily movements in a faithfully represented urban environment. This movement model is then overlaid with an epidemiological model that can be used to simulate the spread of a pathogen through the population. EpiSims has been used as a policy decisionmaking tool in several contexts. In Wang et al. (2014), an ABM is used to study the impact of social norms on obesity and eating behaviors among US school children. At the tissue scale in the human body, a wide variety of problems have been approached through ABMs. In Ziraldo and Solovyev (2015), an ABM is used for a computational study of treatment options for pressure ulcers in patients with spinal cord injuries. In Gong et al. (2015), an ABM of granuloma formation in tuberculosis is used as a platform for the in silico design of combination therapies with different antibiotics. The ABM is combined with a PDE model to accurately represent diffusion of different molecules through tissue. Many other examples can be found in the literature.
3 Toward a Mathematical Approach to ABMs
There is a nearly irresistible pull, when developing an ABM, toward increasing levels of detail and complexity. The enormous flexibility of ABMs allows a modeler to create agents with many attributes operating in an environment that is heterogeneous in multiple dimensions and to build interaction rules that account for rich, complex behaviors and relationships. As the complexity grows, the dimensionality of the parameter space does as well, and the ability to conduct systematic inquiry becomes more challenging. The ABM structure that invites researchers with its developmental simplicity becomes a significant drawback: as noted above, the computational simulation based on logical rules and individual attributes can be quite resistant to formal mathematical analysis or even to experimental insights, if the model rules are too complex. One strategy for approaching these tradeoffs is “patternoriented modeling (Grimm and Railsback 2005; Grimm et al. 2005), in which one balances model complexity against the ability of the model’s output patterns to match those observed in the real world (see also Thorngate and Edmonds 2013, for pattern analysis in ABMs). We suggest that analysis at a system level can benefit greatly from transformation of the ABM into mathematical formalisms that are more accessible to systemlevel analysis.
As stated in our main hypothesis, we assert that ABMs can be used as proxy systems, rather than as the model to be analyzed directly, in an investigatory pathway that can lead to the development of control strategies for highly complex realworld systems. The complexity of aggregate behaviors observed in ABMs, which is seen by many ABM modelers as unapproachable with systemlevel compartmental or aggregated models, offers new and exciting challenges to the systems theory community, calling for the creation of new approaches.
There are several examples in the literature that can be seen as first steps toward a research program of the kind we are advocating. In Roeder et al. (2006), an ABM is used to study the effects of treating chronic myeloid leukemia with the drug imatinib, known as Gleevec, a tyrosine kinase inhibitor that interrupts key signaling pathways in cancer cells, thereby inhibiting cell proliferation. While being very successful in achieving a substantial reduction in the number of malignant cells, this treatment rarely eliminates all such cells, leading to cancer recurrence. In Roeder et al. (2006), the model is used to provide evidence for a new hypothesis explaining lack of complete success, which implicates different effects of imatinib on malignant stem cells, leaving a residual pool that replenishes the repertoire of cancer cells after treatment ends. The model supports this hypothesis, which is also corroborated by patient data. The agents in the ABM are cells of different types. There is no explicit spatial environment; rather, cells are divided into two different environments, representing cell growth and quiescence. Cells can move between these environments, depending on different signals they receive. Imatinib treatment affects several different parameters in the model.
At each ABM time step, the model evaluates a collection of probabilistic rules that affects the state of each cell and its location in one or the other of the compartments. Due to the large number of rules to be evaluated, leading to significant computational cost, it is only possible to use a small fraction of the actual number of cells involved. Still, one simulation run of this large stochastic model requires on the order of 6 h, with hundreds of thousands of rules to be evaluated at each time step. In Kim et al. (2008), the authors developed a deterministic difference equations model, consisting of approximately 6000 equations, that faithfully reproduces the behavior of this ABM and can be simulated in a matter of seconds. Cells are clustered depending on their state, and there is no limit on the size of the clusters, so that the model can represent any number of cells. This clustering approach also allows the model to be deterministic rather than stochastic. Then, in Kim et al. (2008), a (deterministic) PDE model was presented that accomplished the same task, agreeing with both the ABM and the difference equations model in almost all aspects. The main difference is that continuous time causes some aspects of the model to behave differently from either of the discrete time models. We view this progression of models as a case study of how one can move from a complex and hard to execute ABM to a more easily manageable equationbased mathematical model that allows analysis.
4 Optimization and Control
A wellknown family of ABMs known as Sugarscape (Wilensky 2009; Epstein and Axtell 1996) has been used for the study of a variety of control processes in the life sciences, social science, and economics. The stochastic Sugarscape ABMs include agent heterogeneity, environmental heterogeneity, and accumulation of agent resources (i.e., sugar) over time, thus incorporating the main complexities frequently found in ABMs. Agents negotiate a spatial environment in search of a resource called sugar, with higher sugar concentrations represented as elevations in the landscape. Different agents have differing abilities to perceive sugar gradients, leading to different levels of agent fitness. Complete lack of sugar leads to agent death. Control is included as taxation of agents’ sugar resources, with the goal of maximizing a weighted combination of total taxes less a measure of the impact of taxation on the population. Recently, Christley et al. (2015) approximated a Sugarscape model using a system of parabolic PDEs. The goal was to explore optimal control scenarios for Sugarscape, applying mathematical optimization approaches to the PDE model. This approach performed well in scenarios in which the control was assumed to be constant. Optimal controls generated by applying optimal control theory to the PDE system provided timevarying tax rates specific to an agent’s location and current wealth. When implemented in the ABM, the optimal controls performed reasonably well even though some error was introduced between the PDE and ABM systems.
Several different approaches to Sugarscape control are described in Oremland and Laubenbacher (2014a, b), approaches which illustrate the philosophy advocated herein. One approach focuses on dimension reduction of the ABM (by reducing the number of spatial locations, agents, and other aspects), while preserving those model features relevant for a given control objective. This is done by applying a randomly chosen collection of controls to both the original and the reduced ABM and computing the similarity of the relative rankings of the controls for the two models. The user can then choose a level of similarity that is acceptable for a particular control objective, thereby deciding how closely the reduced model needs to fit the original one for the control purpose. Controls are then computed for the reduced model and lifted to the original one. Note that the reduced model might be dramatically different from the original one in other aspects. The advantage of the reduced model might be ease of computation, although care must be taken that this is indeed the case. The optimization method chosen is Pareto optimization. This multiobjective optimization method has the advantage that it computes a collection of controls, each of which has the property that optimality in one objective cannot be improved without losing optimality in another. Thus, it computes optimal control inputs for various weightings of the individual objectives. The user can then decide which to choose. Another approach that can be taken using either the original model, or, possibly more easily, the reduced model, is to approximate the ABM by an equationbased model that can be used for control. Again, the equation model might be adequate for a control objective without preserving many other important features of the ABM, e.g., spatial heterogeneity. In Oremland and Laubenbacher (2014b), for instance, the rabbitsandgrass model is approximated almost perfectly, for the purpose of rabbit control, by a pair of fairly simple difference equations. But these have phenomenological parameters and contain no information about the spatial aspects of the model. This is also illustrated with a Sugarscape example.
Another approach to approximation of ABMs and corresponding optimal control has been developed by Lenhart et al. (2015, 2016), using a system of stochastic partial differential equations with nonlocal terms as an approximation, with corresponding novel optimal control results. The control of this system is motivated by a model for optimal harvesting of a population on a spatial grassland habitat, which is described below.
We interpret these examples as evidence that it is possible to approximate large, complex, stochastic ABMs with mathematical models that are easier to execute and can be analyzed with mathematical methods. We will elaborate on this approach below, using a much simpler ABM as an illustrative proof of concept and further validation of our main hypothesis.
5 A Case Study: Ecological Pest Control
To focus the discussion on mathematical issues, we consider a comparatively simple application problem in which an ABM is a natural and easily implemented model, and for which control policies are of interest. We consider a twospecies consumerresource structure in a twodimensional spatial domain. The spatial domain is divided into discrete patches, and time progresses in discrete steps. The resource, which we call grass, is produced with a commercial goal in mind. The species, called rabbits, consumes the grass and hence degrades the commercial viability of the grass crop. The simple model we examine involves rabbits and grass distributed across a rectangular domain. This model is implemented in the NetLogo framework (Wilensky 1999, 2001) as “Rabbits–Grass–Weeds.” In this simple example, each rabbit’s state is characterized by four quantities: its latlong position in space, the angle it faces, and its energy content. Energy content is measured by the amount of grass the rabbit consumes, and when the rabbit crosses an energy threshold, it produces one rabbit as offspring. Movement is governed by an angular random walk: the rabbit chooses two uniformly distributed angles (left, right), differences them, adds that to its current facing angle, and moves one unit in that direction. The grass state in each position patch is 0 or 1, denoting presence or absence. Grass grows from 0 to 1 at a random rate. Energy content of a rabbit corresponds to the number of grass patches consumed. If a rabbit’s energy level falls below a al threshold, the rabbit dies. There are a small number of parameters in the model: the grass growth probability, the rabbit’s angular field of vision (affecting its movement), as well as birth and death thresholds.
While we focus on this simple version of the model in this paper, the problem is easily made more complex in the presence of a few natural generalizations. First, the grass growth probability may be spatially heterogeneous, dependent on local soil and water conditions. Second, rabbits may be drawn to areas within the region of interest with the highest grass content, bringing about a directed random walk that is chemotactic in nature. Third, the birth and death probabilities of the rabbits may not be identical, potentially leading to “winners” and “losers” within the rabbit community. Even with these potential complexifiers, this model does lack some interesting and important properties that make ABM behavior so rich, such as agent adaptability. Nonetheless, this model does include stochastic variation among agents and modeling features (e.g., the nature of the agent movement and of the conversion of grass to rabbits) that are not easily treated by traditional SLM approaches. Furthermore, because the current goal of this paper is to demonstrate the mapping between ABMs and equationbased systemlevel models in order to apply optimal control methods, we have chosen the simplified version of the rabbit–grass ABM as the initial starting point for the investigation of creating the crossplatform mapping process. By using simplified models, we emphasize the mapping and connection between two different perspectives of representing knowledge about the system, i.e. individualbased knowledge versus aggregated systemlevel knowledge. Our stepwise approach will then attempt to perform this mapping with increasingly sophisticated ABMs and target systemlevel modeling methods (see below).
The available control is that of harvesting, implemented as a probability of being harvested. In each patch and at each time step, the harvesting probability is specified. A random number is generated, and, if that number is less than the harvest rate, any rabbits in the patch are harvested. In its simplest instantiation, this probability may be uniform across the region of interest, but one may also implement harvesting with greater effort in some areas than in others.
It is at this point that we see some difficulty in the agentbased formulation. One may attempt to “wrap an optimization loop around” the ABM in order to devise an optimal harvesting plan. However, the stochastic nature of many ABMs requires careful consideration of optimization algorithms: even with many simulation runs, some variability in output limits the effectiveness of gradientbased search methods. Stochastic approximation techniques may help, and more heuristic global optimization schemes offer potential as well. The issues of obtaining reliable data from repeated simulation and the effect of spatial scale on resulting dynamics were investigated in Oremland and Laubenbacher (2014a). While that paper presents several techniques for control of ABMs directly via simulation, it would seem more appealing in general to apply the welldeveloped systemstheoretic tools of optimal control.
6 Control Techniques
The system may constitute a medical patient receiving treatment, with outputs being physiological measures of health. Another example is a spatially distributed ecosystem of pest predators and their prey, with inputs being trapping strategies implemented over time and distributed spatially, and outputs being observations of total pest population. The system may be a society of agents with different incomes and wealth levels, with a control being taxation, as in Sugarscape.
The inputs can be static, like parameters, or dynamic. Dynamic inputs can include control signals designed to influence the system and disturbances, stochastic or deterministic. The observed output signal represents those things we can measure.
The possible objectives of control are either to keep a system’s output within some desired operating range of interest, or to move a system from its current state to a more desirable one. Designing controls to meet these objectives is generally approached by choosing an optimization criterion or objective functional to be maximized or minimized.
Among the challenges of modeling, particularly modelbased decision and control, is the identification of the state variables needed to capture the dynamics of a complex system with many scales and interacting components. The dynamical system may need to account for spatial heterogeneity (leading often to partial differential equations like diffusions). Other sorts of heterogeneity (e.g., different levels of susceptibility to disease, different levels of metabolism) also make for challenges in state variable modeling approaches. As an example, the effect of simple and straightforward changes in agent movement on the ability to formulate a systemlevel model (SLM) is presented in Oremland and Laubenbacher (2015). To what extent aggregation can be tolerated in an SLM of a complex heterogeneous system and its associated agentbased model is an important topic, the investigation of which offers exciting mathematical research opportunities.
7 Systems Analysis and Control for AgentBased Models
Systemlevel modeling of an ABM might reasonably be viewed as the same activity as developing a model of the real system represented by the ABM. Indeed, one fruitful view of ABMs is that of experimental surrogate. ABM development often centers very directly on translating the process details of a physical, chemical, biological, or social system into behavioral rules that can be instantiated in code. There are, however, a number of fundamental differences.
First and foremost, the ABM may involve a number of control inputs and parametric settings that are not practically realizable in experimentation. This situation is particularly relevant in biological and social systems, for which experiments may involve difficult ethical questions. Second, the ABM may allow one to monitor processes for which current measurement technologies do not exist. Additional measurements in the computational model can provide important insights into control algorithm performance and stimulate innovation in science and engineering of sensor and other measurement technologies. Third, we note that the simulation can be exercised in a wider variety of configurations and spatial and temporal sampling rates than can most physical, biological or social systems.
In order to project down from the ABM to this model, the total amounts of grass and rabbits can be computed from ABM output. However, the reverse direction is not welldefined: lifting up from the simple Lotka–Volterra model to the state of the ABM is not unique. In particular, the harvesting rate must be distributed as a probability for grass growth across the spatial domain.
This issue presents an important general consideration in developing an SLM for system analysis of an ABM: mapping back and forth between the two state spaces of the models. Generally speaking, the mapping from ABM states to SLM states, which we call a projection, is straightforward. Often the SLM states can be viewed as summary statistics of the ABM, be they total abundances, means, or fractions with specific attributes or properties. Summing the number of agents with a certain property (e.g., all the rabbits or all the grass) produces the necessary projection in many situations.
The mapping from the SLM to the ABM, which we call a lifting, requires more care. ABM states tend to involve much more detail than SLM states, but the summary statistic view can help in lifting: ABM states can often be approximated as random samples from a distribution that depends on a simple parameterization that can be characterized using the SLM state. For example, with a known number of rabbits, we can randomly distribute rabbits across the spatial domain. Lifting the state variable itself is not the only consideration here: lifting the control input, which will be used to influence the ABM dynamics, is also an issue.
A partial breakdown of modeling, estimation, and control methods
Modeling approaches  Parameter estimation methods  Control design techniques 

1. Discrete input–output  Output least squares 1,4,5  Pontryagin’s maximum principle 1,4,5 
2. Markov chain  Equation error 4,5  Dynamic programming 1–5 
3. Polynomial dynamical systems  Maximum likelihood 1–5  Large scale constrained optimization 1–5 
4. Difference equations  Maximum a posteriori 1–5  
5. Differential equations 
 (1)
Stochastic optimization applied directly to the ABM;
 (2)
Mechanistically derived SLM constructed as a surrogate for the ABM; and
 (3)
Empirically fit SLM constructed as a surrogate for the ABM.
7.1 Stochastic Optimization Applied Directly to the ABM
Here we view the ABM as an input–output system, much like an experiment with settings and measurements. We consider the homogeneous rabbitsandgrass model, and we apply a harvesting strategy defined as follows. For a full description, see Oremland and Laubenbacher (2014a). At each ABM time step, harvesting may or may not be applied. The harvesting has an effectiveness parameter, modeled as a probability. The probability creates a stochastic mortality for each rabbit. The harvesting remains in effect for a second time step, but its effectiveness is reduced by half. The control is a binary time sequence of apply/do not apply, uniformly distributed across the region.
The objective in this example has two features: minimize a weighted combination of the rabbit population and the level of harvesting. We consider a Pareto approach that allows us to trade off these two objectives, implementing the optimization with a genetic algorithm. In this approach, we construct a set of “seed” controls, which, again, is a set of binary sequences representing the harvesting schedule. Each of these is evaluated in the ABM, with the number of surviving rabbits as output. We estimate the Pareto frontier by removing all controls whose objective pair (surviving rabbits, days of harvesting) is dominated by any other control. Controls that persist are paired to produce offspring by randomly choosing parent sequence components. The details of the procedure can be found in Oremland and Laubenbacher (2014a).
This approach provides a suite of solutions, each of which may be optimal, depending on the preferences of the modeler. Hence, the optimal solution chosen can be thought of as a ‘managerial’ decision. While this approach demonstrates a means by which optimal control results can be obtained directly from ABM execution, it lacks a level of mathematical rigor that may be necessary for conclusive statements.
7.2 Mechanistic SLM Derived for ABM Approximation
In this approach, we also view the ABM as an input–output experimental system. We consider the basic processes captured in the ABM, with an eye toward developing a model amenable to control approaches. We construct a simple consumerresource ODE system with control. For a full description, see Federico et al. (2013).
The solution approach of Federico et al. (2013) is closely related to Pontryagin’s principle: a forwardintime state equation is supplemented with a backwardintime costate equation, the latter being a characterization of Lagrange multipliers in this constrained optimization problem. The equations are solved iteratively, and the converged pair produces the optimal control. This problem, solved by itself without the context of the ABM, leads to a time schedule of harvesting rates.
To lift the resulting control to the ABM, just like the harvesting strategy of the “brute force” optimization above, this rate is used as a random mortality, applied uniformly across the domain. Each rabbit will be harvested with probability u(t).
7.3 Empirical SLM Derived for ABM Approximation
Here, we again view the ABM as an input–output experimental system. We select the output statistics to control: namely the abundance of rabbits. Our goal, once again, is to devise a harvesting strategy, uniformly applied across the domain, to reduce the abundance of the pest rabbit population.
Rather than consider the mechanistic approach of modeling the rabbit/grass dynamics as consumerresource, we simply consider the most desirable state of the system, which is every cell populated by grass with no rabbits present. To achieve this state while controlling the level of harvesting effort is the objective. Toward that end, we consider a rabbitsandgrass model with an initial state of \((r_{0}, g_{0})=(0,N)\) and the desired final state of \((r_{f}, g_{f})\), where N denotes the number of cells in the ABM. We note that this desired final state is an equilibrium: if there are no rabbits to begin with, the grass will eventually populate every cell. It is unstable; however, as soon as a single rabbit is introduced, the population can and will increase according to the rules of the ABM.
8 Discussion
ABMs offer an interesting investigative tool for science, engineering, public health, and public policy. Replicating important features of realworld systems with relatively simple rulebased computational models make ABMs attractive alternatives to traditional mathematical models. Using ABMs as decision tools to improve or optimize realworld systems, however, remain a significant challenge. In this “Perspectives” article, we have examined a number of systemslevel control approaches that might be applied to ABMs, using the simple Rabbits–Grass–Weed ABM as a pest control example.
In approaching an ABM from the point of view of control, a number of key problems arise, especially when SLM approximations are used. Most fundamental is the mapping of inputs and outputs between ABMscale and SLMscale quantities, which we have called “lifting” and “projection.” Next, one must consider the necessary complexity of the SLM required to capture the ABM dynamics relevant to the control inputs and objective.

Methods for dimension reduction;

The extensive heterogeneity possible for individual agents;

Agent adaptation;

Multiple time scales that emerge in complex ABMs;

Taxonomy of ABMs with respect to different SLM paradigms and success of one of the three approaches outlined in this paper (or others);

Rigorous methods for projection and lifting in order to map between analytical and simulation models;

Efficient computational experimental design; and

Uncertainty quantification in assessing SLM–ABM compatibility.
We have presented some possible ways to begin accomplishing this, even though the work to date is ad hoc and does not address many of the obvious questions that remain unanswered. Our investigations have mostly focused on two models, Sugarscape and RabbitsandGrass, in some variations. They were chosen because they include features found in many ABMs in the literature, but they are arguably too simple to draw conclusions about the broader applicability of our methods. Furthermore, the methods themselves are ad hoc and have not been analyzed rigorously. Are there common principles in how an ABM is approximated by a difference equations model or a PDE model? What are the limits of dimension reduction methods, and are there other comparisons of models that should be used to judge the faithfulness of a reduced model? Answers to these and many other questions await further research, and an important motivation for this paper is the hope that it will engage others in work on these problems. One important tool would be a suite of benchmark ABMs, representative of the different ABM types in use, that illustrate the applicability and limitations of the various techniques. And, of course, other, complementary, techniques will need to be developed.
Notes
Acknowledgments
This Project grew out of a NIMBioS (National Institute for Mathematical and Biological Synthesis) workshop in December 2009, followed in 2011 by ongoing work of an interdisciplinary NIMBioS working group (15 members) “Agentbased modeling of biological systems: pathways to control and optimization.” The workshop and working group were funded by NIMBioS, sponsored by the National Science Foundation, the U.S. Department of Homeland Security and the U.S. Department of Agriculture, through NSF Award EF0832858NSF. The work of Lenhart and Fitzpatrick is also partially supported by NIMBioS Award DBI1300426. Fitzpatrick has also been supported in part by the Air Force Office of Scientific Research grants FA95500910524 and FA95501010499. The work of Oremland and Laubenbacher has been supported by Grants N911NF0910538 (U.S. DoD) and DMS1062878 (NSF), and Oremland was partially supported by NSF grant DMS0931642 to the Mathematical Biosciences Institute. An’s work was partially supported by National Institutes of Health Grants R01GM115839 and P30DK42086.
References
 Armaou, A (2004). Continuoustime control of distributed processes via microscopic simulation. In: Proceedings of 2004 ACC, Boston, pp. 933–939Google Scholar
 Christley S, Miller Neilan R, Oremland M, Salinas R, Lenhart S (2015). Optimal control of SugarScape agentbased model via a PDE approximation model, preprint 2015Google Scholar
 Epstein J, Axtell R (1996) Growing artificial societies. Brookings Institution Press, BostonGoogle Scholar
 Eubank S (2005) Network based models of infectious disease spread. Jpn J Infect Dis 58(6):S9–13Google Scholar
 Federico P, Lenhart S, Ryan D, Gross L (2013) Optimal control in individual based models: implication from aggregated methods. Am. Nat. 181:64–71CrossRefGoogle Scholar
 Fisher RA (1926) The design of experiments. Oliver and Boyd, EdinburghGoogle Scholar
 Fitzpatrick B, Scribner R, Ackleh A, Rasul J, Jacquez G, Rommel R, Simonsen N (2012) Forecasting the effect of the Amethyst Initiative on college drinking. ACER 36(9):1608–1613Google Scholar
 Fitzpatrick B, Martinez J, Polidan E, Angelis E (2016) On the effectiveness of social norms intervention in college drinking: the roles of identity verification and peer influence. Alcohol Clin Exp Res 40(1):141–151CrossRefGoogle Scholar
 Fitzpatrick, B. (2016). An empirical systems identification approach to control of agentbased models (in preparation)Google Scholar
 Gilbert N (2008) Agentbased models. Sage, Thousand OaksCrossRefGoogle Scholar
 Gong C, Linderman JJ, Kirschner D (2015) A population model capturing dynamics of tuberculosis granulomas predicts host infection outcomes. Math Biosci Eng 12(3):625–642MathSciNetCrossRefMATHGoogle Scholar
 Grimm V, Railsback S (2005) Individual based modeling and ecology. Princeton University Press, PrincetonGoogle Scholar
 Grimm V, Revilla E, Berger U, Jeltsch F, Mooij WM, Railsback SF, Thulka HH, Weiner J, Wiegand T, DeAngelis DL (2005) Patternoriented modeling of agentbased complex systems: lessons from ecology. Science 310:987–991CrossRefGoogle Scholar
 Hinkelmann F, Murrugarra D, Jarrah A, Laubenbacher R (2011) A mathematical framework for agentbased models of complex biological networks. Bull Math Biol 73(7):1583–1603MathSciNetCrossRefMATHGoogle Scholar
 Kevrekedis IG, Gear CW, Hummer G (2004) Equationfree: the computeraided analysis of complex multiscale systems. AIChE J 50(7):1346–1355CrossRefGoogle Scholar
 Kim PS, Lee PP, Levy D (2008) Modeling imatinibtreated chronic myelogenous leukemia: reducing the complexity of agentbased models. Bull Math Biol 70(3):728–744MathSciNetCrossRefMATHGoogle Scholar
 Kim PS, Lee PP, Levy D (2008b) A PDE model for imatinibtreated chronic myelogenous leukemia. Bull Math Biol 70(7):1994–2016MathSciNetCrossRefMATHGoogle Scholar
 Lenhart S, Xiong J, Yong J (2015) Controlled stochastic partial differential equations for rabbits on a grassland. preprintGoogle Scholar
 Lenhart S, Xiong J, Yong J (2016) Optimal controls for stochastic partial differential equations with an application in population modeling. SIAM J Control Optim 54(2):495–535Google Scholar
 Levy B, Collins C, Lenhart S, Salinas RA, Madden M, Corn JL, Stiver W (2016) A metapopulation model for feral Hogs in Great Smoky Mountains National Park. Nat Resour Model J 29(1):71–97Google Scholar
 McCardell J (2008) The drinking age. The status quo has bombed. US News World Report 145:11Google Scholar
 Morris MD, Mitchell TJ, Ylvisaker D (1993) Bayesian design and analysis of computer experiments: use of derivatives in surface prediction. Technometrics 35(3):243–255MathSciNetCrossRefMATHGoogle Scholar
 Murray J (2011) Mathematical biology ii: spatial models and biomedical applications. Springer, New YorkGoogle Scholar
 Oremland M, Laubenbacher R (2014a) Optimization of agentbased models: scaling methods and heuristic algorithms. J Artif Soc Soc Simul 17(2):6Google Scholar
 Oremland M, Laubenbacher R (2014) Using difference equations to find optimal tax structures on the SugarScape. J Econ Interact Coord 9(2):233–253CrossRefGoogle Scholar
 Oremland M, Laubenbacher R (2015) Optimal harvesting for a predatorprey agentbased model using difference equations. Bull Math Biol 77(3):434–459MathSciNetCrossRefMATHGoogle Scholar
 Peine J, Farmer J (1990) Wild Hog Management Program at Great Smoky Mountain National Park. In: Proceedings of 14th vertebrate pest conference, Paper 67Google Scholar
 President’s Information Technology Advisory Committee (2005) Computational science: ensuring America’s competitiveness. National Coordination Office for Information Technology Research and Development, ArlingtonGoogle Scholar
 Railsback S, Grimm V (2012) Agentbased and individualbased modeling: a practical introduction. Princeton University Press, PrincetonMATHGoogle Scholar
 Roeder I, Horn M, Glauche I, Hochhaus A, Mueller MC, Loeffler M (2006) Dynamic modeling of imatinibtreated chronic myeloid leukemia: functional insights and clinical impliations. Nat Med 12(10):1181–1184CrossRefGoogle Scholar
 Salinas RA, Stiver WH, Corn JL, Lenhart S, Collins C, Madden M, Vercauteren KC, Schmit BB, Kasari E, Adoi A, Hickling G, Mccallum H (2015) An individualbased model for feral hogs in Great Smoky Mountains National Park. Nat Resour Model 28(1):18–29MathSciNetCrossRefGoogle Scholar
 Schelling T (1971) Dynamic models of segregation. J Math Sociol 1(2):143–186CrossRefGoogle Scholar
 Smarr LL (1992) How supercomputers are transforming science, in yearbook of science and the future. Encyclopedia Britannica, ChicagoGoogle Scholar
 Stroud P, Del Valle S, Sydoriak S, Riese J, Mniszewski S (2007) Spatial dynamics of pandemic influenza in a massive artificial society. J Artif Soc Soc Simul 10(4):9Google Scholar
 Thorngate W, Edmonds B (2013) Measuring simulation–observation fit: an introduction to ordinal pattern analysis. JASSS 16(2):4CrossRefGoogle Scholar
 Wald A (1939) Contributions to the theory of statistical estimation and hypothesis testing. Ann Math Stat 10(4):299–326MathSciNetCrossRefMATHGoogle Scholar
 Wang Y, Xue H, Chen HJ, Igusa T (2014) Examining social norm impacts on obesity and eating behaviors among US school children based on an agentbased model. BMC Public Health 14:923CrossRefGoogle Scholar
 Wilensky U (2001) http://ccl.northwestern.edu/netlogo/models/RabbitsGrassWeeds. NetLogo Rabbits Grass Weeds model, Center for connected learning and computerbased modeling. Northwestern University, Evanston
 Wilensky U (2009) http://ccl.northwestern.edu/netlogo/models/community/SugarscapeW. Sugarscape, Center for connected learning and computerbased modeling. Northwestern University, Evanston
 Wilensky U (1999) NetLogo. http://ccl.northwestern.edu/netlogo/. Center for connected learning and computerbased modeling. Northwestern University, Evanston
 Ziraldo C, Solovyev A et al (2015) A computational, tissuerealistic model of presure ulcer formation in individuals with spinal cord injury. PloS Comput Biol 11(6):e1004309CrossRefGoogle Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.