A mathematical analysis of the long-run behavior of genetic algorithms for social modeling
- 685 Downloads
- 1 Citations
Abstract
We present a mathematical analysis of the long-run behavior of genetic algorithms (GAs) that are used for modeling social phenomena. Our analysis relies on commonly used mathematical techniques in the field of evolutionary game theory. We make a number of assumptions in our analysis, the most important one being that the mutation rate is positive but infinitely small. Given our assumptions, we derive results that can be used to calculate the exact long-run behavior of a GA. Using these results, the need to rely on computer simulations can be avoided. We also show that if the mutation rate is infinitely small the crossover rate has no effect on the long-run behavior of a GA. To demonstrate the usefulness of our mathematical analysis, we replicate a well-known study by Axelrod in which a GA is used to model the evolution of strategies in iterated prisoner’s dilemmas. The theoretically predicted long-run behavior of the GA turns out to be in perfect agreement with the long-run behavior observed in computer simulations. Also, in line with our theoretically informed expectations, computer simulations indicate that the crossover rate has virtually no long-run effect. Some general new insights into the behavior of GAs in the prisoner’s dilemma context are provided as well.
Keywords
Genetic algorithm Long-run behavior Social modeling Economics Evolutionary game theory1 Introduction
The field of evolutionary computation is concerned with the study of all kinds of evolutionary algorithms. These algorithms can be used for various purposes. Perhaps the most popular purpose for which they can be used is function optimization (e.g., Gen and Cheng 2000; Goldberg 1989; Michalewicz 1996). In the function optimization context, evolutionary algorithms can be seen as heuristics that serve as alternatives to more traditional techniques from the fields of combinatorial optimization and mathematical programming. Another important purpose for which evolutionary algorithms can be used is the modeling of biological and social phenomena (e.g., Mitchell 1996). This is the topic with which we are concerned in this paper. Our focus is in particular on the use of evolutionary algorithms for modeling social phenomena.
When using evolutionary algorithms in the social modeling context, one of the assumptions one makes is that the agents whose behavior is being modeled are boundedly rational. This basically means that the agents are assumed not to behave in a utility-maximizing manner. There are numerous ways in which boundedly rational behavior can be modeled (e.g., Brenner 2006; Fudenberg and Levine 1998). A popular approach is to rely on an evolutionary metaphor. This is the approach that is taken by evolutionary algorithms. In its simplest form, the evolutionary approach assumes that there is a population of agents and that for each agent in the population the strategy it uses depends on the population-wide past performance of strategies. The better the past performance of a strategy, the more likely the strategy is to be used again. The evolutionary approach also assumes that there always is a small probability that an agent experiments with a new strategy.
The evolutionary approach to modeling boundedly rational behavior has attracted a lot of attention, not only from researchers in the field of evolutionary computation but also from researchers in the social sciences, in particular from economists. Traditionally, economists have typically relied on game-theoretic models to analyze interactions between agents. These models assume agents to behave in a fully rational way. Nowadays, however, the limitations of game-theoretic models are well recognized and many economists have started to study evolutionary models of agent behavior. These models are based on the assumption that the behavior of agents can best be described using some evolutionary mechanism rather than using the idea of full rationality.
In the field of economics, there are two quite separate streams of research that are both concerned with the evolutionary approach to modeling boundedly rational behavior. One stream of research, which is usually referred to as agent-based computational economics (e.g., Tesfatsion 2006), makes use of techniques from the field of evolutionary computation. Especially genetic algorithms (GAs) are frequently used. Early work in this stream of research includes (Andreoni and Miller 1995; Arifovic 1994, 1996; Dawid 1996; Holland and Miller 1991; Marks 1992; Miller 1986), and examples of more recent work are Alkemade et al. (2006, 2009), Georges (2006), Haruvy et al. (2006), Lux and Schornstein (2005), Vriend (2000), Waltman and Van Eck (2009), and Waltman et al. (2011). The other stream of research is more closely related to traditional game theory and is referred to as evolutionary game theory (e.g., Gintis 2000; Maynard Smith 1982; Vega-Redondo 1996; Weibull 1995). Like the traditional game-theoretic approach, the evolutionary game-theoretic approach is model-based and relies heavily on mathematical analysis. The use of computer simulations is not very common in evolutionary game theory.
In this paper, it is not our aim to argue in favor of either the agent-based computational economics approach, which emphasizes algorithms and computer simulations, or the evolutionary game-theoretic approach, which emphasizes models and mathematical analysis. Instead, we want to show how the former approach can benefit from the mathematical techniques used in the latter approach. More specifically, we want to show how evolutionary algorithms that are used for modeling social phenomena can be analyzed mathematically using techniques that are popular in evolutionary game theory. Our focus in this paper is on one particular type of evolutionary algorithm, namely GAs with a binary encoding. However, we emphasize that the approach that we take can be applied to other types of evolutionary algorithms as well. The reason for focusing on GAs with a binary encoding is that this seems to be the type of evolutionary algorithm that is used most frequently for modeling social phenomena (e.g., Alkemade et al. 2006, 2007; Andreoni and Miller 1995; Arifovic 1994, 1996; Ashlock et al. 1996; Axelrod 1987; Crowley et al. 1996; Dawid 1996; Georges 2006; Ishibuchi and Namikawa 2005; Lux and Schornstein 2005; Marks 1992; Miller 1986, 1996; Van Bragt et al. 2001; Vriend 2000; Yao and Darwen 1994).
- 1.
Our mathematical results can be used to calculate the long-run behavior of a GA exactly, while computer simulations can only be used to estimate the long-run behavior of a GA.
- 2.
When using computer simulations, it can be difficult to determine how many iterations of a GA are required to approximate the long-run behavior of the GA reasonably closely. Our mathematical results do not have this problem.
- 3.
Calculating the exact long-run behavior of a GA using our mathematical results requires less computing time than obtaining a reasonably accurate estimate of the long-run behavior of a GA using computer simulations.
Our mathematical results have one important limitation, which is that on most of today’s computers they can only be used if the chromosome length is not greater than about 24 bits. If the chromosome length is greater than about 24 bits, the use of our mathematical results to calculate the long-run behavior of a GA most likely requires a prohibitive amount of computer memory.
Like in Dawid (1996), the mathematical analysis presented in this paper relies on the assumption that the mutation rate is positive but infinitely small. (In other words, the analysis is concerned with the limit case in which the mutation rate approaches zero.) In simulation studies with GAs, researchers typically work with values between 0.001 and 0.01 for the mutation rate. This seems to be a rather pragmatic choice (cf. Dawid 1996). On the one hand, lower values for the mutation rate would lead to very slow convergence and, consequently, very long simulation runs. On the other hand, higher values for the mutation rate would lead to convergence to unstable, difficult to interpret outcomes. We believe that our assumption of an infinitely small mutation rate is justified because an infinitely small mutation rate is less arbitrary than a mutation rate whose value is determined solely based on pragmatic grounds (cf. Foster and Young 1990). The assumption of an infinitely small mutation rate is also in line with the common practice in evolutionary game theory, in which a similar assumption is almost always made. The advantage of assuming an infinitely small mutation rate is that it greatly simplifies the mathematical analysis of the long-run behavior of GAs (see also Dawid 1996). In fact, GAs with an infinitely small mutation rate can be analyzed in a similar way as well-known models in evolutionary game theory (e.g., Foster and Young 1990; Kandori et al. 1993; Vega-Redondo 1997; Young 1993). Like in evolutionary game theory, mathematical results provided by Freidlin and Wentzell (1998) are the key tool for analyzing the long-run behavior to which convergence will take place. We note that, in addition to the assumption of an infinitely small mutation rate, there are some other technical assumptions on which our mathematical analysis relies. Most of these assumptions are not very strong and will probably be satisfied by most GAs.
To demonstrate the usefulness of our mathematical analysis, we replicate a well-known study by Axelrod (1987) (reprinted in Axelrod 1997; see also Dawid 1996; Mitchell 1996). Axelrod used a GA to model the evolution of strategies in iterated prisoner’s dilemmas (IPDs). He showed that an evolutionary mechanism can lead to cooperative behavior. Axelrod’s study has been one of the first and also one of the most influential studies on the use of evolutionary algorithms for modeling social phenomena. Directly or indirectly, his study seems to have inspired many researchers (e.g., Ashlock et al. 1996, 2006; Chong and Yao 2005; Crowley et al. 1996; Fogel 1993; Ishibuchi and Namikawa 2005; Mühlenbein 1991; Thibert-Plante and Charbonneau 2007; Van Bragt et al. 2001; Yao and Darwen 1994). The results obtained by Axelrod are all based on computer simulations. In this paper, we show that more or less the same results can be calculated exactly, with no need to rely on simulations. We also discuss some new insights that exact calculations provide.
The mathematical analysis that we present in this paper also has an important implication for the choice of the parameters of a GA. The analysis indicates that if the mutation rate is infinitely small the crossover rate has no effect on the long-run behavior of a GA. This is a quite remarkable result that, to the best of our knowledge, has not been reported before in the theoretical literature on GAs. The result implies that when GAs are used for modeling social phenomena the crossover rate is likely to be a rather insignificant parameter, at least when one is mainly interested in the behavior of GAs in the long run (for the short run, see Thibert-Plante and Charbonneau 2007). This suggests that in many cases the crossover rate can simply be set to zero, in which case no crossover will take place at all. Simulation results that we report in this paper indeed show no significant effect of the crossover rate on the long-run behavior of a GA.
The remainder of this paper is organized as follows: in Sect. 2, we present a mathematical analysis of the long-run behavior of GAs that are used for modeling social phenomena. Based on the analysis, we derive an algorithm for calculating the long-run behavior of GAs in Sect. 3. In Sect. 4, we demonstrate an application of the algorithm by replicating Axelrod’s study (1987). Finally, we discuss the conclusions of our research in Sect. 5. Proofs of our mathematical results are provided in the Appendix.
2 Analysis
Overview of the mathematical notation
\(\mathcal{C}\) | Set of all chromosomes |
\(\mathcal{G}(i)\) | Set of all chromosomes that have the same binary encoding as chromosome i except that one bit has been changed from one into zero |
\(\mathcal{H}(i)\) | Set of all chromosomes that have the same binary encoding as chromosome i except that one bit has been changed from zero into one |
m | Chromosome length |
n | Population size |
\(\bar{q}(w)\) | Long-run probability of population w |
\(\hat{q}(w)\) | Long-run limit probability of population w |
\(\hat{\mathbf{q}}\) | Long-run limit distribution |
\(\mathcal{U}\) | Set of all uniform populations |
u(i) | Uniform population consisting of n times chromosome i |
\(\mathcal{V}\) | Set of all populations in which there are at most two different chromosomes and in which the binary encodings of chromosomes differ by at most one bit |
v(i, j, λ) | Population consisting of λ times chromosome i and n − λ times chromosome j |
\(\mathcal{W}\) | Set of all populations |
W_{t} | Population at the beginning of iteration t of a GA |
γ | Crossover rate |
δ(i, j) | Hamming distance between chromosomes i and j |
\(\varepsilon\) | Mutation rate |
μ | Number of different chromosomes Number of uniform populations |
ν | Number of combinations of two chromosomes whose binary encodings differ by exactly one bit |
ξ | Number of populations in which there are exactly two different chromosomes and in which the binary encodings of chromosomes differ by at most one bit |
\(\pi(i, j, \lambda, \lambda^{\prime})\) | Probability that the selection operator turns population v(i, j, λ) into population \(v(i,j, \lambda^{\prime})\) in a single iteration of a GA |
Definition 1
A population is said to be uniform if and only if all n chromosomes in the population are identical. A population is said to be non-uniform if and only if some chromosomes in the population are different.
The following definition introduces the notion of almost uniform populations.
Definition 2
Hence, a non-uniform population is almost uniform if and only if no mutation is required to go from the non-uniform population to some uniform population. This is not a very strong condition. In many cases, all non-uniform populations are almost uniform. For example, if a GA uses roulette wheel selection or tournament selection, there is always a possibility that the selection operator selects n times the same chromosome. In other words, the selection operator can turn any non-uniform population into a uniform population in a single iteration. Because of this, when roulette wheel selection or tournament selection is used, all non-uniform populations are almost uniform.
The following two definitions introduce the notion of a connection from one chromosome to another:
Definition 3
Definition 4
A connection from chromosome i to chromosome j is said to exist if and only if there exists a sequence \((i_1,\ldots, i_N)\) such that \(i_1, \ldots, i_N \in \mathcal{C}, i_1 = i,i_N = j,\) and i_{M} is directly connected to i_{M+1} for all \(M\in \{1, \ldots, N - 1\}.\)
Definition 3 states that there is a direct connection from chromosome i to chromosome j if and only if the minimum number of mutations required to go from uniform population u(i) to uniform population u(j) is one. We note that in many cases all chromosomes i and j such that δ(i, j) = 1 have mutual direct connections. This is, for example, the case if a GA uses roulette wheel selection and the fitness of a chromosome is always positive. Definition 4 states that there is a connection from chromosome i to chromosome j if and only if there is a sequence of chromosomes starting at i and ending at j such that each chromosome in the sequence is directly connected to its successor. Clearly, if all chromosomes i and j such that δ(i, j) = 1 have mutual direct connections, then each chromosome is connected to all other chromosomes.
It is well known that the population in the current iteration of a GA has no effect on the behavior of the GA in the long run (e.g., Dawid 1996; Nix and Vose 1992). More specifically, the population an infinite number of iterations in the future is statistically independent of the population in the current iteration. The following lemma states this result in a formal way:
Lemma 1
Proof
See the Appendix.
In our analysis, we are concerned with the long-run behavior of GAs in the limit as the mutation rate \(\varepsilon\) approaches zero. We therefore use the following definition:
Definition 5
For \(w \in \mathcal{W}, \hat{q}(w) =\lim_{\varepsilon \rightarrow 0}\bar{q}(w)\) is called the long-run limit probability of population w.
The following theorem states the main result of our analysis:
Theorem 1
Proof
See the Appendix.
There are three comments that we would like to make on the above theorem. First, the result that under certain assumptions non-uniform populations have a long-run limit probability of zero is not new. A similar result can be found in Dawid (1996, Proposition 4.2.1). Second, under the assumptions of the theorem, the long-run limit probability of a population does not depend on the crossover rate γ. This is a quite remarkable result that, to the best of our knowledge, has not been reported before in the theoretical literature on GAs. It indicates that in the limit as the mutation rate \(\varepsilon\) approaches zero γ has no effect on the long-run behavior of a GA. Third, the theorem can be used to calculate the long-run limit distribution \({\hat{\mathbf{q}}}\) only if the probabilities \(\pi(i, j, \lambda, \lambda^{\prime})\) defined in (6) can be calculated for all i and all j such that δ(i, j) = 1 and for all \(\lambda \in \{1, \ldots, n - 1\}\) and all \(\lambda^{\prime} \in \{0, \ldots, n\}.\) Whether this is possible depends on the way in which the fitness of a chromosome is determined and on the selection operator that is used. This in turn depends heavily on the specific problem that one wants to model using a GA. Because of the dependence on the problem to be modeled, we cannot provide any general results for the calculation of the probabilities \(\pi(i, j, \lambda, \lambda^{\prime}).\) In Sect. 4, however, we demonstrate how the probabilities \(\pi(i, j, \lambda,\lambda^{\prime})\) can be calculated for a GA that is similar to the GA used by Axelrod in his seminal paper on GA modeling (Axelrod 1987).
3 Algorithm
In this section, we present an algorithm for calculating the long-run limit distribution \({\hat{\mathbf{q}}}.\) The algorithm is based on Theorem 1. Like Theorem 1, it assumes that all non-uniform populations are almost uniform and that each chromosome in \(\mathcal{C}\) is connected to all other chromosomes in \(\mathcal{C}.\) It also assumes that the probabilities \(\pi(i, j,\lambda, \lambda^{\prime})\) defined in (6) can be calculated for all i and all j such that δ(i, j) = 1 and for all \(\lambda\in \{1, \ldots, n - 1\}\) and all \(\lambda^{\prime} \in \{0, \ldots,n\}.\)
Matrix \(\mathbf{D}\) has μ^{2} = 2^{2m} elements. Consequently, storing all elements of \(\mathbf{D}\) in a computer’s main memory is possible only if the chromosome length m is not too large. It follows from (28) and (31) that the number of non-zero elements in \(\mathbf{D}\) equals μ(m + 1) = (m + 1)2^{m}. Hence, \(\mathbf{D}\) is a rather sparse matrix and a lot of memory can be saved by storing only its non-zero elements.^{2} In addition to the memory efficiency of the way in which \(\mathbf{D}\) is stored, one should also pay attention to the memory efficiency of the method that is used to solve the linear system given by (23) and (24). Gaussian elimination and other direct (i.e., non-iterative) methods for solving linear systems generally require that at least a large number of elements of the coefficient matrix, including zero elements, are stored in memory. Consequently, when using such a method to solve the linear system given by (23) and (24), it would not be possible to fully exploit the sparsity of \(\mathbf{D}.\) Linear systems can also be solved using iterative methods that require only the non-zero elements of the coefficient matrix to be stored in memory. One such method is the method of successive overrelaxation (e.g., Barrett et al. 1994; Stewart 1994; Tijms 1994, 2003). In the algorithm in Fig. 2, this method is used to solve the linear system given by (23) and (24) (see lines 3–3 of the algorithm). In addition to an initial guess \({\hat{\mathbf{q}}_0}\) for the solution of the linear system, the method of successive overrelaxation also requires a value for the relaxation factor ω. The value of ω, which should be between 0 and 2, may have a large effect on the rate of convergence of the method, and for some values of ω the method may not converge at all. An appropriate value for ω has to be determined experimentally. For ω = 1, the method of successive overrelaxation reduces to the Gauss-Seidel method, which is another iterative method for solving linear systems. We refer to Stewart (1994) for an in-depth discussion of both the method of successive overrelaxation and a number of alternative methods for solving linear systems similar to the one given by (23) and (24). We further note that the amount of main memory in most of today’s computers allows the algorithm in Fig. 2 to be run for chromosomes with length m up to about 24 bits.
4 Application
In this section, we demonstrate an application of the algorithm presented in the previous section. We study the use of a GA for modeling the evolution of strategies in IPDs. The use of GAs in this context was first studied by Axelrod (1987) (reprinted in Axelrod 1997; see also Dawid 1996; Mitchell 1996) and after him by many others (e.g., Ashlock et al. 1996, 2006; Crowley et al. 1996; Ishibuchi and Namikawa 2005; Miller 1996; Mühlenbein 1991; Thibert-Plante and Charbonneau 2007; Van Bragt et al. 2001; Yao and Darwen 1994). The algorithm presented in the previous section is used to analyze the long-run behavior of our GA. The results of the analysis are compared with results obtained using computer simulations (i.e., results obtained simply by running the GA). We emphasize that our primary aim is merely to illustrate the usefulness of the mathematical analysis provided in Sect. 2 and of the algorithm derived from the analysis in Sect. 3. It is not our primary aim to provide new insights into the behavior of GAs in the context of IPDs.
4.1 Genetic algorithm modeling in iterated prisoner’s dilemmas
The way in which we model the evolution of strategies in IPDs is similar to the way in which this was done by Axelrod (1987). However, Axelrod studied two approaches for modeling the evolution of strategies. In one approach, the fitness of a chromosome is determined by the performance of the chromosome in IPD games against a fixed set of opponents. In the other approach, the fitness of a chromosome is determined by the performance of the chromosome in IPD games against other chromosomes in the population. We restrict our attention to the second approach. This is the approach on which almost all studies after Axelrod’s work have focused (an exception is Mittal and Deb 2006).
Payoff matrix for a single period of an iterated prisoner’s dilemma game
Cooperate | Defect | |
---|---|---|
Cooperate | R, R | S, T |
Defect | T, S | P, P |
4.2 Calculation of the long-run limit distribution of the genetic algorithm
The algorithm presented in Sect. 3 also assumes that all non-uniform populations are almost uniform and that each chromosome in \(\mathcal{C}\) is connected to all other chromosomes in \(\mathcal{C}.\) Because of the use of roulette wheel selection, the assumption that all non-uniform populations are almost uniform is satisfied. The assumption that each chromosome in \(\mathcal{C}\) is connected to all other chromosomes in \(\mathcal{C}\) is satisfied if and only if matrix \(\mathbf{D}\) calculated in lines 3–3 of the algorithm in Fig. 2 is irreducible. (\(\mathbf{D} =\left[\begin{array}{l}d(i, j)\end{array}\right]\) is said to be irreducible if and only if there does not exist a non-empty set of chromosomes \(\widetilde{\mathcal{C}} \subset \mathcal{C}\) such that d(i, j) = 0 for all \(i \in \widetilde{\mathcal{C}}\) and all \(j \in\mathcal{C} \setminus \widetilde{\mathcal{C}}.\)) For the particular values that we use for the parameters S, P, R, T, and τ (see the next subsection), \(\mathbf{D}\) turns out to be irreducible. Hence, the assumption that each chromosome in \(\mathcal{C}\) is connected to all other chromosomes in \(\mathcal{C}\) is satisfied.
4.3 Analysis of the long-run behavior of the genetic algorithm
Genetic algorithm parameter settings
Number of runs | 200 or 500 |
Length of a run | \(2 \times 10^ 5\) iterations |
Population size n | 20 chromosomes |
Chromosome length m | 6, 20, or 70 bits |
Selection operator | Roulette wheel with sigma scaling |
Crossover operator | Single point |
Crossover rate γ | 0.0, 0.5, or 1.0 |
Mutation rate \(\varepsilon\) | 10^{−2}, 10^{−3}, 10^{−4}, or 10^{−5} |
Length of an IPD game | 151 periods |
Memory length τ | 1, 2, or 3 periods |
IPD game payoffs | S = 0, P = 1, R = 3, and T = 5 |
The six strategies with the highest long-run limit probability (reported in the first column)
Prob. | Strategy | Chromosomes |
---|---|---|
0.430 | Always defect | 0, 2, 8, 10, 16, 24, 32, 34, 40,42, 48, 50 |
0.147 | Start cooperating; cooperate if and only if both you and your opponent cooperated in the previous period | 56 |
0.139 | Start cooperating; cooperate if and only if your opponent cooperated in the previous period (tit for tat) | 44, 60 |
0.133 | Start defecting; cooperate if and only if you and your opponent played different actions in the previous period | 6, 54 |
0.051 | Start cooperating; cooperate unless you cooperated in the previous period and your opponent did not | 13, 45, 61 |
0.049 | Start defecting; cooperate unless you cooperated in the previous period and your opponent did not | 29 |
In order to check the correctness of the algorithm presented in Sect. 3, we have also used computer simulations to analyze the long-run behavior of our GA. In other words, we have also analyzed the long-run behavior of our GA simply by running the GA. Like above, we first focus on the behavior of the GA for a memory length of τ = 1 period. We performed 500 runs of the GA. The crossover rate was set to γ = 1.0, and the mutation rate was set to \(\varepsilon = 10 ^{-5}.\) Because of the very small value of \(\varepsilon,\) the simulation results should be similar to the results obtained using the algorithm from Sect. 3. (Recall that the latter results hold in the limit as \(\varepsilon\) approaches zero.) Each run of the GA lasted \(2 \times 10 ^ 5\) iterations. This seemed sufficient for the GA to reach its steady state. After the last iteration of a GA run, we almost always observed that the population was uniform. Based on the 500 GA runs that we had performed, we determined for each uniform population the probability of observing that population at the end of a GA run. In this way, we obtained a probability distribution over the uniform populations. This distribution is shown in Fig. 3 (in light gray). Figure 3 allows us to compare the distribution with the long-run limit distribution calculated using the algorithm from Sect. 3. It can be seen that the two distributions are very similar. This confirms the correctness of the algorithm presented in Sect. 3.
Long-run mean fitness and associated 95% confidence interval for various values of the memory length τ, the crossover rate γ, and the mutation rate \(\varepsilon\)
τ = 1 | τ = 2 | τ = 3 | |||||||
---|---|---|---|---|---|---|---|---|---|
γ = 0.0 | γ = 0.5 | γ = 1.0 | γ = 0.0 | γ = 0.5 | γ = 1.0 | γ = 0.0 | γ = 0.5 | γ = 1.0 | |
\(\varepsilon = 10 ^ {-2}\) | 2.76 ± 0.05 | 2.71 ± 0.05 | 2.79 ± 0.04 | 2.64 ± 0.08 | 2.72 ± 0.07 | 2.67 ± 0.07 | 2.67 ± 0.06 | 2.64 ± 0.07 | 2.70 ± 0.06 |
\(\varepsilon = 10 ^ {-3}\) | 2.23 ± 0.08 | 2.24 ± 0.08 | 2.25 ± 0.08 | 2.34 ± 0.12 | 2.41 ± 0.11 | 2.38 ± 0.11 | 2.55 ± 0.09 | 2.60 ± 0.09 | 2.59 ± 0.08 |
\(\varepsilon = 10 ^ {-4}\) | 1.93 ± 0.09 | 1.94 ± 0.09 | 1.90 ± 0.09 | 2.25 ± 0.12 | 2.24 ± 0.12 | 2.32 ± 0.12 | 2.57 ± 0.09 | 2.53 ± 0.09 | 2.50 ± 0.09 |
\(\varepsilon = 10 ^ {-5}\) | 1.85 ± 0.09 | 1.81 ± 0.09 | 1.85 ± 0.09 | 2.28 ± 0.12 | 2.31 ± 0.11 | 2.22 ± 0.12 | 2.58 ± 0.09 | 2.44 ± 0.10 | 2.44 ± 0.10 |
\(\varepsilon \rightarrow 0\) | 1.84 | 1.84 | 1.84 | 2.29 | 2.29 | 2.29 | ? | ? | ? |
Based on the results in Table 5, a number of observations can be made. First, for τ = 1 and τ = 2, the results obtained for \(\varepsilon = 10 ^ {-4}\) and \(\varepsilon = 10 ^ {-5}\) turn out to be very similar to the results obtained for \(\varepsilon \rightarrow0.\) This again confirms the correctness of the algorithm presented in Sect. 3. Second, for τ = 1, we find that the results are quite sensitive to the value of \(\varepsilon.\) Studies on GA modeling sometimes report that the long-run behavior of a GA is relatively insensitive to the value of \(\varepsilon.\) Our results demonstrate that this need not always be the case. Third, for small values of \(\varepsilon,\) it can be seen that increasing τ leads to a higher long-run mean fitness and, hence, to more cooperation. The evolution of cooperative strategies in IPD games therefore seems more likely when players have longer memory lengths. Finally, it can be observed that the value of γ has no significant effect on our results. This is in line with the mathematical analysis provided in Sect. 2. The mathematical analysis implies that for \(\varepsilon \rightarrow 0\) the long-run mean fitness is independent of γ. The results in Table 5 indicate that this is the case not only for \(\varepsilon \rightarrow 0\) but more generally.
5 Conclusions
In this paper, we have presented a mathematical analysis of the long-run behavior of GAs that are used for modeling social phenomena. Under the assumption of a positive but infinitely small mutation rate, the analysis provides a full characterization of the long-run behavior of GAs with a binary encoding. Based on the analysis, we have derived an algorithm for calculating the long-run behavior of GAs. In an economic context, the algorithm can for example be used to determine whether convergence to an equilibrium will take place and, if so, what kind of equilibrium will emerge. Compared with computer simulations, the main advantage of the algorithm that we have derived is that it calculates the long-run behavior of GAs exactly. Computer simulations only estimate the long-run behavior of GAs.
To demonstrate the usefulness of our mathematical analysis, we have replicated a well-known study by Axelrod in which a GA is used to model the evolution of strategies in IPDs (Axelrod 1987). We have used both our exact algorithm and computer simulations to replicate Axelrod’s study. By comparing the results of the two approaches, we have confirmed the correctness of our algorithm. We have also obtained some interesting new insights. For example, when players have a memory length of one period, the tit for tat strategy turns out to be less important than is sometimes claimed (e.g., Axelrod 1984, 1987). In the long run, the strategy is played only 14% of the time. Another finding is that the long-run behavior of a GA can be quite sensitive to the value of the mutation rate. We regard this as a serious problem, since the value of the mutation rate is typically chosen in a fairly arbitrary way without any empirical justification (see also Dawid 1996).
The mathematical analysis that we have presented also reveals that if the mutation rate is infinitely small the crossover rate has no effect on the long-run behavior of a GA. This remarkable result is perfectly in line with the simulation results that we have reported in Sect. 4. For various values of the mutation rate, the simulation results show no significant effect of the crossover rate on the long-run behavior of a GA. Hence, when GAs are used for modeling social phenomena, the crossover rate seems to be a rather unimportant parameter, at least when the focus is on the long run (for the short run, see Thibert-Plante and Charbonneau 2007). It seems likely that in many cases leaving out the crossover operator altogether has no significant effect on the long-run behavior of a GA. Interestingly, leaving out the crossover operator brings GAs quite close to well-known models in evolutionary game theory, such as those studied in Kandori et al. (1993) and Vega-Redondo (1997).
Finally, we note that an analysis such as the one presented in this paper can be performed not only for GAs with a binary encoding but also for other types of evolutionary algorithms. From a modeling point of view, a binary encoding in many cases has the disadvantage that it lacks a clear interpretation (e.g., Dawid 1996). The use of a binary encoding can therefore be difficult to justify and may even lead to artifacts (as shown in Waltman and Van Eck 2009; Waltman et al. 2011). Probably for these reasons, some researchers use evolutionary algorithms without a binary encoding (e.g., Haruvy et al. 2006; Lux and Schornstein 2005). The analysis presented in this paper then does not directly apply. However, when the action space of agents is assumed discrete, the long-run behavior of evolutionary algorithms without a binary encoding can still be analyzed in a similar way as we have done in this paper, namely by relying on mathematical results provided by Freidlin and Wentzell (1998). This indicates that our approach is quite general and can be adapted relatively easily to other types of evolutionary algorithms.
Footnotes
- 1.
In this paper, we use a standard binary encoding. Hence, if m = 2, chromosomes 0, 1, 2, and 3 have binary encodings 00, 01, 10, and 11, respectively. We emphasize that the use of a standard binary encoding is by no means essential for our analysis. Other binary encoding schemes, such as Gray encoding, can be used as well. This does not require any significant changes in our analysis.
- 2.
The non-zero elements of \(\mathbf{D}\) can be stored efficiently by using two arrays: a one-dimensional array of size μ for the diagonal elements of \(\mathbf{D}\) and a two-dimensional array of size m × μ for the non-zero off-diagonal elements of \(\mathbf{D}.\) The element in the κth row and the ith column of the latter array is used to store d(j, i), where j has the same binary encoding as i except that the κth bit is inverted.
- 3.
The software used to obtain the results reported in this subsection is available online at http://www.ludowaltman.nl/ga_analysis/. The software runs in MATLAB and has been written partly in the MATLAB programming language and partly in the C programming language.
- 4.
Notes
Acknowledgments
We would like to thank Uzay Kaymak and Rommert Dekker for their comments on earlier drafts of this paper.
Open Access
This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.
References
- Alkemade F, La Poutré JA, Amman HM (2006) Robust evolutionary algorithm design for socio-economic simulation. Comput Econ 28(4):355–370CrossRefGoogle Scholar
- Alkemade F, La Poutré JA, Amman HM (2007) On social learning and robust evolutionary algorithm design in the Cournot oligopoly game. Comput Intell 23(2):162–175CrossRefGoogle Scholar
- Alkemade F, La Poutré JA, Amman HM (2009) Robust evolutionary algorithm design for socio-economic simulation: a correction. Comput Econ 33(1):99–101CrossRefGoogle Scholar
- Andreoni J, Miller JH (1995) Auctions with artificial adaptive agents. Game Econ Behav 10(1):39–64CrossRefMATHGoogle Scholar
- Arifovic J (1994) Genetic algorithm learning and the cobweb model. J Econ Dyn Control 18(1):3–28CrossRefMATHGoogle Scholar
- Arifovic J (1996) The behavior of the exchange rate in the genetic algorithm and experimental economies. J Political Econ 104(3):510–541CrossRefGoogle Scholar
- Ashlock D, Kim EY, Leahy N (2006) Understanding representational sensitivity in the iterated prisoner’s dilemma with fingerprints. IEEE Trans Syst Man Cybern C 36(4):464–475CrossRefGoogle Scholar
- Ashlock D, Smucker MD, Stanley EA, Tesfatsion L (1996) Preferential partner selection in an evolutionary study of prisoner’s dilemma. Biosystems 37(1–2):99–125CrossRefGoogle Scholar
- Axelrod R (1984) The evolution of cooperation. Basic Books, LondonGoogle Scholar
- Axelrod R (1987) The evolution of strategies in the iterated prisoner’s dilemma. In: Davis L (ed) Genetic algorithms and simulated annealing. Morgan Kaufmann, Menlo Park, pp 32–41Google Scholar
- Axelrod R (1997) The complexity of cooperation: agent-based models of competition and collaboration. Princeton University Press, NJGoogle Scholar
- Barrett R, Berry M, Chan TF, Demmel J, Donato J, Dongarra J, Eijkhout V, Pozo R, Romine C, Van der Vorst H (1994) Templates for the solution of linear systems: building blocks for iterative methods, 2nd edn. SIAM, PhiladelphiaCrossRefGoogle Scholar
- Brenner T (2006) Agent learning representation: advice on modelling economic learning. In: Tesfatsion L, Judd KL (eds) Handbook of computational economics, vol 2. Elsevier, Amsterdam, pp 895–947CrossRefGoogle Scholar
- Chong SY, Yao X (2005) Behavioral diversity, choices and noise in the iterated prisoner’s dilemma. IEEE Trans Evol Comput 9(6):540–551CrossRefGoogle Scholar
- Crowley PH, Provencher L, Sloane S, Dugatkin LA, Spohn B, Rogers L, Alfieri M (1996) Evolving cooperation: the role of individual recognition. Biosystems 37(1–2):49–66CrossRefGoogle Scholar
- Dawid H (1996) Adaptive learning by genetic algorithms: analytical results and applications to economic models. No. 441 in lecture notes in economics and mathematical systems. Springer, BerlinGoogle Scholar
- Fogel DB (1993) Evolving behaviors in the iterated prisoner’s dilemma. Evol Comput 1(1):77–97MathSciNetCrossRefGoogle Scholar
- Foster D, Young P (1990) Stochastic evolutionary game dynamics. Theor Popul Biol 38(2):219–232MathSciNetCrossRefMATHGoogle Scholar
- Freidlin MI, Wentzell AD (1998) Random perturbations of dynamical systems, 2nd edn. Springer, BerlinCrossRefMATHGoogle Scholar
- Fudenberg D, Levine DK (1998) The theory of learning in games. MIT Press, CambridgeMATHGoogle Scholar
- Gen M, Cheng R (2000) Genetic algorithms and engineering optimization. Wiley, New YorkGoogle Scholar
- Georges C (2006) Learning with misspecification in an artificial currency market. J Econ Behav Organ 60(1):70–84CrossRefGoogle Scholar
- Gintis H (2000) Game theory evolving: a problem-centered introduction to modeling strategic interaction. Princeton University Press, NJMATHGoogle Scholar
- Goldberg DE (1989) Genetic algorithms in search, optimization, and machine learning. Addison-Wesley, ReadingMATHGoogle Scholar
- Haruvy E, Roth AE, Utku Ünver M (2006) The dynamics of law clerk matching: an experimental and computational investigation of proposals for reform of the market. J Econ Dyn Control 30(3):457–486CrossRefMATHGoogle Scholar
- Holland JH, Miller JH (1991) Artificial adaptive agents in economic theory. Am Econ Rev (Pap Proc) 81(2):365–370Google Scholar
- Ishibuchi H, Namikawa N (2005) Evolution of iterated prisoner’s dilemma game strategies in structured demes under random pairing in game playing. IEEE Trans Evol Comput 9(6):552–561CrossRefGoogle Scholar
- Kandori M, Mailath GJ, Rob R (1993) Learning, mutation, and long run equilibria in games. Econometrica 61:29–56MathSciNetCrossRefMATHGoogle Scholar
- Kemeny JG, Snell JL (1960) Finite Markov chains. D. Van Nostrand Company, New YorkMATHGoogle Scholar
- Lux T, Schornstein S (2005) Genetic learning as an explanation of stylized facts of foreign exchange markets. J Math Econ 41(1–2):169–196MathSciNetCrossRefMATHGoogle Scholar
- Marks RE (1992) Breeding hybrid strategies: optimal behaviour for oligopolists. J Evol Econ 2(1):17–38MathSciNetCrossRefGoogle Scholar
- Maynard Smith J (1982) Evolution and the theory of games. Cambridge University Press, LondonMATHGoogle Scholar
- Michalewicz Z (1996) Genetic algorithms + data structures = evolution programs, 3rd edn. Springer, BerlinMATHGoogle Scholar
- Miller JH (1986) A genetic model of adaptive economic behavior. Working paper, University of MichiganGoogle Scholar
- Miller JH (1996) The coevolution of automata in the repeated prisoner’s dilemma. J Econ Behav Organ 29(1):87–112CrossRefGoogle Scholar
- Mitchell M (1996) An introduction to genetic algorithms. MIT Press, CambridgeGoogle Scholar
- Mittal S, Deb K (2006) Optimal strategies of the iterated prisoner’s dilemma problem for multiple conflicting objectives. In: Proceedings of the 2006 IEEE symposium on computational intelligence and games, pp 197–204Google Scholar
- Mühlenbein H (1991) Darwin’s continent cycle theory and its simulation by the prisoner’s dilemma. Complex Syst 5(5):459–478MATHGoogle Scholar
- Nix AE, Vose MD (1992) Modeling genetic algorithms with Markov chains. Ann Math Artif Intell 5(1):79–88MathSciNetCrossRefMATHGoogle Scholar
- Rudolph G (1994) Convergence analysis of canonical genetic algorithms. IEEE Trans Neural Netw 5(1):96–101CrossRefGoogle Scholar
- Rudolph G (1998) Finite Markov chain results in evolutionary computation: a tour d’horizon. Fundamenta Informaticae 35(1–4):67–89MathSciNetMATHGoogle Scholar
- Stewart WJ (1994) Introduction to the numerical solution of Markov chains. Princeton University Press, NJMATHGoogle Scholar
- Tesfatsion L (2006) Agent-based computational economics: a constructive approach to economic theory. In: Tesfatsion L, Judd K (eds) Handbook of computational economics, vol 2. Elsevier, Amsterdam, pp 831–880CrossRefGoogle Scholar
- Thibert-Plante X, Charbonneau P (2007) Crossover and evolutionary stability in the prisoner’s dilemma. Evol Comput 15(3):321–344CrossRefGoogle Scholar
- Tijms HC (1994) Stochastic models: an algorithmic approach. Wiley, New YorkMATHGoogle Scholar
- Tijms HC (2003) A first course in stochastic models. Wiley, New YorkCrossRefMATHGoogle Scholar
- Van Bragt DDB, Van Kemenade C, La Poutré JA (2001) The influence of evolutionary selection schemes on the iterated prisoner’s dilemma. Comput Econ 17(2–3):253–263CrossRefMATHGoogle Scholar
- Vega-Redondo F (1996) Evolution, games, and economic behaviour. Oxford University Press, NYCrossRefGoogle Scholar
- Vega-Redondo F (1997) The evolution of Walrasian behavior. Econometrica 65:375–384CrossRefMATHGoogle Scholar
- Vose MD (1999) The simple genetic algorithm: foundations and theory. MIT Press, CambridgeMATHGoogle Scholar
- Vriend NJ (2000) An illustration of the essential difference between individual and social learning, and its consequences for computational analyses. J Econ Dyn Control 24(1):1–19MathSciNetCrossRefMATHGoogle Scholar
- Waltman L, Van Eck NJ (2009) Robust evolutionary algorithm design for socio-economic simulation: some comments. Comput Econ 33(1):103–105CrossRefGoogle Scholar
- Waltman L, van Eck NJ, Dekker R, Kaymak U (2011) Economic modeling using evolutionary algorithms: the effect of a binary encoding of strategies. J Evol Econ 21(5):737–756CrossRefGoogle Scholar
- Weibull JW (1995) Evolutionary game theory. MIT Press, CambridgeMATHGoogle Scholar
- Yao X, Darwen P (1994) An experimental study of N-person iterated prisoner’s dilemma games. Informatica 18(4):435–450MATHGoogle Scholar
- Young HP (1993) The evolution of conventions. Econometrica 61:57–84MathSciNetCrossRefMATHGoogle Scholar