Indirect reciprocity and the evolution of “moral signals”

A simple model of image scoring

The model of Nowak and Sigmund (1998a, b) is simplistic relative to the actual complexity of indirect reciprocity in humans. However, it reveals some important and interesting aspects of indirect reciprocity and its evolution. We can imagine a large population of players randomly paired a fixed number of times in the role of donor or recipient. The donor may choose to incur a cost c to give a benefit b to the recipient with b > c. Each player has an “image score” based on past actions. In the simplest setting, there are only two image scores: “good” (or 0) which means the agent donated on her last action and “bad” (or 1) which means the agent refused donation on her last action. ⁶ There are three strategies: always donate (cooperate), never donate (defect), and the image scoring strategy of donate if and only if the other guy is “good” (discriminate).

A fitness value is assigned to each strategy based on the accumulated payoffs of the interactions. Differences in fitness cause the proportion of each strategy in the population to increase or decrease according to the replicator dynamics (Taylor and Jonker 1978). These dynamics can be interpreted in either a biological context (as differential reproduction) or in a cultural context (as differential imitation or learning of strategies). On this model, cooperative states can evolve provided the initial proportion of discriminators is high enough to drive the defectors to extinction. The resulting cooperative populations are neutrally stable mixes of discriminators and unconditional cooperators. Since these strategies are wholly cooperative in the absence of defectors, each receives an identical payoff in these states. Although this result does not necessitate cooperation, it does show that image scoring strategies can stabilize it. Figure 1 provides a basic picture of the dynamics of indirect reciprocity in the simple image scoring model. ⁷ This dynamical picture closely resembles the situation with the evolution of direct reciprocity.

Open image in new window

These results are promising, but many subtle problems arise when complexities are introduced. For instance, if there are errors in perception or action (i.e. the image scores of other players cannot be reliably accessed or defection sometimes occurs unwillingly), cooperation can be hindered and destabilized (Panchanathan and Boyd 2003; Nowak and Sigmund 2005; Brandt and Sigmund 2005a). ⁸ Additionally, Leimar and Hammerstein (2001) have argued that the image scoring strategy is “paradoxical”: why should discriminators punish others (by defecting) if they pay the cost of having a damaged reputation themselves? To avoid these problems, many have turned to the investigation of “standing strategies.” Standing strategies attend to both the action performed and the circumstances in which it was performed. This allows discriminators to treat defecting on defectors as a good thing rather than a bad thing. ⁹ Many models have been provided which show the success of standing strategies in stabilizing cooperation.

For our purposes here, we will focus only on the basic binary image scoring model from Nowak and Sigmund (1998a). The primary motivation for this focus is simplicity: models which include both second-order standing strategies and signaling will be enormously complex, making any results very difficult to interpret properly. There are also empirical results that emphasize the importance of image scoring. For instance, Milinski et al. (2001) examined indirect reciprocity experimentally, finding that subjects tended to behave in accordance with image scoring strategies.

Setting aside standing strategies, there are two questions in the present context that need to be answered regarding the simple image scoring model. Is it possible to observe a coevolution of indirect reciprocity and the social signals that will serve as a basic image-tracking mechanism? And, if such a coevolution is possible, what implications does this have for the evolution of primitive “moral signals”? To answer these questions, we will introduce a simple signaling game into the model: agents will send signals regarding the most recent behavior of others.

The model

A strategy here is an ordered pair of functions (R,S). R maps signals to actions: {0,1} → {c,d} dictating how the player will act in the stage game. S maps actions to signals: {c,d} → {0,1} dictating what the player will signal about her previous opponent. ¹³ Let Strat represent the set of all strategies. There are 16 strategies in all, which can be represented as ordered quadruples such as (c,d,0,1) meaning “do c if opponent has a 0 image, do d otherwise, send 0 if opponent plays c and send 1 otherwise.” Additionally, each player p in round n will have an image score k _p ⁿ ∈{0,1} that is determined by the signal of their previous opponent q. When players interact, each responds to the image of her opponent and receives a payoff according to the game matrix above. Then, after the interaction, each player gives their opponent a new image before going on to new interactions. For simplicity we will assume that all players have an initial image score of 0. ¹⁴ Note that the signal q sends about p depends on the strategy of q and the action of p (which is a reaction to q’s image). Thus, \(k_p^{n+1}=S_q(R_p(k_q^{n}))\) and the payoff to p for an interaction with q on round n is then \(\pi(R_p(k_q^{n}),R_q(k_p^{n})).\) ¹⁵

The payoff of one strategy type i against another j cannot be calculated for a particular interaction without knowing the image scores of each individual, which depend on the other types in the population. Consequently, each round we calculate the probability that a given type has a particular image based on expected interactions in the previous round of play. We then calculate the expected utility of strategy i against strategy j under all possible image score combinations by weighting the payoffs with the appropriate probabilities. Let u ⁿ (i,j) denote the expected utility of using i against j in round n.

Interpretation of the model

It is important to briefly discuss the interpretation of signals in this model. The signals “0” and “1” do not have any predetermined meaning such as “good” or “bad.” However, in certain populations these signals can come to have primitive content in the sense described above. For instance, in a population of (c,d,0,1) the signal of “1” regarding an individual A could be loosely translated as “A defected; defect on A.” This same population is highly cooperative, able transmit information about individuals, and able to use that information to punish (via defection) any non-cooperative behavior. Thus, the population would be using “moral signals” in the limited sense we are working with.¹⁷

Admittedly, this is an oversimplified model of human moral systems. Real societies differ in several ways; they have more complex interactions, more complex communication, and nuanced differences between reputation and moral standing. Despite these idealizations, however, we can use these models to get a sense of what factors may (or may not) be involved in the evolution of primitive social signals. Furthermore, first attempts at modeling social phenomena should focus on simple settings so as to gain a deeper understanding of the underlying dynamics before further complexities are introduced.

Within this model and various extensions considered below, we can determine how likely it is for indirect reciprocity to evolve alongside a system of “moral signals.” We can also provide an account of the important factors in such a coevolution. This is the topic of the next two sections. If it turns out that “moral signals” evolve only very rarely or that their evolution requires substantial pressure from outside the domain of indirect reciprocity, then this may (in the absence of further modeling) give us a reason to doubt the importance of indirect reciprocity in the evolution of primitive moral language.

Simulation results

Turning to the other possible complication—including different signaling strategies within the population—reveals that the prospects for a coevolution of signaling and indirect reciprocity are very grim. If we suppose that just the defectors signal in a way to disrupt the reputation tracking (d,d,1,0) then cooperation is hurt but still evolves a large portion of the time (48%). However, when all the strategies are included, the results for cooperation by indirect reciprocity are disastrous. Only 5 populations in 1,000 simulations reached cooperative states. To make things even worse, none of these “cooperative” states were efficient (average payoff of 10 per interaction), meaning that there was occasional defection.

The difficulties for cooperative populations are not entirely due to defecting individuals. Cooperative individuals who disrupt the image tracking signals can destabilize indirect reciprocity. For instance, one strategy that is particularly troublesome relative to the strategies in our benchmark case is the “Two-Faced” strategy which always cooperates, but always signals “1” (c,c,1,1). In a population of discriminators of type (c,d,0,1), a player using the Two-Faced strategy undermines the reputation of others without ever harming her own image. This allows the Two-Faced strategy to invade by causing the discriminators to begin defecting on each other and thereby paving the way for defectors to take over. Figure 4 shows the evolutionary dynamics on the face of the simplex which has only Two-Faced cooperators (c,c,1,1), signaling discriminators (c,d,0,1) and signaling defectors (d,d,0,1). Simulations involving only these three strategies result in uncooperative populations every time. This reveals that not only does the presence of communication errors undermine cooperation, as shown by the perception errors examined by Panchanathan and Boyd (2003), but that the situation is much worse: an error-causing strategy can invade a population of error-free discriminators.

Open image in new window

With all possible strategies present, the only populations that reached cooperative states were those that began with a very high proportion of strategies labeling cooperation with “0” and defection with “1”. In this simple model, without some way to police signals and enforce conformity to a separating signaling system, the prospects for the coevolution of reputation tracking signals and indirect reciprocity are bleak. This shows that the introduction of cheap-talk is detrimental to the evolution of cooperation through indirect reciprocity because of reputation-ruining signals. If “moral signals” are to evolve in this setting, we will need to either examine a more complicated model that allows for a way to police the signals or introduce an additional payoff structure into the signaling portion of the game. These extensions will be explored in the next section.

Exogenous payoffs

Here, we consider three possible ways to include an exogenous payoff structure for signals. First, one signal may simply be inherently costly to send. Second, there may be a direct reward associated with using a particular separating strategy such as labeling cooperators with a “0” and defectors with a “1.” Third, we can allow for a coevolving, common interest signaling game to serve as the signaling system for representing the images of players. Each of these possibilities will be discussed below.

Suppose that sending signal 1 is costly. There may be many reasons for this: perhaps sending messages about others is difficult or time consuming (and “0” is simply not saying anything), or perhaps there are social consequences for sending signal 1 that we wish to lump into a single exogenous cost for simplicity. Whatever the reason, altering the model in this way will stabilize strategies such as (c,d,0,1) from the Two-Faced cooperator, since (c,c,1,1) requires sending the costly signal.¹⁸ If we simulate evolution in this setting, we see that efficient cooperative states do sometimes arise, but that it is a rare occurrence. For a cost of 0.5 for every use of signal 1, only 6 of 1,000 populations reached cooperative states and all were mixes of (c,c,0,0), (c,c,0,1), (c,d,0,0) and (c,d,0,1). Results in simulations for other cost amounts (ranging from 0.1 to 1.0) are similar. And, for an even higher cost of 2.0, no cooperative outcomes were observed in 500 simulations.

The third form of exogenous payoffs that we could introduce involves a coevolving signaling game. The basic motivation behind this is that individuals may be unable to distinguish between image signaling and a different game of common interest signaling as in Fig. 2. Thus, each individual’s strategy in the indirect reciprocity setting would double as a strategy for a simple common interest signaling game. Here, there are two states of the world (left and right) that occur with equal probability. One player observes the state and sends a signal (0 or 1). The other player then observes the signal and acts accordingly (c or d). If c is chosen in the left state each get a payoff of y and if d is chosen in the right state each get a payoff y, otherwise each receive no payoff.

All three variations of exogenous payoffs for signaling can stabilize cooperation by indirect reciprocity and plausibly allow an image-tracking signaling system to coevolve with cooperation. Interestingly, we have also seen that some exogenous payoffs to signaling may cause different populations to evolve different uses of “moral signals” and corresponding “moral norms.” One could, however, question the significance of these results, arguing that introducing an exogenous payoff structure merely pushes the questions of coevolution back: where do such exogenous payoffs come from and why should they apply to cases of indirect reciprocity? This is a fair criticism, and surely whether or not introducing a particular additional exogenous payoff structure is justified will depend on what is being modeled, how we interpret the payoffs, etc. Regardless, one conclusion that can be drawn is that an appropriate exogenous payoff structure for signaling, wherever it comes from, would aid the coevolution of signaling and indirect reciprocity. Perhaps it is important that there be such exogenous pressure for a successful system of “moral signals” to evolve.

To avoid the open questions raised by exogenous payoffs, it is important to also examine models that impose a payoff structure on the signals within the setting of indirect reciprocity itself. The next extension considered will model such an endogenous payoff to the signaling strategies. The idea is that sending signals in a manner different from others may elicit a form of signal-retaliation and the signaler herself may put her image at stake when making claims about others. If this additional complication can allow for stable social signaling which enables cooperation through indirect reciprocity, we will have at least one setting where exogenous payoffs are not required for the evolution of a simple system of “moral signals.”

Endogenous payoffs to signaling

On the standard picture of indirect reciprocity, acting cooperatively will elicit future cooperation from others. It is possible that signaling a certain way may be something that also elicits future cooperation (or defection). To explore this possibility, we will introduce an additional layer of signaling into the model. As before, agents meet in one-shot games and send signals regarding the actions of their opponents. Now, an interaction between two agents (say A and B) as well as the signal sent (by B) are observed by another member of the population (C) with probability q. C may then send a signal regarding B, changing B’s image. If B’s signal matches what C would have sent regarding A, C labels B as a cooperator (“0” or “1,” whatever that would be for C). If B’s signal did not match, C gives B the label that corresponds to defection. The idea here is that if agents signal according to your strategy, they are seen as cooperative, and if they signal differently, they are seen as uncooperative. Can this new element of the model stabilize cooperation and the signaling system necessary for image-tracking? Simulations bring both good news and bad news.

The good news is that, with q = 0.1, simulations show that the answer here is yes! This new element does create an endogenous benefit which serves to eliminate strategies that created problems for cooperation, such as the Two-Faced strategy. The cooperative populations that arise are a stable mix of discriminators and altruists, and tend to agree on signaling.²¹ Moreover, all the resulting cooperative populations are optimal. The only other stable states seen in simulations are mixes consisting of various defecting strategies. This result reveals that it is possible for an image-tracking signaling system (“moral signaling”) to evolve in the context of indirect reciprocity, for that system to make indirect reciprocity possible, and for that system to be the mechanism for its own enforcement. However, there is reason to be cautious before concluding much more.

The bad news is that, while this is possible, it does not appear likely. The resulting populations reaching cooperative states (with N = 5 and q = 0.1) was just under 2% of the total simulated populations.²² The benefit to endogenous signaling is not significant enough to push more than a small number of populations to cooperative states. Thus, this way of making the payoffs to signaling endogenous reveals a possible coevolution of signaling and indirect reciprocity. But, without some additional exogenous pressure on signaling, cooperation is far less likely to evolve than in the benchmark image scoring setting.

Possible extensions

There are several possible variants on the model presented in the previous section that would, intuitively, improve the prospects for the coevolution of “moral signals” and indirect reciprocity. A few of these are briefly described below along with simulation results.

Evolution of behavior in Stag Hunts or Coordination Games in standard settings is much more likely to result in cooperation than in the Prisoner’s Dilemma. Using one of these games as our base game in the model above does make the evolution of cooperation more likely. However, the signaling portion of the model has virtually no effect on the evolution; and the resulting populations do not favor one signaling strategy over another. The reason is obvious: in these games, there is no need to “enforce” cooperation once the population is cooperative because it is strictly harmful to behave in another way. Therefore, there is no need to track reputation for enforcement purposes.

Another possibility that could affect the evolution of cooperation and “moral signals” is a bias toward “truth-telling” in the initial populations.²³ Introducing such a bias will, of course, have an effect. However, a small bias only slightly increases the basin of attraction, and a relatively large bias is needed to create substantial changes. Even with truth-telling strategies being, on average, five times more likely than other strategies, only 23.6% of simulations resulted in cooperation. This means that if an initial bias is to help the coevolution of cooperation and “moral signals” it will have to be quite large, which cannot be justified within the scope of this model. ²⁴

Finally, perhaps there are restricted strategy sets, which can be justified, that will dramatically increase the likelihood of the coevolution of cooperation and “moral signals.” Making such modifications certainly has an arbitrary feel, but even if they are justifiable, such restrictions typically do not have a dramatic effect. For example, if we exclude any strategy that responds to “0” with d and “1” with c as well as any strategy that signals “1” for c and “0” for d, we still see only a very small proportion of cooperative signaling populations evolve (only 6 of 500 simulated populations).

In general, the result that the coevolution of indirect reciprocity and moral signaling is unlikely (relative to the benchmark case) seems relatively robust with respect to variations in the model. This can be taken as providing indirect theoretical support, in at least these simple settings, for the following claim: if language is to play the role of reputation tracking, the selective pressure for its evolution must come from outside the arena of indirect reciprocity.