Improved Cooperation by Exploiting a Common Signal

Can artificial agents benefit from human conventions? Human societies manage to successfully self-organize and resolve the tragedy of the commons in common-pool resources, in spite of the bleak prediction of non-cooperative game theory. On top of that, real-world problems are inherently large-scale and of low observability. One key concept that facilitates human coordination in such settings is the use of conventions. Inspired by human behavior, we investigate the learning dynamics and emergence of temporal conventions, focusing on common-pool resources. Extra emphasis was given in designing a realistic evaluation setting: (a) environment dynamics are modeled on real-world fisheries, (b) we assume decentralized learning, where agents can observe only their own history, and (c) we run large-scale simulations (up to 64 agents). Uncoupled policies and low observability make cooperation hard to achieve; as the number of agents grow, the probability of taking a correct gradient direction decreases exponentially. By introducing an arbitrary common signal (e.g., date, time, or any periodic set of numbers) as a means to couple the learning process, we show that temporal conventions can emerge and agents reach sustainable harvesting strategies. The introduction of the signal consistently improves the social welfare (by 258% on average, up to 3306%), the range of environmental parameters where sustainability can be achieved (by 46% on average, up to 300%), and the convergence speed in low abundance settings (by 13% on average, up to 53%).


INTRODUCTION
The question of cooperation in socio-ecological systems and sustainability in the use of common-pool resources constitutes a critical open problem.Classical non-cooperative game theory suggests that rational individuals will exhaust a common resource, rather than sustain it for the benefit of the group, resulting in the 'the tragedy of the commons' [17].The tragedy of the commons arises when it is challenging and/or costly to exclude individuals from appropriating common-pool resources (CPR) of finite yield [37].Individuals face strong incentives to appropriate, which results in overuse and even permanent depletion of the resource.Examples include the degradation of fresh water resources, the over-harvesting of timber, the depletion of grazing pastures, the destruction of fisheries, etc.
In spite of the bleak prediction of non-cooperative game theory, the tragedy of the commons is not inevitable, though conditions under which cooperation and sustainability can be achieved may be more demanding, the higher the stakes.Nevertheless, humans have been systematically shown to successfully self-organize and resolve the tragedy of the commons in CPR appropriation problems, even without the imposition of an extrinsic incentive structure [36].E.g., by enabling the capacity to communicate, individuals have been shown to maintain the harvest to an optimal level [6,37].Though, communication creates overhead, and might not always be possible [44].One of the key findings of empirical field research on sustainable CPR regimes around the world is the employment of boundary rules, which prescribe who is authorized to appropriate from a resource [36].Such boundary rules can be of temporal nature, prescribing the temporal order in which people harvest from a common-pool resource (e.g., 'protocol of play' [3]).The aforementioned rules can be enforced by an authority, or emerge in a self-organized manner (e.g., by utilizing environmental signals such as the time, date, season, etc.) in the form of a social convention.
Many real-world CPR problems are inherently large-scale and partially observable, which further increases the challenge of sustainability.In this work we deal with the most information-restrictive setting: each participant is modeled as an individual agent with its own policy conditioned only on local information, specifically his own history of action/reward pairs (fully decentralized method).Global observations, including the resource stock, the number of participants, and the joint observations and actions, are hiddenas is the case in many real-world applications, like commercial fisheries.Under such a setting, it is impossible to avoid positive probability mass on undesirable actions (i.e., simultaneous appropriation), since there is no correlation between the agents' policies.This leads to either low social welfare, because the agents are being conservative, or, even worse, the depletion of the resource.Depletion becomes more likely as the problem size grows due to the non-stationarity of the environment and the global exploration problem.
We propose a simple technique: allow agents to observe an arbitrary, common signal from the environment.Observing a common signal mitigates the aforementioned problems because it allows for coupling between the learned policies, increasing the joint policy space.Agents, for example, can now learn to harvest in turns, and with varying efforts per signal value, or allow for fallow periods.The benefit is twofold: the agents learn to not only avoid depletion, but also to maintain a healthy stock which allows for large harvest and, thus, higher social welfare.It is important to stress that we do not assume any a priori relation between the signal space and the problem at hand.Moreover, we require no communication, no extrinsic incentive mechanism, and we do not change the underlying architecture, or learning algorithm.We simply utilize a means -common environmental signals that are amply available to the agents [18] -to accommodate correlation between policies.This in turn enables the emergence of ordering conventions of temporal nature (henceforth referred to as temporal conventions) and sustainable harvesting strategies.
1.1 Our Contributions (1) We are the first to introduce a realistic common-pool resource appropriation game for multi-agent coordination, based on bio-economic models of commercial fisheries, and provide theoretical analysis on the dynamics of the environment.
(2) We propose a simple and novel technique: allow agents to observe an arbitrary periodic environmental signal.Such signals are amply available in the environment (e.g., time, date etc.) and can foster cooperation among agents.
(3) We provide a thorough (quantitative & qualitative) analysis on the learned policies and demonstrate significant improvements on sustainability, social welfare, and convergence speed.

Discussion & Related Work
As autonomous agents proliferate, they will be called upon to interact in ever more complex environments.This will bring forth the need for techniques that enable the emergence of sustainable cooperation.Despite the growing interest in and success of multi-agent deep reinforcement learning (MADRL), scaling to environments with a large number of learning agents continues to be a problem [15].A multi-agent setting is inherently susceptible to many pitfalls: non-stationarity (moving-target problem), curse of dimensionality, credit assignment, global exploration, relative overgeneralization [20,33,48] 1 .Recent advances in the field of MADRL deal with only a limited number of agents.It is shown that as the number of agents increase, the probability of taking a correct gradient direction decreases exponentially [20], thus the proposed methods cannot be easily generalized to complex scenarios with many agents.
Our approach aims to mitigate the aforementioned problems of MADRL by introducing coupling between the learned policies.It is important to note that the proposed approach does not change the underlying architecture of the network (the capacity of the network stays the same), nor the learning algorithm or the reward structure.We simply augment the input space by allowing the observation of an arbitrary common signal.The signal has no a priori relation to the problem, i.e., we do not need to design an additional feature; in fact we use a periodic sequence of arbitrary integers.It is still possible for the original network (without the signal) to learn a sustainable strategy.Nevertheless, we show that the simple act of augmenting the input space drastically increases the social welfare, speed of convergence, and the range of environmental parameters in which sustainability can be achieved.Most importantly, the proposed approach requires no communication, creates no additional overhead, it is simple to implement, and scalable.
The proposed technique was inspired by temporal conventions in resource allocation games of non-cooperative game theory.The closest analogue is the courtesy convention of [11], where rational agents learn to coordinate their actions to access a set of indivisible resources by observing a signal from the environment.Closely related is the concept of the correlated equilibrium (CE) [1,35], which, from a practical perspective, constitutes perhaps the most relevant non-cooperative solution concept [18] 2 .Most importantly, it is possible to achieve a correlated equilibrium without a central authority, simply by utilizing meaningless environmental signals [2,7,11].Such common environmental signals are amply available to the agents [18].The aforementioned line of research studies pre-determined strategies of rational agents.Instead, we study the emergent behaviors of a group of independent learning agents aiming to maximize the long term discounted reward.
A second source of inspiration is behavioral conventions; one of the key concepts that facilitates human coordination 3 .A convention is defined as a customary, expected, and self-enforcing behavioral pattern [28,49].It can be considered as a behavioral rule, designed and agreed upon ahead of time [43,46], or it may emerge from within the system itself [34,46].The examined temporal convention in this work falls on the latter category.
Moving on to the application domain, there has been great interest recently in CPR problems (and more generally, social dilemmas [24]) as an application domain for MADRL [21,23,25,26,32,[38][39][40]47].CPR problems offer complex environment dynamics and relate to real-world socio-ecological systems.There are a few distinct differences between the CPR models presented in the aforementioned works and the model introduced in this paper: First and foremost, we designed our model to resemble reality as closely as possible using bio-economic models of commercial fisheries [8,12], resulting in complex environment dynamics.Second, we have a continuous action space which further complicates the learning process.Finally, we opted not to learn from visual input (raw pixels).The problem of direct policy approximation from visual input does not add complexity to the social dilemma itself; it only adds complexity in the feature extraction of the state.It requires large networks because of the additional complexity of extracting features from pixels, while only a small part of what is learned is the actual policy [10].Most importantly, it makes harder to study the policy in isolation, as we do in this work.Moreover, from a practical perspective, learning from a visual input would be meaningless, given that we are dealing with a low observability scenario where the resource stock and the number and actions of the participants are hidden.
In terms of the methodology for dealing with the tragedy of the commons, the majority of the aforementioned literature falls broadly into two categories: Reward shaping [21,23,40], which refers to adding a term to the extrinsic reward an agent receives from the environment, and opponent shaping [25,32,38], which refers to manipulating the opponent (by e.g., sharing rewards, punishments, or adapting your own actions).Contrary to that, we only allow agents to observe an existing environmental signal.We do not modify the intrinsic or extrinsic rewards, design new features, or require a communication network.Finally, boundary rules emerged in [38] as well in the form of spatial territories.Such territories can increase inequality, while we maintain high levels of fairness.

AGENT & ENVIRONMENT MODELS 2.1 Multi-Agent Reinforcement Learning
We consider a decentralized multi-agent reinforcement learning scenario in a partially observable general-sum Markov game [42].At each time-step, agents take actions based on a partial observation of the state space, and receive an individual reward.Each agent learns a policy independently.More formally, let N = {1, . . ., } denote the set of agents, and M be an -player, partially observable Markov game defined on a set of states S.An observation function O : S → R specifies agent 's -dimensional view of the state space.Let A denote the set of actions for agent ∈ N , and = × ∀ ∈N , where ∈ A , the joint action.The states change according to a transition function T : , where Δ(S) denotes the set of discrete probability distributions over S. Every agent receives an individual reward based on the current state ∈ S and joint action .The latter is given by the reward function : Finally, each agent learns a policy : O → Δ(A ) independently through their own experience of the environment (observations and rewards).Let = × ∀ ∈N denote the joint policy.The goal for each agent is to maximize the long term discounted payoff, as given by , where is the discount factor and 0 is the initial state.

The Common Fishery Model
In order to better understand the impact of self-interested appropriation, it would be beneficial to examine the dynamics of realworld common-pool renewable resources.To that end, we present an abstracted bio-economic model for commercial fisheries [8,12].The model describes the dynamics of the stock of a common-pool renewable resource, as a group of appropriators harvest over time.The harvest depends on (i) the effort exerted by the agents and (ii) the ease of harvesting a resource at that point of time, which depends on the stock level.The stock replenishes over time with a rate dependent on the current stock level.
More formally, let N denote the set of appropriators, , ∈ [0, E ] the effort exerted by agent at time-step , and = ∈N , the total effort at time-step .The total harvest is given by Eq. 1, where ∈ [0, ∞) denotes the stock level (i.e., amount of resources) at time-step , (•) denotes the catchability coefficient (Eq.2), and is the equilibrium stock of the resource.
Each environment can only sustain a finite amount of stock.If left unharvested, the stock will stabilize at .Note also that (•), and therefore (•), are proportional to the current stock, i.e., the higher the stock, the larger the harvest for the same total effort.The stock dynamics are governed by Eq. 3, where (•) is the spawner-recruit function (Eq.4) which governs the natural growth of the resource, and is the growth rate.To avoid highly skewed growth models and unstable environments ('behavioral sink' [4,5]), where (•) is the Lambert function (see Section C for details).
We assume that the individual harvest is proportional to the exerted effort (Eq.5), and the revenue of each appropriator is given by Eq. 6, where is the price ($ per unit of resource), and is the cost ($) of harvesting (e.g., operational cost, taxes, etc.).Here lies the 'tragedy': the benefits from harvesting are private ( ℎ , ( , , )), but the loss is borne by all (in terms of a reduced stock, see Eq. 3).
, ( , , ) = ℎ , ( , , ) − 2.2.1 Optimal Harvesting.The question that naturally arises is: what is the 'optimal' effort in order to harvest a yield that maximizes the revenue (Eq.6).We make two assumptions: First, we assume that the entire resource is owned by a single entity (e.g., a firm or the government), which possesses complete knowledge of and control over the resource.Thus, we only have a single control variable, .This does not change the underlying problem since the total harvested resources are linear in the proportion of efforts put by individual agents (Eq.5).Second, we consider the case of zero discounting, i.e., future revenues are weighted equally with current ones.Of course firms (and individuals) do discount the future and bio-economic models should take that into account, but this complicates the analysis and it is out of the scope of this work.We argue we can still draw useful insight into the problem.
Our control problem consists of of finding a piecewise continuous control , so as to maximize the total revenue (max =0 ( , )).The maximization problem can be solved using Optimal Control Theory [13,27].We have proven the following theorem: T 2.1.The optimal control variable * that solves the maximization problem max =0 ( , ) given the model dynamics described in Section 2.2 is given by Eq. 7, where are the adjoint variables of the Hamiltonians: P .(sketch) We formulate the Hamiltonians [13,27], which turn out to be linear in the control variables +1 with coefficients ( +1 − +1 ) ( ( − ( , ))).Thus, the optimal sequence of +1 that maximizes the Hamiltonians is given according to the sign of those coefficients.See Section B for the complete proof.
The optimal strategy is a bang-bang controller, which switches based on the adjoint variable values, stock level, and price.The values for do not have a closed form expression (because of the discontinuity of the control), but can be found iteratively for a given set of environment parameters ( , ) and the adjoint equations [13,27].However, the discontinuity in the control input makes solving the adjoint equations quite cumbersome.We can utilize iterative forward/backward methods as in [13], but this is out of the scope of this paper.
There are a few interesting key points.First, to compute the optimal level of effort we require observability of the resource stock, which is not always a realistic assumption (in fact in this work we do not make this assumption).Second, we require complete knowledge of the strategies of the other appropriators.Third, even if both the aforementioned conditions are met, the bang-bang controller of Eq. 7 does not have a constant transition limit; the limit changes at each time time-step, determined by the adjoint variable +1 , thus finding the switch times remains quite challenging.

2.2.2
Harvesting at Maximum Effort.To gain a deeper understanding of the dynamics of the environment, we will now consider a baseline strategy where every agent harvests with the maximum effort at every time-step, i.e., , = E , ∀ ∈ N, ∀ .This corresponds to the Nash Equilibrium of a stage game (myopic agents).For a constant growth rate and a given number of agents , we can identify two interesting stock equilibrium points ( ): the 'limit of sustainable harvesting', and the 'limit of immediate depletion'.
The limit of sustainable harvesting ( , ) is the stock equilibrium point where the goal of sustainable harvesting becomes trivial: for any > , , the resource will not get depleted, even if all agents harvest at maximum effort.Note that the coordination problem remains far from trivial even for > , , especially for increasing population sizes.Exerting maximum effort in environments with close to , will yield low returns because the stock remains low, resulting in a small catchability coefficient.In fact, this can be seen in Fig. 1 which depicts the social welfare (SW), i.e., sum of utilities, against increasing values ( ∈ [2, 64], E = 1, = 1).Red dots4 denote the , .Thus, the challenge is not only to keep the strategy sustainable, but to keep the resource stock high, so that the returns can be high as well.-axis is in logarithmic scale.
On the other end of the spectrum, the limit of immediate depletion ( , ) is the stock equilibrium point where the resource is depleted in one time-step (under maximum harvest effort by all the agents).The problem does not become impossible for ≤ , , yet, exploration can have catastrophic effects (amplifying the problem of global exploration in MARL).The following two theorems prove the formulas for , and , .T 2.2.The limit of sustainable harvesting , for a continuous resource governed by the dynamics of Section 2.2, assuming that all appropriators harvest with the maximum effort E , is: P .Note that for > , , ( ) < , ∀ , otherwise the resource would be depleted.Moreover, if 0 = -which is a natural assumption, since prior to any intervention the stock will have stabilized on the fixed point -then 5 < 2 , ∀ .Thus, we can re-write Eq. 1 and 2 as: , and = 1 − .The state transition becomes: We write it as a difference equation: At the limit of sustainable harvesting, as the stock diminishes to 6 = → 0, to remain sustainable it must be that Δ ( ) > 0. Thus, it must be that: T 2.3.The limit of immediate depletion , for a continuous resource governed by the dynamics of Section 2.2, assuming that all appropriators harvest with the maximum effort E , is given by: 6 In practice, is enforced by the granularity of the resource.

P
. The resource is depleted if:

Environmental Signal
We introduce an auxiliary signal; side information from the environment (e.g., time, date etc.) that agents can potentially use in order to facilitate coordination and reach more sustainable strategies.Real-world examples include shepherds that graze on particular days of the week or fishermen that fish on particular months.
In our case, the signal can be thought as a mechanism to increase the set of possible (individual and joint) policies.Such signals are amply available to the agents [11,18].We do not assume any a priori relation between the signal and the problem at hand.In fact, in this paper we use a set of arbitrary integers, that repeat periodically.We use G = {1, . . ., } to denote the set of signal values.

Setup
3.1.1Environment Se ings.Let = 1, and = 0, ∀ .We set the growth rate at = 1, the initial population at 0 = , and the maximum effort at E = 1.The findings of Section 2.2.2 provide a guide on the selection of the values.Specifically we simulated environments with given by Eq. 10, where 79 is a constant and ∈ R + is a multiplier that adjusts the scarcity (difficulty).
= 1 corresponds to = , .= 3.1.2Agent Architecture.Each agent uses a two-layer (64 neurons each) neural network for the policy approximation.The input , consisting of the individual effort exerted and reward obtained in the previous time-step and the current signal value.The output is a continuous action value = , ∈ [0, E ] specifying the current effort level.The policies are trained using the Proximal Policy Optimization (PPO) algorithm [41].PPO was chosen because it avoids large policy updates, ensuring a smoother training, and avoiding catastrophic failures.The reward received from the environment corresponds to the revenue, i.e., ( , ) = , ( , , ), and the discount factor was set to = 0.99.

Signal Implementation.
The signal is represented as adimensional one-hot encoded vector, where the high bit is shifted periodically.The initial value was chosen at random at the beginning of each episode to avoid bias towards particular values.Throughout this paper, the term no signal will be used interchangeably to a unit signal size = 1, since a signal of size 1 in one-hot encoding is just a constant input that yields no information.We evaluated signals of varying cardinality (see Section 3.6).

Termination
Condition.An episode terminates when either (a) the resource stock falls below a threshold = 10 −4 , or (b) a fixed number of time-steps = 500 is reached.We trained our agents for a maximum of 5000 episodes, with the possibility of early stopping if both of the following conditions are satisfied: (i) a minimum of 95% of the maximum episode duration (i.e., 475 timesteps) is reached for 200 episodes in a row, and, (ii) the average total reward obtained by agents in each episode of the aforementioned 200 episodes does not change by more than 5%.In case of early stopping, the metric values for the remainder of the episodes are extrapolated as the average of the last 200 episodes, in order to properly average across trials.

Measuring
The Influence of the Signal.It is important to have a quantitative measure of the influence of the introduced signal.As such, we adapted the Causal Influence of Communication (CIC) [31] metric, initially designed to measure positive listening in emergent inter-agent communication.The CIC is calculated using the mutual information between the signal and the agent's action.Please see Section D.3 for a complete description.
3.1.6Reproducibility, Reporting of Results, Limitations.Reproducibility is a major challenge in (MA)DRL due to different sources of stochasticity, e.g., hyper-parameters, model architecture, implementation details, etc. [14,19,20].To minimize those sources, the implementation was done using RLlib 7 , an open-source library for MADRL [29].We refer the reader to Section D for a description of the architecture and hyper-parameters.
All simulations were repeated 8 times and the reported results are the average values of the last 10 episodes over those trials (excluding Fig. 7 which depicts a representative trial).(MA)DRL also lacks common practices for statistical testing [19,20].In this work, we opted to use the Student's T-test [45] due to it's robustness [9].Nearly all of the reported results have p-values < 0.05.
Finally, we strongly believe that the community would benefit from reporting negative results.As such, we want to make clear that the proposed solution is not a panacea for all multi-agent coordination problems, not even for the proposed domain.For example, we failed to find sustainable policies using DDPG [30] -with or without the signal -for any set of environment parameters.This also comes to show the difficulty of the problem at hand.We suspect that the clipping in PPO's policy changes plays an important role in averting catastrophic failures in high-stakes environments.

Results
We present the result from a systematic evaluation of the proposed approach on a wide variety of environmental settings ( ∈ [0.2, 1.2], i.e., ranging from way below the limit of immediate depletion, ≈ 0.63, to above the limit of sustainable harvesting, = 1) and population size ( ∈ [2,64]).Due to lack of space we only present the most relevant results; see Section E for a complete report (e.g., results tables, fairness, small population sizes, etc.).
In the majority of the results, we study the influence of a signal of cardinality = compared to no signal ( = 1).Thus, unless stated otherwise, the term 'with signal' will refer to = .

Sustainability & Social Welfare
3.3.1 Sustainability.We declare a strategy 'sustainable', iff the agents reach the maximum episode duration (500 steps), i.e., they do not deplete the resource.Fig. 2 depicts the achieved episode  ), for environments of decreasing difficulty (increasing equilibrium stock multiplier ).
length -with and without the presence of a signal ( = ) -for environments of decreasing difficulty (increasing ∝ ).The introduction of the signal significantly increases the range of environments ( ) where sustainability can be achieved.Assuming that ∈ [0, 1] -since for ≥ 1 sustainability is guaranteed by definition -we have an increase of 17% − 300% (46% on average) in the range of sustainable values.Moreover, as the number of agents increases ( = 32 & 64), depletion is avoided in non-trivial values only with the introduction of the signal.Finally, note that the value where a sustainable strategy is found increases with , which demonstrates that the difficulty of the problem increases superlinearly to (given that ∝ ).
3.3.2Social welfare.Reaching a sustainable strategy -i.e., avoiding resource depletion -is only one piece of the puzzle; an agent's revenue depends on the harvest (Eq.1), which in turns depends on the catchability coefficient (Eq.2).Thus, in order to achieve a high social welfare (sum of utilities, i.e., ∈N (•)), the agents need to learn policies that balance the trade-off between maintaining a high stock (which ensues a high catchability coefficient), and yielding a large harvest (which results to a higher reward).This problem becomes even more apparent as resources become more abundant (i.e., for = 1 ± , i.e., close to the limit of sustainable harvesting (below or, especially, above), see Section 2.2.2).In these settings, it is easy to find a sustainable strategy; a myopic best-response strategy (harvesting at maximum effort) by all agents will not deplete the resource.Yet, it will result in low social welfare (SW).
Fig. 3 depicts the relative difference in SW, in a setting with and without the signal (( = − =1 )/ =1 , where = denotes the SW achieved using a signal of cardinality ), for environments of decreasing difficulty (increasing ∝ ) and varying population size ( ∈ [8,64]).To improve readability, changes greater than 100% are shown with numbers on the top of the bars.Given the various sources of stochasticity, we opted to omit settings in which agents were not able to reach an episode duration of more than 10 time-steps (either with or without the signal).The presence of the signal results in a significant improvement in SW.Specifically, we have an average of 258% improvement across all the depicted settings8 in Fig. 3, while the maximum improvement is 3306%.These improvements stem from (i) achieving more sustainable strategies, and (ii) improved cooperation.The former results in higher rewards due to longer episodes in settings where the strategies without the signal deplete the resource.The latter allows to avoid over-harvesting, which results in higher catchability coefficient, in settings where both strategies (with, or without the signal) are sustainable.The contribution of the signal is much more pronounced under scarcity: the difference in achieved SW decreases as increases, eventually becoming less than 10% ( > 1 for = 8 & 16, and > 1.2 for = 32 & 64).This suggests that the proposed approach is of high practical value in environments where resources are scarce (like most real-world applications), a claim that we further corroborate in Sections 3.5 and 3.7.

Convergence Speed
The second major influence of the introduction of the proposed signal -besides the sustainability and efficiency of the learned strategies -is on the convergence time.Let the system be considered converged when the global state does not change significantly.As a practical way to pinpoint the time of convergence, we used the 'Termination Criterion' of Section 3.1.4.Fig. 4 depicts the relative difference in convergence time with the introduction of a signal (( = − =1 )/ =1 , where = denotes the time until convergence, in #episodes, when using a signal of cardinality ), for environments of decreasing difficulty (increasing ∝ ) and varying population size ( ∈ [8,64]).We have omitted the settings in which agents were not able to reach an episode duration of more than 10 time-steps (either with or without the signal).
There is a disjoint effect of the signal on the convergence speed.Up to the limit of sustainable harvesting ( ≤ 1), the signal significantly improves the convergence speed (13% improvement on average, across all the depicted settings including the ones with no improvement, and up to 53%).This is vital, as the majority of real-world problems involve managing scarce resources.On the other hand, for > 1, i.e., settings with abundant resources, the system converges faster without the signal (14% slower with the signal on average, across all the depicted settings).One possible explanation is that as resources become more abundant, it is harder (impossible for > 1) for agents to deplete them.Therefore the learning is more efficient -and the convergence is faster -since the episodes tend to last longer (without needing the signal).Moreover, having an abundance of resources decouples the effects of the agents' actions to each other, reducing the variance, and again making easing the learning process without the signal.

Influence of Signal on Agent Strategies
The results presented so far provide a qualitative measure of the influence of the introduced signal through the improvement on sustainability, social welfare, and convergence speed.They also indicate a decrease on the influence of the signal as resources become abundant.The question that naturally arises is: how much do agents actually take the signal into account in their policies?To answer this question, Fig. 5 depicts the CIC values -a quantitative measure of the influence of the introduced signal (see Section 3.1.5)-versus increasing values of (i.e, increasing ∝ , or more abundant resources), for population/signal size = ∈ {8, 16, 32, 64}.The values are averaged across the 8 trials and the agents, and are normalized with respect to the maximum value for each population 9 .Higher CIC values indicate a higher causal influence of the signal.CIC is low for the trials in which a sustainable strategy could not be found ( = 0.2−0.3 for = 8, 16, = 0.2−0.5 for = 32, and = 0.2 − 0.8 for = 64, see Fig. 2).In cases where a sustainable strategy was reached (e.g., ≥ 0.4 for = 8), we see significantly higher CIC values on scarce resource environments, and then the CIC decreases as increases.The harder the coordination problem, the more the agents rely on the environmental signal.

Robustness to Signal Size
Up until now we have evaluated the presence (or lack thereof) of an environmental signal of cardinality equal to the population size ( = ).This requires exact knowledge of , thus it is interesting to test the robustness of the proposed approach under varying signal sizes.As a representative test-case, we evaluated different signals of cardinality = 1, 2 , 23, , 41, and 3  2 for = 32 and moderate scarcity for the resource ( values of 0.7, 0.8 and 0.9).The values 23 and 41 were chosen as they are prime numbers (i.e., not multiples of ).Fig. 6 depicts the achieved social welfare and convergence time under the aforementioned settings.
Starting with Fig. 6a we can see that the SW increases with the signal cardinality.Specifically, we have 263%, 255%, 341%, 416%, and 474% improvement on average across the three values for = 2 , 23, , 41, and 3 2 , respectively.We hypothesize that the improvement stems from an increased joint strategy space that the larger signal size allows.A signal size larger than can also allow the emergence of 'rest' (fallow) periods -signal values where the majority of agents harvests at really low efforts.This would allow the resource to recuperate, and increase the SW through a higher catchability coefficient.See Section 3.7 / Fig. 7b for an example.
Regarding the convergence speed (Fig. 6b), we have 22%, 38%, 36%, 41%, and 36% improvement on average (across values).These results showcase that the introduction of the signal itself -regardless of its cardinality or, more generally, its temporal representative power -provides a clear benefit to the agents in terms of SW and convergence speed.This greatly improves the real-world applicability of the proposed technique, as the the knowledge of the exact population size is not required; instead the agents can opt to select any signal available in their environment 10 .Moreover, the signal cardinality can also be considered as a design choice, depending on the requirements and limitations of the system.alitative Analysis.We have seen that the introduction of an arbitrary signal facilitates cooperation and the sustainable harvesting.But do temporal conventions actually emerge?Fig. 7a presents an example of the evolution of the agents' strategies for each signal value for a population of = 4, signal size = = 4, and equilibrium stock multiplier = 0.5 (smoothed over 50 episodes).Each row represents an agent (agent ), while each column represents a signal value (value ).Each line represents the average effort the agent exerts on that specific signal value -calculated by averaging the actions of the agent in each corresponding signal value across the episode.

Emergence of Temporal Conventions
We can see a clear temporal convention emerging: at signal value 1 (first column), only agents 1 and 3 harvest (first and third row), at 2 , 2 and 4 harvest, at 3 , 1 and 3 harvest, and, finally, at 4 , 2 and 4 .Contrary to that, in a sustainable joint strategy without the use of the signal, every agent harvests at every time-step with an average (across all agents) effort of ≈ 40% (for the same setting of = 4 and = 0.5).Having all agents harvesting at every time-step makes coordination increasingly harder as we increase the population size, mainly due to the non-stationarity of the environment (high variance) and the global exploration problem.

Fallowing.
A more interesting joint strategy can be seen in Fig. 7b ( = 2, = 0.5).In this setting, we have an increased number of available signals, specifically = 3 2 = 3.We can see that agents harvest alternatingly in the first two signal values, and rest on the third (fallow period), potentially to allow resources to replenish and consequently obtain higher rewards in the future due 10 The signal is represented as a one-hot vector, i.e., Fig. 6 shows that a network with 32 inputs can work for population sizes ∈ [16,48], or equivalently, that agents in a population of size = 32 can use networks with 16 − 48 inputs for the signal. 11The setting with = 64 was run with = 2 in both cases (with and without the signal).See Section E for more information.to a higher catchability coefficient.This also resembles the optimal (bang-bang) harvesting strategy of Theorem 2.1.

CONCLUSION
The challenge to cooperatively solve 'the tragedy of the commons' remains as relevant now as when it was first introduced by Hardin in 1968.Sustainable development and avoidance of catastrophic scenarios in socio-ecological systems -like the permanent depletion of resources, or the extinction of endangered species -constitute critical open problems.To add to the challenge, real-world problems are inherently large in scale and of low observability.This amplifies traditional problems in multi-agent learning, such as the global exploration and the moving-target problem.Earlier work in common-pool resource appropriation utilized intrinsic or extrinsic incentives (e.g., reward or opponent shaping).Yet, such techniques need to be designed for the problem at hand and/or require communication or observability of states/actions, which is not always feasible (e.g., in commercial fisheries, the stock or harvesting efforts can not be directly observed).Humans on the other hand show a remarkable ability to self-organize and resolve common-pool resource dilemmas, often without any extrinsic incentive mechanism or communication.Social conventions and the In order to decouple the state ( , the current resource stock) and the control ( ) 13 and simplify the calculations, we resort to a change of variables.We define the new state as the remaining stock after harvest at time-step : − ( , ) Therefore, and Using Eq. 13 and 14, we can write the new state equation as: In the current form of the state equation (Eq.15), the harvested resources appear outside the nonlinear growth function (.), making the following analysis significantly simpler.
Under optimal control, the resource will not get depleted before the end of the horizon , thus ( ) ≤ , ∀ < 14 .We can rewrite the total harvest as: The state equation (Eq.15) becomes: +1 = ( ) − ( ( )) +1 and the optimization problem: . Solving the optimization problem is equivalent to finding the control that optimizes the Hamiltonians [13,27].Let = ( −1 , 0 , . . ., −1 ) denote the adjoint function.The Hamiltonian at time-step is given by: The adjoint equations are given by [13]: 13 In accordance to the literature on Optimal Control Theory [27], 'state' in the context of the proof refers to the variable describing the the behavior of the underlying dynamical system, and 'control' refers to the input function used to steer the state of the system. 14Let us assume this is not the case and the optimal strategy would deplete the stock at certain time-step .That means that rewards are 0 and the optimal is arbitrary for > . Using the modified equation for the total harvest (Eq.16), we allow ( , ) > or − ( , ) < 0. This would lead to +1 = ( − ( , )) < 0 ∀ > , i.e., the stock would become negative.In such a case, any positive effort would decrease the revenue (since it would result in a negative harvest), thus the optimal strategy would be to set = 0. Thus, using the modified equation for the total harvest (Eq.16), does not change the optimal solution.
where is the control input, which corresponds to +1 in our formulation.The last condition corresponds to the maximization of the Hamiltonian H in Eq. 17 for all time-steps [27, Chapter 23].In our case, Eq. 17 is linear in +1 with coefficient ( +1 − +1 ) ( ( )).Therefore, the optimal sequence of +1 that maximizes Eq. 17 is given based on the sign of the coefficient: The optimal strategy therefore implements bang-bang controller, which oscillates based on adjoint variable values , current state , and price .

C GROWTH RATE
The growth rate, , plays a significant role in the stability of the stock dynamics.A high growth rate -and subsequently high population density -can even lead to the extinction of the population due to the collapse in behavior from overcrowding (a phenomenon known as 'behavioral sink' [4,5]).This is reflected by the spawnerrecruit function (Eq.4) in our model.As depicted in Fig. 8, the higher the growth rate, , the more skewed the stock dynamics.Specifically, Fig. 8 plots the spawner-recruit function (•) for various growth rates: = 1, 2, − −1 (− 1 2 ) ≈ 2.678, and 4 (Fig. 8a, 8b, 8c, and 8d, respectively).The -axis denotes the current stock level ( ), while the -axis depicts the stock level on the next time-step, assuming no harvest (i.e., +1 = ( )).The dashed line indicates a stock level equal to 2 .For a growth rate of = 4 for example (Fig. 8d), we have a highly skewed growth model that can lead to the depletion of the resource (high stock values ( ) result to +1 < → 0, i.e., permanent depletion of the resource).For this reason, we want an unskewed growth model, specifically we want the stock to remain below two times the equilibrium stock point, i.e., +1 ≤ 2 15 .For this reason, we need to bound the growth rate according to the following theorem: For a continuous resource governed by the dynamics of Section 2.2, the stock value does not exceed the limit of 2 , if where (•) is the Lambert function.

P
. Let ≤ 2 for a time-step .We want +1 ≤ 2 , thus we need to bound the maximum value of the spawnerrecruit function, ( ) (Eq. 4).

D MODELING DETAILS D.1 Agent Architecture Details
Recent work has demonstrated that code-level optimizations play an important role in performance, both in terms of achieved reward and underlying algorithmic behavior [14].To minimize those sources of stochasticity -and given that the focus of this work is in

D.2 Introduced Signal: Implementation Details
The introduced signal was encoded as a -dimensional one-hot vector of fixed size, in which the high bit is shifted periodically.In particular, its value at index at time-step is given by: where is the random offset determined at the beginning of each episode in order to avoid learning any bias towards certain signal values.

D.3 Causal Influence of Communication (CIC): Implementation Details
The Causal Influence of Communication (CIC) [31] estimates the mutual information between the signal and the agent's action.The mutual information between two random variables ˜ and ˜ is defined as the reduction of uncertainty (measured in terms of entropy (•)) in the value of ˜ with the observation of ˜ : The pseudo-code for calculating the CIC for a single agent is presented in Alg. 1.Note that the CIC implementation in [31] considers a multi-dimensional, one-hot, discrete action space with accessible probabilities for every action, while in our case we have a single, continuous action (specifically, the effort , ).To solve this problem, we discretize our action space into intervals between minimum (E = 0) and maximum (E = 1) effort values, and each interval is assumed to correspond to a single discrete action.Let denote the event of an action , belonging to interval .To calculate the CIC value, we start by generating random 'partial' states (i.e., without signal), − , which are then concatenated with each possible signal value to obtain a 'complete' state, = [ , − ] (Lines 5 -7 of Alg. 1).Then, we estimate the probability of an action given a signal value, ( | ), by generating actions from our policy given the 'complete' state ( ( )), and normalizing the number of instances in which the action belongs to a particular bin with the total number of samples.The remaining aspects of the calculation are the same as in the original implementation.In our calculations we used = = 100.

D.4 Fairness Metrics
We also evaluated the fairness of the final allocation, to ensure that agents are not being exploited by the introduction of the signal.We used two of most established fairness metrics: the Jain index [22] and the Gini coefficient [16]: (a) The Jain index [22]: Widely used in network engineering to determine whether users or applications receive a fair share of system resources.It exhibits a lot of desirable properties such as: population size independence, continuity, scale and metric independence, and boundedness.For an allocation of agents, such that the th agent is alloted , the Jain index is given by Eq. 19.[16]: One of the most commonly used measures of inequality by economists intended to represent the wealth distribution of a population of a nation.For an allocation game of agents, such that the th agent is alloted , the Gini coefficient is given by Eq. 20.G(x) ≥ 0. A Gini coefficient of zero expresses perfect equality, i.e., an allocation is fair iff G(x) = 0.
Both metrics showed that learning both with and without the signal results in fair allocations, with no significant change with the introduction of the signal (see Section E).

E SIMULATION RESULTS IN DETAIL
In this section we provide numerical values of the simulation results presented in the main text.Specifically: Tables (2 and 3 Table 12 presents the CIC values.Tables 13, 14, 15, 16, and 17 present the results on the aforementioned metrics for varying signal size, = {1, 2 , 23, , 41, 3  2 }.We also ran simulations with higher growth rate, specifically = 2.The results can be found in Tables 18, and 19.Every resource has a natural upper limit on the size of the population it can sustain.Fig. 2 shows that as the number of agents grow, we reach sustainable strategies at higher equilibrium stock multipliers ( ).Thus, we expect that as we increase the growth rate, the effect of the signal will be even more pronounced in larger populations ( ) -which is corroborated by the aforementioned Tables.The environment's ability to sustain a population also affects the counts of 'active' agents of Section 3.7.2.As we can see in Fig. 2, for a growth rate of = 1, the resource is too scarce, thus, even with the addition of the signal, the first sustainable strategy is reached at = 0.9.This is a high equilibrium stock multiplier, close to the limit of sustainable harvesting.As a result, the number of 'active' agents is naturally really high because they do not need to harvest in turns.By increasing the growth rate to = 2, we have an environment that can sustain larger populations.Therefore, the first strategy that does not result to an immediate depletion is reached much earlier, and we can observe the emergence of a temporal convention (see Table 20).
All reported results are the average values over 8 trials.Note also that, as was specified in Section 3.1.1, . Therefore, in the settings where the growth rate is = 1, ≈ 0.79, while in the settings where = 2, ≈ 0.58.

Figure 2 :
Figure 2: Episode length, with and without the signal ( =), for environments of decreasing difficulty (increasing equilibrium stock multiplier ).

Figure 3 :
Figure 3: Relative difference in social welfare (SW) when signal of cardinality = is introduced (( = − =1 )/ =1 , where = denotes the SW achieved using a signal of cardinality ), for environments of decreasing difficulty (increasing ∝ ) and varying population size ( ∈ [4, 64]).To improve readability, changes greater than 100% are shown with numbers on the top of the bars.

Figure 4 :
Figure 4: Relative difference in convergence time with the introduction of a signal (( = − =1 )/ =1 , where = denotes the time until convergence when using a signal of cardinality ), for environments of decreasing difficulty (increasing ∝ ) and varying population size ( ∈ [8, 64])

Figure 7 :
Figure 7: Evolution of the agents' strategies for each signal value, smoothed over 50 episodes.Fig. 7a pertains to a population of = 4 and signal size = = 4, while Fig. 7b to a population of = 2 and signal size = 3 2 = 3.In both cases the equilibrium stock multiplier is = 0.5.Each row represents an agent ( ), while each column a signal value ( ).Each line depicts the average effort the agent exerts on that specific signal value -calculated by averaging the actions of the agent in each corresponding signal value across the episode.Shaded areas represent one standard deviation.
), (4 and 5),(6 and 7),(8 and 9), and(10 and 11)  include the results (absolute values with and without the introduced signal, relative difference, and Student's T-test p-values) on social welfare, episode lengths (in time-steps), training time (in number of episodes), Jain index, and Gini coefficient, respectively.

Table 1 :
List of hyper-parameters.All the hyper-parameters were left to the default values specified in Ray and RLlib17.For completeness, Table1presents a list of the most relevant of them.

Table 5 :
Episode Length (Relative difference & p-values) (i) Relative difference in Episode Length when signal of cardinality = is introduced ((Result = − Result =1 )/Result =1 , where Result = denotes the achieved result using a signal of cardinality ), and (ii) Student's T-test p-values, for varying population size ( ∈ [2, 64]) and environments of decreasing difficulty (increasing ∝ ).NaN values in the p-values column are due to having only a single data point; both cases (with and without the signal) have the same episode length in all the trials.

Table 7 :
Training Time (Relative difference & p-values) (i) Relative difference in Training Time when signal of cardinality = is introduced ((Result = − Result =1 )/Result =1 , where Result = denotes the achieved result using a signal of cardinality ), and (ii) Student's T-test p-values, for varying population size ( ∈ [2, 64]) and environments of decreasing difficulty (increasing ∝ ).

Table 19 :
Social Welfare, Episode Length, Training Time, Jain Index, Gini Coefficient (i) Relative difference in the achieved result when signal of cardinality = is introduced ((Result = −Result =1 )/Result =1 , where Result = denotes the achieved result using a signal of cardinality ), and (ii) Student's T-test p-values, Results (averaged over 8 trials) for higher growth rate ( = 2), with ( = ) and without ( = 1) the introduced signal, for environments of decreasing difficulty (increasing ∝ ), and population size = 64.