D-Brane: a diplomacy playing agent for automated negotiations research

Abstract

Existing work on Automated Negotiations commonly assumes the negotiators’ utility functions have explicit closed-form expressions, and can be calculated quickly. In many real-world applications however, the calculation of utility can be a complex, time-consuming problem and utility functions cannot always be expressed in terms of simple formulas. The game of Diplomacy forms an ideal test bed for research on Automated Negotiations in such domains where utility is hard to calculate. Unfortunately, developing a full Diplomacy player is a hard task, which requires more than just the implementation of a negotiation algorithm. The performance of such a player may highly depend on the underlying strategy rather than just its negotiation skills. Therefore, we introduce a new Diplomacy playing agent, called D-Brane, which has won the first international Computer Diplomacy Challenge. It is built up in a modular fashion, disconnecting its negotiation algorithm from its game-playing strategy, to allow future researchers to build their own negotiation algorithms on top of its strategic module. This will allow them to easily compare the performance of different negotiation algorithms. We show that D-Brane strongly outplays a number of previously developed Diplomacy players, even when it does not apply negotiations. Furthermore, we explain the negotiation algorithm applied by D-Brane, and present a number of additional tools, bundled together in the new BANDANA framework, that will make development of Diplomacy-playing agents easier.

Appendices

Appendix A:

Our experiments are largely implemented using the DipGame framework. However, we have implemented a couple of new tools on top of DipGame to make experimentation easier. We have combined these tools into a new extension of DipGame, that we call BANDANA (BAsic eNvironment for Diplomacy playing Automated Negotiating Agents). It includes the following components:

• A new negotiation server.

• A new negotiation language, which we find simpler to use than DipGame’s default language.

• A Notary agent, for the implementation of the Unstructured Negotiation Protocol, as explained in Section 4.

• Several example agents, including D-Brane and DumbBot.

• The strategic component of D-Brane.

• Example code that shows how one can implement a negotiating agent on top of D-Brane’s strategic component.

• An Adjudicator which, given a set of orders for all units, determines which of those orders are successful.

• A Game Builder, which allows users to set up a customized board configuration. This can be useful for testing.

The BANDANA framework can be downloaded from: http://www.iiia.csic.es/~davedejonge/bandana.

More details about BANDANA’s negotiation language and its other tools can be found in the manual which can also be downloaded from the same address.

Appendix B:

We here give a more thorough definition of the one-shot game D i p _𝜖 . We claim that D i p _𝜖 can be modeled as a COG, and that a single round of Diplomacy can be seen as an instance of the negotiation game N ( D i p _𝜖 ).

Definition 9

Let 𝜖 denote a configuration of units on the Diplomacy map. Then D i p _𝜖 is defined by the tuple:

$$(Gr, SC, Pl^{Dip}, (Units_{1}, {\dots} Units_{7}), loc, (f^{Dip_{\epsilon}}_{1}{\dots} f^{Dip_{\epsilon}}_{7})).$$

Here, G r is a symmetric graph, of which the vertices are called provinces. The set of provinces is denoted P r o v and we use the notation a d j(p,q) to state that provinces p and q are adjacent in the graph. The set of Supply Centers SC is a subset of P r o v. The set P l ^Dip represents the 7 players: $Pl^{Dip} = \{\alpha _{1}, {\dots } \alpha _{7}\}$. For each player α _i there is a finite set U n i t s _i , which we call the set of units owned by α _i . These sets are all disjoint: $i\neq j \Rightarrow Units_{i} \cap Units_{j} = \emptyset$. The set of all units is denoted: $\mathit {Units}= \bigcup _{i=1}^{7}\mathit {Units_{i}}$. The state 𝜖 of the game implicitly defines an injective function $loc : Units \rightarrow Prov$ that assigns a province (the location of u) to any unit u. In order to define the utility functions $f^{Dip_{\epsilon }}_{i}$ we first need to define several other concepts.

Given the state 𝜖 we can define the set of possible orders O r d ^𝜖:

$$\begin{array}{@{}rcl@{}} Ord^{\epsilon} &=& Mto^{\epsilon} \bigcup Sup^{\epsilon} \\ Mto^{\epsilon} &=& \{ (u, p) \in Units \times Prov \mid p=loc(u) \lor adj(p, loc(u)) \} \\ Sup^{\epsilon} &= & \{ (u, u^{\prime}) \in Units \times Units \mid u \neq u^{\prime}\} \end{array} $$

The orders in M t o ^𝜖 are called move-to orders ¹¹ and the orders in S u p ^𝜖 are called support orders. We use the notation $Ord_{u}^{\epsilon }$ to denote the subset of O r d ^𝜖 consisting of all possible orders for a given unit u.

$$\begin{array}{@{}rcl@{}} &&Ord_{u}^{\epsilon} = Mto_{u}^{\epsilon} \bigcup Sup_{u}^{\epsilon}\\ &&Mto_{u}^{\epsilon} = \{ (u^{\prime}, p) \in Mto^{\epsilon} \mid u^{\prime} = u\}\\ &&Sup_{u}^{\epsilon} = \{ (u^{\prime}, u^{\prime\prime}) \in Sup^{\epsilon} \mid u^{\prime} = u\} \end{array} $$

Furthermore, we will use $Ord_{i}^{\epsilon }$ to denote the set possible orders for any unit of player α _i :

$$\begin{array}{@{}rcl@{}} &&Ord_{i}^{\epsilon} = \bigcup_{u \in Units_{i}} Ord_{u}^{\epsilon}\\ &&Ord_{-i}^{\epsilon} = Ord^{\epsilon} \setminus Ord_{i}^{\epsilon} \end{array} $$

and for a set of players $C \subset Pl$ we define:

$$\begin{array}{@{}rcl@{}} &&Ord_{C}^{\epsilon} = \bigcup_{\alpha_{i} \in C} Ord_{i}^{\epsilon} \\ &&Ord_{-C}^{\epsilon} = Ord^{\epsilon} \setminus Ord_{C}^{\epsilon} \end{array} $$

We use similar notation conventions for other sets, such as U n i t s, M t o ^𝜖 and S u p ^𝜖.

An action μ _i for a player α _i in D i p _𝜖 is then defined as a set of orders, containing exactly one order for each of α _i ’s units:

$$ \mathcal{M}_{i}^{Dip_{\epsilon}} = \{\mu_{i} \subset Ord_{i}^{\epsilon} \mid \forall u\in Units_{i} : |Ord_{u}^{\epsilon} \cap \mu_{i}| = 1 \} $$

(4)

Definition 10

If α _i plays an action μ _i that contains order o then we say that α _i submits the order o. If μ=(μ ₁,μ ₂,...μ ₇) is an action profile then $\hat {\mu }$ denotes the set of all orders submitted by all players:

$$\hat{\mu} = \bigcup_{i=1}^{7} \mu_{i} $$

Definition 11

A support order $(u,u^{\prime }) \in Sup^{\epsilon }$ is considered valid, for an action profile μ, if $\hat {\mu }$ contains a move-to order $(u^{\prime }, p) \in Mto^{\epsilon }$ where p is adjacent to the location of u. This is denoted by the predicate $val(\mu , u,u^{\prime })$.

$$val(\mu, u,u^{\prime}) \,\,\, \Leftrightarrow \,\,\, \exists p \in Prov : (u^{\prime},p) \in \hat{\mu} \,\,\, \land \,\,\, adj(p, loc(u)) $$

The rules of Diplomacy specify that players may only submit support orders that are valid.

Definition 12

Given an action profile μ, a support $(u,u^{\prime })\in Sup_{u}^{\epsilon }$ in $\hat {\mu }$ is said to be cut if $\hat {\mu }$ also contains a move-to order that moves to the location of u:

$$cut(\mu, u) \,\,\, \Leftrightarrow \,\,\, \exists (u,u^{\prime}) \in \hat{\mu} \,\,\, \land \,\,\, \exists (u^{\prime\prime},p) \in \hat{\mu} \,\,\, \land \,\,\, p = loc(u) $$

Definition 13

The set of successful supports of u in an action profile μ is defined as those orders that support u, and that are valid and not cut:

$$\begin{array}{@{}rcl@{}} SucSup_{\mu, u} = \{(u^{\prime}, u) \in \hat{\mu} \mid val(\mu, u^{\prime}, u) \land \lnot cut(\mu, u^{\prime}) \} \end{array} $$

Definition 14

The force s(μ,u,p) exerted by unit u on province p is defined as:

$$s(\mu, u, p) =\! \left\{\begin{array}{ll} 1.5 + |SucSup_{\mu, u}| & \text{if}~(u, p) \in \hat{\mu} \,\,\, \land \,\,\, p = loc(u)\\ 1 + |SucSup_{\mu, u}| & \text{if}~(u, p) \in \hat{\mu} \,\,\, \land \,\,\, p \neq loc(u)\\ 0 & \text{otherwise} \end{array}\right. $$

Definition 15

We say a player α _i conquers a province p if α _i has a unit that exerts more force on p than any other unit:

$$\begin{array}{@{}rcl@{}} conq(\mu, i, p) &\Leftrightarrow& \exists u \!\in Units_{i}\ \forall u^{\prime} \!\in\! Units \!\setminus\! \{u\} \!:\! s(\mu, u,p)\\ &>& s(\mu, u^{\prime}, p) \end{array} $$

Finally, we can define the utility function $f^{Dip_{\epsilon }}_{i}$ for a player α _i as the number of Supply Centers he or she conquers:

$$ f^{Dip_{\epsilon}}_{i}(\mu) = | \{p \in SC \mid conq(\mu, i, p) \} | $$

(5)

A natural way to play Diplomacy is to determine for each Supply Center separately whether, and how, it can be conquered. However, the decision how to attack one Supply Center may restrict the possibilities to attack another Supply Center if the same units are involved. This is the essence of a COG. Therefore, we will now show how D i p _𝜖 can be modeled as a COG.

We define the set of units of player α _i involved in province p as those units that may move to p, hold in p, support a unit holding in or moving to p, or that may cut any opponent unit that could give support to another opponent unit holding in p or moving to p. This set is denoted by U n i t s _p,i . More precisely, it consists of all units next to or inside p, and all units located next to an opponent’s unit that is located next to p:

Definition 16

The set of units of player α _i involved in province p, denoted U n i t s _p,i is defined as:

$$\begin{array}{@{}rcl@{}} &&Units_{p,i} = MayAttackOrSupport_{p,i} \cup MayCut_{p,i}\\ &&MayAttackOrSupport_{p,i} = \{u \in Units_{i} \mid loc(u) = p \lor adj(p,loc(u)) \\ &&\qquad\qquad\qquad\,\, MayCut_{p,i} = \{u \in Units_{i} \mid \exists u^{\prime} \in Units_{-i} : adj(loc(u), loc(u^{\prime})) \quad \land \\ &&\qquad\qquad\qquad\qquad\qquad\qquad\,\,\,(loc(u^{\prime}) = p \lor adj(p,loc(u^{\prime})))\} \end{array} $$

We model D i p _𝜖 as a COG by defining a micro-game D i p _𝜖,p for each Supply Center p∈S C. An action in such a micro-game consists of a set of orders containing maximally 1 order for each unit of player α _i involved in Supply Center p:

$$\mathcal{M}_{i}^{Dip_{\epsilon,p}} = \left\{\mu_{p,i} \subset \bigcup_{u \in Units_{p,i}} Ord_{u}^{\epsilon} \mid \forall u \in Units_{p,i} : | \mu_{p,i} \cap Ord_{u}^{\epsilon}| \leq 1\right\}$$

Note that in this definition a player is not required to submit an order to each unit involved in p. This is because that unit may instead be used to attack or defend another province $p^{\prime }$.

This definition implies that there are binary constraints between the micro-games, because a unit may be involved in more than one province. Two actions μ _p,i and μ _q,i are incompatible if for any unit involved in both provinces, two different orders are submitted. That is, μ _p,i and μ _q,i are compatible iff the following restriction holds:

$$ \forall u \in Units_{i} : |(\mu_{p,i} \cup \mu_{q,i}) \cap Ord_{u}^{\epsilon} | \leq 1 $$

(6)

We define the utility function of the micro-game D i p _𝜖,p to be:

$$ f^{Dip_{\epsilon,p}}_{i}(\mu_{p}) = \left\{\begin{array}{ll} 1 & \text{if}~ conq(\mu_{p}, i, p) \\ 0 & \text{otherwise} \end{array}\right. $$

(7)

Here, μ _p is an action profile in the micro-game D i p _𝜖,p . The question whether c o n q(μ,i,p) holds only depends on the orders submitted for the units involved in p. Therefore, if $\hat {\mu }_{p}$ is a subset of $\hat {\mu }$ then c o n q(μ _p ,i,p) holds iff c o n q(μ,i,p) holds, which means that (5) and (7) are consistent with (1).

In the following, we use the notation U n i t s _C , for a set of players C (a ‘coalition’) to denote the union of the units of those players:

$$\begin{array}{@{}rcl@{}} &&Units_{C} = \bigcup_{\alpha_{i} \in C} Units_{i}\\ &&Units_{-C} = Units \setminus Units_{C} \end{array} $$

Definition 17

Let C denote a coalition containing player α _i , then a battle plan β for α _i and Supply Center p is a set of orders $\beta \subset Ord^{\epsilon }$ of the form:

$$\beta = \{ (u,p) \} \cup Supports \cup Cuts$$

with u∈U n i t s _i , and $Supports \subseteq Sup_{C}^{\epsilon }$ is a (possibly empty) set of support orders $(u^{\prime }, u)$ that support u and $Cuts \subseteq Mto_{C}^{\epsilon }$ a (possibly empty) set of orders that aim to cut any opponent unit that may support a hostile move into p:

$$(u^{\prime\prime},p^{\prime}) \!\in\! Cuts \,\,\, \Rightarrow \,\,\, \exists u^{\prime} \!\in\! Units_{-C} : \,\, loc(u^{\prime}) \,=\, p^{\prime} \,\,\, \land \,\,\, adj(p^{\prime},p) $$

Furthermore we have the restriction that β can contain at most one order for each unit:

$$\forall u \in Units : |\beta \cap Ord_{u}| \leq 1$$

The set of all battle plans for player α _i on Supply Center p is denoted $B_{i,p}^{\epsilon }$.

Definition 18

Given a set of deals X, an invincible plan is a battle plan $\beta \in B_{i,p}^{\epsilon }$ for some province p and some player α _i that for every action profile that satisfies X and in which all orders of β are submitted α _i will conquer p. That is, β is invincible iff the following holds:

$$\forall \mu \in \mathcal{M}^{Dip_{\epsilon}[X]} : \quad \beta \subset \hat{\mu} \quad \Rightarrow \quad conq(\mu, i, p)$$

Furthermore, we can determine all invincible pairs:

Definition 19

Given a set of deals X, an invincible pair is a pair of battle plans $(\beta _{1} , \beta _{2}) \in B_{i,p}^{\epsilon } \times B_{i,q}^{\epsilon }$ for two provinces p,q and some player α _i that guarantees player α _i to either conquer p or conquer q:

$$\forall \mu \in \mathcal{M}^{Dip_{\epsilon}[X]} : \,\, \beta_{1} \cup \beta_{2} \subset \hat{\mu} \,\, \Rightarrow \,\, conq(\mu, i, p) \lor conq(\mu, i, q)$$