A logarithmic market scoring rule agent-based model to evaluate prediction markets

Abstract

Prediction Markets (PMs) are markets in which agents trade event contingent assets. Enterprises use PMs to forecast revenues and project deadlines. This paper presents an Agent-based model, called Logarithmic Market Scoring Rule-Automated Market Maker (LMSR-ASM), to evaluate Prediction Markets. Our model is capable of testing different types of Automated Market Makers (AMMs), which are mathematical functions or computational mechanisms needed to provide liquidity in Prediction Markets. The model offers insights into how to set parameters in a PM and how profits react to contrasting settings and AMMs. In addition, we simulate different probability processes, distinct AMMs, and agent behaviors. This paper also utilizes the LMSR-ASM to evaluate the impact of choosing initial prices in profits and revenue opportunities regarding AMM computational implementation. We show that we can use the LMSR-ASM to find optimal parameters for maximizing profits in PMs and how different AMMs affect market results under a variety of settings.

Appendices

Appendix A: Overview, design concepts and details protocol

In this appendix, we describe our model in terms of the ODD protocol (Grimm et al. , 2006; Grimm et al. , 2010; Müller et al. , 2013; Grimm et al. , 2020). The ODD protocol provides a “template” to present the model overview, including the purpose and main processes that take place in the model, the Design concepts of the model, and the Details necessary for the implementation of the model.

We split this appendix into three subsections. In Appendix A.1, we present the overview of the model. In Appendix A.2, we show the design of the model. In Appendix A.3, we detail the model in terms of the ODD protocol.

1.1 A.1 Overview

Purpose and patterns

The Logarithmic Market Scoring Rule Artificial Stock Market (LMSR-ASM) is an agent-based model capable of evaluating different Automated Market Makers used in Prediction Markets, namely the Logarithmic Market Scoring Rule (LMSR) (Hanson , 2007) and the Liquidity Sensitive Logarithmic Market Scoring Rule (LS-LMSR) (Othman et al. , 2013). The model builds upon the Santa-Fe Institute Artificial Stock Market (SFI-ASM) (Palmer et al. , 1994; Lebaron , 2002; Johnson , 2002) to evaluate how the two AMMs analyzed differ in relation to profits and liquidity.

Patterns observed that show the ability of the model are:

• Profits decrease with the increase of the parameter b, while volume increases, using LMSR.

• Profits are higher in LS-LMSR than in LMSR, excluding edge cases.

• Volume decreases with the increase of the parameter $\alpha$, using LS-LMSR.

Entities, state variables and scales

The LMSR-ASM includes three types of entities: an Automated Market Maker (AMM), Agents, and Stocks. The AMM sets prices and handles transactions. Agents interact with the AMM, buying and selling according to their own portfolio decisions. Stocks update underlying probabilities and determine the payoff at the end of a simulation.

In Table 3, we present the attributes and state variables associated with the AMM.

Table 3 The attributes and state variables associated with the AMM

In Tables 4 and 5, we present the attributes and state variables associated with the agents and stocks, respectively. Each period does not relate directly to the passing of time but can be interpreted as the opening and closing of a trading session.

Table 4 The attributes and state variables associated with the agents

Table 5 The attributes and state variables associated with the stocks

Process overview and scheduling

The LMSR-ASM follows three different sub-processes: setup, step, and end. At step 0, the model follows the setup, creating initial stocks and setting probabilities. After setup, trading starts, in which all agents interact in order with the AMM. Updates in this phase happen asynchronously. In the last period, the AMM realizes payment for those agents holding the winning stock. Figure 10 presents the full process that rules the LMSR-ASM.

Fig. 10

The LMSR-ASM full process

1.2 A.2 Design

Basic principles

The LMSR-ASM is based on the version presented by Ehrentreich (2008) of the Santa-Fe Institute Artificial Stock Market (SFI-ASM). We use two types of agents: ideal and noisy. Ideal agents know the real probability of an underlying event. As in the SFI-ASM, noisy agents are myopic, meaning they only consider prices for the next period. Noisy agents are also homogeneous with respect to their utility function, determining their optimal stock holdings at each period by maximizing a Constant Absolute Risk Aversion (CARA) utility in the form:

$$\begin{aligned} U(W_{i,t+1})=-e^{-\lambda W_{i,t+1}}, \end{aligned}$$

(A.1)

where $\lambda$ is a strictly positive parameter, representing the degree of risk aversion, and $W_{i,t+1}$ is the expected wealth of agent i for the next period.

Individual decision-making

Ideal agents buy stocks until the price matches the real probability of an event. Noisy agents maximize their utility by buying stocks according to their optimal portfolio, given by

$$\begin{aligned} \widehat{x_{i,t}}=\frac{E_{i,t}[p_{t+1}]-p_t}{\lambda \sigma ^2_{t,p}}, \end{aligned}$$

(A.2)

with $x_{i,t}$ being the amount of stock held by the agent i at period t; $\sigma ^2_{t,p}$ being the variance of the stock, which follows a Bernoulli distribution; $p_t$ is the price at time t and $E_{i,t}[p_{t+1}]$ being agent i’s forecast.

At each period, agents interact in sequence with the market maker, deciding to buy or sell stocks, according to their own expected value calculations of the transaction.

Individual sensing

While ideal agents know exactly the underlying probability of an event, noisy agents make forecasts according to the underlying probability, adjusted by a random factor with normal distribution $\mathcal {N}(0, 0.025)$. Both agents know market prices at each moment of interaction with the market maker.

Individual prediction

Agents predict according to the probability process. Each period, agents update beliefs, which are equal to underlying probabilities (in the case of ideal agents) or the probability plus a noise factor given by a normal distribution $\mathcal {N}(0, 0.025)$ with zero mean and 0.025 variance (in the case of noisy agents).

Interaction

Each agent interacts only with the Automated Market Maker by buying and selling stocks. Trading happens asynchronously at each period.

Learning

Agents do not learn but update beliefs each period according to the underlying probability process.

Adaptation

At each step, agents update their optimal holdings considering the changes in their forecast of the underlying event. If the agent’s forecast of the probability of the underlying event is higher than the current stock price, and it does not hold No stocks, it buys Yes stocks according to its optimal demand. If the agent’s forecast of the probability of the underlying event is lower than the current stock price, and it does not hold Yes stocks, it buys No stocks according to its optimal demand. When the agent’s prediction does not match its stock holdings, the agent changes its current holds according to its optimal portfolio. Table 6 summarizes the agents’ adaptation process.

Table 6 Summary of the agents’ adaptation process

Collectives

The model presents no collective behavior.

Heterogeneity

Heterogeneity in the model is given by the individual forecasts of each agent.

Stochasticity

Stochasticity in the model is given by the probability process of the event and agent behavior. In random walk settings, probabilities are updated at every step, changing in relation to arbitrary shocks that follow a normal distribution $\mathcal {N}(0, 0.025)$.

Noisy agents also similarly update their beliefs, changing beliefs every period (in the case of noisy agents), with their prediction being the underlying probability plus a noise factor given by a normal distribution $\mathcal {N}(0, 0.025)$ with zero mean and 0.025 variance.

Emergence

Emergence in the LMSR-ASM is primarily reflected by market volume and AMM profit. Those emergent variables are the result of trading between agents and the AMM, according to each agent’s portfolio decision and utility-maximization problem.

Non-emergent variables are probabilities and stock prices. Probabilities can be fixed, as set arbitrarily at the start of the simulation, or updated at each period, following a random walk with normal distribution $\mathcal {N}(0, 0.0025)$. Prices closely follow those set probabilities.

Observation

We collect data on trade volume and market profit at the end of the simulation.

Table 7 Initial settings that do not vary among simulations

1.3 A.3 Details

Implementation Details

LMSR-ASM adapts the version of the SFI-ASM due to Ehrentreich (2008) to PM simulations. Ehrentreich (2008) developed the SFI-ASM in Java using the Repast library.

Initialization

Table 7 presents the initial settings of the model that do not vary among simulations. Other initial parameters vary according to the purpose of the analysis done. We use the LMSR-ASM to evaluate optimal parameters; the difference in profits between LMSR and LS-LMSR; the effects of rounding methods on profits; the effects of setting initial prices. Table 8 presents the initial parameters used to find the optimal parameters of the market. Table 9 presents the initial parameters used to find the difference in profits between LMSR and LS-LMSR. Table 10 shows the initial parameters used to find the effects of rounding methods on profits. Table 11 presents the initial parameters used to find the effects of initial prices.

Table 8 Initial parameters used to find the optimal parameters of the market

Table 9 Initial parameters used to find the difference in profits between LMSR and LS-LMSR

Table 10 Initial parameters used to find the effects of rounding methods on profits

Table 11 Initial parameters used to find the effects of initial prices

Input Data

The model does not utilize external data.

Submodels

Stock creation To set initial prices, the AMM creates artificial stocks, which are stocks held by the AMM, and, therefore, do not have to be paid at the last period. During stock creation, the AMM adds new stocks until the price of the marginal stock is greater than the price set. If initial prices are $\$0.50$, no additional stocks are created.

Internal probability updates Before interacting with the AMM, agents observe the underlying probability of the model at that period. For ideal agents, the internal probability is the underlying probability. For noisy agents, the internal probability is the underlying probability plus a noise factor given by a normal distribution $\mathcal {N}(0, 0.025)$.

Internal optimal portfolio updates After updating probabilities, noisy agents update their optimal portfolio calculations, given by

$$\begin{aligned} \widehat{x_{i,t}}=\frac{E_{i,t}[p_{t+1}]-p_t}{\lambda \sigma ^2_{t,p}}, \end{aligned}$$

(A.3)

where $x_{i,t}$ is the amount of stock held by the agent i at period t, $\sigma ^2_{t,p}$ is the variance of the stock, which follows a Bernoulli distribution, $p_t$ is the price at time t and $E_{i,t}[p_{t+1}]$ is agent i’s forecast.

The amount of stocks the agent will buy or sell is given by the difference between their holdings and this portfolio update.

Agent interactions with AMM

Agents send a buy or sell order to the AMM and the AMM provides a price, according to

$$\begin{aligned} P(\overline{q}-q) = C(\overline{q}) - C(q), \end{aligned}$$

(A.4)

where $P(\overline{q}-q)$ is the price of the order, C(q) is the cost function at the time of the order and $C(\overline{q})$ is the cost function after the order.

The cost function for the LMSR is given by

$$\begin{aligned} C(q)=b\log \left( e^{\frac{q_\textsc {Y}}{b}} + e^{\frac{q_\textsc {N}}{b}} \right) , \end{aligned}$$

(A.5)

where $q_\textsc {Y}$ and $q_\textsc {N}$ are the two types of stocks available, and b a strictly positive parameter that controls liquidity in the market.

The cost function for the LS-LMSR is given by

$$\begin{aligned} C(q)=b(q)\log \left( e^{\frac{q_\textsc {Y}}{b(q)}} + e^{\frac{q_\textsc {N}}{b(q)}} \right) , \end{aligned}$$

(A.6)

where

$$\begin{aligned} b(q)=\alpha (q_\textsc {Y}+q_\textsc {N}), \end{aligned}$$

(A.7)

with $\alpha$ being a strictly positive parameter set by the owner of the PM.

Appendix B: Variance and factor analysis

1.1 B.1 Variance analysis

To evaluate how many runs are necessary, we follow the protocol proposed in Lorscheid et al. (2012), presenting a pre-experimental error variance matrix created by analyzing four preliminary design points. Table 12 describes the settings used in the simulations. “N. of Agents” stands for the number of agents. The initial probability and the probability after shock are shown in the columns “Initial Prob.”, and “Prob. after shock”, respectively.

Table 12 Description of the parameters used in the Variance Analysis

We consider Noisy agents, 2-digits rounding methods, and fixed probability process

The parameters represent an assortment of standard and extreme values we use during simulations. These design points aid in the decision of the required number of runs. Table 13 shows the results of these simulations, in an error variance matrix.

Table 13 Error Variance Matrix

Even at a low number of runs, the model presents convergence in the coefficient of variation. However, convergence happens only at a higher number of runs, for some variables. The low number of runs necessary for convergence is intuitive, given stochasticity in the model is derived from a normal distribution with mean 0 and low standard deviation ($\mathcal {N}(0, 0.025)$). In Section 4, we decided to run each specification 100 times since our pre-simulations show convergence in the analyzed parameters at that level. This helps maintain a low computational cost, allowing for the analysis of more simulation specifications. To further demonstrate the validity of the chosen number of runs, we present in Table 14 the results of an F-test between all samples and the result of a t-test between the largest sample and the sample with 100 runs. All t-tests and F-tests failed to reject the null hypothesis that the means are equal.

Table 14 Statistics and p-values for equality of means between number of runs

1.2 B. 2 ANOVA

To evaluate the impact of the parameters on profits and volume, we run a factorial ANOVA, varying multiple parameters simultaneously. Table 15 presents all the different initial settings used. Due to the number of simulations needed, a lower number of 50 runs was used. It is important to notice that this value is still consistent with the process presented in 1. There are a total of 6,000 different initial specifications for the LS-LMSR and 4,000 for the LMSR, with the total number of simulations used in the ANOVA being 500,000. We present ANOVA results for four different specifications. Table 16 are results for volume and profit, using LMSR. Table 17 are results for the same variables, using LS-LMSR.

Table 15 Initial parameters used in the factorial ANOVA

Table 16 Analysis of variance for the LMSR

Table 17 Analysis of variance for the LS-LMSR

ANOVA supports our major findings. In particular, when it comes to the optimal values for parameters b and $\alpha$ we have discussed in Section 4.1, we can infer that they are significant for the LMSR and LS-LMSR markets, respectively. Observe that it holds for volume and market profit. With respect to the analysis based on the variation of the probability after shock and the initial probability provided in Section 4.2, Tables 16 and 17 tell us that – by themselves – the initial probability and probability after shock are not significant. However, these two terms have significant interactions with initial prices, which leads to the interpretation that, in a market where market creators decide initial prices, volumes, and profits are similar even between markets with different overall probabilities, given other characteristics are equal. Also, due to the difference in the effect of these interactions, we can see the final probability has a bigger effect on outcomes, due to it determining payoff. Regarding the rounding methods discussed in Section 4.3, we observe that their effects are not significant for the LMSR but significant for the LS-LMSR. The initial prices discussed in Section 4.4 are significant for the market profit both in the LSMR and LS-LMSR. As prices reflect probabilities, the AMM will have to bear the costs if initial prices are far from their actual value.

1.3 B.3 Effect sizes

We utilize the partial $\eta ^{2}$ (Pierce et al. , 2004), derived from the analysis of variance presented in Section 2, to measure effect sizes of the parameters on volume and profit. This approach is also used in Klingert and Meyer (2018). The results support our main findings in the paper while also revealing new insights. Table 18 shows the partial $\eta ^{2}$ of the varying parameters on volume under LMSR. As expected, the main drivers of variance are the number of agents and the value of parameter b. One other interesting effect is that the initial probability of the event does not affect volumes, and surprisingly, neither does the probability after the shock. This shows that volume is similar in a market with relatively homogeneous traders – regardless of how much probabilities vary over time. It is important to notice that only one shock is possible in the model, so we cannot test how various probability shifts affect market volume.

Table 18 Effect sizes on volume using LMSR

Table 19 shows the effects of the factors on profit, for the LMSR. The main effects are related to choosing b and initial prices. This reinforces our findings regarding the importance of setting initial prices according to the market creator’s beliefs of the probability of the outcome. The effect of b is also intuitive, as the parameter determines the worst-case loss for the market maker, which is given by $b\log n$.

Table 19 Effect sizes on profit using LMSR

The following tables show the results regarding the LS-LMSR. Table 20 presents the effects on volume. Results are similar to the LMSR case, with the number of agents being the main factor. However, in the LS-LMSR, initial prices also affect market volume with a low impact.

Table 20 Effect sizes on volume using LS-LMSR

Results for the effects on profits are also similar to LMSR. However, there is a distinct effect of rounding methods – showing that the way trades are (computationally) handled has a significant impact on LS-LMSR.

Table 21 Effect sizes on profit using LS-LMSR

These measures strengthen the main findings of our paper, such as rounding methods having little to no impact on volume but being an important decision that affects profits under the LS-LMSR. The results also show the importance of defining initial prices, so market makers are advised to study underlying events before creating a market and setting prices to their beliefs.