Reinforcement Learning in a Cournot Oligopoly Model

Abstract

This paper analyzes the learning behavior of firms in a repeated Cournot oligopoly game. Literature shows the degree of information and cognitive capacity of learning firms is a key factor that determines long run outcome of an oligopoly market. In particular, when firms possess the knowledge of market demand and are capable of computing the optimal production quantity given the output of other firms, the resulting market outcome is the Nash equilibrium. On the other hand, imitation that assumes low behavioral sophistication of firms generally favors higher output and converges to the Walrasian equilibrium. In this paper, a reinforcement learning algorithm with low cognitive requirement is adopted to model firms’ learning behavior. Reinforcement learning firms observe past production choices and fine tune them to improve profits. Analytical result shows that the Nash equilibrium is the only fixed point of the reinforcement learning process. Convergence to the Nash equilibrium is observed in computational simulations. When firms are allowed to imitate the most profitable competitor, all states between the Nash equilibrium and the Walrasian equilibrium can be reached. Furthermore, the long run outcome shifts towards the Nash equilibrium as the length of firms’ memory increases.

Appendix

In this section, we change the probability distribution adopted in the reinforcement learning algorithm and perform a robustness check on results obtained in the text. In particular, we consider two alternative probability distributions: a Logistic distribution and a Cauchy distribution.

We first investigate the case where firms select production quantities from a Logistic distribution. The probability density function of production choice q follows

$$\begin{aligned} f(q|s,\theta ) = \frac{\exp \left( -\frac{q-\mu (s,\theta )}{\sigma (s,\theta )}\right) }{\sigma (s,\theta )\left( 1+\exp \left( -\frac{q-\mu (s,\theta )}{\sigma (s,\theta )}\right) \right) ^2}, \end{aligned}$$

(30)

where $\mu (s,\theta ) = \theta _{\mu }^{\prime } s$ is the mean and $\sigma (s,\theta ) = \exp (\theta _{\sigma }^{\prime }s)$ is a scale parameter that determines the variance. $\theta =(\theta _{\mu }^{\prime },\theta _{\sigma }^{\prime })'$ is a vector of parameters that firms learn through the reinforcement learning process in (10).

As an example, we look at the oligopoly economy calibrated above and choose market configurations with $n=2$. When imitation is present, we let $k=50$ be the length of memory of firms. Figure 8 is a representative of common outcome in our simulations. The top panel illustrates the learning dynamics of aggregate production quantity in treatment I. The middle panel and the bottom panel present results for treatment II and III, respectively. These results are similar to those from benchmark simulations based on Gaussian distribution: firms are able to learn the Nash quantity and reinforcement learning converges to the Nash equilibrium (treatment I and III); moreover, long run market output deviates from the Nash value when firms perform imitation (treatment II).

Fig. 8

Learning dynamics of joint production based on Logistic distribution with n = 2 and k = 50 for three treatments. Top: Treatment I. Middle: Treatment II. Bottom: Treatment III

We next consider the learning process with a Cauchy distribution. The probability density function of Logistic distribution is

$$\begin{aligned} f(q|s,\theta ) = \frac{1}{\pi \gamma (s,\theta )\left[ 1+(\frac{q-\mu (s,\theta )}{\gamma (s,\theta )})^2\right] }. \end{aligned}$$

(31)

Here $\mu (s,\theta )$ is the location parameter that determines the mode of the Cauchy distribution, and $\gamma (s,\theta )$ is the scale parameter that specifies the half-width at half-maximum.³ We run the same simulations with this distribution and re-examine our results above. In Fig. 9, we plot dynamics of aggregate market output for three treatments with $n=2$ and $k=50$. We find that our conclusions still hold. In particular, convergence to the Nash equilibrium takes place under reinforcement learning and imitation drives market outcome towards the Walrasian equilibrium. Therefore, we conclude that the reinforcement learning process considered in this paper produce robust outcome that is not sensitive to the choice of probability distributions.

Fig. 9

Learning dynamics of joint production based on Cauchy distribution with n = 2 and k = 50 for three treatments. Top: Treatment I. Middle: Treatment II. Bottom: Treatment III