Monte Carlo Tree Search: a review of recent modifications and applications

Abstract

Monte Carlo Tree Search (MCTS) is a powerful approach to designing game-playing bots or solving sequential decision problems. The method relies on intelligent tree search that balances exploration and exploitation. MCTS performs random sampling in the form of simulations and stores statistics of actions to make more educated choices in each subsequent iteration. The method has become a state-of-the-art technique for combinatorial games. However, in more complex games (e.g. those with a high branching factor or real-time ones) as well as in various practical domains (e.g. transportation, scheduling or security) an efficient MCTS application often requires its problem-dependent modification or integration with other techniques. Such domain-specific modifications and hybrid approaches are the main focus of this survey. The last major MCTS survey was published in 2012. Contributions that appeared since its release are of particular interest for this review.

1 Introduction

Monte Carlo Tree Search (MCTS) is a decision-making algorithm that consists in searching combinatorial spaces represented by trees. In such trees, nodes denote states, also referred to as configurations of the problem, whereas edges denote transitions (actions) from one state to another.

MCTS has been originally proposed in the work by Kocsis and Szepesvári (2006) and by Coulom (2006), as an algorithm for making computer players in Go. It was quickly called a major breakthrough (Gelly et al. 2012) as it allowed for a leap from 14 kyu, which is an average amateur level, to 5 dan, which is considered an advanced level but not professional yet. Before MCTS, bots for combinatorial games had been using various modifications of the minimax alpha–beta pruning algorithm (Junghanns 1998) such as MTD(f) (Plaat 2014) and hand-crafted heuristics. In contrast to them, the MCTS algorithm is at its core aheuristic, which means that no additional knowledge is required other than just rules of a game (or a problem, generally speaking). However, it is possible to take advantage of heuristics and include them in the MCTS approach to make it more efficient and improve its convergence.

Moreover, in practical applications, the given problem often tends to be difficult for the base variant of the algorithm. The utilitarian definition of “too difficult” is that MCTS achieves poor outcomes in the given setting under practical computational constraints. Sometimes, increasing the effective computational budget would help, although in practical applications it may not be possible (e.g., because of strict response times, hardware costs, parallelization scaling). However, often, a linear increase is not enough to tackle difficult problems represented by trees and solving them would require effectively unlimited memory and computational power. There can be various reasons for a given problem being hard for MCTS. To name a few—combinatorial complexity, sparse rewards or other kinds of inherent difficulty. Whenever the vanilla MCTS algorithm, i.e., implemented in its base unmodified form, fails to deliver the expected performance, it needs to be equipped with some kind of enhancements. In this survey, we will focus only on such papers that introduce at least one modification to the vanilla version of the method. Although works that describe the use of standard MCTS in new domains have been published, they are not in the scope of this survey.

More recently, MCTS combined with deep reinforcement learning (RL) has become the backbone of AlphaGo developed by Google DeepMind and described in the article from Silver et al. (2016). It has been widely regarded as not only another major breakthrough in Go, but in artificial intelligence (AI) in general. It is safe to say that MCTS has been tried in most of combinatorial games and even some real-time video games (Farooq et al. 2016; Kim and Kim 2017). Examples include Poker (Van den Broeck et al. 2009), Chess (Silver et al. 2018), Settlers of Catan (Szita et al. 2009), Othello (Robles et al. 2011), Hex (Arneson et al. 2010), Lines of Action (Winands et al. 2010), Arimaa (Syed and Syed 2003), Havannah (Teytaud and Teytaud 2010) and General Game Playing (GGP) (Genesereth and Thielscher 2014; Perez et al. 2014). The latter is particularly interesting because in GGP, computer bots are pitted against each other to play a variety of games without any domain knowledge. This is where MCTS is a commonly applied approach.

Driven by successes in games, MCTS has been increasingly often applied in domains outside the game AI such as planning, scheduling, control and combinatorial optimization. We include examples from these categories in this article as well. Formally, MCTS is directly applicable to problems which can be modelled by a Markov decision process (MDP) (Lizotte and Laber 2016), which is a type of discrete-time stochastic control process. Certain modifications of MCTS make it possible to apply it to partially observable MDPs (POMDPs).

1.1 Methodology

We review only such applications of the MCTS algorithm in which the standard version of the algorithm has been deemed ineffective or unsatisfactory, because of the given problem difficulty or unsuitability for the standard version as it is. This is a common scenario, for instance, in real world problems such as vehicle routing problem (VRP) or real-time video games, which tend to be more complex than combinatorial games due to an enormous branching factor, large state spaces, a continuous environment (time and space) and fast-paced action that requires making constant decisions.

It is useful to distinguish between two classes of environments—single-agent and multi-agent ones. Let us comment on this using the game domain example. The differences between single-player and multi-player games make various AI techniques suitable for a particular environment. For instance, deterministic single-player games are like puzzles to solve. The success in solving them depends only on the agent and not on external factors such as chance or opponents. Multi-player games require some form of opponent’s modelling or cooperation (if they are cooperative). The implementation of MCTS can be optimized appropriately for the given case.

We focus on multi-agent problems, e.g., multi-player games, with a few exceptions. The exception is that we include such approaches of using the MCTS algorithm that, in our opinion, could be easily applied if the given problem was a multi-agent one. This is due to our subjective evaluation. The idea behind this was to provide readers with relatively well-generalizable modifications and not necessarily tailored for a given single-player game.

Moreover, certain problems may be modelled as single-player or multi-player games. For example, VRP can be modelled as a multi-player cooperative game, in which each vehicle has its own game tree and the synchronization between them is treated as in a multi-agent system. However, it can also be modelled as a single-player game, in which there is one coordinator player.

This survey covers contributions made after 2012. The reasoning behind 2012 was to start from the year in which the previous comprehensive MCTS survey was published by Browne et al. (2012). We recommend it to readers interested in earlier developments related to MCTS and as an additional introduction to the method. Section 2 contains an introduction to the MCTS algorithm as well.

To start with, we focused on IEEE Transactions on Games (ToG), which was previously known as IEEE Transactions on Computational Intelligence and AI in Games (TCIAG, until 2018). This is the leading journal concerning scientific, technical, and engineering aspects of games and the main source of high-quality contributions related to MCTS. We have also included the major conference on games—IEEE Conference on Games (CoG). It also changed the name and we included proceedings published under the previous name of IEEE Computational Intelligence and Games (CIG). Given those two sources, we performed an exhaustive search within each issue of ToG/TCIAG and each proceedings of CoG/CIG and analyzed all articles whether they fit the scope of this survey.

The remaining articles were found through scientific bibliography providers (online libraries): dblp, IEEE Xplore Digital Library and Google Scholar. Using them, we performed search over titles and abstracts by keywords. The following keywords were chosen: “Monte Carlo”, “Monte Carlo”, “Tree Search”, “MCTS”, “Upper Confidence Bounds”, “UCT”, “UCB”, “Multi-Armed Bandit”, “Bandit-Based”, “General Game Playing” and “General Video Game Playing”. The results were curated by us with the bar set low (i.e., an acceptance rate was high). Using common sense, we excluded articles which had not been peer reviewed or were available only from unverified sources. The latter applies to Google Scholar which can index such sources too.

1.2 Structure of the survey

The following survey structure is mainly organized by types of extensions of MCTS algorithm. However, those modifications are usually motivated by challenging applications of the algorithm. These two classifications cross each other multiple times, which makes unambiguous navigation through the survey content troublesome. This section is to present the main structure of the text. The complete overview of references is presented in two Tables, grouped by application and grouped by method. It could be found in the last section Conclusions on pages 53, 54, 55.

The main content of the survey is composed of seven sections.

Section 2—Classic MCTS introduces the MCTS algorithm in its classic (vanilla) variant. The basics are recalled for readers with less experience with the method.

Section 3—Games with Perfect Information is devoted to the modifications of MCTS in games with perfect information, which are simpler in the combinatorial aspect. The topic is divided into three subsections:

• Action Reduction (Sect. 3.1)—extensions based on elimination of possible actions in decision states. Usually, obviously wrong actions are skipped. This could be achieved by manipulation of exploration term in the tree policy function.

• UCT Alternatives (Sect. 3.2)—Upper Confidence Bounds for Trees (UCT) is the core equation in the MCTS method. Several modification of UCT are discussed.

• Early Termination (Sect. 3.3)—techniques for improving efficiency by setting cut-off depth for random playouts.

• Opponent Modelling (Sect. 3.4)—chosen aspects of modelling opponent’s moves in perfect information games

Section 4—Games with Imperfect Information is dedicated to imperfect information games also referred to as games with hidden information. We distinguish six different types of MCTS extensions related to this game genre.

• Determinization (Sect. 4.1)—removing randomness from the game by setting a specific value for each unknown feature.

• Information Sets (Sect. 4.2)—a method to deal with imperfect information games, where states, which are indistinguishable from player’s perspective are grouped in information sets.

• Heavy Playouts (Sect. 4.3)—extension of simple, random playouts by adding domain-specific knowledge. Methods from this group proved to be effective; however, one should be aware that utilization of domain knowledge is always at the expense of increased algorithm’s complexity.

• Policy Update (Sect. 4.4)—a general group of methods designed to manipulate policy during the tree building.

• Master Combination (Sect. 4.5)—algorithms taking part in competitions are usually a combination of different MCTS extensions.

• Opponent Modelling (Sect. 4.6)—modelling the game progress requires modelling the opponent’s actions. This is actually true for all multi-player games, but is particularly challenging in imperfect information games.

Section 5—Combining MCTS with Machine Learning summarizes the approaches that combine MCTS with machine learning (ML).

• MCTS + Neural Networks (Sect. 5.3)—AlphaGo approach and other inspired methods.

• MCTS + Temporal Difference Learning (Sect. 5.4)—MCTS combined with temporal difference (TD) methods.

• Advantages and Challenges of Using ML with MCTS (Sect. 5.5)—summary of the section.

Section 6—MCTS with Evolutionary Methods summarizes the approaches that combine MCTS with methods inspired by biological evolution.

• Evolving Heuristic Functions (Sect. 6.1)—evaluation functions for MCTS are evolved.

• Evolving Policies (Sect. 6.2)—the aspects related to performing simulations (roll-outs) undergo the evolution process.

• Rolling Horizon Evolutionary Algorithm (RHEA, Sect. 6.3)—a competitor algorithm to MCTS with elements inspired by evolutionary algorithms (EA).

• Evolutionary MCTS (Sect. 6.4)—combination of MCTS and EA in a single approach.

Section 7—MCTS Applications Beyond Games is focused on applications to “real-world” problems from outside the games domain.

• Planning (Sect. 7.1)—optimal planning in logistics and robotics.

• Security (Sect. 7.2)—finding optimal patrolling schedules for security forces in attacker–defender scenarios.

• Chemical synthesis (Sect. 7.3)—planning in organic chemistry domain.

• Scheduling (Sect. 7.4)—real business challenges, such as scheduling combined with risk management.

• Vehicle Routing (Sect. 7.5)—solutions to various formulations of the VRP.

Section 8—Parallelization discusses parallel MCTS implementations. Both basic approaches and more advanced methods are discussed.

Section 9—Conclusions presents an overview of the discussed methods and applications in the form of two summary tables, as well as conclusions and open problems.

Note—The majority of the literature works analysed in this survey apply some kind of parameter optimization or fine-tuning. In particular, the focus is on the exploration constant C, the maximum search depth, the efficient use of an computational budget, or parameters related to choosing and executing policies (tree policy and default policy). Typically such an optimisation is performed empirically.

Due to a variety of possible parameter-tuning approaches and their direct connections to particular implementations of the MCTS mechanism, we decided to address the topic of parameter optimization alongside introduction and discussion of particular MCTS variants/related papers—instead of collecting them in a separate dedicated section. We believe that such a presentation is more coherent and allows the reader to take a more comprehensive look at a given method.

1.3 Definitions of the considered game genres

In this section we present selected definitions of the most popular genres of games considered in this paper. An application-oriented taxonomy is presented in Table 1 in Sect. 9.

Perfect Information Games. Game is classified as perfect information when every player at any time has a complete knowledge about the history and current state of the game. Chess and Go are prominent representatives of this group.

Imperfect Information Games. In games with imperfect information a game state is not fully available to the players, which means that decisions are taken without the full knowledge of the opponent’s position. There can be hidden elements of the game, for example cards in hand or face down cards. Game mechanics can include also randomness e.g. drawing a starting hand of cards at the beginning of gameplay. Most modern games are classified as imperfect information.

General Game Playing (GGP). GGP (Genesereth and Thielscher 2014) is a research framework that concerns creating computer agents (AI players) capable of playing a variety of games without any prior knowledge of them. Despite a proposal by Thielscher (2010) specifying how to include games of chance and incomplete information, the official GGP Competition that is organized each year in this area has only involved finite combinatorial games with perfect information. Apart from these three conditions, the games can be of any kind, e.g. they do not need to be only board ones. Also, they can involve any finite number of players including one, i.e., single-player games. The agents have to figure out how to play from scratch, being given only rules of the game to play. This is possible due to the fact that rules are written in a formal logic-based language, called game description language (GDL) discussed in details by Świechowski and Mańdziuk (2016b), which is a concise way to represent a state machine of the initial state, terminal states and all possible transitions. Therefore, agents are able to simulate games in order to do reasoning. Since 2007, all GGP winners, such as CadiaPlayer developed by Finnsson and Björnsson (2011) have been simulation-based. An introduction to GGP and review of the related literature can be found in the survey paper by Świechowski et al. (2015b).

General Video Game Playing (GVGP). GVGP is also referred to as General Video Game AI (GVGAI) (Perez-Liebana et al. 2016). The GVGP Competition, first hosted in 2014 at IEEE Conference on Computational Intelligence and Games, has been inspired by a more mature GGP Competition. However, it is not a successor to GGP but rather a different framework where the time to decide on an action is significantly shorter. The goal remains the same—to create agents capable of playing various games based only on rules given to them as a run-time parameter. The games of choice, in GVGP, are simple 2D real-time video games. The examples are Sokoban, Asteroids, Crossfire or Frogger. Similar to GGP, this framework has its own language for game definitions that features concepts such as sprites, levels and it is generally more suitable for said games. The agents have only 40 ms to make a move in the game. Despite such a vast difference in response times, both GGP and GVGP competitions are dominated by simulation-based approaches including ones based on MCTS. The competition is open for entries to various specific tracks such as single-player planning, single-player learning, two-player planning and level generation.

Arcade Video Games. Arcade Video Games is a group of simple computer games that gained popularity in the early 1980s. When the MCTS algorithm had been developed, many researchers validated its applicability for games such as Pac-Man, Mario or Space Invaders. In many cases interesting modifications to standard MCTS were formulated.

Real Time Strategy Games. In Real Time Strategy games players combat each other through building units and simulating battles on a map. In general, there are two types of units which the players can deploy: economy-based and military. The former gather resources from certain places on the map; they can be either renewable or depletable depending on the game. These resources are spent on recruiting military units, which fight in battles controlled by the players. This combination of financial and tactical management results in RTS games posing a significant challenge to tree search algorithms. The three main issues associated with RTS games are:

• huge decision space—Starcraft,¹ the precursor of the genre, with the branching factor about $10^{50}$ or higher Ontañón et al. (2013), whereas Chess about 35 and Go roughly 180. Moreover, the player typically has to manage tens of units simultaneously,

• real-time—actions can be performed simultaneously, at any time, as opposed to turn-based games where players must take actions one at a time, in turns. Typically, actions in strategy games are divided into macro-management (e.g., strategic decisions) and micro-management (e.g., units’ movement). The latter are taken with a higher frequency,

• partial observability—the map is covered with fog of war, meaning that some parts are hidden from the players. Handling partial observability in RTS games is presented in Uriarte and Ontañón (2017). Given the current observation and past knowledge the method estimates the most probable game state. This state is called believe state and is then sampled while performing MCTS.

2 Classic MCTS

2.1 Basic theory

MCTS is an iterative algorithm that searches the state space and builds statistical evidence about the decisions available in particular states. As mentioned in the “Introduction”, formally, MCTS is applicable to MDPs. MDP is a process modelled as a tuple $(S, A_S, P_a, R_a)$, where:

• S—is a set of states that are possible in an environment (state space). A specific state $s_0 \in S$ is distinguished as the initial state.

• $A_s$—denotes a set of actions available to perform in state s. The subscript S can be omitted if all actions are always available in the given environment.

• $P_{a}(s,s')$—is the transition function modelled by a probability that action a performed in state s will lead to state $s'$. In deterministic games, the probability is equal to 1 if the action in state s leads to $s'$, whereas 0 if it does not.

• $R_a(s)$—is the immediate reward (payoff) for reaching state s by action a. In Markov games, where states incorporate all the necessary information (that summarizes the history), the action component can be omitted.

This formalism enables to model a simulated environment with sequential decisions to make in order to receive a reward after certain sequence of actions. The next state of the environment can be derived only using the current state and the performed action. We will use the game-related term action and a broader one—decision—interchangeably. The same applies for the term game and problem.

In non-trivial problems, the state space (e.g. a game tree) cannot be fully searched. In practical applications, MCTS is allotted some computational budget which can be either specified by the number of iterations or the time available for making a decision. MCTS is an anytime algorithm. This property means that it can be stopped at anytime and provide the currently best action (decision) using Eq. 1:

$$\begin{aligned} a^{*} = \arg \max _{a \in A(s)} Q(s,a), \end{aligned}$$

(1)

where A(s) is a set of actions available in state s, in which decision is to be made and Q(s, a) denotes the empirical average result of playing action a in state s. Naturally, the more the iterations, the more confident the statistics are and the more likely it is that the recommended best action is indeed the optimal one.

Fig. 1

Monte Carlo Tree Search phases

Each iteration consists of four phases as depicted in Fig. 1:

• (1) Selection—the algorithm searches the portion of the tree that has already been represented in the memory. Selection always starts from the root node and, at each level, selects the next node according to the selection policy also referred to as tree policy. This phase terminates when a leaf node is visited and the next node is either not represented in the memory yet or a terminal state of the game (problem) has been reached.

• (2) Expansion—unless selection reached a terminal state, expansion adds at least one new child node to the tree represented in memory. The new node corresponds to a state that is reached by performing the last action in the selection phase. When expansion reaches the terminal state, which is extremely rare, then the current iteration skips directly to backpropagation.

• (3) Simulation—performs a complete random simulation of the game/problem, i.e. reaching a terminal state and fetches the payoffs. This is the “Monte Carlo” part of the algorithm.

• (4) Backpropagation—propagates the payoffs (scores), for all modelled agents in the game, back to all nodes along the path from the last visited node in the tree (the leaf one) to the root. The statistics are updated.

2.2 Tree policy

The aim of the selection policy is to maintain a proper balance between the exploration (of not well-tested actions) and exploitation (of the best actions identified so far). The most common algorithm, which has become de facto the enabler of the method is called Upper Confidence Bounds applied for Trees (UCT) introduced by Kocsis and Szepesvári (2006). First, it advises to check each action once and then according to the following formula:

$$\begin{aligned} a^{*} = \arg \max _{a \in A(s)}\left\{ Q(s,a)+C\sqrt{\frac{ln\left[ N(s) \right] }{N(s,a)}} \right\} , \end{aligned}$$

(2)

where A(s) is a set of actions available in state s, Q(s, a) denotes the average result of playing action a in state s in the simulation performed so far, N(s) is a number of times state s has been visited in previous iterations and N(s, a)—a number of times action a has been sampled in state s. Constant C controls the balance between exploration and exploitation. The usually recommended value of C to test first is $\sqrt{2}$, assuming that Q values are normalized to the [0, 1] interval, but it is a game-dependent parameter, in general.

Due to the UCT formula (c.f. 2) and random sampling, MCTS searches the game tree in an asymmetric fashion in contrast to traditional tree search methods such as minimax. The promising lines of play are searched more thoroughly. Figure 2 illustrates the type of asymmetrical MCTS search.

Fig. 2

The root denotes the starting state. The average scores [Q(s, a)] of actions leading to next states are shown inside the circles. The bars denote how many simulations started from a particular action. This is an abstract example—not taken from any particular game

UCT is such a dominant formula for the exploration vs. exploitation trade-off in MCTS, that the approach is often called MCTS/UCT. However, there have been alternatives proposed. One of them is called UCB1-Tuned, which is presented in Eq. 3 from the work by Gelly and Wang (2006). The only new symbol here compared to Eq. 2 is $\sigma _a$ which denotes the variance of the score of action a.

$$\begin{aligned} a^{*} = \arg \max _{a \in A(s)}\left\{ Q(s,a)+C\sqrt{\frac{\ln \left[ N(s) \right] }{N(s,a)} \min \left( \frac{1}{4},\sigma _a+\frac{2*\ln \left[ N(s) \right] }{N(s,a)}\right) } \right\} . \end{aligned}$$

(3)

The Exploration–Exploitation with Exponential weights (EXP3) proposed by Auer et al. (2002) is another established tree policy algorithm. In contrast to UCT, it makes no assumption about the reward distribution. For a fixed horizon, EXP3 provides an expected cumulative regret bound against the best single-arm strategy. It is often used in combination with UCT in games with partial observability and/or simultaneous actions.

Thomson sampling is a different than UCT, heuristic approach to the exploration vs. exploitation problem. It preceded the MCTS algorithm (Thompson 1933) but has found its use as a tree policy as shown by Bai et al. (2013). Thomson sampling is a model that incorporates Bayesian concepts:

• parameterized likelihood function,

• prior distribution,

• set of past rewards,

• posterior distribution.

The model is updated as more information comes in. The expected reward of given action is sampled using the posterior distribution.

Typically, in the expansion phase, only one node is added to the tree which is a practical trade-off based on empirical experiments in lots of games. Adding more nodes visited in simulations would slow down the algorithm and increase its memory usage. The further down the road in a simulation a state is visited, the less likely it is for the actual game to reach it. However, if the states (and therefore potential nodes) belonged to the optimal path, it could be beneficial to add them. To make more use of simulations, Gelly and Silver (2011) proposes a method called Rapid Value Action Estimation (RAVE). In RAVE, an additional score is maintained called $Q_{RAVE}$, which is updated for each action performed in the simulation phase and not only for those chosen in the selection steps. The idea is to minimize the effect of the cold start, because especially at the beginning when there are not many samples, the MCTS algorithm behaves chaotically. One of the standard procedures is to linearly weight the RAVE evaluation with the regular average score [c.f. Q(s, a) in Eq. 2] and reduce the impact of RAVE when more simulations have been performed. The new formula is shown in Eq. 4:

$$\begin{aligned} \sqrt{\frac{k}{3*N(s)+k}} * Q_{RAVE}(s,a) + \left( 1-\sqrt{\frac{k}{3*N(s)+k}} \right) *Q(s,a), \end{aligned}$$

(4)

where k is the so-called RAVE equivalence constant.

There are a few more algorithms that modify or build upon the UCT formula such as Move-Average Sampling Technique (MAST) or Predicate-Average Sampling Technique (PAST). We recommend papers by Finnsson and Björnsson (2010, 2011) for details.

2.3 Transposition tables

The RAVE algorithm is often used with Transposition Tables (Kishimoto and Schaeffer 2002). They have been originally proposed as an enhancement to the alpha–beta algorithm, which reduces size of minmax trees. As shown in paper by Kishimoto and Schaeffer (2002), within the same computational budget, the enhanced algorithm significantly outperforms the basic one without the transposition tables. The term “transpositions” refers to two or more states in the game that can be achieved in different ways. Formally speaking, let $S_i$, $S_j$, $S_k$ be some arbitrary states in the game. If there exists a sequence of actions $(a_i,\ldots ,a_k)$, that defines the path from state $S_i$ to $S_k$, and a sequence of actions $(a_j,\ldots ,a_k)$ which differs from $(a_i,\ldots ,a_k)$ on at least one position, and transforms the state from $S_j$ to $S_k$, then $S_k$ represents transposed states. For simpler management, the MCTS tree is often modeled in such a way, that each unique sequence of actions leads to a state with a unique node in the tree. This may result in a duplication of nodes, even for indistinguishable states. However, one of the benefits of such a duplication, is the fact that, whenever an actual action in the game is performed, the tree can be safely pruned into the sub-tree defined by the state the action lead to. All nodes that are either above the current one or on an alternative branch cannot be visited anymore, so there is no need to store them anymore. The problem is more complicated when transpositions are taken into account, so there is one-to-one mapping between states and nodes. In such a case, the structure is no longer a tree per se, but a directed acyclic graph (DAG). When an action is played in the game, it is non-trivial to decide which nodes can be deallocated and which cannot because they might be visited again. In general, it would require a prediction model that can decide whether a particular state is possible to be encountered again. Storing all nodes, without deallocations, is detrimental not only for performance but mostly for the memory usage, which can go too high very quickly.

2.4 History heuristic

History Heuristic, pioneered by the work of Schaeffer (1989), is an enhancement to game-tree search that dates back to 1989. It dwells on the assumption that historically good actions are likely to be good again regardless of the state they were performed in. Therefore, additional statistics Q(a)—the average payoff for action a and N(a)—the number of times action a has been chosen, are stored globally. There are a few ways of how these statistics can be used. The most common approach is to use the Q(a) score to bias simulations. In a simulation, instead of uniform random choice, the historically best action is chosen with $\epsilon$ probability, whereas with $1-\epsilon$ probability, the default random choice is performed. This method is called $\epsilon -greedy$. An alternative approach is to use the roulette sampling based on Q(a) or a particular distribution such as Boltzmann Distribution, which is the choice in the article from Finnsson and Björnsson (2010). A similar enhancement to History Heuristic is called Last-Good Reply Policy (LGRP), which is presented in Tak et al. (2012). The idea is to detect countermoves to particular actions and store them globally. For each action, the best reply to it is stored. The authors use the LGRP to rank the unexplored actions in the selection step and the simulation step. In the same article (Tak et al. 2012), the idea is further expanded to the so-called N-grams Selection Technique (NST). In NST, the statistics are stored not just for actions but for sequences of actions of length N.

2.5 Section conclusion

There is no “free lunch” (Wolpert and Macready 1997), which in the area of MCTS means that most modifications of the base algorithm introduce a bias. Such a bias may increase the performance in a certain problem but decrease in others. There are, however, certain universal enhancements that can be applied such as parallelization. Opening books, discussed e.g. by Gaudel et al. (2010), are another example of a universally strong enhancement, but they are not always available. MCTS with opening books can be considered a meta-approach, because the first actions are chosen from the opening book and when, at some point in the game, the available actions are not present in the opening book, the remaining ones are chosen according to the MCTS algorithm. MCTS with relaxed time constraints, e.g. run for days, can even be used to generate the opening books for subsequent runs of the MCTS-based agents (Chaslot et al. 2009). Similarly, endgame books can be integrated with MCTS, but this is rarely applied as MCTS is a very strong end-game player and far less efficient at the beginning, where the whole game tree is to be searched.

In this section, we presented the MCTS algorithm as it was originally defined and discussed several standard modifications that were proposed before 2012. In the next sections, we focus exclusively on modifications proposed after 2012.

3 Games with perfect information

Almost any modern game features a huge branching factor, which poses a great challenge for search-based algorithms. In recent years, several MCTS modifications intended to tackle this problem such as action abstractions or rapid value estimations have been developed. Most of the modifications can be divided into the three categories:

• Action Reduction—limiting the number of available actions,

• Early Termination—ending the simulation phase (Fig. 1) prematurely,

• UCT alternatives—other approaches to MCTS selection phase.

In the following subsections, those three types of extension will be discussed with several examples. Another well-known solution, though recently less common, is to simplify the model of the game. One can construct a prediction model capable of anticipating promising moves in a given situation of the game. Graf and Platzner (2014) introduce a prediction model for the game of Go. The model features patterns extracted from common fate graph—a graph representation of the board (Graepel et al. 2001). These patterns are updated incrementally during the game both during the tree search and the playouts. This process allows reduction of the branching factor without affecting the prediction performance. Model simplification appears to be indispensable in video games. It allowed the use of the MCTS technique for campaign bots in a very complex game—Total War: Rome II as discussed in Preuss and Risi (2020). Swiechowski and Slęzak (2018) propose the so-called granular games as a framework for simplification of complex video games in order to apply MCTS to them.

3.1 Action reduction

In the expansion phase, the MCTS algorithm adds new nodes into the tree for states resulting after performing an action in the game. For many problems, the number of possible actions can be too high. In such a case, the tree is growing sideways, with limited chances for the in-depth inspection of promising branches. The aim of the action reduction methods is to limit this effect by eliminating some of the actions.

Sephton et al. (2014) apply heuristic move pruning to a strategy card game Lords of War.² The heuristic knowledge is split into two categories: measuring the fitness of a state card-based statistics and evaluating card positions based on heat maps. The former proves more efficient, but the heat maps make use of state extrapolation.

Justesen et al. (2014) extend the UCT Considering Durations (UCTCD) algorithm for combats in Starcraft³ (Churchill and Buro 2013). Unlike UCTCD, where the algorithm searches for unit actions, the proposed method considers sequences of scripts assigned to individual units. The approach is elaborated even further—the experiments prove that the scripts appointed to groups of units are more effective in terms of branching factor reduction.

Similarly to the scripts, Option MCTS (O-MCTS) is presented by De Waard et al. (2016). The option is a predefined method of reaching a particular subgoal, e.g. conquering an area in an RTS game. In order to do this, options that contain several consecutive actions within the tree—Fig. 3 were introduced. O-MCTS selects options instead of actions as in vanilla MCTS. Once the selected option is finished (the subgoal has been reached), the expansion phase (Fig. 1) is conducted. This process allows a more in-depth search in the same amount of time and diminishes the branching factor. The paper includes an empirical optimisation of the basic parameters of the algorithm such as discount factor, maximum action time, maximum search depth and UCT constant C.

Fig. 3

Representation of options within a tree

Another approach to reducing the branching factor is imposing constraints. Constraints determine situations to be avoided, i.e. actions which result in a defeat, whereas options lead to a specific sub-goal. Subramanian et al. (2016) propose a new technique of applying options and constraints to the search policy called Policy-Guided Sparse Sampling (PGSS). PGSS uses constraints for the possibility of pruning a node and options to bias the search towards the desired trajectories. Endowing MCTS with both techniques provides more accurate value estimation. Different action abstraction methods can be found in Gabor et al. (2019), Moraes et al. (2018), and Uriarte and Ontanón (2014).

Real Time Strategy games are challenging for MCTS due to a huge branching factor. Ontanón (2016) proposes Informed MCTS (IMCTS)—an algorithm dedicated for games with large decision spaces. IMCTS aggregates information of a game using two novel probability distribution models: Calibrated Naive Bayes Model (CNB) and Action-Type Independence Model (AIM). CNB is a Naive Bayes classifier with a simplified calibration of posterior probability. AIM derives the distribution only for legal actions in the current game state. The models are incorporated into the tree policy by computing the prior probability distributions in the exploration phase. The models are trained on the data from bot duels in an RTS testbed. The experiments show that the models outperform other MCTS approaches in the RTS domain.

Fig. 4

A sketch of tree pruning at depth $d=2$ for BMCTS: (a) a beam of the three most visited nodes ($W=3$) is selected, and (b) the rest is pruned

One of the basic techniques of heuristic action reduction is Beam Search (Lowerre 1976). It determines the most promising states using an estimated heuristic. These states are then considered in a further expansion phase, whereas the other ones get permanently pruned. Baier and Winands (2012) combine Beam Search with MCTS (BMCTS). BMCTS features two parameters: beam width W and tree depth d. They describe how many of the most promising nodes should be selected at an arbitrary depth—Fig. 4. The nodes outside the beam are pruned. Manual experimentation is used for finding optimal values of parameters: C—exploration coefficient, L—simulation limit, W—beam width. Different variants of MCTS with a variable depth can be found in Pepels and Winands (2012), Soemers et al. (2016), Zhou et al. (2018).

Ba et al. (2019) present Variable Simulation Periods (VSP-MCTS). The method extends the decision variables with the simulation time. In VSP-MCTS actions are selected with respect to the possible simulation time before the next selection phase. The available set of action is initially limited to the most promising ones and gradually progressively expanded (unpruned). To achieve this, the authors combine progressive widening and Hierarchical optimistic optimization applied to tree (HOOT). This approach supports the balance between the frequent action choice and the time dedicated to simulations.

Progressive Bias (Chaslot et al. 2008b) is another method of heuristic action reduction. Progressive bias extends the UCB function with a heuristic value, which is reduced as the number of node visits increases. Gedda et al. (2018) propose a progressive win bias:

$$\begin{aligned} P = W \frac{H_i}{T_i (1 - \bar{X_i}) + 1}, \end{aligned}$$

(5)

where $H_i$ is the heuristic value, $\bar{X_i}$ is the average reward for the node, $T_i$ is the number of node visits and W is a positive constant controlling the impact of the bias. Unlike the basic approach, in this formula the heuristic value depends on the number of losses. Other extensions to UCB can be found in Liu and Tsuruoka (2015), Mandai and Kaneko (2016), Tak et al. (2014) and Yee et al. (2016). Perick et al. (2012) compare different UCB selection policies. Ikeda and Viennot (2013a) involve Bradley–Terry Model of static knowledge (Coulom 2007) to bias UCB. A Bias in MCTS can affect the performance in terms of suboptimal moves (Imagawa and Kaneko 2015).

Demediuk et al. (2017) introduce a set of Outcome-Sensitive Action Selection formulas to create an MCTS agent in a fighting game. The equations evaluate actions with a score function, which in this case is the difference between the player’s and the enemy agent’s health. The action with the score closest to zero is selected.

A large branching factor can be limited by manipulation of the exploration term in the tree policy function (UCT, Eq. 2), which is covered in Sect. 3.2.

3.2 UCT alternatives

In the vanilla MCTS, every state or state–action pair is estimated separately according to the Eq. 2 (on p. 10). Some games like the game of Go feature actions which do not strictly depend on the state in which they were played.. To handle this the all-moves-as-first (AMAF) heuristic has been developed (Brügmann 1993). It evaluates each move a by the total reward $\bar{W_a}$ divided by the number of simulations in which a was ever played $\bar{N_a}$.

Rapid Value Estimation, which has already been discussed in Sect. 2.2, is a generalized idea of AMAF to search trees. It shares action values among subtrees. The value estimation of action a is performed with respect to all the simulations involving a at the subtrees. This process provides more data for each action in comparison to vanilla MCTS, although the additional data may be needless in some cases. In RTS games, any change in positions of nearby units can mislead the action value. The MCTS–RAVE method handles this observation through interpolating the unbiased Monte Carlo and AMAF value estimate. Thus, the score of state–action pair is as follows (Gelly et al. 2012):

$$\begin{aligned} {{\bar{Z}}}(s,a) = (1 - \beta (s,a)) \frac{W(s,a)}{N(s,a)} + \beta (s,a) \frac{{{\bar{W}}}(s,a)}{{{\bar{N}}}(s,a)}, \end{aligned}$$

(6)

where ${{\bar{N}}}(s,a)$ is the AMAF visit count, ${{\bar{W}}}(s,a)$ is the corresponding total reward and $\beta (s,a)$ is a weighting parameter that decays from 1 to 0 as the visit count $N(s,a)$ increases. In recent years, several RAVE improvements within the GGP domain have been proposed—Sironi and Winands (2016) and Cazenave (2015). Sarratt et al. (2014) and Sironi and Winands (2019) discuss effects of different parameterization policies on convergence.

A case analysis of the classical position in the Go game (Browne 2012) shows that the inclusion of domain knowledge in the form of heavy playouts (see also Sect. 4.3) dramatically improves the performance of MCTS, even in the simplest flat version. Heavy playouts reduce the risk of convergence to a wrong move. Exploration constant has been found empirically to give the best convergence. Incorporating domain-independent RAVE technique can actually have a negative effect on performance, if the information taken from playouts is unreliable.

The non-uniformity of tree shape is another serious challenge for the MCTS method. The Maximum Frequency method (Imagawa and Kaneko 2015) adjusts a threshold used in recalculation of playout scores into wins or losses. The method is based on maximizing the difference between the expected reward of the optimal move and the remaining ones. The Maximum Frequency method observes the histogram of the results and returns the best threshold based on it. Algorithm parameters are manually adjusted in order to minimize failure rate. The proposed solution has been verified on both random and non-random trees.

The playout policy can be changed during a tree search. Graf and Platzner (2016, 2015) show that updating policy parameters (see Algorithm 1) increases the strength of the algorithm. An adaptive improvement of the policy is achieved by the gradient RL technique.

Despite its effectiveness in GGP, there are some domains in which the UCT selection strategy does not work nearly as well. Chess-like games featuring many tactical traps pose a challenging problem for MCTS. One of the tactical issues are optimistic actions—initially promising moves (even leading to a win), but easy to refute in practice (Gudmundsson and Björnsson 2013). Until MCTS determines the true refutation, the move path leading to an optimistic action is scored favorably. To diminish this positive feedback UCT (see Eq. 2) has been extended with sufficiency threshold $\alpha$ (Gudmundsson and Björnsson 2013). It replaces constant C with the following function:

$$\begin{aligned} {{\hat{C}}} = \left\{ \ \begin{array}{ll} C &{} \hbox { when all}\ Q(s,a) \le \alpha ,\\ 0 &{} \hbox { when any}\ Q(s,a) > \alpha . \end{array} \right. \end{aligned}$$

(7)

When the state–action estimate $Q(s,a)$ is sufficiently good (the value exceeds $\alpha$) then the exploration is discounted; otherwise the standard UCT strategy is applied. This approach allows rearrangement of the order of moves so that occasional fluctuations of the estimated value can be taken into account.

Hybrid MCTS-minimax algorithms (Baier and Winands 2014; Baier and Mark 2013) have also been proposed for handling domains with a large number of tactical traps. In this context, the basic drawback of vanilla MCTS is the averaging of the sampled outcomes. It results in missing crucial moves and underestimation of significant terminal states. To overcome this, the authors employ shallow-depth minimax search at different MCTS phases (Sect. 2). Since the minimax search is performed without any evaluation functions, the method does not require domain knowledge and finds its application in any game. Parameters tuning covers exploration constant, minimax depth and visit counter limit.

3.3 Early termination

MCTS in its classical form evaluates game states through simulating random playouts i.e. complete play-throughs to the terminal states. In dynamic domains, where actions are taken in a short period of time, such outcome sampling appears to be a disadvantage of vanilla MCTS. One of the alternatives is to early terminate the playouts—a node is sampled only to an arbitrary depth and then the state is evaluated similarly to the minimax search. One of the key issues is to determine the cut-off depth. Earlier terminations save the simulation time although they result in evaluation uncertainty. On the other hand, later terminations cause the algorithm to behave more like vanilla MCTS.

Lorentz (2016) finds the optimal termination point for diverse evaluation functions in the context of three different board games. The results prove that even a weak function can compare favorably to a long random playout. The evaluation functions can also augment the quality of simulations (Lanctot et al. 2014). The proposed method replaces $Q(s,a)$ in the classic UCB selection—Eq. 2—with the following formula:

$$\begin{aligned} {{\hat{Q}}}(s,a) = (1 - \alpha ) \frac{r_{s,a}^{\tau }}{n_{s,a}} + \alpha v_{s,a}^{\tau }, \end{aligned}$$

(8)

where $r_{s,a}^{\tau }$ is the cumulative reward of the state–action pair with respect to the player $\tau$, $n_{s,a}$ is the visit count, $v_{s,a}^{\tau }$ is the implicit minimax evaluation for that player and $alpha$—weighting parameter. The minimax evaluations are computed implicitly along with standard updates to the tree nodes and maintained separately from the playout outcomes. Hierarchical elimination tournament with 200 games has been used to tune the parameters of the algorithm.

The generation of evaluation functions is a challenging research especially in GGP (see Sect. 1.3). Waledzik and Mańdziuk (2011, 2014) construct a state evaluation function in GGP automatically based solely on the game rules description, and apply this function to guide the UCT search process in their Guided UCT method.

Moreover, evaluation functions can be approximated with neural networks (Wu et al. 2018). Goodman (2019) trains an evaluation network on the data from offline MCTS, which allows omission of the simulation step during the tree search.

3.4 Opponent modelling

Opponent modelling is a challenge that appears in multi-player games. The more information we possess or can infer about the opponent, the better simulation model of their actions we can build. Opponent modelling is a complex topic that is related to games, game theory and psychology. The model of the opponent can be independent of the algorithm an AI agent uses. In this section, we will only scratch a surface and show a few examples of how opponent modelling is incorporated into the MCTS algorithm in the context of games with perfect information.

The paper by Baier and Kaisers (2020) emphasizes the idea of “focusing on yourself” during MCTS simulations. The authors argue that in multi-player games the agent’s own actions have often a significantly greater impact on the final score than appropriate opponent prediction. Naturally, there are exceptions to this, such as complex tactical situations. The proposed approach considers abstractions over opponent moves (OMA) that only consider the history (past moves) of the main agent’s moves. The abstracted moves share the same estimates that are combined with the UCT formula in various ways. The authors report a better win-rate than vanilla MCTS in six perfect-information, turn-taking, board games with 3+ players.

Fighting games pose a special challenge, since they require a short response time. However, players often repeat the same patterns, e.g., sequences of punches and kicks. An algorithm designed for such a game is described by Kim and Kim (2017). An action table is used to describe the opponent’s playing patterns. A default action table has three columns depending on the distance to the opponent. Each column corresponds to a particular distance range. After each round the action table is updated based on the observed action frequency.

Opponent modelling is also present in games such as Pacman. In a paper by Nguyen and Thawonmas (2012), the main enhancement was related to the prediction of the opponent’s moves, which reduced the number of states analyzed. In the backpropagation phase, the node reward scheme combines the final playout score, but also the simulation time. The playouts are not completely random, the space of the analyzed moves is limited by heuristic rules.

4 Games with imperfect information

The success of MCTS applications to games with perfect information, such as Chess and Go, motivated the researchers to apply it to other games such as card games, real-time strategy (RTS) and other video games. The main feature, which challenges the effective applicability of tree search algorithms is imperfect information. For instance, in RTS games the “fog of war” often appears, covering the map including the enemy’s units.

Imperfect information can increase the branching factor significantly, as every hidden element results in a probability distribution of possible game states. Those states should be included within the tree in the form of additional nodes.

In order to handle imperfect information there have been developed several improvements of the methods such as Perfect Information MCTS (PIMC) (see Sect. 4.1) and Information Set MCTS (ISMCTS) (see Sect. 4.2).

Rules of the game and human expert knowledge can be used to increase the reliability of playouts (see Sect. 4.3 for heavy playouts), or to optimize the tree development process (see Sect. 4.4 for policy update). The highest quality results can be achieved by combining several methods. Papers of this type usually concern algorithms created for competitions and is described in Sect. 4.5.

The last issue related to imperfect information discussed in this section is especially relevant for games and concerns modelling the opponent (see Sect. 4.6).

4.1 Determinization

One method of solving the problem of randomness in games is determinization, which involves sampling from the space of possible states and forming games of perfect information for each such sampling, correspondingly. The sampling requires setting a specific instance (value) for each unknown feature, thus determining it, so it is no longer unknown. For instance, determinization may involve guessing the cards an opponent might have in their hand in a card game (see Fig. 5).

Fig. 5

Determinization method. The dice and the cards in the opponent’s hand are sampled in order to form possible subsets. Then each subset rolls out a new subtree

Perfect Information MCTS (PIMC) assumes that all hidden information is determined and then the game is treated as a perfect information one with respect to a certain assumed state of the world. PIMC faces three main challenges (Cowling et al. 2012a):

• strategy fusion—since a given imperfect information state can contain tree nodes with different strategies, a (deterministic) search process may make different decisions in that state depending on particular determinization,

• non-locality—optimal payoffs are not recursively defined over subgames,

• sharing computational budget across multiple trees.

Cowling et al. (2012b) apply this method to deal with imperfect information in the game “Magic: The Gathering” (MtG). MtG is a two-person, collectible card game, in which players draw cards for their starting hand. The cards are not visible to the opponent until they are placed on the table, therefore MtG features partially observable states and hidden information. To approach this game with MCTS, Cowling et al. (2012b) propose multiple perfect information trees. This method considerably increases the branching factor, so a few enhancements are employed such as move pruning with domain knowledge and binary representation of the trees. The improvements reduce CPU time per move, rendering the computational budget more manageable.

4.2 Information sets

The concept of Information Sets has been introduced by Frank and Basin (1998). It is illustrated in Fig. 6. The game starts with a chance node, which has three possible moves (circle) for the first player. The moves of the second player are split into two information sets: red (1) and blue (2). Nodes in each information set are treated as equivalent since the player is only aware of the previous moves, not of the moves below the chance node.

Fig. 6

Two information sets:red (with 1 inside the circles) and blue (with 2 inside the circles). Each set contains nodes indistinguishable from a player’s perspective. (Color figure online)

Cowling et al. (2012a) propose Information Set MCTS (ISMCTS) to handle the determinization problems: strategy fusion and suboptimal computational budget allocation. The information set is an abstract group of states, which are indistinguishable from a player’s point of view (Wang et al. 2015). Instead of a single state, each node of ISMCTS comprises an information set that allows the player to act upon their own observations. Information sets bring all the statistics about moves in one tree, providing the use of the computational budget in the optimal way. Moreover, applying information sets reduces the effects of strategy fusion, as the search is able to exploit moves that are beneficial in many states of the same information set. For an extension of ISMCTS that incorporates advanced opponent modelling, please refer to Sect. 4.6.

In games where non-locality is a key factor, Lisỳ et al. (2015) introduce Online Outcome Sampling (OOS). The algorithm extends Monte Carlo Counterfactual Regret Minimization with two features: building its tree incrementally and targeting samples to more situation relevant parts of the game tree. Performance evaluated on both OOS and ISMCTS shows that only OOS consistently converges close to Nash Equilibrium, and it does so by adapting an offline equilibrium computation to a limited time. This is achieved using the information set about the current position in the game. To tune efficiency of the algorithm sampling technique is used for strategy probabilities parameters.

Furtak and Buro (2013) introduce Recursive Imperfect Information Monte Carlo (IIMCTS) which is used for playouts with a fixed maximum recursive depth. One of the differences to Cowling et al. (2012a) is that ISMCTS implicitly leaks game state information by restricting the playouts to be consistent with the player’s hidden/private information (e.g. cards in the player’s hand in Poker). IIMCTS prevents the leakage by not adapting the players’ playout policies across playouts.

Moreover, the move computation in IIMCTS allows use of the inference mechanism at any node during the lookup, while Information Set MCTS considers nodes only at the root or sets uniform distributions of the hidden state variables.

Another collectible card game with imperfect information in which MCTS was used is Pokemon (Ihara et al. 2018). In this research, authors compare the effectiveness of determinization versus information sets within MCTS. The results indicate a significant advantage of information sets as ISMCTS reduces the effects of strategy fusion and allocates computing resources more effectively. A similar approach has been presented for Scopone game (Di Palma and Lanzi 2018).

Sephton et al. (2015) propose the RobustRoulette extension for the selection mechanism in ISMCTS, which averages across many possible states and has a positive effect on the playing strength.

Uriarte and Ontañón (2017) use a similar method to generate a single believe state—an estimation of the most probable game state. The estimation derives from the current game state, which is not fully observable to the player. This situation is often encountered in Real Time Strategy games (see Sect. 1.3), where enemy units’ positions are unknown (covered with “fog of war”).

The concept of Information Sets, which is inherent in MCTS application to solving imperfect information problems, has been recently utilized also in domains other than games, e.g. in Security (Karwowski and Mańdziuk 2015, 2019a, b, 2020). A more detailed discussion is presented in Sect. 7.2.

4.3 Heavy playouts

Heavy playouts is a term defined in the classical paper by Drake and Uurtamo (2007). Authors define two ways of adding domain-specific knowledge into vanilla MCTS: move ordering and heavy playouts. The first one concerns the issue of creating new nodes during the tree’s expansion phase, whereas the second one uses a heuristic to bias the “random” moves during the rollout.

The implementation of heavy playouts results in more realistic games in the simulation phase. However, it should be remembered that playouts need some degree of randomness to efficiently explore the solution space. Experiments in a card game Magic: the Gathering show the importance of the rollout strategy (Cowling et al. 2012b). A deterministic expert-rule-based rollout strategy provides an inferior performance to utilizing the reduced-rule-based player that incorporates some randomness within its decisions. This is true even though the expert-rule-based algorithm used as a standalone player is more efficient than the reduced-rule-based algorithm. On the other hand, relying on completely random playouts gives poor performance.

GGP defines a special challenge for the AI agents since they are expected to be able to play games with very different rules (see Sect. 1.3). On the one hand, a fully random playout strategy is universal although, it avoids the use of domain knowledge gained from the past simulations. On the other hand, too much heuristics can lead to omitting a potentially important group of moves. Therefore, one has to find a balance between these two extremes. Świechowski and Mańdziuk (2014) present a method based on switching between six different strategies (from simple random or History Heuristic to more advanced ones). Their solution has been implemented in the MINI-Player agent, which reached the final round in the 2012 GGP Competition. The method was further extended towards handling single-player games more efficiently (Świechowski et al. 2016).

Genetic programming (GP) has been successfully used to enhance the default MCTS policy in Pac-Man (Alhejali and Lucas 2013). The GP system operated using seven categories of elements such as IF ELSE statements, numerical operators and comparison operators. Based on these components, a tree policy function was evolved. After sweeping for optimal value of nine different GP and MCTS algorithm parameters, it allowed achievement of a 18% greater average score than the random default policy. Moreover, the constructed function did not introduce a significant computational overhead.

Experiment with different ‘weights’ of heavy playouts are described by Godlewski and Sawicki (2021). Increasing the amount of expert knowledge added to the playout function generally improves the performance of the AI agent. However, for multistage card games like “The Lord of the Rings”, each stage might have a different value of optimal playout weight.

4.4 Policy update

The previous subsection concerned the introduction of knowledge through heavy playouts. In this section, the methods of modification of the tree building policy is presented.

The classical RAVE approach (see Sect. 2.2) has been extended by Kao et al. (2013). They introduce the RIDE method (Rapid Incentive Difference Evaluation) where the default MCTS policy is updated by using differences (9) between action values for the same state s.

$$\begin{aligned} D^{RIDE}(a,b)=E\{Q(s,a)-Q(s,b)|s \in S \}. \end{aligned}$$

(9)

The RAVE and RIDE approaches differ in their sampling strategy. RAVE applies an independent sampling strategy, whereas RIDE applies a pairwise sampling strategy.

The playout policy can be updated by moves, but also based on features of the moves. In the solution presented by Cazenave (2016), the tree is developed exactly as UCT does. However, the playouts have weights for each possible move and choose randomly, proportionally to the exponential of the weight.

Heuristics can be added at the tree development phase or at the playouts phase. Naturally, it is also possible to combine these two approaches (Trutman and Schiffel 2015). The authors propose three schemes (playout heuristics, tree heuristics, combined heuristics) in the GGP framework. They have been compared with MAST (Finnsson and Björnsson 2008), which is used to bias random playouts, and the RAVE technique for trees. None of the proposed heuristics is universally better than the others and their performance depends on a particular type of game.

The default policy update using heuristics is one of the key enhancements in MCTS experiments for Hearthstone. Since the release in 2014, every year the game has attracted thousands of players, quickly becoming the most popular card game, exceeding 100 million of users worldwide in 2018.⁴

The game state includes imperfect information—each player’s hand is hidden for the opponent. Moreover, the game-play imposes random events and complex decisions. Because of this, state evaluation for this game has always been under research. Santos et al. (2017) propose heuristic functions for evaluating subsequent states based on hand-picked features. In addition, they enhance the state search with a database of cards, which contains cards already played by the opponent. The database is used for prediction of cards in the next turn.

State evaluation with a value network (Świechowski et al. 2018) is another approach. The authors employ the network to predict the winner in a given state. To transform the state vector as the input of the network, they use skip-gram model instead of hand-picking. A value network from Zhang and Buro (2017) maps a game state to a vector of probabilities for a card to be played in the next turn. Training data for both networks is generated by bot players.

4.5 Master combination

A special category of research papers are those describing methods developed for game AI competitions (Świechowski 2020). Each year, there are many events in which AI players compete against each other. One of the largest ones among academic community members is the IEEE Conference of Games (CoG) (formerly Computational Intelligence and Games—CIG) with over ten categories for popular games such as Angry Birds, Hearthstone and StarCraft; and more general disciplines (General Video Game Playing—GVGP, MicroRTS, Strategy Card Game) e.g. IEEE-CoG (2020). Open competitions stimulate development of new methods, but particularly good results are achieved by combining multiple methods masterfully. Strong problem-specific enhancements can address different phases of the MCTS algorithm.

One example of such a master combination is the algorithm developed to win the Pac-Man versus Ghost Team competition (Nguyen and Thawonmas 2012). The team of ghosts was controlled by MCTS to catch Pac-Man. It is intrinsically interesting that the winner on the Pac-Man side was also an agent using the MCTS algorithm. The set of several improvements was described by Pepels and Winands (2012) and Pepels et al. (2014). Most of them refer to heuristic changes in the policy:

• a tree with variable length edges (taking the distance between decision points into account),

• saving the history of good decisions to improve the playout policy,

• changes of policy (different modes to eliminate some obviously wrong decisions).

Fig. 7

Comparison of MCTS enhancements for Mario arcade game (Vanilla MCTS—yellow line, Macro Actions—red line, Best combination—blue line). Figure reproduced from Jacobsen et al. (2014). (Color figure online)

Another popular challenge is the Mario arcade game (see Fig. 7). Jacobsen et al. (2014) describe a combination of several improvements to vanilla MCTS that shows excellent results. However, most of them could be classified as an incorporation of domain knowledge into the general algorithm: MixMax (high rewards for good actions), Macro Actions (avoid monsters in a sequence of moves), Partial Expansion (eliminate obvious choices), Hole Detection (extra heuristic to jump over a deadly trap). The paper contains explanation of optimization process for playout depth and exploration constant.

GVGP is a class of AI competitions where agents do not know in advance what game will be played (see Sect. 1.3). The vanilla MCTS is an algorithm which does not rely on domain-specific heuristics, so it is naturally a good candidate for such problems. In the first GVG-AI competition at CIG 2014 (Preuss and Gunter 2015), the vanilla MCTS agent surprisingly came in 3rd place, achieving a win-rate of about 32%. These impressive results attracted attention to the MCTS method, however, Nelson (2016) found out that simply increasing the playout budget is not enough to significantly improve the win-rate.

Soemers et al. (2016) describe several enhancements of the method, such as N-gram Selection Technique (Tak et al. 2012)—a playout policy update introducing a bias in the steps towards playing sequences of actions that performed well in earlier simulations. The idea of Loss Avoidance is particularly interesting, where an agent tries to reduce losses by immediately searching for a better alternative whenever a loss is encountered. A combination of enhancements allows an increase in the average win-rate in over 60 different video games from 31.0 to 48.4% in comparison with a vanilla MCTS implementation.

Frydenberg et al. (2015) show that classical MCTS is too cautious for some games in GVGP. Better results are observed after increasing risk-seeking behavior by modification of the exploitation part of UCT. Undesirable oscillations of the controlled character can be eliminated by adding a reversal penalty. Macro-actions and limiting tree expansion are other important improvements.

Adding a heuristic into the algorithm is a common procedure. Guerrero-Romero et al. (2017) formulate a conclusion that the best strategy is to change the type of utilized heuristic during the game. They investigate four heuristics: winning maximization, exploration maximization, knowledge discovery and knowledge estimation. It is beneficial to start with enhancing knowledge discovery or knowledge estimation and only then add more winning or exploration heuristics. These rules could be described by a high level meta-heuristic extension.

An interesting combination of MCTS enhancements is shown by Goodman (2019) describing their efforts to win the Hanabi competition at CIG in 2018. Hanabi is a cooperative game, so communication is an essential factor. The convention used for communication is a particularly interesting sub-problem for AI. When all players are controlled by the same AI agent, hidden information can be naturally shared. In the case of the cooperation between different agents, the communication convention has to be established. For this task, a novel re-determinizing ISMCTS algorithm has been proposed, supported by adding rules to restrict the action space. The rules of competition give an agent 40 ms time for a decision but these were easily achieved using a supervised neural network with one hidden layer.

Nowadays, AI agents usually outperform human players, which could be perceived as a problem when human enjoyment of the game is necessary. This topic has been studied by Khalifa et al. (2016) in terms of GVGP. Several modifications of the UCT formula are introduced, such as an action selection bias towards repeating the current action, making pauses, and limiting rapid switching between actions. In this way, the algorithm behaves in a manner similar to humans, which increases the subjective feeling of satisfaction with the game in human players.

This section can be concluded by stating that the best results are achieved by combining different types of extensions of the MCTS algorithm. However, there is no one-size-fits-all solution. So this should encourage scientists to experiment with various methods.

4.6 Opponent modelling

Problem of opponent modelling is also relevant for games with imperfect information. This section presents a few examples, which incorporate analysis of the opponent into the MCTS algorithm.

Goodman and Lucas (2020) analyse how accurate the opponent model must be in a simple RTS game called Ground War. Two types of opponents are considered—one that uses the same policy as the MCTS-based agent and a parameterized heuristic model. The authors not only have shown that having an accurate opponent model is beneficial to the performance of the MCTS-based agents but also compared its performance against another agent based on RHEA. The findings show that the latter is more sensitive to the quality (accuracy) of the opponent model.

A game in which opponent modelling is crucial in order to succeed is Poker. Cowling et al. (2015) extend ISMCTS with the ability to perform inference or bluffing. Inference is a method of deriving the probability distribution of opponent moves based on the observed data (actions already done). However, the opponent can assume that his moves are inferred, and then bluffing comes into play—they can take misleading actions deliberately. Bluffing as an ISMCTS extension is realized by allowing sampling of states that are possible only from the opponent’s perspective, the search agent’s point of view is neglected. This technique is called self-determinization. The frequency with which the determinization samples each part of the tree is also recorded, thereby allowing performance of inference.

Another example of opponent modelling is Dynamic Difficulty Adjustment (DDA; Hunicke 2005). DDA is a family of methods focused on creating a challenging environment for the player. In order to keep the player engaged during the whole game, DDA changes the difficulty of encountered opponents according to the player’s actual skill level.

A topic which can be related to opponent modelling is making an MCTS-driven player to model a certain behavior. Here, instead of predicting what other players may do in the game, the goal is to make the agent mimic a certain group of players. Baier et al. (2018) consider the problem of creating a computer agent that imitates a human play-style in a mobile card game Spades. The human-like behavior leads to a more immersive experience for human players, increases retention as well as enables prediction what human players will do, e.g. for testing and QA purposes. The MCTS-based bots are often very strong but rarely feel human-like in their style of playing. The authors train neural networks using approximately 1.2 million games played by humans. The output of the networks is the probability, for each action, that such an action would be chosen by a human player in a given state. Various network architectures were tested, but all of them were shallow, i.e., not falling into the category of deep learning. Next, various biasing techniques, i.e., ways of including the neural networks in the MCTS algorithm were tested. The notable one was based on knowledge bias, in which the UCT formula (c.f. 2) was modified as in Eq. 10:

$$\begin{aligned} a^{*} = \arg \max _{a \in A(s)}\left\{ Q(s,a)+C\sqrt{\frac{\ln \left[ N(s) \right] }{N(s,a)}} +C_{BT}\sqrt{\frac{K}{N(s)+K}} P(m_i) \right\} , \end{aligned}$$

(10)

where $P(m_i)$ is the output from the neural network trained on human data; $C_{BT}$ is a weight of how the bias blends with the UCT score and K is a parameter controlling the rate at which the bias decreases. Values of $C_{BT}$ and K have been experimentally optimized.

5 Combining MCTS with machine learning

5.1 General remarks

ML has been on the rise in the computer science research. It has lead to many breakthrough innovations and its popularity and applicability in practical problems is still growing rapidly. The field of ML consists of techniques such as neural networks, decision trees, random forests, boosting algorithms, Bayesian approaches, support vector machines and many others. Although, MCTS is not typically associated with ML, but rather with the classical AI and search problems, it can be considered a ML approach, and a type of reinforcement learning, specifically. However, MCTS is often combined with other ML models to great success. ML, as a general technique, can be used in various ways and contexts. A good example is creating an evaluation function through ML. We have already referred to such an approach in Sect. 3.3 and works such as authored by Wu et al. (2018) and Goodman (2019). In this section, this topic will be investigated in more details. In Sect. 4, we have also already shown that convolutional neural networks (CNNs) can be useful tools for map analysis in RTS games (Barriga et al. 2017).

5.2 The AlphaGo approach: policy and value

The ancient game of Go has been considered the next Grand Challenge of AI in games since Chess and the famous match between Garry Kasparov and Deep Blue, which was analysed by Campbell et al. (2002). Following Gelly et al. (2012), Go is considerably more difficult for computer programs to play than Chess. Until 2006, computer programs were not stronger than average amateur human player. In 2006–2007, MCTS was introduced and became a quantum leap in terms of bots competence levels. MCTS has become the first major breakthrough in computer Go, whereas MCTS combined with ML techniques have became the second major breakthrough. In 2015, AlphaGo program created by Silver et al. (2016) under the auspices of Google, has become first to beat a professional human player, Fan Hui, on a regular board without a handicap. Subsequently, in 2016, AlphaGo won 4–1 in a match versus Lee Sedol (Silver et al. 2017), one of the strongest Go players of all time. By many scientists, e.g. Wang et al. (2016), this result was considered one of the most important achievements of AI in the recent years. AlphaGo inspired many approaches that combine MCTS and ML models. Because many of them share the same basic structure, we will now outline it. The structure is based on two ML models that represent the so-called value and policy functions, respectively. Such models are often referred to as “heads” if they are constructed using neural networks, which is the most common choice to modelling them.

• Value function—a function $v^{*}(s)$ that approximates the outcome of the game in a given state s. A perfect (non-approximate) value function is often used when solving a game, e.g. Checkers by Schaeffer et al. (2007). This is a similar concept to the state evaluation function. The use of them can be technically the same, however, the term “state evaluation function” is usually used when the function is static, whereas value function is self-improving through ML.

• Policy function—the policy, denoted by p(a|s) informs which action—a—should be chosen given state s. A policy can be deterministic or stochastic. It is often modelled as a probability distribution of actions in a given state.

The AlphaGo approach employs deep convolutional networks for modelling both value and policy functions as depicted in Fig. 8. In contrast to a later iteration of the program called AlphaZero, AlphaGo’s policy function is kick-started by supervised learning (SL) over a corpus of moves from expert human players. Readers interested in the details of the ML pipelines pursued in various versions of AlphaGo and AlphaZero are advised to check the papers from Silver et al. (2018, 2016, 2017). The initial policy is called the SL policy and contains 13 layers (Silver et al. 2016). Such a policy is optimized by a RL policy through self-play and the policy gradient optimization method. This adjusts the SL policy to correctly approximate the final outcome rather than maximize the predictive accuracy of expert moves. Next, the value network is trained to predict the winner in games played by the RL policy.

Fig. 8

Policy and value deep convolutional networks used in AlphaGo. The figure is reproduced from Silver et al. (2016)

The authors propose to use a slightly modified formula for selection in MCTS compared to 2:

$$\begin{aligned} a^{*} = \arg \max _{a \in A(s)}\left\{ Q(s,a)+ \frac{P(s,a)}{1+N(s,a))}\right\} , \end{aligned}$$

(11)

where P(s, a) is the prior probability and the remaining symbols are defined in the same way as in Eq. 2.

When a leaf node is encountered, its evaluation becomes:

$$\begin{aligned} V(s_L)=(1-\lambda )\upsilon _\Theta (s_L)+\lambda z_L, \end{aligned}$$

(12)

where $\upsilon _\Theta (s_L)$ is the value coming from the value network, $z_L$ is the result of a rollout performed by the policy network and $\lambda$ is the mixing parameter.

5.3 MCTS + neural networks

In this section, we outline approaches that use both the value and the policy functions, so they can be considered as inspired by the AlphaGo approach. Chang et al. (2018) propose one such work called the “Big Win Strategy”. It is essentially the AlphaGo approach applied for $6\times 6$ Othello. The authors propose small incremental improvements in the form of an additional network that estimates how many points a player will win/lose in a game as well as using depth rewards during the MCTS search.

5.3.1 AlphaGo inspired approaches

The AlphaGo style training was also applied to the game of Hex. Gao et al. (2017) proposed a player named MoHex-CNN that managed to outperform the previous state-of-the-art agent named MoHex 2.0 (Huang et al. 2013). MoHex-CNN introduced a move-prediction component based on a deep convolutional network that serves as a policy network and is used together with MCTS. Next, Gao et al. (2018) proposed a novel three-head neural network architecture to further significantly improve MoHex-CNN and named the new player MoHex-3HNN. The network outputs include policy, state- and action-values.

Takada et al. (2019) present another approach, called DeepEzo, that has been reported to achieve a winning rate of $79.3\%$ against MoHex 2.0, then world champion. The authors train value and policy functions using RL and self-play, as in the case of AlphaGo. The distinctive feature, however, is that the policy function is trained directly from game results of agents that do not use the policy function. Therefore, the policy function training is not a continuous process of self-improvement. Although DeepEzo won against MoHex 2.0, it lost against MoHex-3HNN in a direct competition during the 2018 Computer Olympiad (Gao et al. 2019).

Yang et al. (2020) build upon the AlphaGo approach to work without the prior knowledge of komi, which is the handicap that can be dynamically applied before a match. AlphaGo as well as most of the programs are trained using a komi fixed beforehand, i.e. equal to 7.5. In the described work, the authors designed the networks that can gradually learn how much komi there is. In addition, they modified the UCT formula to work with the game score directly and not with the win-rate, which is better for unknown komi. Wu et al. (2018) propose another approach to tackle the dynamic komi settings. Here, the authors proposed a novel architecture for the value function called a multi-labelled value network. Not only it supports dynamic komi but also lowers the mean squared error (MSE).

Tang et al. (2016) state in the first sentence of the abstract that their method for creating a Gomoku agent was inspired by AlphaGo. The main differences are that (1) here only one neural network is trained that predicts the probability of players winning the game given the board state, (2) the network is only three-layer deep, and (3) the network is trained using Adaptive Dynamic Programming (ADP).

Barriga et al. (2017) in their work aim at creating AI agents for RTS games in the $\mu$RTS framework. You can find more details about the approach in the RTS games Sect. 1.3 of this survey. The use of ML is very similar to the AlphaGo’s approach as both value and policy functions are modelled as CNNs. However, the networks are only used for the strategic search, which outputs a strategic action. Next, a purely adversarial search replaces the strategic action by lower level tactical actions. This can be considered a two-level hierarchical approach.

Value network	Policy network
Input: $128\times 128$, 25 planes	Input: $128\times 128,$ 26 planes
8 Conv. layers	8 Conv. layers
Global averaging over $16\times 16$ planes
2-Way softmax	4-Way softmax

Hearthstone, which has already been introduced in Sect. 4.4, is a two player, competitive card game in which the information about game state is imperfect. That partial observability in practice means two situations—when player draws a card and when takes combat decisions without knowing the opponent’s cards in hand and the so-called secrets on the board. To handle the first case, Zhang and Buro (2017) introduce chance event bucketing. Chance event is a game mechanics with a random outcome e.g. a player rolls a dice or draws a card. Those tens or hundreds different outcomes produced by a single chance event can be divided into an arbitrary number of buckets. Then at the simulation phase each bucket containing similar outcomes is sampled, that process allows to reduce branching factor in the tree search. Other sampling methods based on chance nodes are described by Choe and Kim (2019) and Santos et al. (2017).

Świechowski et al. (2018) use the MCTS algorithm together with SL for the game of Hearthstone. In this work, the authors describe various ways in which ML-based heuristics can be combined with MCTS. Their variant of choice uses a value network for the early cutoff which happens at the end of the opponent’s turn, therefore the simulation phase of the MCTS algorithm searches the tree only two turns ahead. However, players can make multiple moves within their turns. The policy network is used to bias the simulations. The authors use pseudo-roulette selection in two variants. In the first one, the probability of choosing an action in a simulation was computed using Boltzmann distribution over heuristic evaluations from the value network. In the second variant, the move with the highest evaluation was chosen with the probability of $\epsilon$, whereas a random one (the default policy) was chosen otherwise, i.e., with the probability of $1-\epsilon$.

Dots-and-Boxes is a paper-and-pencil game that involves capturing territory. For actions, players draw horizontal or vertical lines that connect two adjacent dots. Zhuang et al. (2015) present an approach to creating a strong agent for this game. The program is based on the MCTS algorithm and a sophisticated board representation. The MCTS is combined with a traditional, i.e., shallow, neural network that predicts the chance of winning of the players. Although this is effectively a value network, it is used to guide the simulation phase of MCTS. The network is trained using stochastic gradient descent.

Yang and Ontañón (2018) propose a neural network approach to RTS games in the µRTS framework. It relies on learning map-independent features that are either global (e.g. resources, game time left) or belong to one of 15 feature planes. Each map is represented by a tensor of size $15 \times width \times height$. Such tensor are passed to CNNs that are trained to estimate winning probability in the current state. The generalization ability of the trained network then allows extension to larger maps than the ones used for training.

Hu et al. (2019) present work that is another of the many examples of approaches that have been inspired by AlphaGo. Here, a system called Spear has been proposed for dependency-aware task scheduling that considers both task dependencies and task resource demands for various resources. It is based on MCTS in which the simulation phase is replaced by a deep RL model that acts in the environment and chooses actions according to its policy rather than randomly.

5.3.2 MCTS as a trainer for ML models

Soemers et al. (2019) show that is possible to learn a policy in a MPD using the policy gradient method and value estimates directly from the MCTS algorithm. Kartal et al. (2019a) propose a method to combine deep RL and MCTS, where the latter acts as a demonstrator for the RL component. The authors motivate this approach as “safer reinforcement learning” in “hard-exploration” domains, where negative rewards may have profound consequences. The domain of choice is a classic video game called Pommerman. In each state, the agent may move in one of four directions on the map, stay in place or plant a bomb. The MCTS is run with relatively shallow depth for time-constraints. However it is sufficient to learn, for instance, to avoid being killed by agent’s own bomb. Please note that the bombs placed by the agent explode with a delay and this interaction is hard to learn from scratch by pure RL methods. The approach, depicted in Fig. 9, is based on the actor-critic idea.

A relatively similar idea of using MCTS as a trainer for reinforcement learner was proposed by Guo et al. (2014) for simple Atari games. In contrast to the previous approach, the authors use deep CNNs as a model for policy. The input to the CNNs are raw pixel images of what is displayed on the screen. This is essentially the same representation human players observe. The original size of $160 \times 120$ pixels is downgraded to $84 \times 84$ in a preprocessing step. The last four frames, e.g., a tensor of size $84 \times 84\times 4$ is passed to the network. The output is a fully connected linear layer with a single output for each valid action in the game. The authors compare three different variants of how MCTS can train a CNN: UCTtoRegression, UCTtoClassification and UCTtoClassification-Interleaved.

Fig. 9

Overview of the Planner Imitation based A3C framework. There are multiple demonstrators making MCTS actions while keeping track of what action the actor network would take. The figure is reproduced from Kartal et al. (2019b)

Pinto and Coutinho (2018) present another approach in which MCTS and RL are combined. Here, MCTS operates in the “Options Framework”. Options are defined as a tuple $(I, \mu , \beta )$:

• (1) I—is the initiation set that contains states from which options are valid;

• (2) $\mu$—is a policy that tells what actions to make as part of the current option;

• (3) $\beta$—is the termination condition for the option.

The MCTS algorithm searches the combinatorial space of possible options and prepares them for RL. Using options allowed the authors to reformulate the MPD problem into a semi-Markov decision problem (SMDP). In the latter, the Q-learning rule can be defined over options as shown in Eq. 13.

$$\begin{aligned} Q(s,o)=(1-\alpha ) Q(s,o) + \alpha \left[ r + \gamma \max _{o' \in O}Q(s',o') \right] , \end{aligned}$$

(13)

where O denotes the possible options.

5.4 MCTS with temporal difference learning

Temporal Difference (TD) learning was one of the first notable applications of the RL paradigm in the domain of games Tesauro (1994). Here, the game is modeled as time sequences of pairs $(x_t, y_t)$, where t is the time step, $x_t$ denotes a state at this step and $y_t$ is the predicted reward associated with it. The main idea that underpins TD learning is that the reward at the given time step can be approximated by a discounted sum of rewards at future time steps:

$$\begin{aligned} Y_t = y_{t+1}+ \gamma y_{t+2} + \gamma ^2 y_{t+3} + \cdots \end{aligned}$$

(14)

The predictions are iteratively updated to be closer to the prediction of future states, which are more accurate. Please note that the last prediction, which is the final game outcome, is the most accurate. The goal is to learn the prediction for the current state or states available from the current states using the training signal from future predictions. The learning process typically runs for many game repeats. This idea is called bootstrapping.

TD methods can be applied to any pairs of values, i.e., input and output signals. Silver et al. (2012) use it to learn the weights of an evaluation function components for Go rather than for complete states.

The value function update in TD learning is most commonly defined for any state in the game as:

$$\begin{aligned} V(s_t) \leftarrow V(s_t) + \alpha (R_{t+1} + \gamma V(s_{t+1}) - V(s_t)), \end{aligned}$$

(15)

where $\alpha$ is the learning factor; $\gamma$ is the discount factor; $R_{t+1}$ is the reward observed in step $(t+1)$ and $V(s_t)$ is the value of state encountered in step t of the game.

Vodopivec and Šter (2014) incorporate the ideas of TD learning to replace the UCT formula (c.f. Eq. 2) and calculate state estimates in nodes in a different fashion. Using TD learning inside MCTS affects only the backpropagation phase, in which values are updated, and the selection phase—because the tree policy no longer uses the UCT score. The remaining parts of the MCTS algorithm remain unchanged. The most general formula, which has specific variants in the paper, is based on weighted average of the classic UCT estimate and the TD estimate:

$$\begin{aligned} Q_{TD-UCT} = \omega V + (1-\omega )*Q + C\sqrt{\frac{\ln \left[ N(s) \right] }{N(s,a)}}, \end{aligned}$$

(16)

where $\omega$ is the weight, V is the estimate sampled from the TD value function and the remaining symbols are as defined in Eq. 2.

Equation 16 is used instead of the UCT formula as the tree policy. The TD value function update ((V)) is performed in the back-propagation phase. For this task, the authors specifically chose the TD($\lambda$) algorithm, in which a state value $V(s_t)$ is updated based on the immediate reward—$R_{t+1}$—and the value in the following state—$V(s_{t+1})$:

$$\begin{aligned} V(s_t) \leftarrow V(s_t) + \alpha e(s_t) \delta _t, \end{aligned}$$

(17)

where

$$\begin{aligned} \delta _t = R_{t+1} + \gamma V(s_{t+1}) - V(s_t), \end{aligned}$$

(18)

$\delta _t$ is the temporal difference error (TD error) and $e(s_t)$ denotes the eligibility trace. The remaining symbols have already been defined before.

Three variants of the TD($\lambda$) integration have been proposed:

• TD-UCT Single Backup—here, the authors introduce a simplification that all future values of states have already converged to the final value of the playout. The TD error depends on the distance to the terminal state and the value estimated in a leaf node: $\delta _1 = \gamma ^P R_i - V(leaf)$, where P is the number of steps in a playout.

• TD-UCT Weighted Rewards—this variant is even more simplified version of TD-UCT Single Backup obtained by fixing most of the parameters. In particular, they set $\gamma = 1$ (in Eq. 18) and $\omega = 1$ (in Eq. 16) which makes V fully replace the average score estimate Q. However, the exploration part of Eq. 2 remains unchanged, so this variant combines the idea of TD learning with the UCT exploration part.

• TD-UCT Merged Bootstrapping—here, there is no longer the assumption that all future state values converge to the final reward of the playout. Instead, this variant includes the regular TD bootstrapping, i.e. the state value is updated based on the value of the next state.

In addition, the combinations with the AMAF heuristic and RAVE (c.f. Eq. 4) are considered in the experiments. The various combinations were tested on four games: Hex, Connect-4, Gomoku and Tic-Tac-Toe. One variant, namely TD-UCT Merged Bootstrapping with AMAF, achieved the best performance in all four games compared to the other variants as well as compared to the plain UCT.

The TD method can also be used to adapt the policy used in the simulation phase of the MCTS algorithm (Ilhan and Etaner-Uyar 2017). This is especially useful for games (or other domains) in which uniform random simulation performs poorly. This idea is explored in General Video Game AI games: Frogs, Missile Command, Portals, Sokoban and Zelda. The TD method of choice is Sarsa($\lambda$). The authors use a linear value function approximation (as suggested in true online Sarsa($\lambda$)) along with a state representation method. The representation is a vector of numerical features that describe the game state and its weights. Here, the goal is to learn the weights of the state vector and then use a custom rollout procedure that incorporates state evaluation. The outcome of a rollout is calculated as in Eq. 14, so the outcome at step t is a discounted sum of future steps. Each consecutive step is performed by a separate rollout. This is a different approach to a vanilla MCTS, in which there is one rollout that fetches the game score, obtained at the terminal state, and propagates the same score to all nodes visited along the path. Here, states on the visited path receive different scores according to the TD learning idea. The rollouts are performed according to the $\epsilon -$Greedy selection policy that maximizes the weighted linear combination of features in a state over possible actions that lead to it. The authors achieve better results than vanilla MCTS in 4 out 5 tested games.

5.5 Advantages and challenges of using ML with MCTS

We have reviewed approaches that combine MCTS with ML techniques in various ways. Let us conclude this section with a short discussion about the advantages and specific challenges that arise when such a joint approach is pursued.

Advantages

• New quality results. The main factor that motivates researchers to introduce ML into MCTS is to obtain better results than the current state-of-the-art. In the game domain, the common goal is to create a next-level player that will win against the strongest programs or human champions. The examples of AlphaGo (Silver et al. 2016), MoHex-3HNN (Gao et al. 2018) and other show that ML can indeed be an enabler for such spectacular results. In Sect. 7, which is devoted to non-game applications, we will also show an approach that combines ML and MCTS and achieves novel results in chemical synthesis (Segler et al. 2018).

• Faster convergence. Another motivation, and ultimately an advantage, that ML brings to MCTS is that it enables MCTS to converge faster. We have seen three groups of approaches in this category. First, e.g. by Zhuang et al. (2015), Wu et al. (2018) and Świechowski et al. (2018), achieve it by introducing an ML model for the evaluation function in MCTS. Second, which is often combined with the first as in AlphaGo’s inspired approaches, consists in using an ML model as the policy for simulations. Lastly, RL algorithms such as TD($\lambda$) can be combined with the UCT formula or used instead in MCTS as shown by Vodopivec and Šter (2014).

• Approximation capabilities. Let us discuss the last two advantages on a theoretical level. MCTS is a search-based method. In order to properly evaluate the quality of a state, the state must be visited enough number of times (to gain the confidence about the statistics). The MCTS algorithm has no inherent ability to reason about the states rather than based on statistics stored directly for them. Many ML models, however, such as neural networks or logistic regression are universal approximators. They can provide answers for states that have never been observed during simulations as long as the ML models are well-trained, i.e., achieve good performance on the training data and are neither underfit nor overfit.

• Generalization. Continuing on the aspect the approximation—ML models can generalize to some extend the knowledge encoded in MCTS trees. Although the training process usually takes a long time, using the trained models in real-time is by orders of magnitude faster than using MCTS or any search-based method. Moreover, let us imagine a solved end-game position in Chess, in which there is a strategy for a king that involves a traversal around an enemy piece. If we increased the board size from $8\times 8$ to $9\times 9$, then MCTS not only would have to compute everything from scratch but also the complexity of the problem would increase (bigger board). Thanks to generalization and abstraction capabilities, ML models can bring new quality here.

Challenges

• Data volume and quality. The biggest challenge of introducing ML into MCTS is data availability. ML models, especially those based on deep learning such as AlphaZero (Silver et al. 2017) require a lot of data. Moreover, there is a saying “garbage in, garbage out” so the data need to be of high quality. In game AI, it means that it must be generated by strong AI players. However, if ML is to help MCTS in the development of strong players, then we have a situation that strong players are not yet available. This is a major obstacle for using SL algorithms. To tackle this problem, self-play and RL (Kartal et al. 2019a) can be applied.

• Training accurate models. This is a general challenge when creating ML models. In game AI, overfitting may appear, for example, when heuristic-based and non-adaptive AI players are used in training games to generate data. It may lead to a scenario, in which a lot of similar states are present in the training data, whereas a lot of areas are under-represented. A proper variety of positions must be ensured in the training process as well as proper training methodology. Often, experts at game AI, especially in video game development industry, are not experts in ML.

• Technology problems. The most common programming languages used for games are C++ and C#. The most common programming language used for ML is Python, followed by R. Therefore, using ML techniques in games usually requires either custom implementations of the ML models or having complex projects that combine multiple programming languages.

6 MCTS with evolutionary methods

6.1 Evolving heuristic functions

Benbassat and Sipper (2013) describe an approach, in which each individual in the population encodes a board-evaluation function that returns a score of how good the state is from the perspective of the active player. The authors use strongly typed genetic programming framework with boolean and floating-point types. The genetic program, i.e., the evaluation function, is composed of terminal nodes and operations such as logic functions, basic arithmetic functions and one conditional statement. The terminal nodes consist of several domain-independent nodes such as random constant, true, false, one or zero as well as domain-specific nodes such as EnemyManCount, Mobility or IsEmptySquare(X,Y). The paper considers two domains of choice: Reversi and Dodgem games. Figure 10 illustrates exemplar function for Reversi.

Fig. 10

An example of a genetic programming tree used to evolve an evaluation function. The figure was inspired by a depiction of the evaluation function for Reversi in work by Benbassat and Sipper (2013)

The authors incorporate the evolved heuristics in the simulation phase of MCTS. Instead of simulating a random move, the agent randomly chooses a parameter called playoutBranchingFactor in each step. The upper bound of this value is equal to the number of currently available actions. Then, a set of playoutBranchingFactor random moves is selected and the one with the highest heuristic evaluation is chosen to be played in the simulation. This way, there is still stochastic element in simulations, which is beneficial for the exploration of the state-space.

The authors conclude that evolving heuristic evaluation function improves the performance of the MCTS players in both games. However, it is time consuming, therefore it is most suitable in scenarios, when it can be done offline, i.e., before the actual game, which is played by MCTS with an already prepared evaluation function.

Alhejali and Lucas (2013) propose a similar idea to evolving heuristics by means of GP for Ms Pac-Man agent. In this research, the new default policy to carry out simulations is evolved directly.

6.2 Evolving policies

The approaches presented in this section consist in optimizing the policies of MCTS—tree policy or default simulation policy—using evolutionary computation. This is a different case to the previous section, whereby a new evaluation function is constructed.

One of the first papers on this topic was published by Lucas et al. (2014). The authors introduce a weight vector w that is used to influence both tree policy T(w) and default policy D(w). The weight vectors are stored individuals optimized by a (1 + 1) Evolution Strategy (ES). For default policy, a mapping from the state space to a feature space with N features is introduced. Those features are assigned weights that are used to bias actions during a simulation towards states with a greater aggregated sum of weights. To maintain exploration, softmax function is used instead of a greedy selection.

Perez et al. (2014) introduce an approach called Knowledge-Based Fast Evolutionary MCTS (KB Fast-Evo MCTS) which extends the work by Lucas et al. (2014). The environment of choice is GVGP, which as a multi-domain challenge, encourages adaptive algorithms that are not tailored for a particular game. The authors dynamically extract features from a game state, in contrast to a fixed number of features employed in the previous work. The features not only vary between games but also between steps of the same game. Each feature i has a weight $w_{ij}$ assigned for action $a_i$. The weights are evolved dynamically and they are used to bias the MCTS simulations in a similar way as proposed by Lucas et al. (2014). Here, also a soft-max function is used to calculate the probability of choosing an action.

KB Fast-Evo MCTS also introduces a dynamic construction of a knowledge base. The knowledge base consists of knowledge items, for which statistics such as occurrences and average scores changes are maintained. The authors introduce the notions of curiosity and experience and the formula to calculate them for a knowledge base. Next, both the values, i.e. curiosity and experience, are combined to calculate the knowledge change for a knowledge base. Finally, the authors propose a formula to calculate the predicted end-game reward that can be used for the MCTS roll-out. The reward is either equal to the change of the game score or equals to the weighted sum of the knowledge change and the distance change if no game score change is observed. The distance change is a heuristic value introduced for GVGP. This approach was tested using 10 games from the GVGP corpus. The proposed KB Fast-Evo MCTS outperformed the vanilla MCTS in all of the games, whereas the previous Fast-Evo MCTS in 7 out of 10 games.

Pettit and Helmbold (2012) introduced a system called Hivemind that learns how to play abstract strategy games on regular boards. The game of choice here is Hex. The system uses MCTS/UCT and Evolution Strategies to optimize the default policy for simulations. The subject of evolution are individuals that encode:

• Local patterns on a board. For example, the status of a given hex cell with its six neighbors as shown in Fig. 11.

• Move selection strategies (default, uniform local, uniform local with “tenuki”).

Fig. 11

Example of a board pattern encoded in an individual of a population optimized by EA (Pettit and Helmbold 2012)

Thanks to using ES, the authors were able to evolve meaningful patterns that used by move selection strategies outperform baseline default policies, including uniform random selection. The obtained results are stable, i.e., the evolutionary process finds similar efficient policies each time. The authors also show that the learnt policies generalizes to different board sizes.

The tree policy, i.e., in the selection part of MCTS, can also be optimized alone. Bravi et al. (2016) investigated whether it is possible to evolve a heuristic that will perform better than UCB in the field of GVGP. They experimented with five different games from this framework: Boulderdash, Zelda, Missile Command, Solar Fox and Butterflies. In this work, GP is employed with a syntax tree that serves as an encoding of a chromosome. The syntax tree denotes a formula to use instead of UCB.

The possible components are:.

• 16 Predefined constants from the range $[-30,30],$

• Unary functions—square root, absolute value, multiplicative inverse,

• Binary functions—addition, subtraction, multiplication, division and power,

• Variables such as child visits, child depth, parent visits, child max value.

In a similar paper, Bravi et al. (2017), the authors added certain Game Variables as possible components that describe features of the state of the game. The evolved heuristic functions perform on a similar level, i.e., not significantly better or worse, to UCB in four of five tested games. In the last game, they yield poor results. However, these heuristics give a lot of insights about particular games, e.g., whether exploration is an important factor. The authors conclude that this approach can be promising when combined with additional variables and mechanisms.

6.3 Rolling horizon evolutionary algorithm

In this subsection, we will provide a short background of RHEA. In general, RHEA is a competitor technique for creating decision-making AI agents in games. As the name implies, it is a representative of a broad class of evolutionary algorithms (EA). However, it should not be mistaken with Evolutionary-MCTS, which has been inspired by RHEA. Although MCTS and RHEA are different techniques, they are often compared in terms of performance in particular games as well combined within the same agent.

The first application of RHEA was published in Perez et al. (2013) for the task of navigation in single-player real-time games. For a recent summary of this method and its modifications so far, we recommend the article by Gaina et al. (2021).

Instead of doing the traditional selection and simulation steps, the RHEA algorithms evolve sequences of actions starting from the current state up to some horizon of N actions. Each individual in a population encodes one sequence. Only the first action from the best sequence is played in the actual game. To compute the fitness value, the simulator applies encoded actions one by one. If an action is not legal, then it is replaced by a default one (“do nothing”). After applying the last action, the score of the game is computed as shown in Eq. 19:

$$\begin{aligned} h(s) = \left\{ \ \begin{array}{ll} 1 &{} \text{ win } \text{= } \text{ True },\\ 0 &{} \text{ win } \text{= } \text{ False }, \\ score \in [0,1] &{} \text{ otherwise }. \end{array} \right. \end{aligned}$$

(19)

The algorithm incorporates EA operators of crossover and mutation. The selection is typically the tournament selection. Elitism, i.e., directed promotion of the most fit parents to the generation, can be incorporated. In the mentioned article by Gaina et al. (2021), you can find a description of an RHEA framework tested using a set of 20 different games. As this survey is focused on modifications of the vanilla MCTS algorithm, the work by Gaina et al. (2017b) presents enhancements to the vanilla RHEA approach. The online self-adaptation of MCTS (e.g.) has also its counterparts for RHEA as described by Gaina et al. (2020).

Horn et al. (2016) consider a few hybridization schemes of the MCTS and EA. One is called RHEA with rollouts, in which after the application of a sequence of actions stored in a genome, a number of rollouts (Monte Carlo simulations) is performed. The final fitness is the average of the classic RHEA fitness and the average value from additional rollouts. Another hybridization scheme proposed is an ensemble approach, which first runs RHEA for a fixed time and then MCTS to check for alternative paths.

The analysis by Gaina et al. (2017a) compares the MCTS and RHEA approaches in GVGP. In addition, in this works, the impact of various parameters of RHEA is measured. Within the considered corpus of various games used in GVGP, there are cases, in which one method is stronger than the other. The conclusion is that neither algorithm is inherently stronger and their efficacy depends on a particular game.

6.4 Evolutionary MCTS

Fig. 12

Evolutionary MCTS. Instead of performing rollouts, sequences of actions are mutated. Repair operators are introduced in case of possible illegal moves generated this way. The figure was inspired by the work of Baier and Cowling (2018)

The common element of the three approaches described in papers by Baier and Cowling (2018) and Horn et al. (2016) is that EA is responsible for performing simulations. Baier and Cowling (2018) propose Evolutionary MCTS (EMCTS), which builds a different type of tree than the vanilla MCTS. It does not start from scratch, but instead all nodes, even the root, contain sequences of actions of certain length. In the presented study with Hero Academy game, the sequences are 5 action long. Each node is a sequence, whereas edges denote mutation operators as shown in Fig. 12. Instead of performing rollouts, the sequences in leaf nodes are evaluated using a heuristic function.

7 MCTS applications beyond games

7.1 Planning

Automated planning is one of the major domains of application of the MCTS algorithm outside games. The planning problem is typically formulated as MDP, which was defined in Sect. 2, or POMDP (Partially Observable MDP). Similarly to games, in AI planning, there is a simulated model that can be reasoned in. The model consists of an environment with the initial state, the goal states (to achieve) and available actions. The solution is a strategy—either deterministic or stochastic, depending on a particular problem, that transitions the initial state to the goal state, playing by the rules of the environment, in the most efficient way. The most efficient fashion may be, e.g., the shortest transition or having the smallest cost. Particular applications differ between each other with respect to various constraints, extensions and assumptions. In this subsection, we focus on general planning problems. Scheduling, transportation problems and combinatorial optimization have their own dedicated sections.

Vallati et al. (2015) present a summary of the International Planning Competitions up to 2014, in which we can read that winning approaches of the probabilistic track (International Probabilistic Planning Competition) in 2009–2014 were using Monte Carlo Tree Search. It was shown that MCTS-based planners were very strong but they lost ground in the largest problems. They often needed to be combined with other models such as Bayesian RL (Sharma et al. 2019). This is along the motivation behind this survey, which focuses on extensions aimed at tackling the most difficult problems from the combinatorial point of view.

Scheduling is a similar problem to planning, wherein resources are assigned to certain tasks that are performed according to a schedule. Ghallab et al. (2004) point out that both automated planning and scheduling are often denoted as AI planning. There have been several standard implementations of MCTS to scheduling such as of Amer et al. (2013) for activity recognition and by Neto et al. (2020) for forest harvest scheduling.

MCTS has been used in other discrete combinatorial problems in a relatively standard variant, e.g. by Kuipers et al. (2013) for optimizing the Horner’s method of evaluating polynomials. In the work by Jia et al. (2020), MCTS is used for optimizing low latency communication. The authors present a parallelized variant, in which each parallel node is initialized with a different seed. The master processor merges the MCTS trees constructed with different seeds. Shi et al. (2020) apply MCTS to automated design of generating large scale floor plans with adjacency constraints. The only non-standard modification there is discarding the rollout phase entirely. Instead the tree is expanded by each node encountered by the tree policy. The MCTS algorithm has also been used as part of a more general framework, e.g., to the imbalance subproblem in a complex framework based on cooperative co-evolution to tackle large optimization problems.

7.1.1 Simplifications of a problem/model

To tackle large planning problems, Anand et al. (2015) propose an abstraction of state–action pairs for UCT. Later, Anand et al. (2016) came up with a refined version of this idea called “On-the-go abstractions” . The abstraction is based on the equivalence function that maps state–action pairs to equivalent classes. In their work, Anand et al. (2015) consider the following problems:

• How to compute abstractions—the algorithm is presented.

• When to compute abstractions—time units are introduced and when the UCT is sampled up to a certain level, abstractions are computed every t units, where t is computed using a dynamic time allocation strategy.

• Efficient implementation of computing abstractions—let n denote the number of State–Action Pairs (SAPs). The authors reduced the complexity of the naive algorithm of computing the abstractions from $O(n^2)$ to $O(rk^2)$, where r is the number of hash buckets and k is the size of a bucket and $n^2 > rk^2$.

The novel method with abstractions, called ASAP-UCT, has been compared with the vanilla UCT planner and other three algorithms in six problems from three domains. The ASAP-UCT was the clear winner with statistically significantly better results in all but one comparison.

Keller and Eyerich (2012) introduce a system called PROST aimed at probabilistic planning problem and acting under uncertainty. As previously mentioned, in this subproblem of planning, MCTS approaches has been the most successful. PROST is equipped with several modifications of vanilla MCTS/UCT approach such as:

• Action pruning—the model includes the concepts of dominance and equivalence of actions. Dominated actions are excluded, whereas only one action from equivalent sets is considered.

• Search space modification—the tree contains chance nodes and decision nodes and even after action pruning the search space is strongly connected.

• Q-value Initialization—PROST uses single-outcome determinization of the MDP using iterative deepening search (IDS) to initialize Q-values and avoid the cold start.

• Search depth limitation—search depth limit is introduced as a parameter $\lambda$.

• Reward Locks—states, in which no matter what the agent does, it will receive the same reward. The authors explain this is often the case close to goals or dead ends. The method of detecting reward locks is presented in the paper. The MCTS uses such information in two ways: (1) to terminate a rollout earlier and (2) minimize the error induced by limited depth search by changing the simulation horizon.

7.1.2 Modifications of UCT

Painter et al. (2020) modify the standard UCT formula (c.f. Eq. 2) to include the so-called Contextual Regret and adapted to multi-objective settings. The approach has been coined as Convex Hull Monte Carlo Tree Search. In a multi-objective setting the reward function is multi-dimensional and produces vectors instead of single values. The approach was validated in experiments using the Generalized Deep Sea Treasure (GDST) problem in both online and offline planning scenarios. A similar type of modification of UCT in planning problems, that is based on simple regret optimization, can be found in the article published by Feldman and Domshlak (2014)

Hennes and Izzo (2015) consider the problem of interplanetary trajectory planning and introduces a handful of modifications to MCTS that are unique to this problem. One of the modifications, however, is the fifth step of the MCTS routine called by the authors contraction. In the considered problem, the search-tree is often broad, i.e. the branching factor is as high as 500, but shallow, i.e., the depth is limited. In the contraption phase, the algorithm checks whether there is a sub-tree with terminal states. If each nodes of such a subtree have already been visited it can be considered solved.

7.1.3 Hierarchical and distributed MCTS

Vien and Toussaint (2015) propose a hierarchical Monte Carlo planning method to tackle the problem of large search spaces and combinatorial explosion in planning. The hierarchy is introduced at the action level. The top-level tasks are subsequently decomposed into subtasks that cannot be decomposed any further. The authors discuss the concept of a hierarchical policy in POMDPs. This research contains a successful application of macro actions in a non-game domain. The idea is illustrated in Fig. 13, in which there are four actions: $\{a_{1}, a_{2}, Pickup, Putdown\}$ and three macro actions $\{GET, PUT, NAV\}$.

Fig. 13

Hierarchical Monte Carlo Planning. The figure is inspired by Vien and Toussaint (2015)

The vertical dotted lines denote rollouts, whereas solid ones are tree policy executions. The green nodes (i.e., the smallest ones with o1, o2, a1, a2 annotations on the edges to them) denote subtree with only primitive actions. Nodes with other colors denote subtrees with particular macro actions. Each macro action defines a particular subgoal for the algorithm and has a nested routine (subroutine) for action selection. The authors show how to traverse such a MCTS tree with nodes of various hierarchy and how to expand it. The paper contains not only empirical evaluation using the Taxi problem but also theoretical results such as estimation of the bias induced by the Hierarchical UCT (H-UCT). They conclude that the proposed approach aims to mitigate two challenging problems by means of the “curse of dimensionality” and the “curse of history”. Hierarchical MCTS planning is efficient whenever a hierarchy can be found in the problem. It has been featured in recent works as well (Patra et al. 2020).

7.1.4 Planning in robotics

MCTS has also been used for planning in robotics. Best et al. (2019) propose a decentralized variant of MCTS for multi-robot scenario, in which each robot maintains its individual search tree. The trees in compressed forms are periodically sent to other robots, which results in an update of joint distribution over the policy space. The approach is suitable for online replanning in real-time environment. The algorithm was compared to standard MCTS with a single tree with actions of all robots included. Therefore, the action space of the tree was significantly larger than any of the decentralized trees of the Dec-MCTS approach. Apart from theoretical analysis, the experiments were carried out in eight simulated environments based on the team orienteering problem as shown in Fig. 14. The Dec-MCTS achieved a median 7% better score than the standard algorithm.

Fig. 14

Decentralized planning. The figure is drawn inspired by the work of Best et al. (2019)

Wu et al. (2015) introduce the k-Rejection Multi-Agent Markov Decision Process k-RMMDP model in the context of real-world disaster emergency responders. In this model, plans can be rejected under certain conditions e.g. if the human responders decide that they are too difficult. Because their approach is based on the two-pass UCT planning as shown in Algorithm 2, it can be considered hierarchical as well.

Monte Carlo Discrepancy Search (Clary et al. 2018) is another MCTS variant with applications in robotics. The idea is to find and store the current best path out to a receding horizon. Such a path is invalidated either by a change in the horizon or by stochastic factors. The simulations are performed around the best plan from the previous iteration. Then a so-called discrepancy generator selects a new node to branch from in the tree. The method was applied to planning the movement of a self-stable blind-walking robot in a 3D environment, which involved navigating around the obstacles.

7.2 Security

One of very recent areas of MCTS application are Security Games (SG). In short, in SG methods from game theory (GT) are used to find optimal patrolling schedules for security forces (e.g. police, secret service, security guards, etc.) when protecting a set of potential targets against the attackers (criminals, terrorists, smugglers, etc.). Recently, SGs gained momentum due to rising terrorist threats in many places in the world. While the most noticeable applications refer to homeland security, e.g. patrolling LAX airport (Jain et al. 2010) or patrolling US coast (An et al. 2013), SGs can also be applied to other scenarios, for instance poaching preventing (Fang et al. 2015; Gholami et al. 2018; Bondi et al. 2020) or cybersecurity (Wang et al. 2020).

In SG there are two players, the leader (a.k.a. the defender) who represents the security forces, and the follower (a.k.a. the attacker) who represents their opponents. The vast majority of SGs follow the so-called Stackelberg Game (StG), which is an asymmetric game model rooted in GT. In StG each of the two players commits to a certain mixed strategy, i.e. a probability distribution of pure strategies. In a typical patrolling example, a pure strategy would be a deterministic patrolling schedule which happens with certain probability. A set of such deterministic schedules (pure strategies) whose probabilities sum up to 1 constitutes mixed leader’s strategy.

The aforementioned information asymmetry of StG consists in the underlying assumption that first the leader commits to a certain mixed strategy, and only then the follower (being aware of that strategy) commits to their strategy. This assumption reflects common real-life situations in which the attacker (e.g. a terrorist or a poacher) is able to observe the defender’s behavior (police or rangers) so as to infer their patrolling schedules, before committing to an attack. SGs relying on StG model are referred to as Stackelberg SGs (SSG). SSG are generally imperfect-information games as certain aspects of the current game state are not observable by the players. Solving SSG consists in finding the Stackelberg Equilibrium (SE), which is typically approached by solving the Mixed Integer-Linear Program (MILP) corresponding to a given SSG instance (Paruchuri et al. 2008).

Calculation of SE is computationally expensive in terms of both CPU time and memory usage, and therefore, only relatively simple SSG models can be solved (exactly) in practice. For this reason, in many papers various simplifications are introduced to MILP SG formulations, so as to scale better for larger games, at the expense of obtaining approximate solutions. However, these approximate approaches still suffer from certain limitations of the baseline MILP formulation, e.g. the requirement of matrix-based game representation. Consequently, the vast majority of deployed solutions consider only single-act (one-step) games.

The lack of efficient methods for solving multi-act games motivated the research on entirely different approach (Karwowski and Mańdziuk 2015) which, instead of solving the MILP, relies on massive Monte Carlo simulations in the process of finding efficient SE approximation.

In the next couple of years the initial approach was further developed into a formal solution method, called Mixed-UCT (Karwowski and Mańdziuk 2016)

which is generally suitable for a vast range of multi-act games. Mixed-UCT consists in iterative computation of the leader’s strategy, based on the results of MC simulations of the game, played against a gradually improving follower. The currently considered follower is defined as a combination of a certain number of past followers from previous Mixed-UCT iterations. The method scales very well in both time and memory and provides efficient SE approximations (approximate mixed leader’s strategies) for various types of SGs (Karwowski and Mańdziuk 2019a). A general concept of Mixed-UCT is depicted in Fig. 3.

Recently, the same authors have proposed another UCT-based method for SE approximation in general-sum multi-step SGs. The method, called Double Oracle UCT (O2UCT), iteratively interleaves the following two phases: (1) guided MCTS sampling of the attacker’s strategy space; and (2) constructing the defender’s mixed-strategy, for which the sampled attacker’s strategy (1) is an optimal response (Karwowski and Mańdziuk 2019b, 2020).

One of the underlying assumptions of Stackelberg Games (SGs) is perfect rationality of the players. This assumption may not always hold in real-life scenarios in which the (human) attacker’s decisions may be affected by the presence of stress or cognitive biases. There are several psychological models of bounded rationality of human decision-makers, among which the Anchoring Theory (AT; Tversky and Kahneman 1974) is one of the most popular. In short, AT claims that humans have a tendency to flatten probabilities of available options, i.e. they perceive a distribution of these probabilities as being closer to the uniform distribution than it really is. An extension of O2UCT method that incorporates the AT bias into the attacker’s decision process (AT-O2UCT) has been recently considered in Karwowski et al. (2020).

7.3 Chemical synthesis

Segler et al. (2018) in an article published in Nature show an outstanding approach to planning chemical syntheses that combines MCTS with deep neural networks. These networks have been trained on most reactions known from the history of organic chemistry research. The particular type of synthesis considered in this paper is computer-aided retrosynthesis, in which molecules are transformed into increasingly simpler precursors. In this approach, named 3N-MCTS, three neural networks guide the MCTS algorithm:

• Rollout policy network—trained on 17,134 rules describing reactions, which were selected from the set of all published rules with a requirement for each rule to appear at least 50 times before 2015. The actions in the simulation phase of MCTS are sampled from this network.

• In-score filter network—this network provides retroanalysis in the expansion phase and predicts whether a particular transformation resulting by applying an action from the rollout policy is expected to actually work. If not—the action is discarded and a different one has to be chosen. Awais et al. (2018) based their work on a similar idea for synthesis of approximate circuits that discards circuit designs which are not promising in terms of quality. This way, the combinatorial complexity could be drastically reduced.

• Expansion policy network—this network selects the most promising node to add in the MCTS tree. This is a significant change compared to the standard approach, in which the first non-tested action defines the node to expand.

Combining deep neural networks with MCTS has been proven a viable approach to various chemical or physical problems including FluidStructure Topology Optimization by Gaymann and Montomoli (2019). The problem is formulated as a game on a grid, similar to Go.

7.4 Scheduling

Combining MCTS with additional heuristics is a pretty common approach to large computational problems. Asta et al. (2016) use MCTS to generate initial solutions to a Multi-mode Resource-constrained Multi-project Scheduling Problem, which are then modified through a local search guided by carefully crafted hyper-heuristics.

In order to create a test-bed for Computational Intelligence (CI) methods dealing with complex, non-deterministic and dynamic environments Waledzik et al. (2014) introduces a new class of problems, based on the real-world task of project scheduling and executing with risk management. They propose Risk-Aware Project Scheduling Problem (RAPSP) as a significant modification of the Resource-Constrained Project Scheduling Problem (RCPSP).

In Waledzik et al. (2015), they describe a method called Proactive UCT (ProUCT) that combines running heuristic solvers with the UCT algorithm. The heuristic solvers create baseline schedules, whereas UCT is responsible for testing various alternative branches caused by dynamic events. The method is further extended and summarized in Waledzik and Mandziuk (2018).

Wijaya et al. (2013) present an approach to energy consumption scheduling in a smart grid system that is based on MCTS. The application of the algorithm is relatively standard, however, the solutions can vary in terms of the so-called laziness. The lazier the solution, the less frequent consumption schedule changes. Since the lazier solutions are preferable, the UCT formula includes an additional penalty that is inversely proportional to the laziness value of the solution in a node.

An interesting scheduling approach inspired by AlphaGo Silver et al. (2016) has been proposed in Hu et al. (2019). Spear system, briefly summarized in Sect. 5.3.1, was reported to outperform modern heuristics and reduce the maskespan of production workloads by 20%.

7.5 Vehicle routing

Transport problems such as Capacitated VRP (CVRP) are special class of problems that often involve both planning and scheduling. We recommend the surveys by Caceres-Cruz et al. (2014) or Mańdziuk (2019) for the introduction to various problem specifications, the most commonly applied techniques other than MCTS and the latest advances. There are many variants of CVRPs. The common elements include planning routes for vehicles, which resembles solving multiple travelling salesman problems (TSP), as well as allocating goods to the vehicles that need to be delivered to customers, which resembles the knapsack problem. Therefore, even the basic version of CVRP is a cross of two NP-hard problems. The particular CVRP variants may operate with discrete or continuous time, time windows for loading and unloading, traffic, heterogeneous goods, specific client demands, the number of depots the vehicle start in, pickup and delivery, split delivery and many more. The goal, in the order of importance, is to: (1) deliver goods to the clients that meet their demand, (2) minimize the total length of the routes, (3) sometimes minimize the number of vehicles used. At the same time, all the constraints such as not exceeding the capacity of any vehicle must be preserved.

There have been several papers, in which an MCTS-based approach has been applied to a CVRP problem or a similar one such as Multi-Objective Physical TSP. Since we focus on MCTS in this survey and the approaches share many similarities regarding how this algorithm is applied, we grouped them into topics:

Problem simplification by having a “good” initial solution—featured in the articles by Powley et al. (2013b), Mańdziuk and Świechowski (2017, 2016), and Juan et al. (2013). Powley et al. (2013b) use a classical TSP solver for the initial solution and MCTS acted as a steering controller over it. Mańdziuk and Świechowski (2017, 2016) as well as Juan et al. (2013) generate the initial solution with the use of a modified Clarke and Wright savings algorithm. In the mentioned works by Mańdziuk and Świechowski (2017, 2016), the MCTS algorithm was used to test various incremental modifications of the initial solutions. Therefore, an action of the MCTS algorithm resembled a mutation operator in the evolutionary algorithm. Juan et al. (2013) employ the MCTS algorithm to generate a distribution of various starting solutions. Each solution is sent to a parallel processing node for further modification.

Problem factorization—featured in papers by Kurzer et al. (2018), as well as Mańdziuk and Świechowski (2017, 2016) and Mańdziuk (2018). Here instead of maintaining one MCTS tree for the complete solution, each vehicle (route) is attributed with a separate tree that MCTS iterates over. However, when the simulation takes into account effects of applying actions in each tree in a synchronized manner. The final decision, i.e., “the action to play” is a vector of actions from each tree. Macro-actions and action heuristics—featured in Powley et al. (2013b), Mańdziuk (2018), Mańdziuk and Świechowski (2017, 2016), and Kurzer et al. (2018). In each approach, the authors proposed actions that are not the atomic actions available in the model. This modification was commonly referred to in this survey as it drastically reduces the search space. Kurzer et al. (2018) interleave macro actions with regular actions.

Replacement of the UCT formula—featured in Mańdziuk and Nejman (2015) and Kurzer et al. (2018). In the latter work the UCT formula is replaced by a solution assessment formula designed for the CVRP problem to better distinguish between good and poor routes. Three gradation patterns were tested - hyperbolic, linear and parabolic. Kurzer et al. (2018) replace the UCT formula by the $\epsilon -Greedy$ strategy.

Depth limited search—featured in Powley et al. (2013b). The fitness function was introduced to evaluate non-terminal states.

7.6 Multi-domain MCTS with heuristics

Sabar and Kendall (2015) present a relatively general framework that combines MCTS with heuristics. Herein, MCTS tree is composed of nodes that correspond to low level heuristics applied to given states. Therefore, the whole approach can be considered as a hierarchical hyper- or meta-heuristic. The feasibility of the approach was tested across six different domains. The results were competitive with the state-of-the-art methods.

8 Parallelization

The MCTS algorithm consists of iterations shown in Fig. 1. The iterations as well as the respective phases within an iteration are performed sequentially in the classical variant. However, there have been many approaches to parallelization of the algorithm predominantly aimed at increasing the overall number of iterations per second. The more statistics gathered, the more confident the MCTS/UCT evaluation. In many traditional approaches to the parallel execution of algorithms such as matrix multiplication ones, the result of sequential and parallel versions must be exactly the same. However, in contrast to them, MCTS implementations drift away from the classic version of the algorithm in order to enable parallelism. Therefore, they can be considered variants or modifications of MCTS. For instance, the action selection formula may no longer be optimal, but multiple selections can be done at the same time and, as a result, the overall performance of the algorithm might be better.

8.1 Base approaches

There have been three major approaches to parallel MCTS proposed—Leaf Parallelization, Root Parallelization and Tree Parallelization.

• (1) Leaf Parallelization (Cazenave and Jouandeau 2007)—the simplest variant, in which there is one MCTS/UCT tree, which is traversed in the regular way in the selection phase. Once a new node is expanded, there are K independent playouts performed in parallel instead of just one. All of them start with the same newly added node. The results of simulations are aggregated, e.g. added or averaged, and back-propagated in the regular fashion as in the sequential algorithm. In Leaf Parallelization, the tree does not grow quicker compared to the non-parallel variant. However, the evaluation of new nodes is more accurate.

• (2) Root Parallelization (Chaslot et al. 2008a)—maintains K parallel processes. Each process builds its independent MCTS/UCT as the sequential algorithm would do. At certain synchronization points, e.g. just before making the actual (non-simulated) decision, the statistics from the first level in the tree are aggregated from the parallel processes. Typically, weighted aggregation is performed based on the number of visits. Although, this method does not result in deeper trees, the combined statistics of nodes are more confident. Moreover, Root Parallelization comes with minimal communication overhead.

• (3) Tree Parallelization (Cazenave and Jouandeau 2008)—like in Leaf Parallelization and in contrast to Root Parallelization, there is one MCTS/UCT tree. The tree is shared, iterated and expanded by K parallel processes. In contrast to Leaf Parallelization, the whole iteration is done by such process and not only the playout. Therefore, synchronization mechanisms such as locks and mutexes are used. In order to avoid selecting the same node by subsequent processes, the so-called Virtual Loss is usually used. It tells to assign the worst possible outcome for the active players as soon as the node is visited and correct this result in the backpropagation phase. Such a mechanism reduces the UCT score and discourages the algorithm to follow the previous choices until the correct result is back-propagated.

The algorithms that dynamically modify policies are particularly difficult to implement in parallel. Graf and Platzner (2015) show that manipulation of a playout policy can reduce the efficiency of parallelization. The authors use adaptive weights (c.f. Fig. 1 in Sect. 3) that are updated after each playout and shared among all threads. Although they lead to a stronger player, they require synchronization using a global lock which is harmful for computational performance. The performance ratio between adaptive and static playouts in case of 16 threads computations has been measured as 68%.

8.2 Using specialized hardware

Barriga et al. (2014) propose a parallel UCT search that runs on modern GPUs. In their approach, the GPU constructs several independent trees that are combined using the idea of Root Parallelism. In addition, each three is assigned a multiple of 32 GPU threads for leaf parallelism, i.e., simulations done in parallel from the same node. The method was tested using the game Ataxx. Various setups parameterized by the number of trees and threads have been tested . The best variant achieved a 77.5% win-rate over the sequential counterpart. However, there are many challenges to such an approach. Firstly, the game simulator must be implemented on the GPU, which is often infeasible depending on the complexity of the game. Secondly, the programmer has no control over thread scheduling and CPU/GPU communication, which can make the approach risky to apply in tournament scenarios that require precise time controls.

Mirsoleimani et al. (2015) study the MCTS method running in parallel on Intel Xeon Phi, which is a processing unit designed for parallel computations. Each unit allows shared memory scaling up to 61 cores and 244 threads. In their paper, there are three contributions outlined: (1) the analysis of the performance of three common threading libraries on the new type of hardware (Xeon Phi), (2) a novel scheduling policy and (3) a novel parallel MCTS with grain size control. The authors report 5.6 times speedup of the modified MCTS variant compared to the sequential version on the Xeon E5-2596 processor. The proposed variant is based on Tree Parallelization. The iterations are split into chunks, which are tasks to be executed serially.

Finally, Hufschmitt et al. (2015) present a parallelization method aimed for Multi-Purpose Processor Arrays (MPPA). Each MPPA chip contains 256 processing cores organized into 16 clusters. The authors investigate parallelization of various elements and their impact on the overall performance of the resulting player. Firstly, distributed propnet evaluation is proposed. The propnet is an inference engine responsible for game-state manipulations, i.e., computing the next state in the simulation. This type of parallelization ended up introducing too much of the synchronization overhead. Secondly, playouts parallelization was evaluated requiring fewer synchronization layers. The authors report “good” scaling in three games (TicTacToe, Breakthrough and EightPuzzle) with the number of clusters ranging from 1 to 16 and the maximum number of 16 threads per cluster. However, relatively high synchronization overhead was still reported and authors conclude that there are many ways, in which the approach could be further improved.

8.3 Lock-free parallelization

A “lock” is a mechanism that allows to give a thread the exclusive access to certain code section. It is to prevent deadlocks, races, crashes and inconsistent memory state. In the MCTS algorithm, there are two critical moments that are usually placed within the lock—expansion of a new node and back-propagation. The shared selection phase can also be a potential race condition scenario depending on the implementation. Mirsoleimani et al. (2018) propose a novel lock-free algorithm that takes advantage of the C++ multi-threading-aware memory model and the atomic keyword. The authors shared their work as an open source library that contains various parallelization methods implemented in the lock-free fashion. The efficacy was shown using the game of Hex and both Tree Parallelization and Root Parallelization.

8.4 Root–tree parallelization

The approach proposed by Świechowski and Mańdziuk (2016a) defines a way of combining Root- and Tree Parallelization algorithms that leads to a very high scalability. The setup consists of the so-called worker nodes, hub nodes and the master node, for the purpose of message passing and scalability. Each worker node maintains its own MCTS/UCT tree that is expanded using Leaf Parallelization. On top of them, there is a Root Parallelization method (running on hub and master nodes) that gathers and aggregates statistics from worker nodes in a hierarchical manner. In addition, the authors propose a novel way, called Limited Root–Tree Parallelization, of splitting the tree into certain subtrees to be searched by worker nodes. The subtrees are not entirely disjoint as some level of redundancy is desirable for Root Parallelization. This allows to push the scaling boundary further when compared with just the hierarchical combination of Root and Tree Parallelization methods. The resulting hybrid approach is shown to be more effective than either method applied separately in a corpus of nine GGP games.

8.5 Game specific parallelization

Parallelism can become an enabler to solving games, which are not too combinatorially complex. One such example is Hex (Arneson et al. 2010). Liang et al. (2015) propose an approach to solving Hex in a parallel fashion. The work builds upon the Scalable Parallel Depth-First Proof-Number Search (SPDFPN) algorithm, which has the limitation that the maximum number of threads that can be utilized cannot be greater than the number of CPU cores. The proposed Job-Level UCT search no longer has this limitation. The authors introduced various techniques aimed at optimizing the workload sharing and communication between the threads. The resulting solver is able to solve four openings faster than the previous state-of-the-art approach. However, solving the game universally is not yet feasible.

Schaefers and Platzner (2014) introduce a novel approach to parallelization of MCTS in Go. The novelty includes efficient parallel transposition table and dedicated compute nodes that support broadcast operations. The algorithm was termed UCT-Treesplit and involves ranking nodes, load balancing and certain amount of node duplication (sharing). It is targeted at HPC clusters connected by Infiniband. The MPI library is employed for message passing. The major advantage of using the UCT-Treesplit algorithm is its great scalability and rich configuration options that enable to optimize the algorithm for the given distributed environment.

8.6 Information set parallelization

Sephton et al. (2014) report a study aimed at testing the base parallelization methods, i.e. Leaf, Root and Tree, adapted to Information Set MCTS (ISMCTS). In addition, the Tree Parallelization was included in two variants—with and without the so-called Virtual Loss. The game of choice for the experiments is a strategic card game Lords of War. The authors conclude that the Root Parallelization method is the most efficient approach when combined with ISMCTS.

9 Conclusions

Table 1 References grouped by application

Table 2 References grouped by method, part 1/2, continued on Table 3

Table 3 References grouped by method, part 2/2, continuation from Table 2

MCTS is a state-of-the-art tree-search algorithm used mainly to implement AI behavior in games, although it can be used to support decision-making processes in other domains as well. The baseline algorithm, described in Sect. 2, was formulated in 2006, and since then multitude of enhancements and extensions to its vanilla formulation have been published. Our main focus in this survey is on works that have appeared since 2012, which is the time of the last major MCTS survey authored by Browne et al. (2012). Our literature analysis yielded 240 papers cited and discussed in this review, the vast majority of which fell within the above-mentioned time range. An overview of the considered papers grouped by application domains and by enhancements introduced to baseline MCTS are presented in Tables 1 and 2, respectively.

In their 2012 survey, Browne et al. (2012) concluded that MCTS would be “extensively hybridised with other search and optimisation algorithms”. In the consecutive 8 years this prediction was positively validated in many papers and, based on their analysis, the following main trends of such hybridized approaches can be distinguished:

• (1) MCTS + ML—combining MCTS with ML models, in particular deep neural networks trained with RL (Silver et al. 2016; Anthony et al. 2017), has become a significant research trend which is discussed in Sect. 5. Its popularity is to a large extent due to the unprecedented success of AlphaGo (Silver et al. 2016) and the rapid increase of interest in the ML field, in general. Further extensive growth in this direction can be safely envisaged.

• (2) AutoMCTS—an enhancement of MCTS with self-adaptation mechanisms (Świechowski and Mańdziuk 2014; Gaina et al. 2020), which rely on finding optimal hyperparameters or the most adequate policies for a given problem being solved. As discussed in Sect. 5, MCTS is often combined with optimization techniques (e.g. evolutionary approaches) that are executed offline in order to find the set of parameters for the online MCTS usage (Horn et al. 2016; Lucas et al. 2014; Pettit and Helmbold 2012; Bravi et al. 2016).

• (3) MCTS + Domain Knowledge—an inclusion of domain knowledge into MCTS operation, either in the form of the heavy playouts, or action prioritization, or exclusion of certain paths in the search space (action reduction), which are not viable or plausible due to the possessed expert knowledge. Other possibilities include building a light evaluation function for action assessment or early MCTS termination due to certain problem-related knowledge (Lorentz 2016).

Elaborating more on the last point, let us observe that MCTS has become the state-of-the-art technique for making strong computer players in turn-based combinatorial games. It has led to many breakthroughs, e.g. in Go and Chess. One of its main strengths is no need for game-related expert knowledge. In the vanilla form, only the simulation engine, i.e. rules of the game, is needed. Hence, it has been widely applied in multi-game environments such as GGP and General Video Game AI.

At the same time, embedding knowledge into MCTS without introducing too much bias is a potential way of improving its performance.

We expect that injecting knowledge into MCTS, especially in a semi-automatic fashion, is one of the future directions. We consider a direct inclusion of a human knowledge into the MCTS algorithm as a special case of domain knowledge utilization:

• (4) Human-in-the-Loop—a direct implosion of human expert domain knowledge to the MCTS operational pattern by means of a direct, online steering of the MCTS by the human operator (Świechowski et al. 2015a). A human expert has the power of excluding parts of the search tree, disable/enable certain actions or increase the frequency of visits in certain search space states (tree search nodes). In this context, it may be worth trying to develop some frameworks for semi-automatic translation of expert knowledge into MCTS steering or adjusting so as to make the above Human–Machine model of cooperation as effortless and efficient as possible. We believe that such direct teaming of the human expert (knowledge) with (the search speed of) MCTS has potential to open new research path in various complex domains, extending beyond combinatorial games or deterministic problems.

The fifth MCTS related research prospect that we envisage is

• (5) MCTS in New Domains—in principle, MCTS can be applied to any sequential decision-making problem that can be represented in the form of a tree whose leaf nodes (solutions) can be assessed using some form of an evaluation/reward function. This includes combinatorial optimization (Sabar and Kendall 2015), planning (Feldman and Domshlak 2014), and scheduling (Neto et al. 2020) problems encountered in a variety of fields, e.g. logistics (Kurzer et al. 2018), chemistry (Segler et al. 2018), security (Karwowski and Mańdziuk 2019a), or in UAV (Unmanned Aerial Vehicle) navigation (Chen et al. 2016). We are quite confident that MCTS will gradually become more and more popular in the areas other than combinatorial games.

Finally, one of the potential research prospect, which we regard more as an opportunity than a definite prediction, is

• (6) MCTS in Real Time Games—although MCTS has been applied to some real-time video games such as Pac-Man (Pepels et al. 2014) or Starcraft (Uriarte and Ontañón 2016), it is not as popular a tool of choice in this field as it is in combinatorial games. Rabin (2013) summarizes the techniques that are predominantly used in real-time video games, which include Behaviour Trees, Goal-Oriented Action Planning, Utility AI, Finite State Machines and even simple scripts. The main reason for using them is the limited computational budget. Video games tend to have quite complex state representations and high branching factors. Moreover, the state can change rapidly in real-time. In research environment, it is often possible to allocate all available resources to the AI algorithms. In commercial games, the AI module is allowed to use only a fraction of computational power of a single consumer-level machine.

Besides computational complexity, the other reason for MCTS not to be the first choice in video games is the degree of unpredictability introduced by stochastic factors. The game studios often require all interactions to be thoroughly tested before a game is released. Developing more formal methods of predicting and controlling the way MCTS-driven bots will behave in new scenarios in the game seems to be another promising future direction.

The third impediment of MCTS application to video games is the difficulty of implementing a forward model able to simulate games in a combinatorial-like fashion (Preuss and Risi 2020). This issue has already been addressed by using simplified (proxy) game models and letting the MCTS algorithm operate either on abstraction (simplification) of the actual game or on some subset of the game (using a relaxed model).

In the view of the above, we believe that some methodological and/or computational breakthroughs may be required before MCTS becomes a state-of-the-art (or sufficiently efficient) approach in RTS games domain.

In summary, the main advantages of MCTS which differentiate this method from the majority of other search algorithms can be summarized as follows. First of all, the problem tree is built asymmetrically, with the main focus on the most promising lines of action and exploring options that may most likely become promising. Second of all, the UCT formula is the most popular way of assuring the exploration vs. exploitation balance. In practice, the method has proven efficient in various (though not all) search problems. Its additional asset is the theoretical justification of convergence. Furthermore, the method is inherently easy to parallelize (cf. Sect. 8). And finally, MCTS is the so-called anytime algorithm, i.e. it can be stopped at any moment and still return a reasonably good solution (the best one found so-far).

On the downside, since MCTS is a tree-search algorithm, it shares some weaknesses with other space-search. For instance, it is prone to search space explosion. MCTS modifications, such as model simplifications or reducing the actions space are one way to tackle this problem. A different idea is to use MCTS as an offline teacher that can be used for sufficiently long time and generate high-quality training data. Yet another method that is not search-based but rather learning-based and inherently capable of generalizing knowledge (e.g. an ML model) is trained on these MCTS-generated training samples. Consequently, a computationally heavy process is run just once (offline) and then this time-efficient problem representation can be used in subsequent online applications. The approach of combining an MCTS trainer with a fast learning-based representation can be hybridised in various ways, specific to a particular problem/domain of interest (Kartal et al. 2019a; Guo et al. 2014; Soemers et al. 2019).

Finally, we would like to stress the fact that MCTS, originally created for Go, was for a long time devoted to combinatorial games and only recently started to “expand” its usage beyond this domain. We hope that the researchers working with other genres of games or in other fields will make more frequent attempts at MCTS utilization in their domains, perhaps inspired by the MCTS modifications discussed in this survey.