Strategy Optimization in Sports via Markov Decision Problems

Abstract

In this paper, we investigate a sport strategic question related to the Olympics final in beach volleyball: Should the German team play most aggressively to win many direct points or should it rather play safer to avoid unforced errors? This paper introduces the foundations of our new two-scale approach for the optimization of sport strategic decisions. We present possible answers to the benchmark question above based on a direct approach and the presented two-scale method. A comparison shows the benefits of the new paradigm.

Appendices

Appendix 1

In this section, we give the details of the proofs for Theorems 2 and 1. One important observation is the following plausible lemma that holds for both special cases of s-MDPs. It says that it is no disadvantage for us when we have more points or the opponent has fewer points.

Lemma 1

The optimal expected reward-to-go $v^*(x,y,k,\ell )$ satisfies $v^*(x,y,k, \ell ) \le v^* (x+1,y,k, \ell )$ and $v^*(x,y,k, \ell ) \ge v^* (x,y+1,k, \ell )$.

Proof

We prove this by comparing all possible realizations of the game separately. First of all, the outcome of future rallies does not depend on the score. Each winning scenario starting from state $(x,y,k,\ell )$ corresponds to a winning scenario with identical transitions starting from state $(x+1,y,k,\ell )$ with one stage less that has at least the same probability. Thus, the total winning probability starting from $(x+1,y,k,\ell )$ is no smaller than the one starting in $(x,y,k,\ell )$. Moreover, each losing scenario starting from state $(x,y,k,\ell )$ corresponds to a losing scenario with identical transitions starting from state $(x,y+1,k,\ell )$ with one stage less that has at least the same probability. Thus, the total losing probability starting from $(x,y+1,k,\ell )$ is no smaller than the one starting in $(x,y,k,\ell )$. The claim expresses exactly this in terms of the optimal reward-to-go in the respective states.

In the previous lemma we compared the winning probabilities in states with identical service components. We now explain why the winning probability increases when we win the next point.

Lemma 2

The optimal expected reward-to-go satisfies $v^*(x+1,y,P,1) \ge v^*(x,y+1,Q,1)$.

Proof

Team $P$, in order to win starting at state $(x,y+1,Q,1)$, first has to reach a score of $x+1$ at some point in time. Thus, the main observation, denoted by ($*$), is that all winning scenarios starting from state $(x,y+1,Q,1)$ pass through exactly one of the states $(x+1,y+z,P,1)$, $z = 1, \dots , 21-y$. Let W be the event that $P$ wins, let E be the event that state $(x,y+1,Q,1)$ is passed, and for $z = 1, \dots , 21-y$ let $E_z$ be the event that state $(x+1,y+z,P,1)$ is passed. Then we compute:

$$\begin{aligned} v^*(x,y+1,Q,1)&= \text {Prob}(W|E)\nonumber \\&= \sum _{z=1}^{21-y} \text {Prob}(E_z|E) \text {Prob}(W|E_z) \quad \text {Markov-Property and}~*\\&= \sum _{z=1}^{21-y} \text {Prob}(E_z|E) v^*(x+1,y+z,P,1)\nonumber \\&\le \sum _{z=1}^{21-y} \text {Prob}(E_z|E) v^*(x+1,y,P,1) \quad \text {Lemma~1 and induction}\\&\le v^*(x+1,y,P,1) \qquad \text {by}~*. \end{aligned}$$

Thus, an optimal policy is myopic:

Corollary 1

The policy that always maximizes the probability to win the next point is optimal for the s-MDP for beach volleyball.   $\square$

Appendix 2

This appended section defines the details for the infinite-horizon, stationary g-MDP for a beach volleyball rally that was sketched in Sect. 6.

Let $P$ and $Q$ be the teams participating in the game. $P_1$ and $P_2$ are the players of $P$; $Q_1$ and $Q_2$ are the players of $Q$. Team $P$ is the team for which we want to choose an optimal playing strategy, whereas team $Q$ is the uncontrolled opposing team. That means, as in the s-MDP, team $P$ is the decision making team, and the behaviour of team $Q$ is part of the system disturbance in the transition probabilities. We have decision epochs $T = \{1, 2, 3, \ldots \}$, and $t \in T$ is the total number of ball contacts minus the blocking contacts in the rally so far.

A state in the g-MDP is a tuple that contains the players’ positions, the ball’s position, a counter of the number of contacts, the information which player last contacted the ball, a Boolean variable that indicates the hardness of the last hit, and the designated blocking player of the defending team for the next attack. A general formulation for a state is

$$\begin{aligned} (\text {\textit{pos}}(P_1), \text {\textit{pos}}(P_2),\text {\textit{pos}}(Q_1), \text {\textit{pos}}(Q_2),\text {\textit{pos}}(\text {\textit{ball}}), \text {\textit{counter}},\text {\textit{lastContact}},\textit{hard},blocker ). \end{aligned}$$

The function $\text {\textit{pos}}(\cdot )$ returns the position of a player or the ball. A position on the court is defined on basis of the grid presented in Fig. 9.

Fig. 9

Court grid

The components $\text {\textit{counter}}$ and $\text {\textit{lastContact}}$ are needed to implement the three-hits and the double contact rule respectively. The state variable $\text {\textit{counter}}$ can take values from the set $\{-1, 0,1,2,3\}$. The case “$-1$’ marks a service state. This way it is possible to forbid a blocking action on services. The counter stays $-1$ if the ball crosses the net after a serve. This helps to distinguish between a reception or defence action. Consequently, if the counter is 0, the ball crossed the net via an attack-hit performed in a field attack. The information which player last contacted the ball is needed to implement the double-contact fault into the model. The state variable $\text {\textit{lastContact}}$ takes values in $\{P_1, P_2, Q_1, Q_2, \emptyset \}$. If the ball has just crossed the net or the state is a serving state, a $\emptyset$-symbol shows that both players are allowed to execute the next hit. The Boolean state variable $\textit{hard}$ indicates the power of the last hit. If $\textit{hard}= 1$, then the ball has a high speed when reaching the field, else the ball has normal speed. Finally, the state variable $blocker$ takes values in $\{P_1, P_2, Q_1, Q_2\}$ and indicates the designated blocking player of the currently defending team. It is necessary to save it in the state since the decision who blocks is made once at the beginning of the opponents attack plan and followed more than one time step. Besides these generic states, the g-MDP contains the absorbing states $\text {\textit{point}}$ and $\text {\textit{fault}}$, where $\text {\textit{point}}$ and $\text {\textit{fault}}$ is denoted from the perspective of team $P$. The resulting g-MDP has around one billion different states. As an example $(P02, P33, Q12, Q13, P02, -1, \emptyset , 0,-)$ is a typical serving state for team $P$.

Of course, some of the states occur more often in practice than others. Depending on the current state, there are different actions available to each player. The individual player actions of a player $\rho$ consist of a hit $h$ and a move $\mu$. We distinguish between a one-field and a two-field movement. Also, the direction (f:= forward, fr:= forward-right, ...) of the movement matters. A blocking action belongs to the group of movements since ball possession is not required to perform a block. A blocking action can only be performed if the player is in a field at the net. All possible moves for team $P$ are listed in Table 13. The moves of the players that belong to team $Q$ are defined analogously.

Table 13 Move specification for $\rho$ belonging to team $P$

Depending on the position of a player and on the position of the ball relative to the player, each player has a set of available hits. Sometimes, this set can consist solely of the hit no hit. A hit $h^{\text {\textit{tech}}}_{\text {\textit{field}}}$ is defined by a hitting technique $\text {\textit{tech}}$ and a target field $\text {\textit{field}}$. Depending on the hit’s degree of complexity, there are different requirements such that the hit is allowed in the model. The function $\text {\textit{neighbour}}(\text {\textit{field}})$ returns a set of all neighbouring fields of $\text {\textit{field}}$ according to the grid presented in Fig. 9 and the field itself. All hitting techniques with their possible target fields and requirements are listed in Table 14. The hitting techniques for a player of team $Q$ are defined analogously.

There are rules in the model that restrict the possible combinations of a hit with a move to a player action as well as rules that restrict the possible combinations of two player actions to a team action. Reasons for these restrictions are practical considerations. There are three rules on combining a hit with a movement to a player action. The first one is: If a player makes a real hit, i.e., a hit that is not no hit, due to timing reasons only a one-field movement is allowed. The second one is: If a player makes a hit that is performed with a jump, like, e.g., a jump serve, only a one-field movement in forward direction (i.e., towards the net) can follow. The third one is: If the hit requires a movement before executing the hit, no additional movement afterwards is allowed. This is, e.g., the case for a reception that takes place in a neighbouring field of the hitting player. We incorporate one restriction to the combination of player actions: If two player actions are combined to a team action, only one player may make a real hit. Team actions that themselves or whose player actions do not follow these rules are not available in the model—for both teams. Further conceivable restrictions could be easily implemented in the model whenever they only depend on the current state.

Table 14 Hit specification for player $\rho$ of team $P$ and ball $\text {\textit{ball}}$; requires always $\rho \ne \text {\textit{lastContact}}$ except the action no hit ($^\star$ if $\text {\textit{pos}}(\text {\textit{ball}}) \ne \text {\textit{pos}}(\rho )$ then no movement afterwards allowed)

Transition probabilities determine the evolution of the system if a certain action in a certain state is chosen. Assume, we know for each player $\rho$ and each hitting technique $h^{\text {\textit{tech}}}_{\textit {target }}$ the probability

$$\begin{aligned} p_{\text {\textit{succ}},\rho }\left( \text {\textit{pos}}(\rho ),\ h^{\text {\textit{tech}}}_{\textit {target }} \right) := \mathbb {P}\bigl (\text {\textit{pos}}^{t+1} (\text {\textit{ball}}) = \textit {target }\ | \ \text {\textit{pos}}^t(\rho ), h^{\text {\textit{tech}}}_{\textit {target }}\bigr ), \end{aligned}$$

i.e., the probability that the specified target field $\textit {target }$ from $\rho$’s position at time t is met. In the notation used above, the terms $\text {\textit{pos}}(\rho )$ and $h^{\text {\textit{tech}}}_{\textit {target }}$ show the dependence on the position of the hitting player and the hit he uses. The probability is time-independent. The t on the right-hand side of the last equation is only used to indicate that $\text {\textit{pos}}^{t}(\rho )$ is the position of $\rho$ at time t while $\text {\textit{pos}}^{t+1}(\text {\textit{ball}})$ is the position of the ball in the subsequent state. Similarly, assume, we know the probability of an execution fault

for player $\rho$ using hit $h^{\text {\textit{tech}}}_{\textit {target }}$ from position $\text {\textit{pos}}(\rho )$. An execution fault includes hits where the ball is not correctly hit such that the referee terminates the rally. For serves and attack-hits an execution fault also includes that the ball is hit into the net.

Furthermore, assume that we know the blocking skills of each player. The parameter $p_{\textit{block}, \rho }$ denotes the probability that player $\rho$ touches the ball when performing the block b against an adequate attack-hit from the opponent’s side of the court. The probability $p_{\textit{block}, \rho }$ is independent of the skills of the attacking player. There are three possible outcomes of that block. The block can be so strong that it is impossible for the opponent team to get the returned ball, and the blocking team wins the rally. This probability is denoted by $p_{\textit{block},\rho ,point }$. Also, the block can result in a fault with probability $p_{\textit{block},\rho ,fault }$. That happens if the ball is blocked into the net and can not be regained or the blocking player touches the net, which is an execution fault. None of the above happens with probability $p_{\textit{block},\rho ,ok }:= p_{\textit{block}, \rho }- p_{\textit{block},\rho ,point }- p_{\textit{block},\rho ,fault }$. This is called an “ok”-block, and the ball lands in one random field on the opponents or own court side. We define $p_{\textit{no block}, \rho }:=1-p_{\textit{block}, \rho }$ as the probability that the blocking player fails to get his hands at the ball. In this case, the landing field of the ball is not affected by the block. In total, the blocking probabilities are

$$\begin{aligned} p_{\textit{no block}, \rho }+ \underbrace{p_{\textit{block},\rho ,point }+ p_{\textit{block},\rho ,fault }+ p_{\textit{block},\rho ,ok }}_{p_{\textit{block}, \rho }}= 1. \end{aligned}$$

From all these input probabilities, we generate all transition probabilities in the g-MDP@. We explain how the next state evolves from the current state and the played team actions: The next player’s position depends only on the current position and the movement the player makes. An allowed movement will always be successful. The crucial component is the next position of the ball. Here, the individual skills of the hitting player enter the model. Assume first, no player of the opposing team is blocking. Then with probability $p_{\text {\textit{succ}},\rho }\left( \text {\textit{pos}}(\rho ),\ h^{\text {\textit{tech}}}_{\textit {target }} \right)$ the ball’s next position will be the desired target field, and with probability $p_{\text {\textit{fault}},\rho }\left( \text {\textit{pos}}(\rho ),\ h^{\text {\textit{tech}}}_{\textit {target }} \right)$ the hitting player makes an execution fault. The remaining probability

$$\begin{aligned} 1-p_{\text {\textit{succ}},\rho }\left( \text {\textit{pos}}(\rho ),\ h^{\text {\textit{tech}}}_{\textit {target }} \right) -p_{\text {\textit{fault}},\rho }\left( \text {\textit{pos}}(\rho ),\ h^{\text {\textit{tech}}}_{\textit {target }} \right) =: p_{\text {\textit{dev}},\rho }\left( \text {\textit{pos}}(\rho ),\ h^{\text {\textit{tech}}}_{\textit {target }} \right) \end{aligned}$$

will be the probability that the ball lands in a neighbouring field of the target field. We assume each neighbouring field is equally probable.

If the hit is an attacking hit to the opponent’s court side, then the ball may be blocked. The blocking action must be made from an adequate position³ such that the block can have an impact. If all preconditions are fulfilled, we first evaluate whether the hit is successful. A hit is successful if no execution fault occurs, the ball crosses the net, and approaches the target field or one of its neighbours with the respective probabilities. Given a successful attack, we evaluate in the next step the result of the block. If the blocking player does not touch the ball, then the next position of the ball will not be affected by the block. Otherwise, the outcome of the block is evaluated according to the blocking skill of that player and may be a point, fault or a different position of the ball. This need not automatically mean a point for the attacking team, since the defending team may perform a successful defence action in the next time step. Finally, in case of an execution fault or if the ball is not hit by any player, then the next state will be $\text {\textit{point}}$ or $\text {\textit{fault}}$, respectively, from the perspective of team $P$.

Appendix 3

Table 15 Direct estimation of s-MDP probabilities for team Q