TVVS: A top-view visualization system from broadcasting soccer video

Abstract

A holy grail for sports analytics is the top-view visualization of the game. The top-view visualization provides the actual between-player distances as opposed to the between-player distances calculated from the side and/or oblique view of a match. Related work in this area relies on multiple camera installations in the stadium or directly derive the registration map between a broadcasting video and the top-view model. Aberrating the state-of-the-art, a factor theory based approach is presented to derive the top-view visualization of the game from the broadcasting sports video. It is theoretically proved that the proposed factor theory based approach is more efficient than the state-of-the-art approach for the top-view visualization. In addition, as per the proposed approach, a model is presented for the top-view visualization by transforming the broadcasting video into a single and static camera visualization. In order to generate the single-camera visualization, the view of the entire ground is needed which is expressed as a solution to a convex optimization function, devised to explore putative matrix completions. To give pristine empirical evidence, the benchmark dataset is used and a soccer dataset has been introduced towards the end. The proposed top-view approach brings atleast 7% and 10% gains over the state-of-the-art on the benchmark and the proposed dataset respectively.

Appendices

Appendix 1

Let F^∗ be an arbitrary approximation of F directly derived by a model from (b) into (d) in Fig. 2. Then for any F^∗, there exists a H and T such that (HT) is a better approximation of F.

Proof

Let we have the groundtruth F that registers the players of the video frame into the top view model. Given a video frame, the problem is to approximate F. Let, the state-of-the-art method approximates F as F^∗ defined from (b) into (d). F^∗ registers players from the video frame into the top-view model with approximation error 𝜖 i.e. ∥F − F^∗∥ = 𝜖, where ∥.∥ is the Frobenius matrix norm. Given arbitrary F^∗, we aim to construct T and H such that (HT) is a better approximation of F. More precisely, given arbitrary F^∗, there exists T and H such that (HT) has approximation error less than 𝜖. i.e. ∥F − (HT)∥ < 𝜖. The proof is comprised of three steps. First, we construct H and T for a given F which depends on the value of a variable s. Then we derive the error bound for the proposed approximation (HT) involving s. Finally, the value of s is computed for the given 𝜖. Using the value of s, H and T can be computed specifically with approximation error less than 𝜖. Next we expound the details.

Construction of H and T The task of computing a factor form approximation of F can be divided into two steps. The first is to construct a subspace that captures the range of F. The second is to restrict the F to the subspace and compute a standard factorization of the reduced F with the help of H. Next, we discuss how to accomplish the proposed steps.

The first step can be executed with random sampling methods [27]. To understand how randomness works, let us consider F = B + E, where B captures the range of F and E is a small perturbation error during groundtruth generation process. Our aim is to obtain a basis of exact rank r that covers as much of the range of B as possible.

Let us consider the dimension of F, B and E are (a × b). In order to obtain r rank approximation of B, we fix a small number s. Then (r + s) random vectors {α_i} can be generated such that:

$$ F(\alpha_{i})= B(\alpha_{i}) + E(\alpha_{i}), $$

(6)

for i = 1,..., (r + s). The perturbation E deviates the direction of each {α_i} outside the range of B. Therefore, the extra s vectors enhance the chance of spanning the required subspace. Overall the general randomized algorithm to derive the H is comprised of three steps [11] as follows.

First, a random (b × (r + s)) matrix ${\mathscr{G}}$ is generated whose columns are Gaussian vectors. Thereafter compute $(F{\mathscr{G}})$. Finally, construct a matrix H whose columns form an orthonormal basis of the range $(F{\mathscr{G}})$. Once we get the H, then we can compute the other factor (H^∗F). i.e. F ≈ H(H^∗F). Considering T = H^∗F, we approximate F in factor form of (HT). Next, we compute the bound of approximation error of ∥F − HT∥. There after, we determine the value of s that is needed to compute H and T so that the approximation error is less than 𝜖.

Computing the error bound We aim to show:

$$ E(\|{F-H(H^{*}F)}\|)) \leq \left( 1+ \frac{r}{s-1}\right) \left( {\sum}_{i=r+1}^{\min{(a,b)}} {\sigma_{i}^{2}}\right) $$

where E is the expectation, ${\sum }_{i=r+1}^{\min \limits {(a,b)}} \sigma _{i}^2$ is the theoretically minimal error in approximating F by a matrix of rank r [11].

First, consider the singular value decomposition of F as $F= U_{1} {\Sigma }_{1} V_{1}^{*}$, where U₁ is a (a × r) orthonormal matrix, Σ₁ is a diagonal matrix containing the non negative singular values of F and V₁ is a (r × n) orthonormal matrix. We call U₁ and V₁ as left unitary factor and right unitary factor respectively. First partition the Σ₁ = [Σ₂|Σ₃], where the Σ₂ and Σ₃ are the diagonal matrix containing the first r and (b − r) singular values respectively. Thereafter, partition V₁ = [V₂|V₃] into blocks containing r and b − r columns respectively. Define ${\mathscr{G}}_{2} = V^{*}_{2}{\mathscr{G}}$ and ${\mathscr{G}}_{3} = V^{*}_{3}{\mathscr{G}}$. Since, V₂ and V₃ are orthonormal, then ${\mathscr{G}}_{2}$ and ${\mathscr{G}}_{3}$ are also Gaussian. We denote the pseudoinverse of ${\mathscr{G}}_{2}$ and ${\mathscr{G}}_{3}$ as $\hat {{\mathscr{G}}_{2}}$ and $\hat {{\mathscr{G}}_{3}}$ respectively. ${\mathscr{G}}_{2}$ and ${\mathscr{G}}_{3}$ are non overlapping, so they are stochastically independent. Applying Holder’s inequality, we can write:

$$ E(\|{F-H(H^{*}F)}\|))\leq (E(\|{F-H(H^{*}F)}\|^{2}))^{1/2} $$

(7)

It is proved in [11] that:

$$ E(\|{F-H(H^{*}F)}\|^{2}) \leq (\|{{\Sigma}_{3}}\|^{2}_{F} + E(\|{{\Sigma}_{3}\mathscr{G}_{3} \hat{\mathscr{G}_{2}})}\|^{2}) $$

(8)

Therefore, using (7) and (8), we can write:

$$ E(\|{F-H(H^{*}F)}\|))\leq (\|{{\Sigma}_{3}}\|^{2}_{F} + E(\|{{\Sigma}_{3}\mathscr{G}_{3} \hat{\mathscr{G}_{2}})}\|^{2})^{1/2} $$

(9)

We are interested in the r ranks of the matrix. Therefore we compute $E(\|{{\Sigma }_{3}{\mathscr{G}}_{3} \hat {{\mathscr{G}}_{2}}}\|^{2})$ by conditioning on the value of ${\mathscr{G}}_{2}$ as follows:

$$ E(\|{{\Sigma}_{3}\mathscr{G}_{3} \hat{\mathscr{G}_{2}}}\|^{2}) = E(E(\|{{\Sigma}_{3}\mathscr{G}_{3} \hat{\mathscr{G}_{2}}}\|^{2})|\mathscr{G}_{2}) $$

(10)

The Frobenious norm is unitarily invariant. i.e. for any two orthonormal matrices U₁ and V₁, we can write ∥U₁Σ₁V₁∥ = ∥Σ₁∥. In addition, the distribution of a Gaussian matrix is invariant under orthogonal transformations. Therefore, we can write:

$$ \begin{array}{@{}rcl@{}} E(E(\|{{\Sigma}_{3}\mathscr{G}_{3} \hat{\mathscr{G}_{2}}}\|^{2})|\mathscr{G}_{2}) & =& E(E({\Sigma}_{jk}(\sigma_{jj} [\mathscr{G}_{3}]_{jk} [\hat{\mathscr{G}_{2}}]_{kk})) \\ & =& E({\Sigma}_{jk}(\sigma^{2}_{jj} [\hat{\mathscr{G}_{2}}]^{2}_{kk})) \\ & =& E (\|{{\Sigma}_{3}}\|_{F}^{2} \|{\hat{\mathscr{G}_{2}}}\|^{2}) \\ & =& \|{{\Sigma}_{3}}\|_{F}^{2} E(\|{\hat{\mathscr{G}_{2}}}\|^{2}) \\ & =& \frac{r}{s-1} \|{{\Sigma}_{3}}\|_{F}^{2} \\ & =& \frac{r}{s-1}\left( \sum\limits_{i=r+1}^{\min{(a,b)}}{\sigma_{i}^{2}}\right) \\ \end{array} $$

(11)

Therefore, putting the expression of $E(\|{{\Sigma }_{3}{\mathscr{G}}_{3} \hat {{\mathscr{G}}_{2}}}\|^{2})$ in the (9), we can write:

$$ E(\|{F-H(H^{*}F)}\|)) \leq \left( 1+ \frac{r}{s-1}\right) \left( \sum\limits_{i=r+1}^{\min{(a,b)}} {\sigma_{i}^{2}}\right) $$

(12)

In (12), ${\sum }_{i=r+1}^{\min \limits {(a,b)}} \sigma _{i}^2$ is the theoretically minimal error in approximating F by rank r [11]. Therefore, the optimal bound is missed by a factor of $(1+ \frac {r}{s-1})$. Now, our objective is to determine the value of s for a given 𝜖.

Computing the value of s Our proposed error for factorized approximation is less than 𝜖. The target rank r is strictly greater than 1. Therefore from (12), we can write:

$$ \begin{array}{@{}rcl@{}} \left( 1+ \frac{r}{s-1}\right) \left( \sum\limits_{i=r+1}^{\min{(a,b)}} {\sigma_{i}^{2}}\right) & <& \epsilon \\ \left( 1+ \frac{r}{s-1}\right) & <& \frac{\epsilon}{({\sum}_{i=r+1}^{\min{(a,b)}} {\sigma_{i}^{2}})} \\ \frac{r}{s-1} & <& \frac{\epsilon}{({\sum}_{i=r+1}^{\min{(a,b)}} {\sigma_{i}^{2}})} -1 \\ \frac{r}{s-1} & <& \frac{\epsilon-({\sum}_{i=r+1}^{\min{(a,b)}} {\sigma_{i}^{2}})}{({\sum}_{i=r+1}^{\min{(a,b)}} {\sigma_{i}^{2}})} \\ \frac{r({\sum}_{i=r+1}^{\min{(a,b)}} {\sigma_{i}^{2}})}{\epsilon-({\sum}_{i=r+1}^{\min{(a,b)}} {\sigma_{i}^{2}})} & <& s-1 \\ \frac{r({\sum}_{i=r+1}^{\min{(a,b)}} {\sigma_{i}^{2}})}{\epsilon-({\sum}_{i=r+1}^{\min{(a,b)}} {\sigma_{i}^{2}})}+1 & <& s \\ \end{array} $$

(13)

Therefore, we can choose $s= \Bigg \lceil {\frac {r({\sum }_{i=r+1}^{\min \limits {(a,b)}} \sigma _{i}^2)}{\epsilon -({\sum }_{i=r+1}^{\min \limits {(a,b)}} \sigma _{i}^2)}+1}\Bigg \rceil$, where ⌈.⌉ function gives the least integer greater than or equal to the given input.

The upshot Putting the value of s we can compute the H and T = H^∗F such that HT is an approximation of F with approximation error less than 𝜖. Thus we prove that for a given F and an arbitrary state-of-the-art approximation of F say F^∗, we can always find an approximation in factor form of (HT) which is better than F^∗. □

Appendix 2

Lemma 1: Let $X^{*}_{\kappa }$ be the solution to (3) and $X_{\infty }$ be the minimum Frobenius norm solution to (1) defined as $X_{\infty } := \{\underset {X}{ argmin}\|{X}\|^2_F: X \text {is a solution of}$ (1)}. Then $\lim _{\kappa \to \infty } \|{X^{*}_{\kappa }-X_{\infty }}\|_F=0$.

Proof

From the definition of $X^{*}_{\kappa }$, we can write

$$ \|{X^{*}_{\kappa}}\|_{*} + \|{X^{*}_{\kappa}}\|^{2}_{F} \leq \|{X_{\infty}}\|_{*} + \|{X_{\infty}}\|^{2}_{F}. $$

(14)

From the definition of $X_{\infty }$ we can write:

$$ \|{X_{\infty}}\|_{*} \leq \|{X^{*}_{\kappa}}\|_{*}. $$

(15)

From the 14 and 15, we can write:

$$ \|{X^{*}_{\kappa}}\|^{2}_{F} \leq \|{X_{\infty}}\|^{2} $$

(16)

The (16) implies that $X^{*}_{\kappa }$ is uniformly bounded. Now, the theorem is proved if we can show that any convergent subsequence $\{X^{*}_{\kappa _{i}}\}_{i \geq 1}$ must converge to $X_{\infty }$.

Consider an arbitrary converging subsequence $\{X^{*}_{\kappa _{i}}\}$ and set $X_c= \lim _{i \to \infty } X^{*}_{\kappa _{i}}$. Since $X^{*}_{\kappa }$ is uniformly bounded, we can write:

$$ \lim_{\kappa \to \infty} sup \|{X^{*}_{\kappa}}\| \leq \|{X_{\infty}}\|_{*} $$

(17)

and

$$ \|{X^{*}_{\kappa}}\|_{*} \leq \lim_{\kappa \to \infty} inf \|{X_{\infty}}\|_{*} $$

(18)

From Eq. 17 and 18, we can write $\lim _{\kappa \to \infty } \|{X^{*}_{\kappa }}\| = \|{X_{\infty }}\|_{*}$, therefore, $\|{X_{c}}\|_{*}= \|{X_{\infty }}\|_{*}$. This shows that X_c is a solution of $\underset {X}{ argmin}\|{X}\|_{*}$. Now it follows from the definition of $X_{\infty }$ that $\|{X_c}\|_F \geq \|{X_{\infty }}\|_{F}$, while we also have $\|{X_c}\|_F \leq \|{X_{\infty }}\|_F$ because of (18). Therefore, we conclude that $\|{X_c}\|_F = \|{X_{\infty }}\|_{F}$ and thus $X_{c} = X_{\infty }$ since $X_{\infty }$ is unique. □

Lemma 2: For each κ > 0 and an (n₁ × n₂) image Y , algo. 2 converges to: $\underset {X}{ argmin} \|{X-Y}\|^2_F+\kappa \|{X}\|_{*}$.

Proof

Essentially, we have to prove that:

$O_{\kappa }(Y)=\underset {X}{ argmin} \|{X-Y}\|^2_F+\kappa \|{X}\|_{*}$. Let us assume $M(X)= \underset {X}{ argmin} \|{X-Y}\|^2_F+\kappa \|{X}\|_{*}$. M is strictly convex. So there exists a unique minimizer of M. Therefore, we need to prove that the minimizer is equal to O_κ(Y ). To do this, the definition of a subgradient of a convex function is as follows [2]: Z is a subgradient of M at X₀, denoted as Z ∈ ∂M(X₀), if

$$ M(X) \geq M(X_{0}) + \langle Z, (X - X_{0}) \rangle $$

(19)

for all X. Now O_κ(Y ) minimizes M if and only if 0 is a subgradient of the M at the point O_κ(Y ). i.e.

$$ 0 \in O_{\kappa}(Y)-Y + \kappa \partial \|{O_{\kappa}(Y)}\|_{*}, $$

(20)

where ∂∥O_κ(Y )∥_∗ is the set of subgradients of the nuclear norm. Let X be an arbitrary image and SVD of X = UΣV^T. Then we can write [23] [42]:

$\partial \|{X}\|_{*} = \{UV^{T} + W : W \in \mathbb {R}^{n_1 \times n_2}, U^{T}W = 0, WV = 0, \|{W}\|_{F} \leq 1\}$.

In order to show that O_κ(Y ) obeys (20), decompose the SVD of Y as: $Y = U_{3} {\Sigma }_{3} {V_{3}^{T}} + U_{4} {\Sigma }_{4} {V_{4}^{T}}$, where U₃, V₃ (resp. U₄, V₄) are the singular vectors associated with singular values greater than κ (resp. smaller than or equal to κ ). With these notations, we have $O_{\kappa }(Y) = U_{3} ({\Sigma }_{3} - \kappa I) {V_{3}^{T}}$.

Therefore, $Y - O_{\kappa }(Y) = \kappa (U_{3} {V_{3}^{T}} + W)$, and thus $W = \kappa ^{-1} U_{4} {\Sigma }_{1} {V_{4}^{T}}$.

By definition, ${U^{T}_{0}}W = 0$, WV₃ = 0 and since the diagonal elements of ${\sum }_{1}$ have magnitudes bounded by κ, we also have ∥W∥₂ ≤ 1. Hence Y − O_κ(Y ) ∈ κ∂∥O_κ(Y )∥_∗, which concludes the proof. □