Backtesting VaR and expectiles with realized scores

Abstract

Several statistical functionals such as quantiles and expectiles arise naturally as the minimizers of the expected value of a scoring function, a property that is called elicitability (see Gneiting in J Am Stat Assoc 106:746–762, 2011 and the references therein). The existence of such scoring functions gives a natural way to compare the accuracy of different forecasting models, and to test comparative hypotheses by means of the Diebold–Mariano test as suggested in a recent work. In this paper we suggest a procedure to test the accuracy of a quantile or expectile forecasting model in an absolute sense, as in the original Basel I backtesting procedure of value-at-risk. To this aim, we study the asymptotic and finite-sample distributions of empirical scores for normal and uniform i.i.d. samples. We compare on simulated data the empirical power of our procedure with alternative procedures based on empirical identification functions (i.e. in the case of VaR the number of violations) and we find an higher power in detecting at least misspecification in the mean. We conclude with a real data example where both backtesting procedures are applied to AR(1)–Garch(1,1) models fitted to SP500 logreturns for VaR and expectiles’ forecasts.

Appendix

We report in this Appendix the explicit computation of the expected values and variances of piecewise linear and quadratic scores and identification function in the normal and uniform cases. That is, we compute the quantities $E[S^{(v)}(x,Y)]$, $E\left[ S^{(e)}(x,Y)\right]$, $E[I^{(v)}(x,Y)]$, $E[I^{(e)}(x,Y)]$ and the corresponding variances when $Y\sim N(0,1)$ and $Y \sim U(0,1)$. We denote with $\phi (x)$, $\varPhi (x)$ and ${\overline{\varPhi }}(x)$ the density, the cumulative and the retro-cumulative function of a standard normal r.v.

The following identities will be repeatedly used:

$$\begin{aligned} \int _{-\infty }^{x}y\phi (y)dy&=-\,\phi (x), \int _{x}^{+\infty }y\phi (y)dy=\phi (x) \\ \int _{-\infty }^{x}y^{2}\phi (y)dy&=\varPhi (x)-x\phi (x), \int _{x}^{+\infty }y^{2}\phi (y)dy={\overline{\varPhi }}(x)+x\phi (x) \\ \int _{-\infty }^{x}y^{3}\phi (y)dy&=-\,x^{2}\phi (x)-2\phi (x), \int _{x}^{+\infty }y^{3}\phi (y)dy=x^{2}\phi (x)+2\phi (x) \\ \int _{-\infty }^{x}y^{4}\phi (y)dy&=-\,x^{3}\phi (x)+3\left[ \varPhi (x)-x\phi (x) \right] \\ \int _{x}^{+\infty }y^{4}\phi (y)dy&=x^{3}\phi (x)+3\left[ {\overline{\varPhi }} (x)+x\phi (x)\right] \end{aligned}$$

Piecewise linear score, normal case

$$\begin{aligned} E[S^{(v)}(x,Y)]= & {} \alpha \int _{x}^{+\infty }(y-x)\phi (y)dy+(1-\alpha )\int _{-\infty }^{x}(x-y)\phi (y)dy\\= & {} \alpha [\phi (x)-x\overline{\varPhi }(x)]+(1-\alpha )[x\varPhi (x)+\phi (x)]\\= & {} \phi (x)+x[\varPhi (x)-\alpha ]. \end{aligned}$$

When $x=z_{\alpha }$, we get

$$\begin{aligned} E[S^{(v)}(z_{\alpha },Y)]= & {} \phi (z_{\alpha })\text {.}\\ E\left[ S^{(v)}(x,Y)^2\right]= & {} \int _{-\infty }^{+\infty }\left[ \alpha (y-x)_{+}+(1-\alpha )(y-x)_{-}\right] ^{2}\phi (y)dy \\= & {} \alpha ^{2}\int _{x}^{+\infty }(y-x)^{2}\phi (y)dy+(1-\alpha )^{2}\int _{-\infty }^{x}(y-x)^{2}\phi (y)dy \\= & {} \alpha ^{2}\left\{ {\overline{\varPhi }}(x)-x\phi (x)+x^{2}{\overline{\varPhi }} (x)\right\} \\&+(1-\alpha )^{2}\left\{ \varPhi (x)+x\phi (x)+x^{2}\varPhi (x)\right\} \\= & {} \left[ 1+x^{2}\right] \left\{ \alpha ^{2}+(1-2\alpha )\varPhi (x)\right\} +(1-2\alpha )x\phi (x)\text {.} \end{aligned}$$

When $x=z_{\alpha }$, we get

$$\begin{aligned} E\left[ S^{(v)}(x,Y)^2\right]&=\left[ 1+z_{\alpha }^{2}\right] \left\{ \alpha -\alpha ^{2}\right\} +(1-2\alpha )z_{\alpha }\phi (z_{\alpha })\\ { Var}[(S^{(v)}(z_{\alpha },Y)]&=E\left[ S^{(v)}(x,Y)^2\right] -\phi ^{2}(z_{\alpha }) \\&=\alpha (1-\alpha )(1+z_{\alpha }^{2})+(1-2\alpha )z_{\alpha }\phi (z_{\alpha })-\phi ^{2}(z_{\alpha }). \end{aligned}$$

Piecewise quadratic score, normal case

$$\begin{aligned} E\left[ S^{(e)}(x,Y)\right]&=\alpha \int _{x}^{+\infty }(y-x)^{2}\phi (y)dy+(1-\alpha )\int _{-\infty }^{x}(x-y)^{2}\phi (y)dy \\&= \alpha [{\overline{\varPhi }}(x)+x\phi (x)+x^{2}\overline{\varPhi } (x)-2x\phi (x)] \\&\quad +(1-\alpha )[\varPhi (x)-x\phi (x)+x^{2}\varPhi (x)+2x\phi (x)] \\&=(1-2\alpha )x\phi (x)+(1+x^{2})[\alpha \overline{\varPhi }(x)+(1-\alpha )\varPhi (x)].\\ E\left[ S^{(e)}(x,Y)^2\right]&=\int _{-\infty }^{+\infty }\left[ \alpha (y-x)_{+}^{2}+(1-\alpha )(y-x)_{-}^{2}\right] ^{2}\phi (y)dy \\&=\int _{x}^{+\infty }\alpha ^{2}(y-x)^{4}\phi (y)dy+\int _{-\infty }^{x}(1-\alpha )^{2}(y-x)^{4}\phi (y)dy.\\ \end{aligned}$$

Now we have for the first term:

$$\begin{aligned} \int _{x}^{+\infty }(y-x)^{4}\phi (y)dy= & {} x^{3}\phi (x)+3\left[ {\overline{\varPhi }}(x)+x\phi (x)\right] -4x\left[ x^{2}\phi (x)+2\phi (x)\right] \\&+\,6x^{2}\left[ {\overline{\varPhi }}(x)+x\phi (x) \right] -4x^{3}\phi (x)+x^{4}{\overline{\varPhi }}(x)\\= & {} {\overline{\varPhi }}(x)\left[ 3+6x^{2}+x^{4}\right] -5x\phi (x)-x^{3}\phi (x). \end{aligned}$$

Similarly for the second term:

$$\begin{aligned} \int _{-\infty }^{x}(y-x)^{4}\phi (y)dy= & {} -x^{3}\phi (x)+3\left[ \varPhi (x)-x\phi (x)\right] -4x\left[ -x^{2}\phi (x)-2\phi (x)\right] \\&+\,6x^{2}\left[ \varPhi (x)-x\phi (x)\right] -4x^{3}\left[ -\phi (x)\right] +x^{4}\varPhi (x) \\= & {} \varPhi (x)\left[ 3+6x^{2}+x^{4}\right] +x^{3}\phi (x)+5x\phi (x). \end{aligned}$$

Summing up, we get

$$\begin{aligned} E\left[ S^{(e)}(x,Y)^2\right]= & {} \left( 3+6x^{2}+x^{4}\right) \left( \alpha ^{2}-2\alpha \varPhi (x)+\varPhi (x)\right) \\&+\,(1-2\alpha )\left( 5x\phi (x)+x^{3}\phi (x)\right) \end{aligned}$$

When $x=e_{\alpha }$, we get

$$\begin{aligned} { Var}\left[ S^{(e)}(e_{\alpha },Y)\right]= & {} \left( 3+6e_{\alpha }^{2}+e_{\alpha }^{4}\right) \left( \alpha ^{2}-2\alpha \varPhi (e_{\alpha })+\varPhi (e_{\alpha })\right) \\&+(1-2\alpha )\left( 5 e_{\alpha } \phi (e_{\alpha })+e_{\alpha }^{3}\phi (e_{\alpha })\right) +\\&- \left( (1-2\alpha ) e_{\alpha } \phi (e_{\alpha })+\left( 1+e_{\alpha }^{2}\right) \left( \alpha {\overline{\varPhi }}(e_{\alpha })+(1\!-\!\alpha )\varPhi (e_{\alpha })\right) \right) ^{2}. \end{aligned}$$

Identification function, normal case. Recall that

$$\begin{aligned} I^{(e)}(Y,e)=\alpha (Y-e)_{+}-(1-\alpha )(Y-e)_{-}. \end{aligned}$$

Since $E[I^{(e)}(Y,e_\alpha )] =0$, we get

$$\begin{aligned} { Var}[I^{(e)}(Y,e_\alpha )]&=\alpha ^{2}\int _{\varepsilon }^{+\infty }(y-e_\alpha )^{2}\phi (y)dy+(1-\alpha )^{2}\int _{-\infty }^{e_\alpha }(e_\alpha -y)^{2}\phi (y)dy\\&=\alpha ^{2}\left[ {\overline{\varPhi }}(e_\alpha )+e_\alpha \phi (e_\alpha )-2e_\alpha \phi (e_\alpha )+e_\alpha ^{2}(1-\varPhi (e_\alpha ))\right] \\&\quad +(1-\alpha )^{2}\left[ \varPhi (e_\alpha )-e_\alpha \phi (e_\alpha )+2e_\alpha \phi (e_\alpha )+e_\alpha ^{2}\varPhi (e_\alpha )\right] \\&=(1+e_\alpha ^{2})\left[ \alpha ^{2}+\varPhi (e_\alpha )(1-2\alpha )\right] +e_\alpha \phi (e_\alpha )\left[ 1-2\alpha \right] . \end{aligned}$$

Let now $Y\sim U(0,1)$, $\phi (y)=1_{[0,1]}$, $\varPhi (y)=y1_{[0,1]}$ and $z_{\alpha }=\alpha$.

Piecewise linear score, uniform case.

$$\begin{aligned} E[S^{(v)}(z_{\alpha },Y)]&=\alpha \int _{\alpha }^{1}(y-\alpha )\phi (y)dy+(1-\alpha )\int _{0}^{1}(\alpha -y)\phi (y)dy \\&=\frac{\alpha (1-\alpha )}{2}. \\ E[S^{(v)}(z_{\alpha },Y)^{2}]&=\alpha ^{2}\int _{\alpha }^{1}(y-\alpha )^{2}dy+(1-\alpha )^{2}\int _{0}^{\alpha }(\alpha -y)^{2}dy \\ { Var}[S^{(v)}(z_{\alpha },Y)]&=E[S^{(v)}(z_{\alpha },Y]^{2})-[E[S^{(v)}(z_{\alpha },Y)]]^{2} \\&=\frac{1}{12}\alpha ^{2}\left( \alpha -1\right) ^{2} \end{aligned}$$

Piecewise quadratic score, uniform case. Recall that if $Y\sim U(0,1)$

$$\begin{aligned} e_{\alpha }=\frac{\alpha -\sqrt{\alpha -\alpha ^{2}}}{2\alpha -1}\text {.} \end{aligned}$$

(9)

$$\begin{aligned} E\left[ S^{(e)}(e_{\alpha },Y)\right]= & {} \int _{0}^{1}\left[ \alpha (y-e_{\alpha })_{+}^{2}+(1-\alpha )(y-e_{\alpha })_{-}^{2}\right] dy \\= & {} \alpha \int _{e_{\alpha }}^{1}(y-e_{\alpha })^{2}dy+(1-\alpha )\int _{0}^{e_{\alpha }}(e_{\alpha }-y)^{2}dy \\= & {} \frac{1}{3}e_{\alpha }^{3}\left( 1-\alpha \right) +\alpha \left( e_{\alpha }^{2}-e_{\alpha }-\frac{1}{3}e_{\alpha }^{3}+\frac{1}{3}\right) \end{aligned}$$

Substituting (9) we get

$$\begin{aligned} E\left[ S^{(e)}(e_{\alpha },Y)\right]= & {} \frac{1}{3}\alpha (1-\alpha )\frac{1-2\sqrt{\alpha (1-\alpha )}}{\left( 2\alpha -1\right) ^{2}}.\\ E[S^{(e)}(e_{\alpha },Y)^2]= & {} \int _{0}^{1}\left[ \alpha (y-e_{\alpha })_{+}^{2}+(1-\alpha )(y-e_{\alpha })_{-}^{2}\right] ^{2}dy \\= & {} \alpha ^{2}\int _{e_{\alpha }}^{1}(y-e_{\alpha })^{4}dy+(1-\alpha )^{2}\int _{0}^{e_{\alpha }}(e_{\alpha }-y)^{4}dy \\= & {} \frac{1}{5}e_{\alpha }^{5}\left( 1-\alpha \right) ^{2}+\alpha ^{2}\left( 2e_{\alpha }^{2}-e_{\alpha }-2e_{\alpha }^{3}+e_{\alpha }^{4}-\frac{1}{5} e_{\alpha }^{5}+\frac{1}{5}\right) \\ E[S^{(e)}(e_{\alpha },Y)^2]= & {} -\frac{11\alpha ^{4}-2\alpha ^{3}-\alpha ^{2}-12\alpha ^{5}+4\alpha ^{6}-\left( \alpha -\alpha ^{2}\right) ^{\frac{5}{2}}}{5\left( 2\alpha -1\right) ^{4}}+ \\&-\frac{5\alpha ^{2}\sqrt{\alpha -\alpha ^{2}}-10\alpha ^{3}\sqrt{\alpha -\alpha ^{2}}+5\alpha ^{4}\sqrt{\alpha -\alpha ^{2}}}{5\left( 2\alpha -1\right) ^{4}} \\= & {} -\frac{1}{5}\alpha ^{2}\left( -4\alpha +4\alpha ^{2}-1\right) \frac{\left( \alpha -1\right) ^{2}}{\left( 2\alpha -1\right) ^{4}}-\frac{4}{5}\frac{ \left( \alpha -\alpha ^{2}\right) ^{\frac{5}{2}}}{\left( 2\alpha -1\right) ^{4}} \\= & {} \frac{\alpha ^{2}\left( \alpha -1\right) ^{2}}{5\left( 2\alpha -1\right) ^{4} }\left( 4\alpha -4\alpha ^{2}+1-4\sqrt{\alpha -\alpha ^{2}}\right) \\= & {} \frac{ \alpha ^{2}\left( 1-\alpha \right) ^{2}}{5\left( 2\alpha -1\right) ^{4}} \left( 1-2\sqrt{\alpha (1-\alpha )}\right) ^{2}.\\ { Var}[S^{e}(e_{\alpha },Y)]= & {} \frac{\alpha ^{2}\left( 1-\alpha \right) ^{2}}{ 5\left( 2\alpha -1\right) ^{4}}\left( 1-2\sqrt{\alpha (1-\alpha )}\right) ^{2}+\\&- \left( \frac{1}{3}\alpha (1-\alpha )\frac{1-2\sqrt{\alpha (1-\alpha )}}{ \left( 2\alpha -1\right) ^{2}}\right) ^{2} \\= & {} \frac{4}{45}\frac{\alpha ^{2}\left( 1-\alpha \right) ^{2}}{\left( 2\alpha -1\right) ^{4}}\left( 1-2\sqrt{\alpha (1-\alpha )}\right) ^{2} \end{aligned}$$

Note the relationship

$$\begin{aligned} { SD}[S^{e}(e_{\alpha },Y)]=\frac{2}{3\sqrt{5}}\frac{\alpha \left( 1-\alpha \right) }{\left( 2\alpha -1\right) ^{2}}\left( 1-2\sqrt{\alpha (1-\alpha )} \right) =\frac{2}{\sqrt{5}}E[S^{e}(e_{\alpha },Y)]. \end{aligned}$$

Identification function, uniform case.

$$\begin{aligned} { Var}[I^{e}(Y, e_\alpha )]&=\alpha ^{2}\int _{e_\alpha }^{1}(y-e_\alpha )^{2}dy+(1-\alpha )^{2}\int _{0}^{e_\alpha }(y-e_\alpha )^{2}dy\\&=\frac{\alpha ^{2}}{3}\left( 3e_\alpha ^{2}-3e_\alpha -e_\alpha ^{3}+1\right) +(1-\alpha )^{2} \frac{1}{3}e_\alpha ^{3} \\&=\frac{\alpha ^{2}}{3}\left( 1-e_\alpha \right) ^{3}+\frac{1}{3}(1-\alpha )^{2} e_\alpha ^{3}. \end{aligned}$$