Theory and applications of proper scoring rules

Abstract

A scoring rule $S(x; q)$ provides a way of judging the quality of a quoted probability density $q$ for a random variable $X$ in the light of its outcome $x$. It is called proper if honesty is your best policy, i.e., when you believe $X$ has density $p$, your expected score is optimised by the choice $q = p$. The most celebrated proper scoring rule is the logarithmic score, $S(x; q) = -\log {q(x)}$: this is the only proper scoring rule that is local, in the sense of depending on the density function $q$ only through its value at the observed value $x$. It is closely connected with likelihood inference, with communication theory, and with minimum description length model selection. However, every statistical decision problem induces a proper scoring rule, so there is a very wide variety of these. Many of them have additional interesting structure and properties. At a theoretical level, any proper scoring rule can be used as a foundational basis for the theory of subjective probability. At an applied level a proper scoring can be used to compare and improve probability forecasts, and, in a parametric setting, as an alternative tool for inference. In this article we give an overview of some uses of proper scoring rules in statistical inference, including frequentist estimation theory and Bayesian model selection with improper priors.