Multi-target prediction: a unifying view on problems and methods

Abstract

Many problem settings in machine learning are concerned with the simultaneous prediction of multiple target variables of diverse type. Amongst others, such problem settings arise in multivariate regression, multi-label classification, multi-task learning, dyadic prediction, zero-shot learning, network inference, and matrix completion. These subfields of machine learning are typically studied in isolation, without highlighting or exploring important relationships. In this paper, we present a unifying view on what we call multi-target prediction (MTP) problems and methods. First, we formally discuss commonalities and differences between existing MTP problems. To this end, we introduce a general framework that covers the above subfields as special cases. As a second contribution, we provide a structured overview of MTP methods. This is accomplished by identifying a number of key properties, which distinguish such methods and determine their suitability for different types of problems. Finally, we also discuss a few challenges for future research.

Appendix A: James–Stein Estimation

In the late sixties, James and Stein discovered that the best estimator of the mean of a multivariate Gaussian distribution is not necessarily the maximum likelihood estimator. More formally, assume that $\theta$ is the unknown mean of a multivariate Gaussian distribution with dimension $m>2$ and a diagonal covariance matrix. Consider a single observation $\mathbf {y}$ randomly drawn from that distribution:

$$\begin{aligned} \mathbf {y} \sim {\mathscr {N}}(\theta , \sigma ^2 I) \,. \end{aligned}$$

Using only this observation, the maximum-likelihood estimator for $\theta$ would be ${\hat{\theta }}_{ML} = \mathbf {y}$. James and Stein discovered that the maximum likelihood estimator is suboptimal in terms of mean squared error

$$\begin{aligned} {\mathbb {E}} \big [||\theta - {\hat{\theta }}||^2 \big ] \,, \end{aligned}$$

where the expectation is over the distribution of $\mathbf {y}$. (In general, the expectation is taken over all samples that contain a single observation $\mathbf {y}$. Later on we will shortly discuss a situation in which we draw more than one observation to compute the value of the estimator). An estimator with lower squared error can be obtained by applying a regularizer to the maximum likelihood estimator. In case $\sigma ^2$ is known, the James–Stein estimator is defined as follows:

$$\begin{aligned} {\hat{\theta }}_{JS} = \left( 1 - \frac{(m-2)\sigma ^2}{||\mathbf {y}||^2} \right) \mathbf {y} \,. \end{aligned}$$

From a machine learning perspective, a regularizer is introduced that shrinks the estimate towards the zero vector, and hence reduces variance at the cost of introducing a bias. It has been shown that this biased estimator outperforms the maximum likelihood estimator in terms of mean squared error. The result even holds when the covariance matrix is non-diagonal, but in view of the discussion concerning target dependence, it is most remarkable for diagonal covariance matrices. In fact, in the latter case, it means that joint target regularization will be beneficial even if targets are intrinsically independent. This is somewhat in contradiction with what is commonly assumed in the machine learning literature.

Let us notice, however, that the advantage of the James–Stein estimate over the maximum likelihood estimate will vanish for larger samples (of more than one observation). In the second term in parentheses, $\sigma ^2$ is then divided by the size of the sample, so that the James–Stein estimate converges to the maximum likelihood estimate when the sample size grows to infinity.

The James–Stein paradox analyzes a very simple estimation setting, for which suboptimality of the maximum likelihood estimator can be proved analytically, but the principle extends to various multi-target prediction settings. By interpreting each component of $\theta$ as an individual target (and omitting the instance space, or reducing it to a single point), the maximum likelihood estimator coincides with independent model fitting, whereas the James–Stein estimator adopts a regularization mechanism that is very similar to most of the regularization techniques used in the machine learning literature. For some specific multivariate regression models, connections of that kind have been discussed in the statistical literature (Breiman and Friedman 1997). As long as mean squared error is considered as a loss function and errors follow a Gaussian distribution, one can immediately extend the James–Stein paradox to multivariate regression settings by assuming that target vectors $\mathbf {y}$ are generated according to the following statistical model:

$$\begin{aligned} \mathbf {y} \sim {\mathscr {N}}(\theta (\mathbf {x}), \sigma ^2 I) \, , \end{aligned}$$

where the mean is now conditioned on the input space. For other loss functions, we are not aware of any formal analysis of that kind, but it might be expected that similar conclusions can be drawn.