Multi-period classification: learning sequent classes from temporal domains

Abstract

As the majority of real-world decisions change over time, extending traditional classifiers to deal with the problem of classifying an attribute of interest across different time periods becomes increasingly important. Tackling this problem, referred to as multi-period classification, is critical to answer real-world tasks, such as the prediction of upcoming healthcare needs or administrative planning tasks. In this context, although existing research provides principles for learning single labels from complex data domains, less attention has been given to the problem of learning sequences of classes (symbolic time series). This work motivates the need for multi-period classifiers, and proposes a method, cluster-based multi-period classification (CMPC), that preserves local dependencies across the periods under classification. Evaluation against real-world datasets provides evidence of the relevance of multi-period classifiers, and shows the superior performance of the CMPC method against peer methods adapted from long-term prediction for multi-period tasks with a high number of periods.

Appendix: Complementary metrics

Multi-period classifiers can be evaluated when the attribute under classification is either nominal or ordinal. In the paper, we targeted nominal attributes on the learning codomain and adopted simple loss functions (based on matching operators) to evaluate the performance of the proposed methods. However, three additional views are included in this appendix. First, loss functions to deal with ordinal labels. Second, meaningful evaluation metrics based on compact confusion matrices when a high number of labels is available. Third, distance metrics that can account for misalignements, such as temporal shifts.

Multi-period classification with ordinal labels

Multi-period accuracy $Acc_j$ can be derived from loss functions applied along the horizon of prediction. Representative loss functions include the simple, average normalized or relative root mean squared error. To draw comparisons with literature results, we suggest the use of Normalized Root Mean Squared Error, NRMSE (5) and of Symmetric Mean Absolute Percentage of Error, SMAPE (6) (Ben Taieb et al. 2010).

$$\begin{aligned} {\hbox {Acc}}_j(\varvec{y}_j,\hat{\varvec{y}}_j)= 1-{\hbox {NRMSE}}(\varvec{y}_j,\hat{\varvec{y}}_j)=1-\frac{\sqrt{\frac{1}{h}\Sigma _{i=1}^{h}(y_j^i-\hat{y}_j^i)^2}}{y_{\max }-y_{\min }}\in [0,1]\end{aligned}$$

(5)

$$\begin{aligned} {\hbox {Acc}}_j(\varvec{y}^j,\hat{\varvec{y}}^j)=1-{\hbox {SMAPE}}(\varvec{y}_j,\hat{\varvec{y}}_j)=1-\frac{1}{h}\Sigma _{i=1}^h\frac{\mid y_j^{i}-\hat{y}_j^{i}\mid }{(y_j^{i}+\hat{y}_j^{i})/2}\in [0,1] \end{aligned}$$

(6)

Evaluation using compact confusion matrices

In order to account for further critical performance views, a classic confusion matrix can be computed for each period. This solution, illustrated in Fig. 7, has the undesirable property of not offering compact views to study performance. For instance, multiple metrics need to be computed for each label and period in order to obtain a global view of the multi-period classifier sensitivity. A simple option, similarly to (3) and (4), would be to average the values for an instance across the $h$ periods. However, for the ordinal setting, instead of simply computing the matchings, a normalized distance needs to be applied between each pair of observed and estimated labels.

Fig. 7

Confusion matrices in multi-period classification settings. A confusion matrix in multi-period settings is the composition of classic confusion matrices per label and period, which results in a total of ${\mid }\Sigma {\mid }\times h$ views

However, with this option we loose the ability to understand which periods are affecting the score. A second option is to collapse the labels’ axis by defining a predicate. For this goal, we can rely on a mapping function $T$ to map a set of observed $h$ labels as a single label. An illustrative function is one that decides whether an instance is of interest (positive) or not based on the observed values. For example, relevant patients can be defined as having at least one hospitalization across the horizon of prediction. Still, this option requires the computation of each metric for the $h$ periods. Thus, we propose the use of this option with a simple test (based on a fixed $\beta$-threshold) to evaluate the adequacy of the $h$ predictions for a particular instance, ${\hbox {Acc}}(y,\hat{y})\ge \beta$ ((7) and (8)). Understandably, this option comes at a cost of defining a new labeling function $T$ and of working with $\beta$-threshold levels. Table 6 presents the revised confusion matrix for multi-period classification when two classes are considered. Resulting sensitivity (7) and specificity (8) metrics for this setting are computed as follows:

$$\begin{aligned} {\hbox {Sensitivity}}_c&= \frac{\Sigma _{j=1}^{m}(c=T(\varvec{y}_j))\wedge Acc(\varvec{y}_j,\hat{\varvec{y}}_j)\ge \beta }{\Sigma _{j=1}^{m}c=T(\varvec{y}_j)}, \end{aligned}$$

(7)

$$\begin{aligned} {\hbox {Specificity}}_c&= \frac{\Sigma _{j=1}^{m} (c\ne T(\varvec{y}_j))\wedge Acc(\varvec{y}_j,\hat{\varvec{y}}_j)\ge \beta }{\Sigma _{j=1}^{m} c\ne T(\varvec{y}_j)}. \end{aligned}$$

(8)

Table 6 Multi-period confusion matrix

Complementary evaluation metrics

Understandably, the distance functions used to evaluate the performance of multi-period classifiers are conservative for the cases where mismatches are caused by temporal shifts. To avoid a significant penalization of the performance of multi-period classifiers when misalignments occur on the time or cardinality axes, their evaluation can rely on more expressive time series’ similarity functions.

Ding et al. (2008) and Batista et al. (2011) compare the properties of alternative similarity functions when the attribute under classification is ordinal or numeric. Dynamic Time Warping (DTW) treats misalignments, which becomes critical when dealing with long horizons of prediction. Longest Common Subsequence deals with gap constraints. Pattern-based functions consider shifting and scaling in both the temporal and the amplitude axes.

When the output attribute is nominal, similarity functions proposed to compare biomolecular sequences based significant functional or structural similarity can be applied (Mantaci et al. 2008). These functions are also able to identify temporal shifts as they rely on sequence alignment operators. Moreover, they are able to deal with shifts on the amplitude axis by detecting character level differences.

On one hand, these similarity functions have the advantage of smoothing error accumulation by allowing temporal misalignments. On the other hand, their use can mask the structural accuracy of multi-period classifiers and lead to more optimistic results.