Using machine learning to detect misstatements

Abstract

Machine learning offers empirical methods to sift through accounting datasets with a large number of variables and limited a priori knowledge about functional forms. In this study, we show that these methods help detect and interpret patterns present in ongoing accounting misstatements. We use a wide set of variables from accounting, capital markets, governance, and auditing datasets to detect material misstatements. A primary insight of our analysis is that accounting variables, while they do not detect misstatements well on their own, become important with suitable interactions with audit and market variables. We also analyze differences between misstatements and irregularities, compare algorithms, examine one-year- and two-year-ahead predictions and interpret groups at greater risk of misstatements.

Appendices

Appendix A: Omitted Tables

Table 1 Sample selection

Table 2 Key summary statistics

Table 3 Distribution of restated firms by year

Table 4 Frequency of firm-year restatements

Table 5 Variable definitions

Table 6 Summary statistics of predictors in Table 7

Table 7 Importance of predictors (importance higher than 1.5%)

Table 8 Models using subgroups of variables

Table 9 Difference of means test between restatement and nonrestatement firm-years

Table 10 Fit and accuracy of the different models

Table 11 Catch rate when using small bandwidths

Table 12 Performance of the models for the top 1/3 predicted probabilities of firm-years

Table 13 Performance of alternative methods

Table 14 AAERs sample selection and description

Table 15 AAERs catch rates on test dataset (11,323 firm years)

Table 16 Top 10 variables (bold: same variable across panels; italics: same variable in at least two models)

Table 17 Performance on catching AAERs

Table 18 Results from predicting ahead models

Table 19 Importance of predictors from predicting ahead model (top 15)

Table 20 InTrees for top 10 and all variables

Appendix B: An illustration of GBRT

We now offer a more formal presentation of the main steps of the gradient-boosting algorithm. Let us hold the depth of the tree as fixed and denote $\phi (x;\mathcal {S})$ as a mapping that yields a vector of predicted values for some variable to be predicted z, given x, under the regression tree applied to sample $\mathcal {S}=(z_i,x_i)_{i=1}^N$, where $(z_i)_{i=1}^N$ is a set of observations to be predicted. Note that we focus here on the first-order aspects as we use it in this study; the reader should refer to Friedman (2001) for a general description that would apply to a broader class of datasets.

Let us subsequently set a loss function $L(y,\hat {y})$ for the boosting algorithm, where y is any variable that could be predicted and $\hat {y}$ is a prediction. For a typical application with continuous variables, we could set a quadratic loss $L^q(y,\hat {y})=(y-\hat {y})^2/2$; for our purpose, given that we will predict probabilities, we will use a logistic loss function $L(y, \hat {y})=log(1+exp(-2y\hat {y}))$. Note that, in the quadratic case, the residual $y-\hat {y}$ is also given by the derivative in the second component of the loss function $L_2^q(y,\hat {y})=y-\hat {y}$; gradient boosting builds on this intuition to compute a local version of the residual for a nonquadratic loss function by computing $L_2(y,\hat {y})=2y/(1+exp(2 y \hat {y}))$. Note here that a correct prediction $y=\hat {y}$ reduces L₂, as it would in the case of a residual in a linear regression. So, for later use, let us interpret $L_2(y,\hat {y})$ as a measure of the component of y unexplained by $\hat {y}$.

In pseudo-code, the algorithm will operate as follows.

• 1. Initialize the analysis with a single prediction for the entire sample that minimizes the logistic loss function; that is,

  $$ \hat{y}_{i}^{0}=\frac{1}{2}\log\frac{1+\overline{y}}{1-\overline{y}}, $$

  where $\overline {y}$ is the sample mean.

• 2. For each step m ∈ [1, M], where M is the number of trees:

  • 2.1. Compute the residuals of the previous step and redefine this residual as the variable to be explained:

    $$ {z_{i}^{m}}=L_{2}(y_{i}^{},\hat{y}_{i}^{m-1}). $$

  • 2.2. Fit a regression tree to $\mathcal {S}^m=(z_i^m,x_i)_{i=1}^N$, where $\mathcal {S}$ is a subset drawn randomly according to parameter (d). The prediction is updated to

    $$ \hat{y}_{i}^{m}=\hat{y}_{i}^{m-1}+\nu \phi(x_{i};\mathcal{S}), $$

    where ν is the speed of learning parameter (c).

We illustrate next how gradient boosting trees work using a simple numerical example, where we set the number of trees to M = 2 and the tree depth to 2. In Table 21, we create 16 firm-year observations, coding a misstatement as 1 and nonmisstatements as − 1 (column 1). Of course, this is only meant to carry the analysis by paper and pencil and is in no way a reasonable sample size to conduct a machine learning analysis. The dependent variable will be whether there is a misstatement, and we use two explanatory variables: earnings-to-price ratio and % of soft assets (columns 2 and 3).

Table 21 Example - sample 16 observations, nine of which have misstatements, with two observable explanatory variables E/P and Soft

Note that, in this created data set, the pattern that ties the variables to misstatements is visually self-evident. Plotting this data in Fig. 7, it is clear that misstatements have a monotonic relationship with the two variables. But this relationship is not linear, and there is an interaction between the two variables such that the cutoff for soft assets doesn’t change at low levels of earnings to price. Our objective will be to show how machine learning can uncover this pattern that we have ourselves “learned” from visual inspection over the graph of observations.

Fig. 7

Graphical analysis: misstatements as a function of explanatory variables

Following the pseudo code, we first calculate the sample mean as $\overline {y}=0.125$ and initialize our first predictor at a predicted probability for the whole sample given by

$$ \hat{y}_{i}^{0}=\frac{1}{2}\log\frac{1+0.125}{1-0.125}=0.125657. $$

Then we calculate the residuals for each of the 16 observations given by

$$ {z_{i}^{1}}=L_{2}(y_{i}^{},0.125657)=\frac{2 y_{i}}{1+exp(2 y_{i} \hat{y}_{i}^{0})}, $$

as calculated in column 4 of Table 21. The next step is to fit the regression tree to $z_i^1$, which involves finding one variable and one cutoff, such that all observations where the variable is at one side of the cutoff are given the same prediction. Of note, the regression tree does not use the same loss function as the boosting because the fitted $z_i^1$ are derivatives that need not be bounded. In particular, the loss function used for this step is simply the quadratic loss function L^q and the best cutoff minimizes this loss.

In this example, there are two variables, and each has four values, so we could choose to place the cutoff on one of two variables and at three locations per variable, for six possible choices of the cutoff. For each possible cutoff, we compute a prediction $\overline {z}_i^1$ as the average for all observations at one side of the cutoff and sum over the loss function $L^q(z_i^1,\overline {z}_i^1)$. Table 22 provides the loss function for every possible cutoff. Here, the best cutoff is to partition as a function of soft assets greater or equal than 0.4, with 4 observations below this cutoff and 12 above.

Table 22 Squared loss for various cutoffs (first branch)

Given that the number of leaves is set to be equal to three, we need to apply a second (and final) cutoff to obtain three end leaves. We proceed similarly in Table 23, except that there are now two choices as to whether to apply the cutoff on the left or right side of Soft ≥ 0.4 and then five choices for the cutoff for the two variables if we place the cutoff on the right node versus three if we place the cutoff on the left node. Note that, for the left node, all outcome variables have the same value equal to zero so that the cutoff itself does not increase explanatory power; hence we know, for this case, that the best cutoff must be on the right node.

Table 23 Squared loss for various cutoffs (second branch)

Note that a variable that has been used in a prior cut can be used again in later steps, thus creating potentially complex interactions or nonlinearities. For this example, the best cutoff point is located at EP ≥ 0.07, which leads to a partition of the sample into three regions, as illustrated in Fig. 8.

Fig. 8

First regression tree - complete

Having completed the first tree, we compute an adjustment to our initial prediction from step 2.2, that is,

$$ \hat{y}_{i}^{m}=\hat{y}_{i}^{m-1}+ \phi(x_{i};\mathcal{S}), $$

where ϕ(.) is based on the logistic functional

$$ \phi(x_{i};\mathcal{S}_{0})=\frac{{\sum}_{(z_{i},x_{i})\in R^{1}(x_{i}) } {z_{i}^{0}}}{{\sum}_{(z_{i},x_{i})\in R^{1}(x_{i})} |{z_{i}^{0}}|(2-|{z_{i}^{0}}|)}, $$

where R¹(x_i) indicates the set of observations classified in the same region by the tree. Column 5 in Table 21 provides an updated estimate after the first tree. Note that these refer to the term inside of the logistic term. To make a prediction, we need to convert this term into a probability (column 6) using a standard logistic transformation.¹⁷

Since we have set the number of trees to two, we need to repeat these steps a second time, using an updated set of residuals

$$ {z_{i}^{2}}=L_{2}(y_{i},\hat{y}_{i}^{1}). $$

Skipping the calculations (which are conceptually identical), the second tree illustrated in Fig. 9 is now given by a cutoff on both high E/P and soft assets, which unsurprisingly correspond to the area of predictors that were partly misclassified by the first tree.

Fig. 9

Second tree

After computing the corresponding $\phi (x_i;\mathcal {S}_1)$, where $\mathcal {S}_1=(z_i^2,x_i)$ is the sample for the second tree, and mapping back to an updated value for $\hat {y}_i^2$ and the implied probability, we compute in Column 8 of Table 21 the resulting prediction. As a result of the two trees, the data is now classified into seven regions.

To further illustrate in Fig. 10 the tree component of the analysis, we estimate a single regression tree by partitioning the probability of misstatements into 10 nodes. The probability of misstatements across nodes varies from 2% to 50%. The first branch of the tree, the most important variable is usually whether the auditor has issued a qualified opinion. As expected, qualified opinions tend to be more frequent in the presence of an ongoing misstatement. Conditional on a qualified opinion, the firms with higher book-to-market and higher deferred tax expenses tend to be the most likely to misstate, thus suggesting disagreements about deferred taxes. Conditional on an unqualified opinion, by contrast, a very different set of variables predicts misstatement, with firms with high non-audit fees and high stock market volatility having a high probability to misstate.

Fig. 10

Regression tree