Measuring productivity and efficiency: a Kalman filter approach

Abstract

In the Kalman filter setting, one can model the inefficiency term of the standard stochastic frontier composed error as an unobserved state. In this study a panel data version of the local level model is used for estimating time-varying efficiencies of firms. We apply the Kalman filter to estimate average efficiencies of U.S. airlines and find that the technical efficiency of these carriers did not improve during the period 1999–2009. During this period the industry incurred substantial losses, and the efficiency gains from reorganized networks, code-sharing arrangements, and other best business practices apparently had already been realized.

Appendices

Appendix 1

In this appendix we provide further details about the Kalman filter estimation. Consider two stochastic frontier models that are nested by the general setting that we described:

$$\begin{array}{rcl}{y_{it}} &=& {X_{it}}\beta + Z{B_{it}} + {\varepsilon _{it}},\quad {\varepsilon _{it}} \sim {\bf NID}( 0,{\sigma _\varepsilon ^2} )\\ {B_{i,t + 1}} &=& T{B_{i,t}} + R{e_{it}},\quad {e_{it}} \sim {\bf NID}( 0,{\sigma _e^2} ) \end{array}$$

(12)

where

$${B_{it}} = \displaystyle\left[\begin{array}{c}{\mu _{it}} \\ {\tau _{it}}\end{array}\right],\ T = \displaystyle\left[\begin{array}{cc} 1 & 1 \\ 0 & 1\end{array} \right],\,{\rm and}\ Z = \Big[ 1 \quad 0 \Big] $$

and

$$\begin{array}{rcl} {y_{it}} &=& {X_{it}}\beta + Z{B_{it}} + {\varepsilon _{it}},\quad {\varepsilon _{it}} \sim {\bf{NID}}\left( {0,\sigma _\varepsilon ^2} \right)\\ {B_{i,t + 1}} &=& T{B_{i,t}} + {e_{it}},\quad {e_{it}} \sim {\bf{NID}}\left( {0,\sigma _e^2} \right)\end{array}$$

(13)

where

$${B_{it}} = {\mu _{it}},\,T = \rho ,\,\,{\rm{and}}\,\,Z = 1$$

The first model assumes a random walk with deterministic trend effects term and the second model assumes a potentially non-stationary AR(1) process for the effects term. Both of these models can be estimated by using the recursive equations provided below. The estimation consists of two steps. Kalman filtering and smoothing. In the first step, the following recursive Kalman filter equations are applied:

$$\begin{array}{rcl} {\eta _{it}} &=& {y_{it}} - {X_{it}}\beta - Z{b_{it}}\\ \ \ \ \, {F_{it}} &=& Z{P_{it|t - 1}}Z\prime + \sigma _\varepsilon ^2\\ \ \ \ {M_{it}} &=& {P_{it|t - 1}}Z\prime \\ \kern0.8pc {b_{it|t}} &=& {b_{it|t - 1}} + {M_{it}}F_{it}^{ - 1}{\eta _{it}}\\ \ \ \, {P_{it|t}} &=& {P_{it|t - 1}} - {M_{it}}F_{it}^{ - 1}{{M'}_{it}}\\ {b_{it|t - 1}} &=& T{b_{i,t - 1|t - 1}}\\ {P_{it|t - 1}} &=& T{P_{i,t - 1|t - 1}}T\prime + \sigma _e^2 \end{array}$$

(14)

In the second step, the smoothing is applied by using the following recursive equations:

$$\begin{array}{rcl} \kern1pc{L_{it}} &=& T - T{M_{it}}F_{it}^{ - 1}Z\\ \kern0.2pc{r_{i,t - 1}} &=& Z\prime F_{it}^{ - 1}{\eta _{it}} + {{L'}_{it}}{r_{it}}\\ {N_{i,t - 1}} &=& Z\prime F_{it}^{ - 1}Z + {{L'}_{it}}{N_{it}}{L_{it}}\\ {{\tilde b}_{it|t - 1}} &=& {b_{it|t - 1}} + {P_{it|t - 1}}{r_{i,t - 1}}\\ \kern0.9pc{V_{it}} &=& {P_{it|t - 1}} - {P_{it|t - 1}}{N_{i,t - 1}}{P_{it|t - 1}}\end{array}$$

(15)

where ${r_{i{n_t}}}\, = \,0$ and ${N_{i{n_t}}}\, = \,0$.²¹ The log-likelihood is given by:

$${ln\,(L) = {\sum}_{i\, = \,1}^{n_i} {L_i} = {\rm constant} - \frac{1}{2}\mathop {\sum}\limits_{i\, = \,1}^{n_i} \mathop {\sum}\limits_{t\, = \,{d_i}\, + \,1}^{n_t} \left( {ln\left( {F_{it}} \right) + \frac{{\eta _{it}^2}}{{F_{it}}}} \right)}$$

(16)

where d _i is the number of diffuse states for firm i. The number of diffuse priors (per panel unit) for the first model is two. The number of diffuse priors (per panel unit) for the second model is one. If we assume that ρ = 1, the second model would still have one diffuse prior per panel unit. However, obviously, the number of parameters to be estimated would be smaller. If we assume that |ρ| < 1, the second model would not have any diffuse priors.

Let m be the number of state variables (per panel unit) and q be the number of state variables with diffuse priors (per panel unit). For diffuse initialization we assume that:

$${b_{i0}} = A\delta + S{\eta _{i0}}$$

$${\eta _{i0}} \sim {\bf{NID}}\left( {0,{Q_0}} \right)$$

where δ is a q × 1 vector of unknown quantities and the m × q matrix A and m × (m−q) matrix S are selection matrices that consist of columns of identity matrix. Then, matrix for initialization is:

$${P_{i0}} = \kappa {P_{i\infty 0}} + {P_{ * 0}}$$

where κ→∞, P _∞= A′A, and P _*= SQ ₀ S′. As it can be seen from the log-likelihood, the first d _i observation(s) for panel unit i are burnt out for the sake of initialization and are not considered in the log-likelihood. Hence, for example, for the second model the first observation of each panel unit is used for initialization. The reason for this is that as long as t ≥ d _i + 1, we would have P _i∞t = 0. The variance matrix can be estimated using the standard maximum likelihood procedures. For the estimations we used the standard BFGS optimization method.

Appendix 2

In this appendix we present additional results based on the full translog model and our truncation scheme when calculating the efficiency estimates for KFE and CSSW estimator. We also provide estimates for alternative Kalman filter models.

The full translog estimates are given in Table 10. The parameter estimates are generally not significant even at 10 % significance level. The median of the returns to scale values for the KFE, CSSW, and BC estimators are 0.8625, 1.1478, and 1.0184, respectively. The corresponding returns to scale estimates from the restricted model were 0.883, 0.94, and 1.034, respectively. Hence, for the KFE and BC estimator the returns to scale estimates are robust to the choice of the functional form. Nevertheless, for both restricted and unrestricted translog production models the constant returns to scale value of 1 lies within one sample standard deviation away from the median value of returns to scale estimates from each of these estimates. All the estimators satisfy the monotonicity conditions at the median values of the regressors at each time period. In contrast to the restricted translog production model where only KFE satisfied the regularity conditions at the median values of the regressors, KFE and CSSW estimator satisfies the curvature conditions at each time period. BC estimator failed to satisfy the regularity conditions at four of the time periods. The estimates for the production function parameters and average efficiencies for the KFE and the BC and CSSW estimators are given in Table 10 and Fig. 2. The overall average efficiencies for the KFE, CSSW, and BC are 0.637, 0.458, and 0.605, respectively. These values are not substantially different from their restricted counterparts, i.e., 0.577, 0.438, and 0.632. The average efficiencies for the full translog model are provided in Fig. 2. In line with the restricted translog model, KFE predicts decrease in efficiency in first few years of the study period and relatively stable efficiency levels for the last couple of years.

Table 10 Full translog production model estimates

Fig. 2

Efficiency estimates for translog model

Now, we present the efficiency estimates when the trimming for KFE and CSSW are done for top–bottom 5 % (rather than 7.5 %) of the effects term when calculating the efficiencies. The BC estimates remain the same as they are not subject to such trimming. The average efficiency estimates for 5 % trimming case are provided in Figs. 3 and 4.

Fig. 3

Efficiency estimates for restricted translog model

Fig. 4

Efficiency estimates for translog model

Figure 5 presents the average of efficiency estimates for restricted translog model. Due to outliers the KFE and CSS model estimates are low.

Fig. 5

Efficiency estimates for restricted translog model without trimming

Finally, we provide our estimates for alternative Kalman filter models in Table 11. As we mentioned we failed to estimate the full Kalman filter model that we presented in Eq. 1. We rather estimated the models given in Eq. 12 (random walk model with deterministic trend) and Eq. 13 (AR(1) model).²² Based on the BIC values the random walk model with deterministic trend is not preferred. In particular, BIC values for random walk with deterministic trend and random walk models are 2.1 vs. 1.7, respectively. The second model seems to be subject to the pile up problem as one of the variance parameters is collapsed to zero. Hence, we only provide the results for the sake of completeness. Nevertheless, the estimate of ρ = 1.0315 parameter indicates that our random walk assumption for the trend term is sensible for our empirical example. Hence, these findings support our choice for using the random walk model as our benchmark model.

Table 11 Alternative Kalman filter model estimates