# Sparse Kalman filtering approaches to realized covariance estimation from high frequency financial data

## Abstract

Estimation of the covariance matrix of asset returns from high frequency data is complicated by asynchronous returns, market microstructure noise and jumps. One technique for addressing both asynchronous returns and market microstructure is the Kalman-Expectation-Maximization (KEM) algorithm. However the KEM approach assumes log-normal prices and does not address jumps in the return process which can corrupt estimation of the covariance matrix. In this paper we extend the KEM algorithm to price models that include jumps. We propose a sparse Kalman filtering approach to this problem. In particular we develop a Kalman Expectation Conditional Maximization algorithm to determine the unknown covariance as well as detecting the jumps. In order to promote a sparse estimate of the jumps,we consider both Laplace and the spike and slab jump priors. Numerical results using simulated data show that each of these approaches provide for improved covariance estimation relative to the KEM method in a variety of settings where jumps occur.

## Keywords

Spike and slab ECM Kalman filtering \(\ell _{1}\) regularization## Mathematics Subject Classification

90C26 62P05## Notes

## Supplementary material

## References

- 1.Aravkin, A., Bell, B., Burke, J., Pilonetto, G.: An \(\ell _{1}\) Laplace robust Kalman smoother. IEEE Trans. Autom. Control
**56**(12), 2898–2911 (2011)MathSciNetzbMATHGoogle Scholar - 2.Aït-Sahalia, Y., Fan, J., Xiu, D.: High-frequency covariance estimates with noisy and asynchronous financial data. J. Am. Stat. Assoc.
**105**(492), 1504–1517 (2010)MathSciNetzbMATHGoogle Scholar - 3.Aït-Sahalia, Y., Myklank, P., Zhang, L.: How often to sample a continuous-time process in the presence of market microstructure noise. Rev. Financ. Stud.
**100**, 1394–1411 (2005)Google Scholar - 4.Bandi, F., Russell, J.: Separating microstructure noise from volatility. J. Financ. Econ.
**79**, 655–692 (2006)Google Scholar - 5.Barndorff-Nielsen, O., Hansen, P., Lunde, A., Shephard, N.: Multivariate realised kernels: consistent positive semi-definite estimators of the covariation of equity prices with noise and non-synchronous trading. J. Econom.
**162**, 149–169 (2011)MathSciNetzbMATHGoogle Scholar - 6.Barry, C.B.: Portfolio analysis under uncertain means, variances and covariances. J. Finance
**29**, 515–522 (1974)Google Scholar - 7.Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Imaging Sci.
**2**(1), 183–202 (2009)MathSciNetzbMATHGoogle Scholar - 8.Bollerslev, T.: Modeling the coherence in short-run nominal exchange rates: a multivariate generalized ARCH approach. Rev. Econ. Stat.
**72**, 498–505 (1990)Google Scholar - 9.Boudt, K., Croux, C., Laurent, S.: Outlyingness weighted covariation. J. Financ. Econom.
**9**(4), 657–684 (2011)Google Scholar - 10.Boudt, K., Zhang, J.: Jump robust two time scale covariance estimation and realized volatility budgets. Quant. Finance
**15**(6), 1041–1054 (2015)MathSciNetzbMATHGoogle Scholar - 11.Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn.
**3**(1), 1–122 (2011)zbMATHGoogle Scholar - 12.Campbell, J., Lo, A., MacKinlay, A.C.: The Econometrics of Financial Markets. Princeton University Press, Princeton (1996)zbMATHGoogle Scholar
- 13.Candès, E., Wakin, M., Boyd, S.: Enhancing sparsity by reweighted \(\ell _{1}\) minimzation. J. Fourier Anal. Appl.
**14**, 877–905 (2008)MathSciNetzbMATHGoogle Scholar - 14.Chan, W., Maheu, J.: Conditional jump dynamics in stock market returns. J. Bus. Econ. Stat.
**20**(3), 377–389 (2002)MathSciNetGoogle Scholar - 15.Corsi, F., Peluso, S., Audrino, F.: Missing in asynchronicity: a Kalman-EM approach for multivariate realized covariance estimation. J. Appl. Econom.
**30**(3), 377–397 (2015)MathSciNetGoogle Scholar - 16.DeMiguel, V., Garlappi, L., Uppal, R.: Optimal versus Naive diversification: how inefficient is the 1/n portfolio strategy? Rev. Financ. Stud.
**22**(5), 1915–1953 (2009)Google Scholar - 17.Dempster, A., Laird, N., Rubin, D.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. Ser. B
**39**(1), 1–38 (1977)MathSciNetzbMATHGoogle Scholar - 18.Fan, J., Li, Y., Yu, K.: Vast volatility matrix estimation using high frequency data for portfolio selection. J. Am. Stat. Assoc.
**107**(497), 412–428 (2012)MathSciNetzbMATHGoogle Scholar - 19.Fan, J., Wang, Y.: Multi-scale jump and volatility analysis for high-frequency financial data. J. Am. Stat. Assoc.
**102**(480), 1349–1362 (2007)MathSciNetzbMATHGoogle Scholar - 20.Fink, D.: A compendium of conjugate priors. Technical Report, Montana State Univeristy (1997)Google Scholar
- 21.Ghahramani, Z., Hinton, G.E.: Variational learning for switching state-space models. Neural Comput.
**12**(4), 963–996 (2000)Google Scholar - 22.Goldstein, T., Osher, S.: The split Bregman method for \(\ell _{1}\) regularized problems. SIAM J. Imaging Sci.
**2**(2), 323–343 (2009)MathSciNetzbMATHGoogle Scholar - 23.Jobson, J.D., Korkie, B.: Estimation for Markowitz efficient portfolios. J. Am. Stat. Assoc.
**75**, 544–554 (1980)MathSciNetzbMATHGoogle Scholar - 24.Karpoff, J.: The relation between price changes and trading volume: a survey. J. Financ. Quant. Anal.
**22**, 109–126 (1987)Google Scholar - 25.Liu, C., Tang, C.Y.: A quasi-maximum likelihood approach for integrated covariance matrix estimation with high frequency data. J. Econom.
**180**, 217–232 (2014)MathSciNetzbMATHGoogle Scholar - 26.Lo, A., MacKinlay, A.C.: An econometric analysis of nonsynchronous trading. J. Econom.
**45**, 181–211 (1990)MathSciNetzbMATHGoogle Scholar - 27.Lunenberger, D., Ye, Y.: Linear and Nonlinear Programming. Addison-Wesley, New York (2008)Google Scholar
- 28.Maheu, J.M., McCurdy, T.H.: News arrival, jump dynamics, and volatility components for individual stock returns. J. Finance
**59**(2), 755–793 (2004)Google Scholar - 29.Mattingley, J., Boyd, S.: Real-time convex optimization in signal processing. IEEE Signal Process. Mag.
**27**, 50–61 (2010)zbMATHGoogle Scholar - 30.Meng, X.L., Rubin, D.: Maximum likelihood estimation via the ECM algorithm: a general framework. Biometrika
**80**, 267–278 (1993)MathSciNetzbMATHGoogle Scholar - 31.Peluso, S., Corsi, F., Mira, A.: A Bayesian high-frequency estimator of the multivariate covariance of noisy and asynchronous returns. J. Financ. Econom.
**13**(3), 665–697 (2015)Google Scholar - 32.Roll, R.: A simple implicit measure of the effective bid-ask spread in an efficient market. J. Finance
**39**(4), 1127–1139 (1984)Google Scholar - 33.Seeger, M.W.: Bayesian inference and optimal design for the sparse linear model. J. Mach. Learn. Res.
**9**, 759–813 (2008)MathSciNetzbMATHGoogle Scholar - 34.Shumway, R., Stoffer, D.: Time Series Analysis and its Applications with R Examples. Springer, Berlin (2011)zbMATHGoogle Scholar
- 35.Shumway, R.H., Stoffer, D.S.: An approach to time series smoothing and forecasting using the EM algorithm. J. Time Ser. Anal.
**3**(4), 253–264 (1982)zbMATHGoogle Scholar - 36.Wu, C.F.: On the convergence properties of the EM algorithm. Ann. Stat.
**11**(1), 95–103 (1983)MathSciNetzbMATHGoogle Scholar - 37.Zangwill, W.: Nonlinear Programming: A Unified Approach. Prentice-Hall, Upper Saddle River (1969)zbMATHGoogle Scholar
- 38.Zhang, L.: Estimating covariation: Epps effect and microstructure noises. J. Econ.
**160**(1), 33–47 (2011)MathSciNetzbMATHGoogle Scholar - 39.Zhang, M., Russel, J., Tsay, R.: Determinants of bid and ask quotes and implications for the cost of trading. J. Empir. Finance
**15**(4), 656–678 (2008)Google Scholar