Computational Economics

, Volume 53, Issue 3, pp 991–1017 | Cite as

Finite Gaussian Mixture Approximations to Analytically Intractable Density Kernels

  • Natalia Khorunzhina
  • Jean-François RichardEmail author


The objective of the paper is that of constructing finite Gaussian mixture approximations to analytically intractable density kernels. The proposed method is adaptive in that terms are added one at the time and the mixture is fully re-optimized at each step using a distance measure that approximates the corresponding importance sampling variance. All functions of interest are evaluated under Gaussian product rules. Since product rules suffer from an obvious curse of dimensionality, the proposed algorithm as presented is only applicable to models whose non-linear and/or non-Gaussian subspace is of dimension up to three. Extensions to higher-dimensional applications would require the use of sparse grids, as discussed in the paper. Examples include a sequential (filtering) evaluation of the likelihood function of a stochastic volatility model where all relevant densities (filtering, predictive and likelihood) are closely approximated by mixtures.


Finite mixture Distance measure Gaussian quadrature Importance sampling Adaptive algorithm Stochastic volatility Density kernel 



The authors have benefited from discussions with Dave DeJong and Roman Liesenfeld.


  1. Adcock, C. J. (2004). Capital asset pricing in uk stocks under the multivariate skew-normal distribution. In M. G. Genton (Ed.), Skew-elliptical distributions and their applications (pp. 191–204). London: Chapman & Hall/CRC.Google Scholar
  2. Adcock, C. J. (2010). Asset pricing and portfolio selection based on the multivariate extended skew-student-\(t\) distribution. Annals of Operations Research, 176(1), 221–234.Google Scholar
  3. Ardia, D., Hoogerheide, L., & van Dijk, H. (2009). Adaptive mixture of student-t distributions as a flexible candidate distribution for efficient simulation: The r package admit. Journal of Statistical Software, 29(1), 1–32.Google Scholar
  4. Azzalini, A., & Valle, A. D. (1996). The multivariate skew-normal distribution. Biometrika, 83(4), 715–726.Google Scholar
  5. Basturk, N., Hoogerheide, L., Opschoor, A., & van Dijk, H. (2012). The R package MitISEM: Mixture of student-\(t\) distributions using importance sampling weighted expectation maximization for efficient and robust simulation. Tinbergen Institute Discussion Paper 12-096/III, Tinbergen InstituteGoogle Scholar
  6. Bornkamp, B. (2011). Approximating probability densities by iterated laplace approximations. Journal of Computational and Graphical Statistics, 20(3), 656–669.Google Scholar
  7. Bradley, P. S., & Fayyad, U. M. (1998). Refining initial points for k-means clustering. In: J. W. Shavlik (Ed.) Proceedings of the fifteenth international conference on machine learning, Morgan Kaufmann, San Francisco, CA, USA, pp. 91–99Google Scholar
  8. Bungartz, H. J., & Griebel, M. (2004). Sparse grids. Acta Numerica, 13, 147–269.Google Scholar
  9. Cameron, S. V., & Heckman, J. J. (2001). The dynamics of educational attainment for black, hispanic, and white males. Journal of Political Economy, 109(3), 455–499.Google Scholar
  10. Cappé, O., Guillin, A., Marin, J. M., & Robert, C. P. (2004). Population monte carlo. Journal of Computational and Graphical Statistics, 13, 907–929.Google Scholar
  11. Cappé, O., Godsill, S. J., & Moulines, E. (2007). An overview of existing methods and recent advances in sequential Monte Carlo. Proceedings of the IEEE, 95(5), 899–924.Google Scholar
  12. Cappé, O., Douc, R., Guillin, A., Marin, J. M., & Robert, C. P. (2008). Adaptive importance sampling in general mixture classes. Statistics and Computing, 18(4), 447–459.Google Scholar
  13. DeSarbo, W. S., Degeratu, A. M., Wedel, M., & Saxton, M. (2001). The spatial representation of market information. Marketing Science, 20(4), 426–441.Google Scholar
  14. Domínguez-Molina, J., González-Farías, G., & Ramos-Quiroga, R. (2004). Skew-normality in stochastic frontier analysis. In M. G. Genton (Ed.), Skew-elliptical distributions and their applications (pp. 223–242). London: Chapman & Hall/CRC.Google Scholar
  15. Domínguez-Molina, J., González-Farías, G., Ramos-Quiroga, R., & Gupta, A. K. (2007). A matrix variate closed skew-normal distribution with applications to stochastic frontier analysis. Communications in Statistics: Theory and Methods, 36(9), 1691–1703.Google Scholar
  16. Douc, R., Guillin, A., Marin, J. M., & Robert, C. P. (2007). Minimum variance importance sampling via population monte carlo. ESAIM: Probability and Statistics, 11, 427–447.Google Scholar
  17. Doucet, A., de Freitas, N., & Gordon, N. (2001). Sequential Monte Carlo methods in practice. New York: Springer.Google Scholar
  18. Duffie, D., & Pan, J. (1997). An overview of value at risk. The Journal of Derivatives, 4(3), 7–49.Google Scholar
  19. Duong, T. (2007). ks: Kernel density estimation and kernel discriminant analysis for multivariate data in r. Journal of Statistical Software, 21(1), 1–16.Google Scholar
  20. Everitt, B. S., & Hand, D. J. (1981). Finite mixture distributions. London: Chapman and Hall.Google Scholar
  21. Ferguson, T. S. (1973). A bayesian analysis of some nonparametric problems. The Annals of Statistics, 1(2), 209–230.Google Scholar
  22. Ferreira, J. T., & Steel, M. F. (2007). Model comparison of coordinate-free multivariate skewed distributions with an application to stochastic frontiers. Journal of Econometrics, 137(2), 641–673.Google Scholar
  23. Frühwirth-Schnatter, S. (2006). Finite mixture and Markov switching models. New York: Springer.Google Scholar
  24. Geweke, J. (1996). Monte carlo simulation and numerical integration. In H. M. Amman, D. A. Kendrick, & J. P. Rust (Eds.), Handbook of computational economics (pp. 731–800). Amsterdam: North Holland.Google Scholar
  25. Gilks, W. R., Roberts, G. O., & Sahu, S. K. (1998). Adaptive markov chain monte carlo through regeneration. Journal of the American Statistical Association, 93(443), 1045–1054.Google Scholar
  26. Giordani, P., & Kohn, R. (2010). Adaptive independent metropolis-hastings by fast estimation of mixtures of normals. Journal of Computational and Graphical Statistics, 19(2), 243–259.Google Scholar
  27. Hamerly, G., & Elkan, C. (2002). Alternatives to the k-means algorithm that find better clusterings. In C. Nicholas, D. Grossman, K. Kalpakis, S. Qureshi, H. van Dissel, L. Seligman (Eds.) Proceedings of the eleventh international conference on information and knowledge management, ACM, New York, NY, USA, pp. 600–607Google Scholar
  28. Hamilton, J. D. (1989). A new approach to the economic analysis of nonstationary time series and the business cycle. Econometrica, 57(2), 357–384.Google Scholar
  29. Han, B., Comaniciu, D., Zhu, Y., & Davis, L. S. (2008). Sequential kernel density approximation and its application to real-time visual tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(7), 1186–1197.Google Scholar
  30. Heiss, F., & Winschel, V. (2008). Likelihood approximation by numerical integration on sparse grids. Journal of Econometrics, 144(1), 62–80.Google Scholar
  31. Holmes, G. K. (1892). Measures of distribution. Publications of the American Statistical Association, 3(18/19), 141–157.Google Scholar
  32. Hoogerheide, L., Opschoor, A., & van Dijk, H. K. (2012). A class of adaptive importance sampling weighted EM algorithms for efficient and robust posterior and predictive simulation. Journal of Econometrics, 171(2), 101–120.Google Scholar
  33. Hoogerheide, L. F., Kaashoek, J. F., & van Dijk, H. K. (2007). On the shape of posterior densities and credible sets in instrumental variable regression models with reduced rank: An application of flexible sampling methods using neural networks. Journal of Econometrics, 139(1), 154–180.Google Scholar
  34. Hull, J., & White, A. (1998). Incorporating volatility updating into the historical simulation method for value-at-risk. Journal of Risk, 1, 5–19.Google Scholar
  35. Kasahara, H., & Shimotsu, K. (2015). Testing the number of components in normal mixture regression models. Journal of the American Statistical Association, 110(512), 1632–1645.Google Scholar
  36. Keane, M. P., & Wolpin, K. I. (1997). The career decisions of young men. Journal of Political Economy, 105(3), 473–522.Google Scholar
  37. Kim, S., Shephard, N., & Chib, S. (1998). Stochastic volatility: Likelihood inference and comparison with arch models. Review of Economic Studies, 65(3), 361–393.Google Scholar
  38. Kon, S. J. (1984). Models of stock returns-a comparison. Journal of Finance, 39(1), 147–165.Google Scholar
  39. Kurtz, N., & Song, J. (2013). Cross-entropy-based adaptive importance sampling using gaussian mixture. Structural Safety, 42, 35–44.Google Scholar
  40. Liesenfeld, R., & Richard, J. F. (2006). Classical and Bayesian analysis of univariate and multivariate stochastic volatility models. Econometric Reviews, 25(2–3), 335–360.Google Scholar
  41. Marchenko, Y. V., & Genton, M. G. (2012). A heckman selection-\(t\) model. Journal of the American Statistical Association, 107(497), 304–317.Google Scholar
  42. Mazzuco, S., & Scarpa, B. (2015). Fitting age-specific fertility rates by a flexible generalized skew normal probability density function. Journal of the Royal Statistical Society Series A, 178(1), 187–203.Google Scholar
  43. McLachlan, G. J., & Peel, D. (2000). Finite mixture models. New York: Wiley.Google Scholar
  44. Moe, W. W., & Fader, P. S. (2002). Using advance purchase orders to forecast new product sales. Marketing Science, 21(3), 347–364.Google Scholar
  45. Newcomb, S. (1886). A generalized theory of the combination of observations so as to obtain the best result. American Journal of Mathematics, 8(4), 343–366.Google Scholar
  46. Ogundimu, E. O., & Hutton, J. L. (2016). A sample selection model with skew-normal distribution. Scandinavian Journal of Statistics, 43(1), 172–190.Google Scholar
  47. Oh, M. S., & Berger, J. O. (1993). Integration of multimodal functions by monte carlo importance sampling. Journal of the American Statistical Association, 88(422), 450–456.Google Scholar
  48. Omori, Y., Chib, S., Shephard, N., & Nakajima, J. (2007). Stochastic volatility with leverage: Fast and efficient likelihood inference. Journal of Econometrics, 140(2), 425–449.Google Scholar
  49. Pearson, K. (1894). Contributions to the mathematical theory of evolution. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 185, 71–110.Google Scholar
  50. Pitt, M. K., & Shephard, N. (1999). Filtering via simulation: Auxiliary particle filters. Journal of the American Statistical Association, 94(446), 590–599.Google Scholar
  51. Richard, J. F., & Zhang, W. (2007). Efficient high-dimensional importance sampling. Journal of Econometrics, 141(2), 1385–1411.Google Scholar
  52. Ristic, B., Arulampalam, S., & Gordon, N. (2004). Beyond the Kalman filter: Particle filters for tracking applications. Boston: Artech House.Google Scholar
  53. Scott, D. W. (1992). Multivariate density estimation. New Jersey: Wiley.Google Scholar
  54. Scott, D. W., & Szewczyk, W. F. (2001). From kernels to mixtures. Technometrics, 43(3), 323–335.Google Scholar
  55. Smolyak, S. A. (1963). Quadrature and interpolation formulas for tensor products of certain class of functions. Soviet Mathematics, Doklady, 148(5), 1042–1045.Google Scholar
  56. Stock, J. H., & Watson, M. W. (2007). Why has U.S. inflation become harder to forecast? Journal of Money, Credit and Banking, 39(1), 3–33.Google Scholar
  57. Titterington, M., Smith, A. F., & Makov, U. E. (1985). Statistical analysis of finite mixture distributions. Chichester: Wiley.Google Scholar
  58. Tucker, A. L. (1992). A reexamination of finite- and infinite-variance distributions as models of daily stock returns. Journal of Business and Economic Statistics, 10(1), 73–81.Google Scholar
  59. Venkataraman, S. (1997). Value at risk for a mixture of normal distributions: The use of quasi- bayesian estimation techniques. Economic Perspectives, 21, 2–13.Google Scholar
  60. Wang, X., & Wang, Y. (2015). Nonparametric multivariate density estimation using mixtures. Statistics and Computing, 25(2), 349–364.Google Scholar
  61. Weldon, W. F. (1892). Certain correlated variations in Crangon vulgaris. Proceedings of the Royal Society of London, 51, 1–21.Google Scholar
  62. Weldon, W. F. (1893). On certain correlated variations in Carcinus maenas. Proceedings of the Royal Society of London, 54, 318–329.Google Scholar
  63. West, M. (1992). Modelling with mixtures. In J. M. Bernardo, J. O. Berger, M. H. DeGroot, & A. F. Smith (Eds.), Bayesian statistics 4 (pp. 503–524). Oxford: Oxford University Press.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2017

Authors and Affiliations

  1. 1.Department of EconomicsCopenhagen Business SchoolFrederiksbergDenmark
  2. 2.Department of EconomicsUniversity of PittsburghPittsburghUSA

Personalised recommendations