Statistics and Computing

, Volume 25, Issue 5, pp 893–911 | Cite as

High-dimensional regression with gaussian mixtures and partially-latent response variables

  • Antoine Deleforge
  • Florence Forbes
  • Radu Horaud


The problem of approximating high-dimensional data with a low-dimensional representation is addressed. The article makes the following contributions. An inverse regression framework is proposed, which exchanges the roles of input and response, such that the low-dimensional variable becomes the regressor, and which is tractable. A mixture of locally-linear probabilistic mapping model is introduced, that starts with estimating the parameters of the inverse regression, and follows with inferring closed-form solutions for the forward parameters of the high-dimensional regression problem of interest. Moreover, a partially-latent paradigm is introduced, such that the vector-valued response variable is composed of both observed and latent entries, thus being able to deal with data contaminated by experimental artifacts that cannot be explained with noise models. The proposed probabilistic formulation could be viewed as a latent-variable augmentation of regression. Expectation-maximization (EM) procedures are introduced, based on a data augmentation strategy which facilitates the maximum-likelihood search over the model parameters. Two augmentation schemes are proposed and the associated EM inference procedures are described in detail; they may well be viewed as generalizations of a number of EM regression, dimension reduction, and factor analysis algorithms. The proposed framework is validated with both synthetic and real data. Experimental evidence is provided that the method outperforms several existing regression techniques.


Regression Latent variable Mixture models Expectation-maximization Dimensionality reduction 



The authors wish to thank the anonymous reviewers for their constructive remarks and suggestions which helped organizing and improving this article.


  1. Adragni, K.P., Cook, R.D.: Sufficient dimension reduction and prediction in regression. Philos. Trans. R. Soc. A 367(1906), 4385–4405 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  2. Agarwal, A., Triggs, B.: Learning to track 3D human motion from silhouettes. In: International conference on machine learning, pp. 9–16. Banff, Canada (2004)Google Scholar
  3. Agarwal, A., Triggs, B.: Recovering 3D human pose from monocular images. IEEE Trans. Pattern Anal. Mach. Intell. 28(1), 44–58 (2006)CrossRefGoogle Scholar
  4. Bach, F.R., Jordan, M.I.: A probabilistic interpretation of canonical correlation analysis. Tech. Rep. 688, Department of Statistics, University of California, Berkeley (2005)Google Scholar
  5. Bernard-Michel, C., Douté, S., Fauvel, M., Gardes, L., Girard, S.: Retrieval of Mars surface physical properties from OMEGA hyperspectral images using regularized sliced inverse regression. J. Geophys. Res. 114(E6), (2009)Google Scholar
  6. Bibring, J.P., Soufflot, A., Berthé, M., Langevin, Y., Gondet, B., Drossart, P., Bouyé, M., Combes, M., Puget, P., Semery, A., et al.: Omega: observatoire pour la minéralogie, l’eau, les glaces et l’activité. Mars express: the scientific payload 1240, 37–49 (2004)Google Scholar
  7. Bishop, C.M., Svensén, M., Williams, C.K.I.: GTM: the generative topographic mapping. Neural Comput 10(1), 215–234 (1998)CrossRefGoogle Scholar
  8. Bouveyron, C., Celeux, G., Girard, S.: Intrinsic dimension estimation by maximum likelihood in isotropic probabilistic PCA. Pattern Recognit. Lett. 32, 1706–1713 (2011)CrossRefGoogle Scholar
  9. Cook, R.D.: Fisher lecture: dimension reduction in regression. Stat. Sci. 22(1), 1–26 (2007)CrossRefzbMATHGoogle Scholar
  10. de Veaux, R.D.: Mixtures of linear regressions. Comput. Stat. Data Anal. 8(3), 227–245 (1989)CrossRefzbMATHGoogle Scholar
  11. Deleforge, A., Horaud, R.: 2D sound-source localization on the binaural manifold. In: IEEE workshop on machine learning for signal processing, Santander, Spain, (2012)Google Scholar
  12. Deleforge, A., Forbes, F., Horaud, R.: Acoustic space learning for sound-source separation and localization on binaural manifolds. Int. J. Neural Syst., (2014)Google Scholar
  13. Douté, S., Deforas, E., Schmidt, F., Oliva, R., Schmitt, B.: A comprehensive numerical package for the modeling of Mars hyperspectral images. In: The 38th Lunar and Planetary Science Conference, (Lunar and Planetary Science XXXVIII), (2007)Google Scholar
  14. Fusi, N., Stegle, O., Lawrence, N.: Joint modelling of confounding factors and prominent genetic regulators provides increased accuracy in genetical genomics studies. PLoS Comput. Biol. 8(1):e1002, 330, (2012)Google Scholar
  15. Gershenfeld, N.: Nonlinear inference and cluster-weighted modeling. Ann. N. Y. Acad. Sci. 808(1), 18–24 (1997)CrossRefGoogle Scholar
  16. Ghahramani, Z., Hinton, G.E.: The EM algorithm for mixtures of factor analyzers. Tech. Rep. CRG-TR-96-1, University of Toronto, (1996)Google Scholar
  17. Ingrassia, S., Minotti, S.C., Vittadini, G.: Local statistical modeling via a cluster-weighted approach with elliptical distributions. J. Classif. 29(3), 363–401 (2012)MathSciNetCrossRefGoogle Scholar
  18. Jedidi, K., Ramaswamy, V., DeSarbo, W.S., Wedel, M.: On estimating finite mixtures of multivariate regression and simultaneous equation models. Struct. Equ. Model. 3(3), 266–289 (1996)CrossRefGoogle Scholar
  19. Kain, A., Macon, M.W.: Spectral voice conversion for text-to-speech synthesis. IEEE International Conference on Acoustics, Speech, and Signal Processing, Seattle, WA, USA 1, 285–288 (1998)Google Scholar
  20. Kalaitzis, A., Lawrence, N.: Residual component analysis: Generalising pca for more flexible inference in linear-gaussian models. In: International Conference on Machine Learning, Edinburgh, Scotland, UK, (2012)Google Scholar
  21. Lawrence, N.: Probabilistic non-linear principal component analysis with gaussian process latent variable models. J. Mach. Learn. Res. 6, 1783–1816 (2005)MathSciNetzbMATHGoogle Scholar
  22. Li, K.C.: Sliced inverse regression for dimension reduction. J. Am. Stat. Assoc. 86(414), 316–327 (1991)CrossRefzbMATHGoogle Scholar
  23. McLachlan, G.J., Peel, D.: Robust cluster analysis via mixtures of multivariate t-distributions. In: Lecture Notes in Computer Science, pp 658–666. Springer, Berlin (1998)Google Scholar
  24. McLachlan, G.J., Peel, D., Bean, R.: Modelling high-dimensional data by mixtures of factor analyzers. Comput. Stat. Data Anal. 41(3–4), 379–388 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  25. Meng, X.L., Rubin, D.B.: Maximum likelihood estimation via the ECM algorithm: a general framework. Biometrika 80(2), 267–278 (1993)Google Scholar
  26. Meng, X.L., Van Dyk, D.: The EM algorithm: an old folk-song sung to a fast new tune. J. R. Stat. Soc. B 59(3), 511–567 (1997)CrossRefzbMATHGoogle Scholar
  27. Naik, P., Tsai, C.L.: Partial least squares estimator for single-index models. J. R. Stat. Soc. B 62(4), 763–771 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  28. Qiao, Y., Minematsu, N.: Mixture of probabilistic linear regressions: a unified view of GMM-based mapping techiques. In: IEEE international conference on acoustics, speech, and signal processing, pp 3913–3916, (2009)Google Scholar
  29. Quandt, R.E., Ramsey, J.B.: Estimating mixtures of normal distributions and switching regressions. J. Am. Stat. Assoc. 73(364), 730–738 (1978)MathSciNetCrossRefzbMATHGoogle Scholar
  30. Rosipal, R., Krämer, N.: Overview and recent advances in partial least squares. In: Saunders C, Grobelnik M, Gunn S, Shawe-Taylor J (eds) Subspace, latent structure and feature selection, lecture notes in computer science, vol 3940, pp 34–51. Springer, Berlin (2006)Google Scholar
  31. Smola, A.J., Schölkopf, B.: A tutorial on support vector regression. Stat. Comput. 14(3), 199–222 (2004)MathSciNetCrossRefGoogle Scholar
  32. Talmon, R., Cohen, I., Gannot, S.: Supervised source localization using diffusion kernels. In: Workshop on Applications of Signal Processing to Audio and Acoustics, pp 245–248, (2011)Google Scholar
  33. Thayananthan, A., Navaratnam, R., Stenger, B., Torr, P., Cipolla, R.: Multivariate relevance vector machines for tracking. In: European conference on computer vision, pp 124–138. Springer, Heidelberg (2006)Google Scholar
  34. Tipping, M.: Sparse Bayesian learning and the relevance vector machine. J. Mach. Learn. Res. 1, 211–244 (2001)MathSciNetzbMATHGoogle Scholar
  35. Tipping, M.E., Bishop, C.M.: Mixtures of probabilistic principal component analyzers. Neural Comput. 11(2), 443–482 (1999a)CrossRefGoogle Scholar
  36. Tipping, M.E., Bishop, C.M.: Probabilistic principal component analysis. J. R. Stat. Soc. B 61(3), 611–622 (1999b)MathSciNetCrossRefzbMATHGoogle Scholar
  37. Vapnik, V., Golowich, S., Smola, A.: Support vector method for function approximation, regression estimation, and signal processing. In: Mozer, M., Jordan, M.I., Petsche, T. (eds.) Advances in neural information processing, pp. 281–287. MIT Press, Cambridge (1997)Google Scholar
  38. Wang, C., Neal, R.M.: Gaussian process regression with heteroscedastic or non-gaussian residuals. Computing Research Repository. (2012)Google Scholar
  39. Wedel, M., Kamakura, W.A.: Factor analysis with (mixed) observed and latent variables in the exponential family. Psychometrika 66(4), 515–530 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  40. Wu, H.: Kernel sliced inverse regression with applications to classification. J. Comput. Graph. Stat. 17(3), 590–610 (2008) Google Scholar
  41. Xu, L., Jordan, M.I., Hinton, G.E.: An alternative model for mixtures of experts. In: Tesauro, G., Touretzky, D.S., Leen, T.K. (eds.) Advances in neural information processing systems, pp. 633–640. MIT Press, Cambridge (1995)Google Scholar
  42. Zhao, J.H., Yu, P.L.: Fast ML estimation for the mixture of factor analyzers via an ECM algorithm. IEEE Trans. Neural Netw. 19(11), 1956–1961 (2008)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Antoine Deleforge
    • 1
  • Florence Forbes
    • 1
  • Radu Horaud
    • 1
  1. 1.INRIA Grenoble Rhône-AlpesMontbonnot Saint-MartinFrance

Personalised recommendations