Gaussian process classification: singly versus doubly stochastic models, and new computational schemes

  • Jens Röder
  • Raimon Tolosana-Delgado
  • Fred A. HamprechtEmail author
Original Paper


The aim of this paper is to compare four different methods for binary classification with an underlying Gaussian process with respect to theoretical consistency and practical performance. Two of the inference schemes, namely classical indicator kriging and simplicial indicator kriging, are analytically tractable and fast. However, these methods rely on simplifying assumptions which are inappropriate for categorical class labels. A consistent and previously described model extension involves a doubly stochastic process. There, the unknown posterior class probability f(·) is considered a realization of a spatially correlated Gaussian process that has been squashed to the unit interval, and a label at position x is considered an independent Bernoulli realization with success parameter f(x). Unfortunately, inference for this model is not known to be analytically tractable. In this paper, we propose two new computational schemes for the inference in this doubly stochastic model, namely the “Aitchison Maximum Posterior” and the “Doubly Stochastic Gaussian Quadrature”. Both methods are analytical up to a final step where optimization or integration must be carried out numerically. For the comparison of practical performance, the methods are applied to storm forecasts for the Spanish coast based on wave heights in the Mediterranean Sea. While the error rate of the doubly stochastic models is slightly lower, their computational cost is much higher.


Model-based geostatistics Logistic transformation Classical indicator kriging Simplicial indicator kriging Doubly stochastic process Aitchison maximum posterior Doubly stochastic Gaussian quadrature 



We gratefully acknowledge financial support by the Robert Bosch GmbH (J.R., F.A.H.), by the Deutsche Forschungsgemeinschaft (DFG) under grant no. HA4364/2-1 (F.A.H.) as well as by the Spanish Ministry of Science and Innovation through a “Juan de la Cierva” grant JCI-2008-1835 (R.T.), supported by the European Social Fund (ESF-FSE).


  1. Abramowitz M, Stegun IA (1965) Handbook of mathematical functions. Dover, New YorkGoogle Scholar
  2. Aitchison J (1982) The statistical analysis of compositional data (with discussion). J R Stat Soc B Stat Methodol 44(2):139–177Google Scholar
  3. Aomoto K (1977) Analytic structure of the Schläfli function. Nagoya Math J 68:1–16Google Scholar
  4. Billheimer D, Guttorp P, Fagan WF (2001) Statistical interpretation of species composition. J Am Stat Assoc 96:1205–1214CrossRefGoogle Scholar
  5. Bogaert P (2002) Spatial prediction of categorical variables: the Bayesian maximum entropy approach. Stoch Environ Res Risk Assess 16:425–448CrossRefGoogle Scholar
  6. Carle S, Fogg G (1996) Transition probability-based indicator geostatistics. Math Geol 28(4):453–476CrossRefGoogle Scholar
  7. Carr J, Mao N (1993) A general-form of probability kriging for estimation of the indicator and uniform transforms. Math Geol 25(4):425–438CrossRefGoogle Scholar
  8. Chilès JP, Delfiner P (1999) Geostatistics: modeling spatial uncertainty. Wiley and Sons, New YorkGoogle Scholar
  9. Christakos G (1990) A Bayesian/maximum entropy view to the spatial estimation problem. Math Geol 22(7):763–777CrossRefGoogle Scholar
  10. Diggle PJ, Tawn JA, Moyeed RA (1998) Model-based geostatistics (with discussion). J Royal Stat Soc C Appl Stat 47(3):299–350CrossRefGoogle Scholar
  11. Genz A, Bretz F (2009) Computation of multivariate normal and t probabilities. In: Lecture notes in statistics, vol 195. Springer, HeidelbergCrossRefGoogle Scholar
  12. Gibbs MN (1997) Bayesian Gaussian processes for classification and regression. Dissertation, University of CambridgeGoogle Scholar
  13. Gibbs MN, MacKay DJC (2000) Variational Gaussian process classifiers. IEEE Trans Neural Netw 11(6):1458–1464CrossRefGoogle Scholar
  14. Hsu Y, Tung Y, Kuo J (2010) Evaluation of dam overtopping probability induced by flood and wind. Stoch Environ Res Risk Assess 25(1):35–49CrossRefGoogle Scholar
  15. Journel AG (1983) Nonparametric estimation of spatial distributions. Math Geol 15(3):445–468CrossRefGoogle Scholar
  16. Journel AG, Posa D (1990) Characteristic behavior and order relations for indicator variograms. Math Geol 22(8):1011–1025CrossRefGoogle Scholar
  17. Kazianka H, Pilz J (2010) Copula-based geostatistical modeling of continuous and discrete data including covariates. Stoch Environ Res Risk Assess 24(5):661–673CrossRefGoogle Scholar
  18. Kuss M, Rasmussen CE (2005) Assessing approximate inference for binary Gaussian process classification. J Mach Learn Res 6:1679–1704Google Scholar
  19. Lim YB, Sacks J, Studdeen W, Welch W (2002) Design and analysis of computer experiments when the output is highly correlated over the input space. Can J Stat 30(1):109–126CrossRefGoogle Scholar
  20. Minka TP (2001) A family of algorithms for approximate Bayesian inference. PhD thesis, Massachusetts Institute of Technology, CambridgeGoogle Scholar
  21. Neal RM (1999) Regression and classification using Gaussian process priors. In: Bernardo JM, Berger JO, Dawid AP, Smith AFM (eds) Bayesian statistics vol 6. Oxford University Press, Oxford, pp 475–501Google Scholar
  22. Opper M, Winther O (2000) Gaussian processes for classification: mean field algorithms. Neural Comput 12(11):2655–2684CrossRefGoogle Scholar
  23. Pardo-Igúzquiza E, Dowd P (2005) Multiple indicator cokriging with application to optimal sampling for environmental monitoring. Comp Geosci 31(1):1–13CrossRefGoogle Scholar
  24. Pawlowsky-Glahn V (2003) Statistical modelling on coordinates. In: Thió-Henestrosa S, Martín-Fernández JA (eds) Compositional data analysis workshop—CoDaWork’03. Universitat de Girona, Girona.
  25. Pawlowsky-Glahn V, Egozcue JJ (2001) Geometric approach to statistical analysis on the simplex. Stoch Environ Res and Risk Assess 15:384–398CrossRefGoogle Scholar
  26. Petersen KB, Pedersen MS (2008) The matrix cookbook. Accessed 11 February 2010
  27. Rasmussen CE (1996) Evaluation of Gaussian processes and other methods for non-linear regression. Dissertation, Graduate Department of Computer Science, University of Toronto, TorontoGoogle Scholar
  28. Rasmussen CE, Williams CKI (2006) Gaussian processes for machine learning. MIT Press, CambridgeGoogle Scholar
  29. Rue H, Martino S, Chopin N (2009) Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations. J R Soc B 71:319–392CrossRefGoogle Scholar
  30. Suro-Perez V, Journel AG (1991) Indicator principal component kriging. Math Geol 23(5):759–788CrossRefGoogle Scholar
  31. Tolosana-Delgado R (2006) Geostatistics for constrained variables: positive data, compositions and probabilities. Application to environmental hazard monitoring. Dissertation, Universitat de Girona, GironaGoogle Scholar
  32. Tolosana-Delgado R, Pawlowsky-Glahn V, Egozcue JJ (2008) Indicator kriging without order relation violations. Math Geosci 40:327–347CrossRefGoogle Scholar
  33. Varma S, Simon R (2006) Bias in error estimation when using cross-validation for model selection. BMC Bioinform 7:91CrossRefGoogle Scholar
  34. Wiener N (1949) Extrapolation, interpolation and smoothing of stationary time series with engineering applications. Wiley and Sons, New YorkGoogle Scholar
  35. Williams CKI, Barber D (1998) Bayesian classification with Gaussian processes. IEEE Trans Pattern Anal Mach Intell 20(12):1342–1351CrossRefGoogle Scholar
  36. Williams CKI, Rasmussen CE (1996) Gaussian processes for regression. In: Touretzky DS, Mozer MC, Hasselmo ME (eds) Advances in neural information processing systems, vol 8. MIT Press, London, pp 514–520Google Scholar
  37. Yu H, Yang S, Yen H, Christakos G (2010) A spatio-temporal climate-based model of early dengue fever warning in southern Taiwan. Stoch Environ Res Risk Assess 25(4):485–494CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  • Jens Röder
    • 1
  • Raimon Tolosana-Delgado
    • 2
  • Fred A. Hamprecht
    • 3
    Email author
  1. 1.Corporate Sector Research and Advance Engineering Multimedia, Telematic and Surround Sensing Systems (CR/AEM)Robert Bosch GmbHHildesheimGermany
  2. 2.Maritime Engineering Laboratory (LIM)Universitat Politécnica de Catalunya (UPC)BarcelonaSpain
  3. 3.Heidelberg Collaboratory for Image Processing (HCI)University of HeidelbergHeidelbergGermany

Personalised recommendations