Sparse Approximation for Gaussian Process with Derivative Observations

  • Ang YangEmail author
  • Cheng Li
  • Santu Rana
  • Sunil Gupta
  • Svetha Venkatesh
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11320)


We propose a sparse Gaussian process model to approximate full Gaussian process with derivatives when a large number of function observations t and derivative observations \(t'\) exist. By introducing a small number of inducing point m, the complexity of posterior computation can be reduced from \(\mathcal {O}((t+t')^{3})\) to \(\mathcal {O}((t+t')m^{2})\). We also find the usefulness of our approach in Bayesian optimisation. Experiments demonstrate the superiority of our approach.


Sparse Gaussian process model Bayesian optimisation Derivative-based 



This research was partially funded by the Australian Government through the Australian Research Council (ARC). Prof Venkatesh is the recipient of an ARC Australian Laureate Fellowship (FL170100006).


  1. 1.
    Riihimäki, J., Vehtari, A.: Gaussian processes with monotonicity information. In: Proceedings of the Thirteenth International Conference on AIStat (2010)Google Scholar
  2. 2.
    Siivola, E., Vehtari, A., Vanhatalo, J., González, J.: Bayesian optimization with virtual derivative sign observations. arXiv preprint arXiv:1704.00963 (2017)
  3. 3.
    Solak, E., Murray-Smith, R., Leithead, W.E., Leith, D.J., Rasmussen, C.E.: Derivative observations in Gaussian process models of dynamic systems. In: NIPS (2003)Google Scholar
  4. 4.
    Jauch, M., Peña, V.: Bayesian optimization with shape constraints. arXiv preprint arXiv:1612.08915 (2016)
  5. 5.
    Li, C., Rana, S., Gupta, S., Nguyen, V., Venkatesh, S.: Bayesian optimization with monotonicity information (2017)Google Scholar
  6. 6.
    Yang, A., Li, C., Rana, S., Gupta, S., Venkatesh, S.: Efficient Bayesian optimisation using derivative meta-model. In: Geng, X., Kang, B.-H. (eds.) PRICAI 2018. LNCS (LNAI), vol. 11013, pp. 256–264. Springer, Cham (2018). Scholar
  7. 7.
    Snelson, E., Ghahramani, Z.: Sparse gaussian processes using pseudo-inputs. In: Advances in Neural Information Processing Systems, pp. 1257–1264 (2006)Google Scholar
  8. 8.
    Snelson, E., Ghahramani, Z.: Variable noise and dimensionality reduction for sparse Gaussian processes. arXiv preprint arXiv:1206.6873 (2012)
  9. 9.
    Quiñonero-Candela, J., Rasmussen, C.E.: A unifying view of sparse approximate Gaussian process regression. J. Mach. Learn. Res. 6(Dec), 1939–1959 (2005)MathSciNetzbMATHGoogle Scholar
  10. 10.
    Li, C., Gupta, S., Rana, S., Nguyen, V., Venkatesh, S., Shilton, A.: High dimensional Bayesian optimization using dropout. arXiv preprint arXiv:1802.05400 (2018)
  11. 11.
    Rana, S., Li, C., Gupta, S., Nguyen, V., Venkatesh, S.: High dimensional Bayesian optimization with elastic gaussian process. In: International Conference on Machine Learning, pp. 2883–2891 (2017)Google Scholar
  12. 12.
    Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13(4), 455–492 (1998)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Rasmussen, C.E., Williams, C.K.: Gaussian Processes for Machine Learning. MIT press, Cambridge (2006)zbMATHGoogle Scholar
  14. 14.
    Brochu, E., Cora, V.M., De Freitas, N.: A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599 (2010)
  15. 15.
    Mockus, J.: Application of Bayesian approach to numerical methods of global and stochastic optimization. J. Glob. Optim. 4(4), 347–365 (1994)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Kushner, H.J.: A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. J. Basic Eng. 86(1), 97–106 (1964)CrossRefGoogle Scholar
  17. 17.
    Srinivas, N., Krause, A., Kakade, S.M., Seeger, M.: Gaussian process optimization in the bandit setting: no regret and experimental design. arXiv preprint arXiv:0912.3995 (2009)
  18. 18.
    Finkel, D.E.: Direct optimization algorithm user guide. CRSC 2, 1–14 (2003)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Ang Yang
    • 1
    Email author
  • Cheng Li
    • 1
  • Santu Rana
    • 1
  • Sunil Gupta
    • 1
  • Svetha Venkatesh
    • 1
  1. 1.Center for Pattern Recognition and Data AnalyticsDeakin UniversityGeelongAustralia

Personalised recommendations