Abstract
We apply the Aggregating Algorithm to the problem of online regression under the square loss function. We develop an algorithm competitive with the benchmark class of generalized linear models (our “experts”), which are used in a wide range of practical tasks. This problem does not appear to be analytically tractable. Therefore, we develop a prediction algorithm using the Markov chain Monte Carlo method, which is shown to be fast and reliable in many cases. We prove upper bounds on the cumulative square loss of the algorithm. We also perform experiments with our algorithm on a toy data set and two real world ozone level data sets and give suggestions about choosing its parameters.
Chapter PDF
Similar content being viewed by others
References
Andrieu, C., de Freitas, N., Doucet, A., Jordan, M.: An introduction to MCMC for machine learning. Machine Learning 50, 5–43 (2003)
Banerjee, A.: An analysis of logistic models: exponential family connections and online performance. In: Proceedings of the 2007 SIAM International Conference on Data Mining, pp. 204–215 (2007)
Beckenbach, E.F., Bellman, R.: Inequalities. Springer, Berlin (1961)
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)
Cawley, G.C., Janacek, G.J., Talbot, N.L.C.: Generalised kernel machines. In: Proceedings of the International Joint Conference on Neural Networks, pp. 1720–1725 (2007)
Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines (2001)
Dalalyan, A., Tsybakov, A.B.: Sparse regression learning by aggregation and Langevin Monte-Carlo. In: Proceedings of the 22nd Annual Conference on Computational Learning Theory (2009)
Forster, J.: On relative loss bounds in generalized linear regression. In: Ciobanu, G., Păun, G. (eds.) FCT 1999. LNCS, vol. 1684, pp. 269–280. Springer, Heidelberg (1999)
Harville, D.A.: Matrix Algebra From a Statistician’s Perspective. Springer, New York (1997)
Hazan, E., Agarwal, A., Kale, S.: Logarithmic regret algorithms for online convex optimization. Machine Learning 69, 169–192 (2007)
Herbster, M., Warmuth, M.K.: Tracking the best expert. Machine Learning 32, 151–178 (1998)
Kakade, S.M., Ng, A.Y.: Online bounds for Bayesian algorithms. In: Proceedings of the 18th Annual Conference on Neural Information Processing Systems (2004)
Kivinen, J., Warmuth, M.K.: Relative loss bounds for multidimensional regression problems. Machine Learning 45, 301–329 (2001)
McCullagh, P., Nelder, J.: Generalized Linear Models, 2nd edn. Chapman and Hall, Boca Raton (1989)
Neal, R.M.: Regression and classification using gaussian process priors. In: Bernardo, J.M., Berger, J.O., Dawid, A.P., Smith, A.F.M. (eds.) Bayesian Statistics, vol. 6, pp. 475–501. Oxford University Press, Oxford (1999)
Neal, R.M.: Slice sampling. The Annals of Statistics 31, 705–741 (2003)
Vovk, V.: Derandomizing stochastic prediction strategies. Machine Learning 35, 247–282 (1999)
Vovk, V.: Competitive on-line statistics. International Statistical Review 69, 213–248 (2001)
Zhang, K., Fan, W.: Forecasting skewed biased stochastic ozone days: analyses, solutions and beyond. Knowledge and Information Systems 14, 299–326 (2008)
Zinkevich, M.: Online convex programming and generalized infinitesimal gradient ascent. In: Proceedings of the 20th International Conference on Machine Learning, pp. 928–936 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhdanov, F., Vovk, V. (2010). Competitive Online Generalized Linear Regression under Square Loss. In: Balcázar, J.L., Bonchi, F., Gionis, A., Sebag, M. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2010. Lecture Notes in Computer Science(), vol 6323. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15939-8_34
Download citation
DOI: https://doi.org/10.1007/978-3-642-15939-8_34
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-15938-1
Online ISBN: 978-3-642-15939-8
eBook Packages: Computer ScienceComputer Science (R0)