Prediction with Expert Evaluators’ Advice

  • Alexey Chernov
  • Vladimir Vovk
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5809)

Abstract

We introduce a new protocol for prediction with expert advice in which each expert evaluates the learner’s and his own performance using a loss function that may change over time and may be different from the loss functions used by the other experts. The learner’s goal is to perform better or not much worse than each expert, as evaluated by that expert, for all experts simultaneously. If the loss functions used by the experts are all proper scoring rules and all mixable, we show that the defensive forecasting algorithm enjoys the same performance guarantee as that attainable by the Aggregating Algorithm in the standard setting and known to be optimal. This result is also applied to the case of “specialist” experts. In this case, the defensive forecasting algorithm reduces to a simple modification of the Aggregating Algorithm.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Cesa-Bianchi, N., Lugosi, G.: Prediction, Learning, and Games. Cambridge University Press, Cambridge (2006)CrossRefMATHGoogle Scholar
  2. 2.
    Chernov, A., Kalnishkan, Y., Zhdanov, F., Vovk, V.: Supermartingales in prediction with expert advice. In: Freund, Y., Györfi, L., Turán, G., Zeugmann, T. (eds.) ALT 2008. LNCS (LNAI), vol. 5254, pp. 199–213. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  3. 3.
    Freund, Y., Schapire, R.E., Singer, Y., Warmuth, M.K.: Using and combining predictors that specialize. In: Proceedings of the Twenty Ninth Annual ACM Symposium on Theory of Computing, New York, Association for Computing Machinery, pp. 334–343 (1997)Google Scholar
  4. 4.
    Chernov, A., Vovk, V.: Prediction with expert evaluators’ advice. Technical Report arXiv:0902.4127 [cs.LG], arXiv.org e-Print archive (2009)Google Scholar
  5. 5.
    Haussler, D., Kivinen, J., Warmuth, M.K.: Sequential prediction of individual sequences under general loss functions. IEEE Transactions on Information Theory 44, 1906–1925 (1998)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Vovk, V.: A game of prediction with expert advice. Journal of Computer and System Sciences 56, 153–173 (1998)MathSciNetCrossRefMATHGoogle Scholar
  7. 7.
    Vovk, V.: Derandomizing stochastic prediction strategies. Machine Learning 35, 247–282 (1999)CrossRefMATHGoogle Scholar
  8. 8.
    Dawid, A.P.: Probability forecasting. In: Kotz, S., Johnson, N.L., Read, C.B. (eds.) Encyclopedia of Statistical Sciences, vol. 7, pp. 210–218. Wiley, New York (1986)Google Scholar
  9. 9.
    Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102, 359–378 (2007)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Press, W.H., Flannery, B.P., Teukolsky, S.A., Vetterling, W.T.: Numerical Recipes in C, 2nd edn. Cambridge University Press, Cambridge (1992)MATHGoogle Scholar
  11. 11.
    Vovk, V.: Defensive forecasting for optimal prediction with expert advice. Technical Report arXiv:0708.1503 [cs.LG], arXiv.org e-Print archive (August 2007)Google Scholar
  12. 12.
    Shafer, G., Vovk, V.: Probability and Finance: It’s Only a Game! Wiley, New York (2001)CrossRefMATHGoogle Scholar
  13. 13.
    Dawid, A.P., Vovk, V.: Prequential probability: principles and properties. Bernoulli 5, 125–162 (1999)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Levin, L.A.: Uniform tests of randomness. Soviet Mathematics Doklady 17, 337–340 (1976)MATHGoogle Scholar
  15. 15.
    Vovk, V., Takemura, A., Shafer, G.: Defensive forecasting. In: Cowell, R.G., Ghahramani, Z. (eds.) Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, Savannah Hotel, Barbados, Society for Artificial Intelligence and Statistics, January 6-8, pp. 365–372 (2005), http://www.gatsby.ucl.ac.uk/aistats/

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Alexey Chernov
    • 1
  • Vladimir Vovk
    • 1
  1. 1.Computer Learning Research Centre, Department of Computer ScienceRoyal Holloway, University of LondonEghamUK

Personalised recommendations