Prediction with Expert Evaluators’ Advice
Abstract
We introduce a new protocol for prediction with expert advice in which each expert evaluates the learner’s and his own performance using a loss function that may change over time and may be different from the loss functions used by the other experts. The learner’s goal is to perform better or not much worse than each expert, as evaluated by that expert, for all experts simultaneously. If the loss functions used by the experts are all proper scoring rules and all mixable, we show that the defensive forecasting algorithm enjoys the same performance guarantee as that attainable by the Aggregating Algorithm in the standard setting and known to be optimal. This result is also applied to the case of “specialist” experts. In this case, the defensive forecasting algorithm reduces to a simple modification of the Aggregating Algorithm.
Keywords
Loss Function Expert Advice Specialist Expert Performance Guarantee Cumulative LossPreview
Unable to display preview. Download preview PDF.
References
- 1.Cesa-Bianchi, N., Lugosi, G.: Prediction, Learning, and Games. Cambridge University Press, Cambridge (2006)CrossRefzbMATHGoogle Scholar
- 2.Chernov, A., Kalnishkan, Y., Zhdanov, F., Vovk, V.: Supermartingales in prediction with expert advice. In: Freund, Y., Györfi, L., Turán, G., Zeugmann, T. (eds.) ALT 2008. LNCS (LNAI), vol. 5254, pp. 199–213. Springer, Heidelberg (2008)CrossRefGoogle Scholar
- 3.Freund, Y., Schapire, R.E., Singer, Y., Warmuth, M.K.: Using and combining predictors that specialize. In: Proceedings of the Twenty Ninth Annual ACM Symposium on Theory of Computing, New York, Association for Computing Machinery, pp. 334–343 (1997)Google Scholar
- 4.Chernov, A., Vovk, V.: Prediction with expert evaluators’ advice. Technical Report arXiv:0902.4127 [cs.LG], arXiv.org e-Print archive (2009)Google Scholar
- 5.Haussler, D., Kivinen, J., Warmuth, M.K.: Sequential prediction of individual sequences under general loss functions. IEEE Transactions on Information Theory 44, 1906–1925 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
- 6.Vovk, V.: A game of prediction with expert advice. Journal of Computer and System Sciences 56, 153–173 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
- 7.Vovk, V.: Derandomizing stochastic prediction strategies. Machine Learning 35, 247–282 (1999)CrossRefzbMATHGoogle Scholar
- 8.Dawid, A.P.: Probability forecasting. In: Kotz, S., Johnson, N.L., Read, C.B. (eds.) Encyclopedia of Statistical Sciences, vol. 7, pp. 210–218. Wiley, New York (1986)Google Scholar
- 9.Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102, 359–378 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
- 10.Press, W.H., Flannery, B.P., Teukolsky, S.A., Vetterling, W.T.: Numerical Recipes in C, 2nd edn. Cambridge University Press, Cambridge (1992)zbMATHGoogle Scholar
- 11.Vovk, V.: Defensive forecasting for optimal prediction with expert advice. Technical Report arXiv:0708.1503 [cs.LG], arXiv.org e-Print archive (August 2007)Google Scholar
- 12.Shafer, G., Vovk, V.: Probability and Finance: It’s Only a Game! Wiley, New York (2001)CrossRefzbMATHGoogle Scholar
- 13.Dawid, A.P., Vovk, V.: Prequential probability: principles and properties. Bernoulli 5, 125–162 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
- 14.Levin, L.A.: Uniform tests of randomness. Soviet Mathematics Doklady 17, 337–340 (1976)zbMATHGoogle Scholar
- 15.Vovk, V., Takemura, A., Shafer, G.: Defensive forecasting. In: Cowell, R.G., Ghahramani, Z. (eds.) Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, Savannah Hotel, Barbados, Society for Artificial Intelligence and Statistics, January 6-8, pp. 365–372 (2005), http://www.gatsby.ucl.ac.uk/aistats/