Abstract
We organized a competition on Autonomous Lifelong Machine Learning with Drift that was part of the competition program of NeurIPS 2018. This data driven competition asked participants to develop computer programs capable of solving supervised learning problems where the i.i.d. assumption did not hold. Large data sets were arranged in a lifelong learning and evaluation scenario and CodaLab was used as the challenge platform. The challenge attracted more than 300 participants in its 2 months duration. This chapter describes the design of the challenge and summarizes its main results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
References
J. Q. Candela, N. Lawrence, and A. Schwaighofer. Pascale 2 challenge: Learning when test and training inputs have different distributions challenge, 2005.
Zhiyuan Chen and Bing Liu. Lifelong Machine Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2016.
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2962–2970. Curran Associates, Inc., 2015.
Isabelle Guyon, Kristin Bennett, Gavin C. Cawley, Hugo Jair Escalante, Sergio Escalera, Tin Kam Ho, Núria Macià, Bisakha Ray, Mehreen Saeed, Alexander R. Statnikov, and Evelyne Viegas. Design of the 2015 chalearn automl challenge. In 2015 International Joint Conference on Neural Networks, IJCNN 2015, Killarney, Ireland, July 12–17, 2015, pages 1–8, 2015.
Isabelle Guyon, Imad Chaabane, Hugo Jair Escalante, Sergio Escalera, Damir Jajetic, James Robert Lloyd, Núria Macià, Bisakha Ray, Lukasz Romaszko, Michèle Sebag, Alexander R. Statnikov, Sébastien Treguer, and Evelyne Viegas. A brief review of the chalearn automl challenge: Any-time any-dataset learning without human intervention. In Proceedings of the 2016 Workshop on Automatic Machine Learning, AutoML 2016, co-located with 33rd International Conference on Machine Learning (ICML 2016), New York City, NY, USA, June 24, 2016, pages 21–30, 2016.
Isabelle Guyon, Gideon Dror, Vincent Lemaire, Daniel L. Silver, Graham Taylor, and David W. Aha. 2012 special issue: Analysis of the ijcnn 2011 utl challenge. Neural Networks, 32,:174–178, 2012.
Isabelle Guyon, Lisheng Sun-Hosoya, Marc Boullé, Hugo Jair Escalante, Sergio Escalera, Zhengying Liu, Damir Jajetic, Bisakha Ray, Mehreen Saeed, Michéle Sebag, Alexander Statnikov, WeiWei Tu, and Evelyne Viegas. Analysis of the automl challenge series 2015–2018. In AutoML, Springer series on Challenges in Machine Learning, 2018.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren, editors. Automatic Machine Learning. Springer Series on Challenges in Machine Learning. Springer, 2019.
Haifeng Jin, Qingquan Song, and Xia Hu. Auto-keras: Efficient neural architecture search with network morphism. Technical report, 2018.
Zhengying Liu, Olivier Bousquet, André Elisseeff, Sergio Escalera, Isabelle Guyon, Julio Jacques, Adrien Pavao, Danny Silver, Lisheng Sun-Hosoya, Sebastien Treguer, Wei-Wei Tu, Jingsong Wang, and Quanming Yao. AutoDL Challenge Design and Beta Tests-Towards automatic deep learning. In CiML workshop @ NeurIPS2018, Montreal, Canada, December 2018.
Jorge G Madrid, Hugo Jair Escalante, Eduardo F Morales, Wei-Wei Tu, Yang Yu, Lisheng Sun-Hosoya, Isabelle Guyon, and Michèle Sebag. Towards AutoML in the presence of Drift: first results. In Workshop AutoML 2018 @ ICML/IJCAI-ECAI, Stockholm, Sweden, July 2018. Pavel Brazdil, Christophe Giraud-Carrier, and Isabelle Guyon.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Trans. on Knowl. and Data Eng., 22(10):1345–1359, October 2010.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
John R. Rice. The algorithm selection problem. In Computer Science Technical Reports. Purdue University. Paper 99, 1975.
John R. Rice. The algorithm selection problem. In Morris Rubinoff and Marshall C. Yovits, editors, Advances in Computers, volume 15, pages 65–118. Elsevier, 1976.
J. Schlimmer and R. Granger. Beyond incremental processing: Tracking concept drift. In T. Kehler and S. Rosenschein, editors, Proceedings of the Fifth National Conference on Artificial Intelligence, pages 502–507. San Francisco, CA: Morgan Kaufmann, 1986.
Daniel L. Silver and Qiang Yang amd Lianghao Li. Lifelong machine learning systems: Beyond learning algorithms. In Lifelong Machine Learning: Papers from the 2013 AAAI Spring Symposium, pages 49–55, 2013.
Acknowledgements
The authors are grateful with the challenge sponsors 4Paradigm, Microsoft Research and ChaLearn, as well as the NeurIPS2018 competition chairs.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Escalante, H.J. et al. (2020). AutoML @ NeurIPS 2018 Challenge: Design and Results. In: Escalera, S., Herbrich, R. (eds) The NeurIPS '18 Competition. The Springer Series on Challenges in Machine Learning. Springer, Cham. https://doi.org/10.1007/978-3-030-29135-8_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-29135-8_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-29134-1
Online ISBN: 978-3-030-29135-8
eBook Packages: Computer ScienceComputer Science (R0)