Abstract
Time series forecasting attempts to predict future events by analyzing past trends and patterns. Although well researched, certain critical aspects pertaining to the use of deep learning in time series forecasting remain ambiguous. Our research primarily focuses on examining the impact of specific hyperparameters related to time series, such as context length and validation strategy, on the performance of the state-of-the-art MLP model in time series forecasting. We have conducted a comprehensive series of experiments involving 4800 configurations per dataset across 20 time series forecasting datasets, and our findings demonstrate the importance of tuning these parameters. Furthermore, in this work, we introduce the largest metadataset for time series forecasting to date, named TSBench, comprising 97200 evaluations, which is a twentyfold increase compared to previous works in the field. Finally, we demonstrate the utility of the created metadataset on multi-fidelity hyperparameter optimization tasks.
K. Madhusudhanan and S. Jawed—This authors contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alexandrov, A., et al.: Gluonts: probabilistic time series models in python. ArXiv (2019)
Arango, S.P., Jomaa, H.S., Wistuba, M., Grabocka, J.: Hpo-b: a large-scale reproducible benchmark for black-box hpo based on openml. In: NeurIPS Datasets and Benchmarks Track (2021)
Borchert, O., Salinas, D., Flunkert, V., Januschowski, T., Gunnemann, S.: Multi-objective model selection for time series forecasting. ArXiv (2022)
Clevert, D.A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (elus). In: ICLR (2015)
Deng, D., Karl, F., Hutter, F., Bischl, B., Lindauer, M.: Efficient automated deep learning for time series forecasting. In: ECML PKDD, pp. 664–680. Springer, Heidelberg (2023). https://doi.org/10.1007/978-3-031-26409-2_40
Falkner, S., Klein, A., Hutter, F.: Bohb: robust and efficient hyperparameter optimization at scale. In: ICML, pp. 1437–1446. PMLR (2018)
Godahewa, R., Bergmeir, C., Webb, G.I., Hyndman, R.J., Montero-Manso, P.: Monash time series forecasting archive. In: NeurIPS Datasets and Benchmarks (2021)
Jawed, S., Jomaa, H., Schmidt-Thieme, L., Grabocka, J.: Multi-task learning curve forecasting across hyperparameter configurations and datasets. In: ECML PKDD, pp. 485–501 (2021)
Jomaa, H.S., Schmidt-Thieme, L., Grabocka, J.: Dataset2vec: learning dataset meta-features. Data Min. Knowl. Disc. 35, 964–985 (2021)
Kadra, A., Lindauer, M., Hutter, F., Grabocka, J.: Well-tuned simple nets excel on tabular datasets. In: NeurIPS, vol. 34, pp. 23928–23941 (2021)
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., Talwalkar, A.: Hyperband: a novel bandit-based approach to hyperparameter optimization. JMLR 18(1), 1–52 (2017)
Lindauer, M., et al.: Smac3: a versatile bayesian optimization package for hyperparameter optimization. JMLR 23(54), 1–9 (2022)
Madhusudhanan, K., Burchert, J., Duong-Trung, N., Born, S., Schmidt-Thieme, L.: U-net inspired transformer architecture for far horizon time series forecasting. In: ECML/PKDD (2021)
Nie, Y., Nguyen, N.H., Sinthong, P., Kalagnanam, J.: A time series is worth 64 words: long-term forecasting with transformers. In: ICLR (2023)
Oreshkin, B.N., Carpov, D., Chapados, N., Bengio, Y.: N-BEATS: neural basis expansion analysis for interpretable time series forecasting. In: ICLR (2020)
Rasul, K., Sheikh, A.S., Schuster, I., Bergmann, U.M., Vollgraf, R.: Multivariate probabilistic time series forecasting via conditioned normalizing flows. In: ICLR (2021)
Salinas, D., Flunkert, V., Gasthaus, J., Januschowski, T.: Deepar: probabilistic forecasting with autoregressive recurrent networks. JMLR 36(3), 1181–1191 (2020)
Shah, S.Y., et al.: Autoai-ts: autoai for time series forecasting. In: SIGMOD, pp. 2584–2596 (2021)
Ullah, I., et al.: Meta-album: multi-domain meta-dataset for few-shot image classification. In: NeurIPS, vol. 35, pp. 3232–3247 (2022)
Wu, H., Xu, J., Wang, J., Long, M.: Autoformer: decomposition transformers with auto-correlation for long-term series forecasting. In: NeurIPS, vol. 34, pp. 22419–22430 (2021)
Zeng, A., Chen, M., Zhang, L., Xu, Q.: Are transformers effective for time series forecasting? In: AAAI (2023)
Zhou, H., et al.: Informer: beyond efficient transformer for long sequence time-series forecasting. In: AAAI, vol. 35, pp. 11106–11115 (2021)
Zimmer, L., Lindauer, M., Hutter, F.: Auto-pytorch tabular: multi-fidelity metalearning for efficient and robust autodl. IEEE Trans. Pattern Anal. Mach. Intell. 43(9), 3079–3090 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Madhusudhanan, K., Jawed, S., Schmidt-Thieme, L. (2024). Hyperparameter Tuning MLP’s for Probabilistic Time Series Forecasting. In: Yang, DN., Xie, X., Tseng, V.S., Pei, J., Huang, JW., Lin, J.CW. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2024. Lecture Notes in Computer Science(), vol 14650. Springer, Singapore. https://doi.org/10.1007/978-981-97-2266-2_21
Download citation
DOI: https://doi.org/10.1007/978-981-97-2266-2_21
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-2265-5
Online ISBN: 978-981-97-2266-2
eBook Packages: Computer ScienceComputer Science (R0)