Abstract
Machine learning and deep learning (ML-DL) based models are widely used for rainfall-runoff prediction and they have potential to substitute process-oriented physics based numerical models. However, developing an ML model has also performance uncertainty because of inaccurate choices of hyperparameters and neural networks architectures. Thus, this study aims to search for best optimization algorithms to be used in ML-DL models namely, RMSprop, Adagrad, Adadelta, and Adam optimizers, as well as dropout techniques to be integrated into the Long Short Term Memory (LSTM) model to improve forecasting accuracy of rainfall-runoff modeling. A deep learning LSTMs were developed using 480 model architectures at two hydro-meteorological stations of the Mekong Delta, Vietnam, namely Chau Doc and Can Tho. The model performance is tested with the most ideally suited LSTM optimizers utilizing combinations of four dropout percentages respectively, 0%, 10%, 20%, and 30%. The Adagrad optimizer shows the best model performance in the model testing. Deep learning LSTM models with 10% dropout made the best prediction results while significantly reducing overfitting tendency of the forecasted time series. The findings of this study are valuable for ML-based hydrological models set up by identifying a suitable gradient descent (GD) optimizer and optimal dropout ratio to enhance the performance and forecasting accuracy of the ML model.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of Data and Materials
The data that support the findings of this study are available from the first author, [Duong Tran Anh, duong.trananh@vlu.edu.vn], upon reasonable request.
References
Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S (2016) Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467
Chau KW (2017) Use of meta-heuristic techniques in rainfall-runoff modelling. Water 9(3). https://doi.org/10.3390/w9030186
Dang NM, Anh DT, Dang TD (2019) ANN optimized by PSO and Firefly algorithms for predicting scour depths around bridge piers. Eng Comput 1–11
Dean J, Corrado G, Monga R, Chen K, Devin M, Mao M, Ranzato MA, Senior A, Tucker P, Yang K, Le QV (2012) Large scale distributed deep networks. Adv Neural Inf Process Syst 1223–1231
Deng L, Li J, Huang JT, Yao K, Yu D, Seide F, Seltzer M, Zweig G, He X, Williams J, Gong Y (2013) Recent advances in deep learning for speech research at Microsoft. IEEE Int Conf Acoust Speech Signal Process 8604–8608
Dogo EM, Afolabi OJ, Nwulu NI, Twala B, Aigbavboa CO (2018) December. A comparative analysis of gradient descent-based optimization algorithms on convolutional neural networks. In 2018 International Conference on Computational Techniques, Electronics and Mechanical Systems (CTEMS) (pp. 92–99). IEEE
Dozat T (2016) Incorporating nesterov momentum into adam. ICLR Workshop 1:2013–2016
Duchi J, Hazan E, Singer Y (2011) Adaptive subgradient methods for online learning and stochastic optimization. J Mach Learn Res 12:2121–2159
Duong TA, Song PV, Dang TD, Hoang LP (2019) Downscaling rainfall using deep learning long short-term memory and feedforward neural network. Int J Climatol 39(10):4170–4188
Fang L, Shao D (2022) Application of long short-term memory (LSTM) on the prediction of rainfall-runoff in karst area. Front Phys 685
Fu M, Fan T, Ding ZA, Salih SQ, Al-Ansari N, Yaseen ZM (2020) Deep learning data-intelligence model based on adjusted forecasting window scale: Application in daily streamflow simulation. IEEE Access 8:32632–32651. https://doi.org/10.1109/ACCESS.2020.2974406
Gal Y, Ghahramani Z (2016) A theoretically grounded application of dropout in recurrent neural networks. Adv Neural Inf Process Syst 29:1019–1027
Gao W, Raftari M, Rashid ASA, Mu’azu MA, Jusoh WAW (2020) A predictive model based on an optimized ANN combined with ICA for predicting the stability of slopes. Eng Comput 36(1):325–344
Gauch M, Kratzert F, Klotz D, Nearing G, Lin J, Hochreiter S (2021) Rainfall–runoff prediction at multiple timescales with a single Long Short-Term Memory network. Hydrol Earth Syst Sci 25(4):2045–2062
Hinton G, Srivastava N, Swersky K (2012a) Neural networks for machine learning. Lecture 6a: Overview of mini-batch gradient descent. Retrieved from https://www.coursera.org/ on 30 Apr 2020
Hinton G, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR (2012b) Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580
Hinton G, Deng L, Yu D, Dahl GE, Mohamed AR, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath TN, Kingsbury B (2012c) Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Process Mag IEEE 29(6):82–97
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780
Hu C, Wu Q, Li H, Jian S, Li N, Lou Z (2018) Deep learning with a long short-term memory networks approach for rainfall-runoff simulation. Water 10(11):1543
Hussain D, Hussain T, Khan AA, Naqvi SAA, Jamil A (2020) A deep learning approach for hydrological time-series prediction: A case study of Gilgit River Basin. Earth Sci Inf 13(3):915–927. https://doi.org/10.1007/s12145-020-00477-2
Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML
Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarranma S, Darrell T (2014) Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia 675–678
Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
Kratzert F, Klotz D, Brenner C, Schulz K, Herrnegger M (2018a) Rainfall–runoff modelling using long short-term memory (LSTM) networks. Hydrol Earth Syst Sci 22(11):6005–6022
Kratzert F, Klotz D, Brenner C, Schulz K, Herrnegger M (2018b) Rainfall–runoff modelling using long short-term memory (LSTM) networks. Hydrol Earth Syst Sci 22:6005–6022. https://doi.org/10.5194/hess-22-6005-2018
Krizhevsky A, Sutskever I, Hinton G (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25
Lawrence S, Giles CL, Tsoi AC (1997) Lessons in neural network training: Overfitting may be harder than expected. In AAAI/IAAI 540–545
Lin Yongen, Wang Dagang, Wang Guiling, Qiu Jianxiu, Long Kaihao, Yi Du, Xie Hehai, Wei Zhongwang, Shangguan Wei, Dai Yongjiu (2021) A hybrid deep learning algorithm and its application to streamflow prediction. J Hydrol 601:126636. https://doi.org/10.1016/j.jhydrol.2021.126636
Liu D, Jiang W, Lin Mu, Wang Si (2020) Streamflow prediction using deep learning neural network: case study of Yangtze River. IEEE Access 8:90069–90086. https://doi.org/10.1109/ACCESS.2020.2993874
Man Y, Yang Q, Shao J, Wang G, Bai L, Xue Y (2022) Enhanced LSTM model for daily runoff prediction in the upper Huai River Basin, China. Engineering
Ouma YO, Cheruyot R, Wachera AN (2022) Rainfall and runoff time-series trend analysis using LSTM recurrent neural network and wavelet neural network with satellite-based meteorological data: case study of Nzoia hydrologic basin. Complex Intell Syst 8(1):213–236
Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Köpf A, Yang E, DeVito A, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chitala S (2019) PyTorch: An imperative style, high-performance deep learning library. Adv Neural Inf Process Syst 8024–8035
Pham V, Bluche T, Kermorvant C, Louradour J (2014) Dropout Improves Recurrent Neural Networks for Handwriting Recognition. 14th International Conference on Frontiers in Handwriting Recognition 285–290. https://doi.org/10.1109/ICFHR.2014.55
Reddi SJ, Kale S, Kumar S (2018) On the convergence of adam and beyond. Proc ICLR
Robbins H, Monro S (1951) A stochastic approximation method. Ann Math Stat 400–407
Ruder S (2016) An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747
Sha J, Li X, Zhang M, Wang Z-L (2021) Comparison of forecasting models for real-time monitoring of water quality parameters based on hybrid deep learning neural networks. Water. https://doi.org/10.3390/w13111547
Song PV, Hoang ML, Dat VT, Thanh DD, Loc HH, Duong TA (2020) Deep learning convolutional neural network in rainfall-runoff modelling. J Hydroinf 22(3):541–561. https://doi.org/10.2166/hydro.2020.095
Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958
Wilson AC, Roelofs R, Stern M, Srebro N, Recht B (2017) The marginal value of adaptive gradient methods in machine learning. Adv Neural Inf Process Syst 4148–4158
Xiang Z, Yan J, Demir I (2020) A Rainfall-Runoff Model With LSTM-Based Sequence-to-Sequence Learning. Water Resour Res 56(1)
Yaseen ZM, El-shafie A, Jaafar O, Afan HA, Sayl KN (2015) Artificial intelligence based models for stream-flow forecasting: 2000–2015. J Hydrol 530:829–844
Zeiler MD (2012) ADADELTA: An adaptive learning rate method. arXiv e-prints, arXiv:1212.5701
Acknowledgements
The first author acknowledges the financial support from the Fulbright Visiting Scholar program at the University of South Florida, USA. We also thank the Southern Regional Hydro-meteorological Center and National meteorological center for providing daily rainfall and runoff data in this study.
Author information
Authors and Affiliations
Contributions
Duong Tran Anh, Dat Vi Thanh: Project administration, Conceptualization, Writing- original draft, Software, Formal analysis, Visualization. Hoang Minh Le, Bang Tran Sy, Ahad Hasan Tanim: Formal analysis; Writing- original draft, Visualization. Quoc Bao Pham, Thanh Duc Dang, Son T. Mai: Data curation, Writing, Review, and editing. Nguyen Mai Dang: Supervision, Writing, Review, and Editing.
Corresponding author
Ethics declarations
Ethical Approval
Not applicable.
Consent to Participate
Not applicable.
Consent to Publish
Not applicable.
Competing Interests
This manuscript has not been published or presented elsewhere in part or in entirety and is not under consideration by another journal. There are no conflicts of interest to declare.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Anh, D.T., Thanh, D.V., Le, H.M. et al. Effect of Gradient Descent Optimizers and Dropout Technique on Deep Learning LSTM Performance in Rainfall-runoff Modeling. Water Resour Manage 37, 639–657 (2023). https://doi.org/10.1007/s11269-022-03393-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11269-022-03393-w