Abstract
Satellite-based precipitation (SBP) is emerging as a reliable source for high-resolution rainfall estimates over the globe. However, uncertainty in SBP is still significant, limiting their use without evaluation and often without bias correction. The bias correction of SBP remains a challenge for atmospheric scientists. The present study evaluated the performance of six SBPs, namely, SM2RAIN-ASCAT, IMERG, GSMaP, CHIRPS, PERSIANN-CDS and PERSIANN-CSS, in replicating observed daily rainfall at 364 stations over Peninsular Malaysia. The bias of the most suitable SBP was corrected using a novel machine learning (ML)-based bias-correction method. The proposed bias-correction method consists of an ML classifier to correct the bias in estimating rainfall occurrence and an ML regression model to correct the rainfall amount during rainfall events. Besides, the study evaluated the performance of different widely used ML algorithms for classification and regression to select the most suitable algorithms for bias correction. IMERG showed better performance, showing a higher correlation coefficient (R2) of 0.57 and Kling-Gupta Efficiency (KGE) of 0.5 compared to the other products. The performance of random forest (RF) was better than the k-nearest neighbourhood (KNN) for both classification and regression. RF classified the rainfall events with a skill score of 0.38 and estimated the rainfall amount during rainfall events with the modified index of agreement (md) of 0.56. Comparison of IMERG and bias-corrected IMERG (BIMERG) revealed an average reduction in RMSE by 55% in simulating observed rainfall. The proposed bias correction method performed much better when compared with the conventional bias correction methods such as linear scaling and quantile regression. The BIMERG could reliably replicate the spatial distribution of heavy rainfall events, indicating its potential for hydro-climatic studies like flood and drought monitoring in the study area.
Similar content being viewed by others
Data availability
The observed rainfall data is not available to be shared with a third party as per instruction from the Department of Irrigation and Drainage Malaysia. However, the GCM and Satellite data sets are freely available on the website/references given in the article.
Code availability
The codes used in the current study are available from the corresponding author on reasonable request.
References
Abdulkareem J, Pradhan B, Sulaiman W, Jamil N (2018) Review of studies on hydrological modelling in Malaysia. Modeling Earth Systems and Environment 4:1577–1605
Abera W, Brocca L, Rigon R (2016) Comparative evaluation of different satellite rainfall estimation products and bias correction in the Upper Blue Nile (UBN) basin. Atmos Res 178:471–483
Ahmed F, Schumacher C (2015) Convective and stratiform components of the precipitation-moisture relationship. Geophys Res Lett 42:10453–10462
Ahmed K, Shahid S, Chung E-S, Ismail T, Wang X-J (2017) Spatial distribution of secular trends in annual and seasonal precipitation over Pakistan. Climate Res 74:95–107
Alharbi R (2019) Bias Adjustment of Satellite-Based Precipitation Estimation Using Limited Gauge Measurements and Its Implementation on Hydrologic Modeling. UC Irvine
Badron K, Ismail AF, Asnawi A, Nordin M, a. W., Alam, A. Z. & Khan, S. (2015) Classification of precipitation types detected in Malaysia. Springer, Theory and Applications of Applied Electromagnetics
Beck HE, Wood EF, Mcvicar TR, Zambrano-Bigiarini M, Alvarez-Garreton C, Baez-Villanueva OM, Sheffield J, Karger DN (2020) Bias Correction of Global High-Resolution Precipitation Climatologies Using Streamflow Observations from 9372 Catchments. J Clim 33:1299–1315
Bhatti HA, Rientjes T, Haile AT, Habib E, Verhoef W (2016) Evaluation of Bias Correction Method for Satellite-Based Rainfall Data. Sensors (Basel, Switzerland) [Online], 16. Available: https://doi.org/10.3390/s16060884 [Accessed 2016/06//].
Bitew MM, Gebremichael M (2011) Evaluation of satellite rainfall products through hydrologic simulation in a fully distributed hydrologic model. Water Resour Res 47
Breiman L (2001a) Random Forests Machine Learning 45:5–32
Breiman L (2001b) Random Forests, Vol. 45. Mach Learn, 1
Brocca L, Ciabatta L, Massari C, Moramarco T, Hahn S, Hasenauer S, Kidd R, Dorigo W, Wagner W, Levizzani V (2014) Soil as a natural rain gauge: Estimating global rainfall from satellite soil moisture data. J Geophys Res Atmos 119:5128–5141
Brocca L, Filippucci P, Hahn S, Ciabatta L, Massari C, Camici S, Schüller L, Bojkov B, Wagner W (2019) SM2RAIN-ASCAT (2007–2018): Global daily satellite rainfall from ASCAT soil moisture. Earth Syst. Sci. Data Discuss, 1–31
Caruana R, Karampatziakis N, Yessenalina A (2008) An empirical evaluation of supervised learning in high dimensions. Proceedings of the 25th international conference on Machine learning. Helsinki, Finland: Association for Computing Machinery
Chaudhary S, Dhanya C (2020) Decision tree-based reduction of bias in monthly IMERG satellite precipitation dataset over India. H2Open Journal 3 236–255
Chaudhary S, Dhanya C (2019) Investigating the performance of bias correction algorithms on satellite-based precipitation estimates. Remote Sensing for Agriculture, Ecosystems, and Hydrology XXI. International Society for Optics and Photonics, 111490Z
Ciabatta L, Brocca L, Massari C, Moramarco T, Puca S, Rinollo A, Gabellani S, Wagner W (2015) Integration of satellite soil moisture and rainfall observations over the Italian territory. J Hydrometeorol 16:1341–1355
Ciabatta L, Marra AC, Panegrossi G, Casella D, Sanò P, Dietrich S, Massari C, Brocca L (2017) Daily precipitation estimation through different microwave sensors: Verification study over Italy. J Hydrol 545:436–450
Ciabatta L, Massari C, Brocca L, Gruber A, Reimer C, Hahn S, Paulik C, Dorigo W, Kidd R, Wagner W (2018) SM2RAIN-CCI: a new global long-term rainfall data set derived from ESA CCI soil moisture. Earth System Science Data 10:267
Dasilva NA, Webber BG, Matthews AJ, Feist MM, Stein TH, Holloway CE, Abdullah MFAB (2021) Validation of GPM IMERG extreme precipitation in the Maritime Continent by station and radar data. Earth and Space Science, e2021EA001738
Dewan A, Hu K, Kamruzzaman M, Uddin MR (2019) Evaluating the spatiotemporal pattern of concentration, aggressiveness and seasonality of precipitation over Bangladesh with time–series Tropical Rainfall Measuring Mission data. Extreme Hydroclimatic Events and Multivariate Hazards in a Changing Environment. Elsevier
Fawagreh K, Gaber MM, Elyan E (2014) Random forests: from early developments to recent advancements. Systems Science & Control Engineering: an Open Access Journal 2:602–609
Fernández-Delgado M, Cernadas E, Barro S, Amorim D (2014) Do we need hundreds of classifiers to solve real world classification problems? The Journal of Machine Learning Research 15:3133–3181
Funk C, Peterson P, Landsfeld M, Pedreros D, Verdin J, Shukla S, Husak G, Rowland J, Harrison L, Hoell A, Michaelsen J (2015) The climate hazards infrared precipitation with stations—a new environmental record for monitoring extremes. Scientific Data 2:150066
Gadelha AN, Coelho VHR, Xavier AC, Barbosa LR, Melo DC, Xuan Y, Huffman GJ, Petersen WA, Almeida CDN (2019) Grid box-level evaluation of IMERG over Brazil at various space and time scales. Atmos Res 218:231–244
Gerrity JP Jr (1992) A note on Gandin and Murphy’s equitable skill score. Mon Weather Rev 120:2709–2712
Ghaffari A, Abdollahi H, Khoshayand M, Bozchalooi IS, Dadgar A, Rafiee-Tehrani M (2006) Performance comparison of neural network training algorithms in modeling of bimodal drug delivery. Int J Pharm 327:126–138
Gupta V, Jain MK, Singh PK, & Singh V (2020). An assessment of global satellite‐based precipitation datasets in capturing precipitation extremes: A comparison with observed precipitation dataset in India. International Journal of Climatology, 40(8), 3667-3688. https://doi.org/10.1002/joc.6419
Hashemi H, Nordin M, Lakshmi V, Huffman GJ, Knight R (2017) Bias correction of long-term satellite monthly precipitation product (TRMM 3B43) over the conterminous United States. J Hydrometeorol 18:2491–2509
Heidke P (1926) Berechnung des Erfolges und der Güte der Windstärkevorhersagen im Sturmwarnungsdienst. Geogr Ann 8:301–349
Heung B, Bulmer CE, Schmidt MG (2014) Predictive soil parent material mapping at a regional-scale: A Random Forest approach. Geoderma 214–215:141–154
Hsu K-L, Gao X, Sorooshian S, Gupta HV (1997) Precipitation estimation from remotely sensed information using artificial neural networks. J Appl Meteorol 36:1176–1190
Huang M, Lin R, Huang S, Xing T (2017) A novel approach for precipitation forecast via improved K-nearest neighbor algorithm. Adv Eng Inform 33:89–95
Huffman GJ, Bolvin DT, Braithwaite D, Hsu K-L, Joyce RJ, Kidd C, Nelkin EJ, Sorooshian S, Stocker EF, Tan J (2020) Integrated multi-satellite retrievals for the Global Precipitation Measurement (GPM) mission (IMERG). Satellite precipitation measurement. Springer, Cham
Huffman GJ, Bolvin DT, Braithwaite D, Hsu K, Joyce R, Xie P, Yoo S-H (2015) NASA global precipitation measurement (GPM) integrated multi-satellite retrievals for GPM (IMERG). Algorithm Theoretical Basis Document (ATBD) Version, 4, 26
Iqbal Z, Shahid S, Ahmed K, Ismail T, Khan N, Virk ZT, Johar W (2020) Evaluation of global climate models for precipitation projection in sub-Himalaya region of Pakistan. Atmospheric Res 105061
Katiraie-Boroujerdy P-S, Rahnamay Naeini M, Akbari Asanjan A, Chavoshian A, Hsu K-L, Sorooshian S (2020) Bias Correction of Satellite-Based Precipitation Estimations Using Quantile Mapping Approach in Different Climate Regions of Iran. Remote Sensing 12:2102
Khan N, Pour SH, Shahid S, Ismail T, Ahmed K, Chung ES, Nawaz N, Wang X (2019) Spatial distribution of secular trends in rainfall indices of Peninsular Malaysia in the presence of long-term persistence. Meteorol Appl 26:655–670
King F, Erler AR, Frey SK, Fletcher CG (2020) Application of machine learning techniques for regional bias correction of SWE estimates in Ontario, Canada. Hydrology and Earth System Sciences Discussions, 1–26
Kummerow CD, Randel DL, Kulie M, Wang N-Y, Ferraro R, Joseph Munchak S, Petkovic V (2015) The Evolution of the Goddard Profiling Algorithm to a Fully Parametric Scheme. J Atmos Oceanic Tech 32:2265–2280
Le X-H, Lee G, Jung K, An H-U, Lee S, Jung Y (2020) Application of Convolutional Neural Network for Spatiotemporal Bias Correction of Daily Satellite-Based Precipitation. Remote Sensing 12:2731
Liu YY, Dorigo WA, Parinussa R, De Jeu RA, Wagner W, Mccabe MF, Evans J, Van Dijk A (2012) Trend-preserving blending of passive and active microwave soil moisture retrievals. Remote Sens Environ 123:280–297
Liu YY, Parinussa RM, Dorigo WA, De Jeu RAM, Wagner W, van Dijk AIJM, McCabe MF, and Evans JP (2011) Developing an improved soil moisture dataset by blending passive and active microwave satellite-based retrievals, Hydrol. Earth Syst. Sci., 15, 425–436, https://doi.org/10.5194/hess-15-425-2011.
Ma Y, Yang Y, Han Z, Tang G, Maguire L, Chu Z, Hong Y (2018) Comprehensive evaluation of ensemble multi-satellite precipitation dataset using the dynamic bayesian model averaging scheme over the Tibetan Plateau. J Hydrol 556:634–644
Mahmud MR, Hashim M, Reba MNM (2017) How effective is the new generation of GPM satellite precipitation in characterizing the rainfall variability over Malaysia? Asia-Pac J Atmos Sci 53:375–384
Mastrantonas N, Bhattacharya B, Shibuo Y, Rasmy M, Espinoza-Dávalos G, Solomatine D (2019) Evaluating the benefits of merging near-real-time satellite precipitation products: a case study in the Kinu basin region, Japan. J Hydrometeorol 20:1213–1233
Mega T, Ushio T, Takahiro M, Kubota T, Kachi M, Oki R (2019) Gauge-Adjusted Global Satellite Mapping of Precipitation. IEEE Trans Geosci Remote Sens 57:1928–1935
Misra S, Li H (2020) Chapter 9 - Noninvasive fracture characterization based on the classification of sonic wave travel times. In: MISRA, S., LI, H. & HE, J. (eds.) Machine Learning for Subsurface Characterization. Gulf Professional Publishing
Nashwan M, Shahid S (2019) Symmetrical uncertainty and random forest for the evaluation of gridded precipitation and temperature data. Atmospheric Res 230:104632
Nashwan M, Shahid S, Dewan A, Ismail T, Alias N (2019) Performance of five high resolution satellite-based precipitation products in arid region of Egypt: An evaluation. Atmospheric Research 236:104809
Nashwan MS, Shahid S, Chung E-S (2020) High-Resolution Climate Projections for a Densely Populated Mediterranean Region. Sustainability 12:3684
Nguyen P, Ombadi M, Sorooshian S, Hsu K, Aghakouchak A, Braithwaite D, Ashouri H, Thorstensen AR (2018) The PERSIANN family of global satellite precipitation data: A review and evaluation of products. Hydrol Earth Syst Sci 22:5801–5816
Noor M, Ismail TB, Ullah S, Iqbal Z. Nawaz N, Ahmed K (2020) A non-local model output statistics approach for the downscaling of CMIP5 GCMs for the projection of rainfall in Peninsular Malaysia. Journal of Water and Climate Change 1 December 2020; 11 (4): 944–955. https://doi.org/10.2166/wcc.2019.041
Noor M, Ismail TB, Shahid S, Ahmed K, Chung E-S, Nawaz N (2019) Selection of CMIP5 multi-model ensemble for the projection of spatial and temporal variability of rainfall in peninsular Malaysia. Theoret Appl Climatol 138:999–1012
Paredes-Trejo F, Barbosa HA, Rossato Spatafora L (2018) Assessment of SM2RAIN-derived and state-of-the-art satellite rainfall products over Northeastern Brazil. Remote Sensing 10:1093
Peña-Arancibia JL, Van Dijk AIJM, Renzullo LJ, Mulligan M (2013) Evaluation of Precipitation Estimation Accuracy in Reanalyses, Satellite Products, and an Ensemble Method for Regions in Australia and South and East Asia. J Hydrometeorol 14:1323–1333
Pour SH, Abd Wahab AK, Shahid S, Ismail ZB (2020) Changes in reference evapotranspiration and its driving factors in peninsular Malaysia. Atmospheric Res 105096
Pratama AW, Buono A, Hidayat R, Harsa H (2018) Estimating parameter of nonlinear bias correction method using nsga-ii in daily precipitation data. Telkomnika 16:241–249
Rauniyar S, Protat A, Kanamori H (2017) Uncertainties in TRMM-Era multisatellite-based tropical rainfall estimates over the Maritime Continent. Earth and Space Science 4:275–302
Sa’adi Z, Shahid S, Chung E-S, Ismail TB (2017) Projection of spatial and temporal changes of rainfall in Sarawak of Borneo Island using statistical downscaling of CMIP5 models. Atmos Res 197:446–460
Sáadi Z, Shiru MS, Shahid S, Ismail T (2020) Selection of general circulation models for the projections of spatio-temporal changes in temperature of Borneo Island based on CMIP5. Theoretical and Applied Climatology 139:351–371
Schroeer K, Kirchengast G, O, S. (2018) Strong dependence of extreme convective precipitation intensities on gauge network density. Geophys Res Lett 45:8253–8263
Serrat-Capdevila A, Merino M, Valdes JB, Durcik M (2016) Evaluation of the performance of three satellite precipitation products over Africa. Remote Sensing 8:836
Shankar T, Bandyopadhyay S (2007) Prediction of extrudate properties using artificial neural networks. Food Bioprod Process 85:29–33
Shiru MS, Shahid S, Chae ST, and Chung ES (2022) Replicability of Annual and Seasonal Precipitation by CMIP5 and CMIP6 GCMs over East Asia. KSCE J Civ Eng (2022). https://doi.org/10.1007/s12205-022-0992-6
Shiru MS, Shahid S, Dewan A, Chung E-S, Alias N, Ahmed K, Hassan QK (2020) Projection of meteorological droughts in Nigeria during growing seasons under climate change scenarios. Sci Rep 10:10107
Sivapragasam C, Arun VM, Giridhar D (2010) A simple approach for improving spatial interpolation of rainfall using ANN. Meteorol Atmos Phys 109:1–7
Soo EZX, Jaafar WZW, Lai SH, Islam T, Srivastava P (2019) Evaluation of satellite precipitation products for extreme flood events: case study in Peninsular Malaysia. J Water Climate Change 10:871–892
Soo EZX, Wan Jaafar WZ, Lai SH, Othman F, Elshafie A, Islam T, Srivastava P, Othman Hadi HS (2020) Precision of raw and bias-adjusted satellite precipitation estimations (TRMM, IMERG, CMORPH, and PERSIANN) over extreme flood events: case study in Langat river basin, Malaysia. J Water Climate Change 11:322–342
Su J, Lü H, Zhu Y, Cui Y, Wang X (2019) Evaluating the hydrological utility of latest IMERG products over the Upper Huaihe River Basin, China. Atmos Res 225:17–29
Suliman AHA, Awchi TA, Al-Mola M, Shahid S (2020) Evaluation of remotely sensed precipitation sources for drought assessment in Semi-Arid Iraq. Atmospheric Res 105007
Sun Q, Miao C, Duan Q, Ashouri H, Sorooshian S, Hsu KL (2018) A review of global precipitation data sets: Data sources, estimation, and intercomparisons. Rev Geophys 56:79–107
Tan ML, Duan Z (2017) Assessment of GPM and TRMM precipitation products over Singapore. Remote Sensing 9:720
Tan ML, Ficklin DL, Ibrahim AL, Yusop Z (2014) Impacts and uncertainties of climate change on streamflow of the Johor River Basin, Malaysia using a CMIP5 General Circulation Model ensemble. Journal of Water and Climate Change 5:676–695
Tan ML, Ibrahim AL, Duan Z, Cracknell AP, Chaplot V (2015) Evaluation of six high-resolution satellite and ground-based precipitation products over Malaysia. Remote Sensing 7:1504–1528
Tan ML, Santo H (2018) Comparison of GPM IMERG, TMPA 3B42 and PERSIANN-CDR satellite precipitation products over Malaysia. Atmos Res 202:63–76
Urbanczik R (1996) Learning in a large committee machine: worst case and average case. EPL (europhysics Letters) 35:553
Ushio T, Sasashige K, Kubota T, Shige S, Okamoto K, Aonashi K, Inoue T, Takahashi N, Iguchi T, Kachi M, Oki R, Morimoto T, Kawasaki ZI (2009) A kalman filter approach to the global satellite mapping of precipitation (GSMaP) from combined passive microwave and infrared radiometric data. J Meteorol Soc Japan, 87 A 137–151
Valdés-Pineda R, Demaría E, Valdés JB, Wi S, Serrat-Capdevilla A (2016) Bias correction of daily satellite-based rainfall estimates for hydrologic forecasting in the Upper Zambezi, Africa. Hydrology and Earth System Sciences Discussions, 1–28
Wasko C, Sharma A, Westra S (2016) Reduced spatial extent of extreme storms at higher temperatures. Geophys Res Lett 43:4026–4032
Webster PJ, Magana VO, Palmer T, Shukla J, Tomas R, Yanai M, Yasunari T (1998) Monsoons: Processes, predictability, and the prospects for prediction. J Geophys Res Oceans 103:14451–14510
Wei G, Lü H, Crow WT, Zhu Y, Wang J, Su J (2018) Comprehensive evaluation of GPM-IMERG, CMORPH, and TMPA precipitation products with gauged rainfall over mainland China. Advances in Meteorology, 2018
WMO (1994) Guide to hydrological practices. Data acquisition and processing, analysis, forecasting and other applications (WMO-No. 168). World Meteorological Organization, Geneva. http://www.innovativehydrology.com/WMO-No.168-1994.pdf
Xie P, Xiong AY (2011) A conceptual model for constructing high‐resolution gauge‐satellite merged precipitation analyses. J Geophys Res Atmos 116
Yang Z, Hsu K, Sorooshian S, Xu X, Braithwaite D, Verbist KM (2016) Bias adjustment of satellite-based precipitation estimation using gauge observations: A case study in Chile. Journal of Geophysical Research: Atmospheres 121:3790–3806
Yong B, Ren LL, Hong Y, Wang JH, Gourley JJ, Jiang SH, Chen X, Wang W (2010) Hydrologic evaluation of Multisatellite Precipitation Analysis standard precipitation products in basins beyond its inclined latitude band: A case study in Laohahe basin, China. Water Resour Res 46
Zafar BJ, Chandrasekar V (2004) Classification of precipitation type from space borne precipitation radar data and 2D wavelet analysis. IGARSS 2004. 2004 IEEE International Geoscience and Remote Sensing Symposium IEEE, 3570–3573.
Ziarh GF, Asaduzzaman M, Dewan A, Nashwan MS, Shahid S (2021) Integration of catastrophe and entropy theories for flood risk mapping in peninsular Malaysia. J Flood Risk Manag 14:e12686
Acknowledgements
The authors would like to acknowledge the Belt and Road Special Foundation of the State Key Laboratory of Hydrology-Water Resources and Hydraulic Engineering (No. 2019491311, 2020491011), Young Top-Notch Talent Support Program of National High-level Talents Special Support Plan and the Ministry of Higher Education Malaysia (FRGS) (No. R.J130000.78515F092) for providing financial support to conduct this research. We also acknowledge the Department of Irrigation and Drainage Malaysia for providing the rainfall data of entire Peninsular Malaysia.
Funding
Belt and Road Special Foundation of the State Key Laboratory of Hydrology-Water Resources and Hydraulic Engineering (No. 2019491311, 2020491011), Young Top-Notch Talent Support Program of National High-level Talents Special Support Plan and the Ministry of Higher Education Malaysia (FRGS) (No. R.J130000.78515F092).
Author information
Authors and Affiliations
Contributions
Zafar Iqbal: Conceptualization, methodology, software, writing original draft. Shamsuddin Shahid: Formal analysis, conceptualization, validation, software, project administration: Kamal Ahmed: Draft Preparation, writing review and editing. Tarmizi Ismail: Resources, technical proofreading, funding acquisition, Hamza Farooq Gabriel: Data Curation. Xiaojun Wang: Writing—review and editing.
Corresponding author
Ethics declarations
Ethics approval
All authors provided ethical approval to submit the manuscript.
Consent to participate
The author has the consent to participate in the review process of this manuscript.
Consent for publication
All authors provided consent to the publication of the article in TAAC.
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Significance Statement
A two-stage novel bias correction algorithm is proposed to correct the bias in the best suitable satellite-based precipitation product of Peninsular Malaysia to obtain a high-resolution, reliable precipitation dataset for hydrological modelling.
Rights and permissions
About this article
Cite this article
Iqbal, Z., Shahid, S., Ahmed, K. et al. Bias correction method of high-resolution satellite-based precipitation product for Peninsular Malaysia. Theor Appl Climatol 148, 1429–1446 (2022). https://doi.org/10.1007/s00704-022-04007-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00704-022-04007-6