Streamflow Data Infilling Techniques Based on Concepts of Groups and Neural Networks

  • U. S. Panu
  • M. Khalil
  • A. Elshorbagy
Chapter
Part of the Water Science and Technology Library book series (WSTL, volume 36)

Abstract

For planning, management, and effective control of water resource systems, a considerable amount of data on numerous hydrologic variables such as rainfall, streamflow, evapotranspiration, temperature, etc. is required. Data sets of various hydrologic variables are at times not only short, but also often have gaps because of missing observations. Such deficiencies in hydrologic time series are attributable, among others, to the malfunctioning of monitoring equipment, the effects of natural phenomena, such as earthquakes, hurricanes, or landslides, and problems with data transmission, storage and retrieval processes. Deficiencies in hydrologic data series vary from 5 to 10 percent in the case of runoff data [Correll et al. (1998)] and up to 25 percent in the case of oceanic storm surges [Zhang et al. (1997)] . Time series methods, among others, do not tolerate missing observations, and thus numerous data infilling techniques have evolved in various scientific disciplines to deal with incomplete data sets.

Keywords

Relative Mean Error Streamflow Data Seasonal Group Streamflow Time Series Average Percent Improvement 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Afza, N., and Panu, U.S. (1992) Infilling Missing Monthly Streamflow Data for Rivers with Seasonal Runoff, Civil Eng. Technical Report, No. CE-92–3, Lakehead University, Ontario, Canada.Google Scholar
  2. Alley, W.M. Burns, A.W. (1983) Mixed-Station Extension of Monthly Streamflow Records, Jour. of Hydraulic Engineering,Vol. 109 (10), 1272–1284.CrossRefGoogle Scholar
  3. Beale, E.M. and Little, R.J. (1975) Missing Values in Multivariate Analysis. J.R. Stat. B, 37(1), 129–145.Google Scholar
  4. Beard, L. R. (1962) Statistical Methods in Hydrology, U.S. Army Engineers, Calif., pp. 5.01 - 5.05.Google Scholar
  5. Beard, L.R., Fredrich, A.J., and Hawkins, E.F. (1970) Estimating Monthly Streamflows within a Region. Technical paper 18, HEC, U.S. Army Corps of Engineers, 14 pages.Google Scholar
  6. Beauchamp, J. J., Dowing, D. J., and Railsback, S. F. (1989) Comparison of Regression and Time-Series Methods for Synthesizing Missing Streamflow Records, Water Resources Bulletin, 25(5), 961 - 975.CrossRefGoogle Scholar
  7. Gyau-Boakye, P. G., and Schultz, G. A. (1994) Filling Gaps in Runoff Time Series in West Africa, Hydrological Science Journal, 39(6), 621 - 636.CrossRefGoogle Scholar
  8. Box, G. E. P. and Jenkins, G. M. (1976) Time Series Analysis: Forecasting and Control, Holden-Day, San Francisco, California, Revised Edition.Google Scholar
  9. Chakraborty, K., Mehrotra, K., Mohan, C. K., and Ranka, S. (1992) Forecasting the Behavior of Multi-variate Time Series Using Neural Networks, Neural Networks, Vol. 5, 961–970.CrossRefGoogle Scholar
  10. Chow, V. T. (1964) Handbook of Applied Hydrology. McGraw-Hill, New York.Google Scholar
  11. Correll, D. L., Jordan, T. E., and Weller, D. E. (1998) Effect of Inter-annual Variation of Precipitation on Stream Discharge from Rhode River Sub-watersheds, AWRA (in printing).Google Scholar
  12. Dax, A. (1985) Completing Missing Groundwater Observations by Interpolation. J. Hydrol., 81, 375–399.CrossRefGoogle Scholar
  13. Elshorbagy, A., Simonovic, S. P. and Panu, U. S. (1998) Performance Evaluation of Artificial Neural Networks for Runoff Prediction. ASCE Journal of Hydrologic Engineering (under review).Google Scholar
  14. Elshorbagy, A., Panu, U. S. and Simonovic, S. P. (1999) Investigations into Group Based Data Infilling Techniques. CSCE Annual Conference, Regina, Canada.Google Scholar
  15. Fiering, M. B. (1962) On the Use of Correlation to Augment Data. J. Amer. Statist. Assoc., 57(297), 20–32.CrossRefGoogle Scholar
  16. French, M. N., Krajewski, W. F., and Cuykendall, R. R. (1992) Rainfall Forecasting in Space and Time Using a Neural Network, J. Hydrol., Vol. 137, 1–31.CrossRefGoogle Scholar
  17. Giiroy, E. I. (1971) Reliability of a Variance Estimate Obtained from a Sample Augmented by Multivariate Regression, Water Resources Research, Vol. 6(6), 1595–1600.Google Scholar
  18. Gnanadesikan, R. (1977) Methods for Statistical Data Analysis of Multivariate Observation, John Wiley, New YorkGoogle Scholar
  19. Goodier, C. and Panu, U.S. (1993) Applications of a Multivariate Approach for Infilling of Missing Monthly Streamflows, Civil Engineering Technical Report No. CE-93–3, Lakehead Univ., Thunder Bay, Ontario.Google Scholar
  20. Goodier, C. and Panu, U.S. (1994) Infilling Missing Monthly Streamflow Data Using a Multivariate Approach, Stochastic and Statistical Methods in Hydrology &; Environmental Engineering, 3, 191–202.Google Scholar
  21. Granger, C. W., and Newbold, P. (1986) Forecasting Economic Time Series. Orlando, Academic Press.Google Scholar
  22. Griffith, D. A., Haining, R. P. and Bennett, R. J. (1985) Estimating Missing Values in Space-time Data Series. in Time Series Analysis: Theory and Practice 6. Anderson, O. D., Ord, J. K. and Robinson, E.A., eds., Elsevier Science Publishers B.V., North-Holland.Google Scholar
  23. Grygier, J.C., Stedinger, J. R., Yin, H.B. (1989) A Generalized Maintenance of Variance Extension Procedure for Extending Correlated Series, Water Resources Research, Vol. 25(3), 345–349.CrossRefGoogle Scholar
  24. Gupta, A. and Lam, M. (1996) Estimating Missing Values Using N.Networks. J. Oper. Res., 47(2), 229–238.Google Scholar
  25. Hirsch, R.M. (1979) An Evaluation of Record Reconstruction Techniques. WRR, 15(6), 1781–1790.CrossRefGoogle Scholar
  26. Hirsch, R. M. (1982) A Comparison of Four Streamflow Extension Techniques. WRR, 18(4), 1081–88.CrossRefGoogle Scholar
  27. Hirsch, R.M. and Gilroy, E.J. (1984) Methods of Fitting a Straight Line to Data: Examples in Water Resources, Water Resources Bulletin, Vol. 20(5), 705–711.CrossRefGoogle Scholar
  28. Hsu, Kuo-lin, Gupta, H. V., and Sorooshian, S. (1995) Artificial Neural Network Modeling of the Rainfall-Runoff Process, WRR, Vol. 31(10), 2517–2530.CrossRefGoogle Scholar
  29. Hughes, D. A. and Smakhtin, V. (1996) Daily Flow Time Series Patching or Extension: A Spatial Interpolation Approach Based on Flow Duration Curves. Hydrol. Sci. J., 41(6), 851–871.CrossRefGoogle Scholar
  30. Ishibuchi, H., Miyazaki, A., Kwon, K. and Tanaka, H. (1993) learning from incomplete training data with missing values and medical application. Proc. Int. Joint Conf. On Neu. Net., Japan, V.2, 1871–74.Google Scholar
  31. Johnson, R. A., and Wichern, D. W. (1988) Applied Multivariate Statistical Analysis, Prentice Hall, N.Y.Google Scholar
  32. Kang, K. W., Park, C. Y., and Kim, J. H. (1993) Neural Network and its Application to Rainfall-Runoff Forecasting, Korean J. Hydrosci., Vol. 4, 1–9.Google Scholar
  33. Karunanithi, N., Grenney, W. J., Whitley, D., and Bovee, K. (1994) Neural Networks for River Flow Prediction. J. Computing in Civ. Eng., ASCE, 8(2), 201–220.CrossRefGoogle Scholar
  34. Khalil, M., Panu, U. S. Panu, and Lennox, W.C. (1998) Infilling of Missing Streamflow Values Based on Concepts of Groups and Neural Networks, Civil Engineering Technical Report No CE-98–2, Lakehead University, Thunder Bay, Ontario, Canada.Google Scholar
  35. Khalil, M., Panu, U. S., and Lennox, W. (1999) Streamflow Data Infilling Procedures Based on Concept of Groups and Neural Networks. 1. Development of Models, Journal of Hydrology (under review).Google Scholar
  36. Kuczera, G. (1987) On Maximum Likelihood Estimators for the Multisite Lag-one Streamflow Models: Complete and Incomplete Data Cases. WRR, 23(4), 641–645.CrossRefGoogle Scholar
  37. Lachtermacher, G., and Fuller, J. D. (1994) Back-Propagation in Hydrological Time Series Forecasting, Stochastic and Statistical Methods in Hydrology and Environment Engineering, Vol. 3, 229–242.Google Scholar
  38. Lettenmaier, D. P., and Burges, S. J. (1979) Operational Assessment of Hydrologic Models of Long-term Persistence, WRR, Vol. 13 (1), 113–124.CrossRefGoogle Scholar
  39. Makhuvha, T., Pegram, G., Sparks, R. and Zucchini, W. (1997) Patching Rainfall Data Using Regression Methods. 2. Comparisons of Accuracy, Bias and Efficiency. J. Hydrol., 198, 308–318.Google Scholar
  40. Matalas, N. C. (1967) Mathematical Assessment of Synthetic Hydrology, WRR,Vol. 3, 937 - 945.CrossRefGoogle Scholar
  41. Matalas, N.C., and Jacobs, B. (1964) A Correlation Procedure for Augmenting Hydrologic Data, U.S. Geol. Surv. Prof. Pap., 434-E, E 1–E7.Google Scholar
  42. McCuen, R.A.(1993) Statistical Hydrology. Prentice-Hall, Englewood Cliffs, N.J., 306 pages.Google Scholar
  43. Moran, M. A. (1974) On Estimators Obtained from a Sample Augmented by Multiple Regression. WRR, 10(1), 81–85.CrossRefGoogle Scholar
  44. Mott, P., Sammis, T.W., and Southward, G. M. (1994). Climate Data Estimation Using Climate Information from Surrounding Climate Stations. Appl. Eng. In Agric., 10(1), 41–44.Google Scholar
  45. Panu, U. S. (1991) Application of Some Entropic Measures in Hydrologic Data Infilling Procedures. In Entropy and Energy Dissipation in Water Resources. Singh, V. P. and Fiorentino, M., eds., Kleuwer Academic Publishers, the Netherlands, 175–192.Google Scholar
  46. Panu, U.S., Unny, T.E. and Ragade, R.K., 1978. (1978) A Feature Prediction Model in Synthetic Hydrology Based on Concepts of Pattern Recognition. WRR, 14(2), 335–344.CrossRefGoogle Scholar
  47. Panu, U. S. and Unny, T. E. (1980a) Extension and Application of Feature Prediction Model for Synthesis of Hydrologic Records. WRR, 16(1), 77–96.CrossRefGoogle Scholar
  48. Panu, U.S. and Unny, T.E. (1980b) Stochastic Synthesis of Hydrologic Data Based on Concepts of Pattern Recognition, l-. General Methodology of the Approach, Journal of Hydrology, Vol. 46, 5–34.CrossRefGoogle Scholar
  49. Panu, U.S. and Unny, T.E. (1980c) Stochastic Synthesis of Hydrologic Data Based on Concepts of Pattern Recognition, 2-. Application to Natural Watersheds, Journal of Hydrology, Vol. 46, 197–217.CrossRefGoogle Scholar
  50. Panu, U.S. and Unny, T.E. (1980d) Stochastic Synthesis of Hydrologic Data Based on Concepts of Pattern Recognition, 3-. Performance Evaluation of the Methodology, Journal of Hydrology, Vol. 46, 219–237.CrossRefGoogle Scholar
  51. Panu, U. S., and Afza, N. (1993) Entropic Evaluation of Streamflow Data Infilling Procedures, Proc. of Stochastic and Statistical Methods in Hydrology and Environmental Engineering, pp. 410–412.Google Scholar
  52. Panu, U. S. and A. Ku (1997) Forecasting Monthly Streamflow Patterns for Reservoir Operations. CSCE Annual Conference, Vol. 3, 159–168.Google Scholar
  53. Panu, U. S. and Afza, N. (1998) Development of Feature Infilling Procedures for Hydrologic Data Series. Journal of Hydrology, (to appear).Google Scholar
  54. Pedreira, C.E. and Parente, E. (1995) Neural networks with missing values attributes. Proc. IEEE Int. Conf. Neu. Net., V.6, 3021–23.CrossRefGoogle Scholar
  55. Pegram, G. (1997) Patching Rainfall Data Using Regression Methods. 3. Grouping, Patching and Outlier Detection. J. Hydrol., 198, 319–334.CrossRefGoogle Scholar
  56. Raman, H., Sunilkumar, N. (1995) Multivariate Modeling of Water Resources Time Series Using Artificial Neural Networks, Hydrological Sciences Journal, Vol.40(2), 145–163.CrossRefGoogle Scholar
  57. Salas, J.D. (1993) Analysis and Modeling of Hydrologic Time Series. In Handbook of Hydrology, Maidment, D.R. (ed.), McGraw-Hill, Inc., USA.Google Scholar
  58. Salas J. D., Delleur, J. W., Yevjevich, V., and Lane, W. L. (1980) Applied Modeling of Hydrologic Time Series. Water Resour. Pub. Colorado, 46 1–473.Google Scholar
  59. Salas, J. D., Obeysekera, J. T. B. (1992) Conceptual Basis of Seasonal Streamflow Time Series Models, Journal of Hydraulic Engineering, Vol. 118 (8), 1186–194.CrossRefGoogle Scholar
  60. Shih, S. F., and Cheng, K. S. (1989) Generation of Synthetic and Missing Climatic Data for Puerto Rico, Water Resources Bulletin, Vol. 25 (4), 829–836.CrossRefGoogle Scholar
  61. SPSS (1995) Base Systems User’s Guide (Part-II), SPSS Inc., Chicago, Illinois, USAGoogle Scholar
  62. Srikanthan, R., McMahan, T. A., Codner, G. P., and Mein, R.G. (1983) Practical Aspects of Multi-site generation of stream flow data, Paper presented at Proceeding Hydrology and Water Resources Symposium, Inst. Of Eng., Hobart, Australia.Google Scholar
  63. Stedinger, J. R., Lettenmaier, D. P., and Vogal, R. M (1985) Multi-site ARMA (1,1) and Disaggregation Models for Annual Streamflow Generation, WRR, Vol. 21 (4), 497–510.CrossRefGoogle Scholar
  64. Streit, R. L., and Luginbuhl, T. E. (1994) Maximum Likelihood Training of Probabilistic Neural Networks, IEEE Trans., Neural Networks, Vol. 5(5), 764–783.CrossRefGoogle Scholar
  65. Tanaka, M. (1996) Identification of Nonlinear Systems with Missing Data Using Stochastic Neural Network, Decision and Control: Proceedings of the 35th IEEE Conference; Journal: Vol. 1, 933–934.Google Scholar
  66. Tang, W. Y., Kassim, A. H. M., Abubakar, S. H. (1996) Comparative studies of Various Data Treatment Methods—Malaysian Experience. Atmospheric Research Journal, Vol. 42, 247–262.CrossRefGoogle Scholar
  67. Terry, W. R., Lee, J. B., Kumar, A. (1986) Time Series Analysis in Acid Rain Modeling: Evaluation of Filling Missing Values By Linear Interpolations, Atmospheric Environment, Vol. 20 (10), 1941–1945.CrossRefGoogle Scholar
  68. Tiao, G. C., and Tsay, R. S. (1989) Model Specification in Multivariate Time Series, Journal of the Royal Statistical Society, B 51, 157–213.Google Scholar
  69. Tokar, A. S. (1996) Rainfall—Runoff Modeling in an Uncertain Environment. Ph.D. Thesis. University of Maryland. UMI Dissertation Service. Bell and Howell Company.Google Scholar
  70. Tong, H. (1983) Threshold Models in Nonlinear Time Series Analysis, Lecture Notes in Statistics, 21, New York: Springer-Verlag.CrossRefGoogle Scholar
  71. Tong, H. (1990) Nonlinear Time Series: A Dynamical System Approach, Oxford: Oxford University Press.Google Scholar
  72. Unny, T.E., Panu, U.S., McInnes, C.D., and Wong, A.K.C. (1981) Pattern Analysis and Synthesis of Time Dependent Hydrologic Data. Advances in Hydrosciences, Academic Press, Vol. 12, 222–244,Google Scholar
  73. Vogel, R. M. and Stedinger, J. R. (1985) Minimum Variance Streamtlow Record Augmentation Procedures. WRR, 21(5), 715–723.CrossRefGoogle Scholar
  74. Wong, I., Lam, D., Storey, A., and Fong, P. (1994) A Neural Network Approach to predict Missing Environmental Data, World Congress in Neural Networks, Vol. 1, Conference, San Diego, CA.Google Scholar
  75. Young, G.K., Orlob, G.T. and Roesner, L. A. (1970) Decision Criteria for Using Stochastic Hydrology. Journal of the Hydraulics Division, ASCE, 96 (HY4), 911–926.Google Scholar
  76. Zhang, K., Douglas, B. C., and Leatherman, S. P. (1997) East Coast Storm Surges Provide Unique Climate Record, EOS, Trans AGU, Vol. 78(37), 389–397.Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2000

Authors and Affiliations

  • U. S. Panu
    • 1
  • M. Khalil
    • 1
  • A. Elshorbagy
    • 1
  1. 1.Department of Civil EngineeringLakehead UniversityThunder BayCanada

Personalised recommendations