Constraint-Based Neural Network Learning for Time Series Predictions

  • Benjamin W. Wah
  • Minglun Qian
Chapter

Abstract

In this chapter, we have briefly surveyed previous work in predicting noise-free piecewise chaotic time series and noisy time series with high frequency random noise. For noise-free time series, we have proposed a constrained formulation for neural network learning that incorporates the error of each learning pattern as a constraint, a new cross-validation scheme that allows multiple validations sets to be considered in learning, a recurrent FIR neural network architecture that combines a recurrent structure and a memory-based FIR structure, and a violation-guided back propagation algorithm for searching in the constrained space of the formulation. For noisy time series, we have studied systematically the edge effect due to low-pass filtering of noisy time series and have developed an approach that incorporates constraints on predicting low-pass data in the lag period. The new constraints enable active training in the lag period that greatly improves the prediction accuracy in the lag period. Finally, experimental results show significant improvements in prediction accuracy on standard benchmarks and stock price time series.

Keywords

Time Series Recurrent Neural Network Time Series Prediction Nonlinear Time Series Chaotic Time Series 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 17.1
    M. Aoki: State Space Modeling of Time Series ( Springer-Verlag, Nerlin, 1987 )MATHCrossRefGoogle Scholar
  2. 17.2
    A. Aussem: Dynamical recurrent neural networks towards prediction and modeling of dynamical systems. Neurocomputing, 28, 207–232 (1999)CrossRefGoogle Scholar
  3. 17.3
    S.D. Balkin, J.K. Ord: Automatic neural network modeling for univariate time series. Int’l J. of Forecasting, 16, 509–515 (2000)CrossRefGoogle Scholar
  4. 17.4
    G.E.P. Box, G.M. Jenkins. Time Series Analysis: Forecasting and Control, 2nd ed. ( Holden-Day, San Francisco, 1976 )Google Scholar
  5. 17.5
    C. Brooks, S.P. Burke, G. Persand: Benchmarks and the accuracy of GARCH model estimation. Int’l J. of Forecasting, 17, 45–56 (2001)CrossRefGoogle Scholar
  6. 17.6
    R.G. Brown: Smoothing, Forecasting and Prediction ( Prentice Hall, Englewood Cliffs, NJ, 1963 )Google Scholar
  7. 17.7
    C. Chatfield: The analysis of time series-an introduction (Chapman & Hall, London, 5 edition, 1996 )Google Scholar
  8. 17.8
    C. Chatfield: Time-series forecasting (Chapman & Hall/CRC, Boca Raton, Florida, 2001 )Google Scholar
  9. 17.9
    S. Chen, S. Billings, P. Grant: Non-linear system identification using neural networks. Int’l J. of Control, 51, 1191–1214 (1990)MathSciNetMATHCrossRefGoogle Scholar
  10. 17.10
    D. Drossu, Z. Obradovic: Regime signaling techniques for non-stationary time series forecasting. In: Proc. 30th Hwaii Int’l Conf. on System Sciences ( Wailea, HI, USA, 1997 ) 5, pp. 530–538Google Scholar
  11. 17.11
    R.O. Duda, P.E. Hart: Pattern Classification and Scene Analysis (John Wiley and Sons, 1973 )Google Scholar
  12. 17.12
    R.D. Edwards, J. Magee: Technical Analysis of Stock Trends (John Magee, Springfield, MA, 5 edition, 1966 )Google Scholar
  13. 17.13
    J.L. Elman: Finding structure in time. Cognitive Science, 14, 179-211 (1990) 17.14 B.E. Flores, S.L. Pearce: The use of an expert system in the M3 competition. Int’l J. of Forecasting, 16, 485–496 (2000)Google Scholar
  14. 17.15
    A.B. Geva: ScaleNet–multiscale neural-network architecture for time series prediction. IEEE Trans. on Neural Networks, 9 (5), 1471–1482 (1998)CrossRefGoogle Scholar
  15. 17.16
    C.W.J. Granger, A.P. Andersen: Introduction to Bilinear Time Series Models ( Vandenhoeck & Ruprect, Göittingen, 1978 )MATHGoogle Scholar
  16. 17.17
    S. Gutjahr, M. Riedmiller, J. Klingemann: Daily prediction of the foreign exchange rate between the us dollar and the german mark using neural networks. In: Proc. of SPICES pp. 492–498 (1997)Google Scholar
  17. 17.18
    M. Hallas, G. Dorffner: A comparative study on feedforward and recurrent neural networks in time series prediction using gradient descent learning. In: Proc. of 14th European Meeting on Cybernetics and Systems Research 2, pp. 646–647 (1998)Google Scholar
  18. 17.19
    S. Haykin: Neural Networks: A Comprehensive Foundation (Prentice Hall, NJ, 2 edition, 1999 )Google Scholar
  19. 17.20
    R. Hegger, T. Schreiber: The TISEAN software package. http://www.mpipks-dresden.mpg.de tisean (2002)
  20. 17.21
    T. Hellstrm: Predicting a rank measure for stock returns. Theory of Stochastic Processes, 6 (20), 64–83 (2000)Google Scholar
  21. 17.22
    T. Hellstrom, K. Holmstrom: Predicting the Stock Market. Technical Report Series IMa-TOM-1997-07, Malardalen University, Vasteras, Sweden, 1997Google Scholar
  22. 17.23
    B.G. Horne, C.L. Giles: An experimental comparison of recurrent neural networks. In: G. Tesauro, D. Touretzky, T. Leen (eds.), Neural Information Processing Systems ( MIT Press, Cambridge, MA, 1995 ) pp. 697–704Google Scholar
  23. 17.24
    A.K. Jain, M.N. Murty, P.J. Flynn: Data clustering: A review. ACM Computing Surveys, 31 (3), 264–323 (1999)CrossRefGoogle Scholar
  24. 17.25
    E.S. Gardner Jr., E.A. Anderson-Fletcher, A.M. Wicks: Further results on focus forecasting vs. exponential smoothing. Int’1 J. of Forecasting, 17, 287–293 (2001)CrossRefGoogle Scholar
  25. 17.26
    B.H. Juang, L.R. Rabiner: Hidden Markov models for speech recognition. Technometrics, 33, 251–272 (1991)MathSciNetMATHCrossRefGoogle Scholar
  26. 17.27
    J. Kohlmorgen, K.R. Müller, K. Pawelzik: Analysis of drifting dynamics with neural network hidden markov models. Advances in Neural Information Processing Systems, 10 (1998)Google Scholar
  27. 17.28
    T. Koskela, M. Lehtokangas, J. Saarinen, K. Kaski: Time series prediction with multilayer perceptron, FIR and Elman neural networks. In: Proc. of the World Congress on Neural Networks pp. 491–496 (1996)Google Scholar
  28. 17.29
    K.J. Lang, G.E. Hinton: The development of the time-delayed neural network architecture for speech recognition. Technical Report #CMU-CS-88-152, Carnegie-Mellon University, Pittsburgh, PA, 1988Google Scholar
  29. 17.30
    S. Makridakis, A. Andersen, R. Carbone, R. Fildes, M. Hibon, R. Lewandowski, J. Newton, E. Parzen, R. Winkler: The accuracy of extrapolation (time series) methods: results of a forecasting competition. Int’l J. of Forecasting, 1, 111–153 (1982)CrossRefGoogle Scholar
  30. 17.31
    S. Makridakis, C. Chatfield, M. Hibon, M. Lawrence, T. Mills, K. Ord, L.F. Simmons: The M2-Competition: a real-time judgementally based forecasting study. Int’l J. of Forecasting, 9, 5–23 (1993)CrossRefGoogle Scholar
  31. 17.32
    S. Makridakis, M. Hibon: The M3-Competition: results, conclusions and implications. Int’l J. of Forecasting, 16, 451–476 (2000)CrossRefGoogle Scholar
  32. 17.33
    T. Masters: Neural, Novel and Hybrid Algorithms for Time Series Prediction ( John Wiley & Sons, Inc., NY, 1995 )Google Scholar
  33. 17.34
    N. Meade: A comparison of the accuracy of short term foreign exchange forecasting methods. Int’l J. of Forecasting, 18, 67–83 (2002)MathSciNetCrossRefGoogle Scholar
  34. 17.35
    G. Melard, J.M. Pasteels: Automatic arima modeling including interventions, using time series expert software. Int’l J. of Forecasting, 16, 497–508 (2000)CrossRefGoogle Scholar
  35. 17.36
    J. Moody, C. Darken: Fast learning in networks of locally-tuned processing units. Neural Computation, 1 (2), 281–294 (1989)CrossRefGoogle Scholar
  36. 17.37
    K. Müller, A. Smola, G. Rätsch, B. Schölkopf, J. Kohlmorgen, and V. Vapnik: Predicting time series with support vector machines. In: ICANN pp. 999–1004 (1997)Google Scholar
  37. 17.38
    F. Murtagh, A. Aussem: Using the wavelet transform for multivariate data analysis and time series forecasting. In: C. Hayashi, H.H. Bock, K. Yajima, Y. Tanaka, N. Ohsumi, Y. Baba, editors, Data Science, Classification and Related Methods pp. 617-624 (Springer-Verlag, 1998 )Google Scholar
  38. 17.39
    D.F. Nicholls, A.R. Pagan: Varying coefficient regression. In: E.J. Hannan, P.R. Krishnaiah, M.M. Rao (eds.), Handbook of Statistics ( North-Holland, Amsterdam, 1985 ) pp. 413–449Google Scholar
  39. 17.40
    J.R. Quinlan: Induction of decision trees. Machine Learning, 1, 81–106 (1986).Google Scholar
  40. 17.41
    S. Ramaswamy: One-step prediction of financial time series, BIS Working Pa-per No. 57. Technical report, Bank for Interal Settlements, Basle, Switzerland, 1998Google Scholar
  41. 17.42
    Y. Shang, B.W. Wah: Global optimization for neural network training. IEEE Computer, 29, 45–54 (March 1996)CrossRefGoogle Scholar
  42. 17.43
    H. Tong: Nonlinear Time Series: A Dynamical System Approach ( Oxford University Press, Oxford, 1990 )Google Scholar
  43. 17.44
    B.W. Wah, Y.X. Chen: Constrained genetic algorithms and their applications in nonlinear constrained optimization. In: Proc. Int’l Conf. on Tools with Artificial Intelligence ( IEEE, November 2000 ) pp. 286–293Google Scholar
  44. 17.45
    B.W. Wah, M.L. Qian: Constrained formulations for neural network training and their applications to solve the two-spiral problem. In: Proc. Fifth Int’l Conf. on Computer Science and Informatics 1, pp. 598–601 (February 2000)Google Scholar
  45. 17.46
    B.W. Wah, M.L. Qian: Time-series predictions using constrained formulations for neural-network training and cross validation. In: Proc. Int’l Conf. on Intelligent Information Processing, 16th IFIP World Computer Congress ( Kluwer Academic Press, August 2000 ) pp. 220–226Google Scholar
  46. 17.47
    B.W. Wah, M.L. Qian: Violation-guided learning for constrained formulations in neural network time series prediction. In: Proc. Int’l Joint Conference on Artificial Intelligence ( IJCAI, Aug. 2001 ) pp. 771–776Google Scholar
  47. 17.48
    B.W. Wah, M.L. Qian: Violation guided neural-network learning fo constrained formulations in time-series predictions. Int’l Journal on Computational Intelligence and Applications, 1 (4), 383–398 (December 2001)CrossRefGoogle Scholar
  48. 17.49
    B.W. Wah, M.L. Qian: Constrained formulations and algorithms for stock price predictions using recurrent FIR neural networks. In: Proc. 2002 National Conf. on Artificial Intelligence (AAAI, 2002)(accepted to appear)Google Scholar
  49. 17.50
    B.W. Wah, Z. Wu: The theory of discrete Lagrange multipliers for nonlin-ear discrete optimization. Principles and Practice of Constraint Programming, pp. 28–42 (October 1999)Google Scholar
  50. 17.51
    E.A. Wan: Temporal backpropagation for FIR neural networks. IEEE Int’l Joint Conf. on Neural Networks, 1, pp. 575-580 ( San Diego, CA., 1990 )Google Scholar
  51. 17.52
    E.A. Wan: Finite Impulse Response Neural Networks with Applications in Time Series Prediction. Ph.D. Thesis, Standford University, 1993Google Scholar
  52. 17.53
    C.J. Watkins: Models of Delayed Reinforcement Learning. Ph.D. thesis, Cam-bridge University (Cambridge, UK, 1989 )Google Scholar
  53. 17.54
    A.S. Weigend, N.A. Gershenfeld (eds.): Time Series Prediction: Forecasting the future and understanding the past (Addison-Wesley, 1994 )Google Scholar
  54. 17.55
    R.J. Williams, D. Zipser: A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1, 270–280 (1989)CrossRefGoogle Scholar
  55. 17.56
    Z. Wu: The Theory and Applications of Nonlinear Constrained Optimization using Lagrange Multipliers. Ph.D. Thesis, Dept. of Computer Science, Univ. of Illinois, Urbana, IL (May 2001)Google Scholar
  56. 17.57
    B.L. Zhang, R. Coggins, M.A. Jabri, D. Dersch, B. Flower: Multiresolution forecasting for future trading using wavelet decompositions. IEEE Trans. on Neural Networks, 12, 766–775 (2001)Google Scholar
  57. 17.58
    G. Zheng, J.L. Starck, J.G. Campbell, F. Murtagh: Multiscale transforms for filtering financial data streams. J. of Computational Intelligence in Finance, 7, 18–35 (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Benjamin W. Wah
    • 1
  • Minglun Qian
    • 1
  1. 1.Urbana-ChampaignUniversity of IllinoisUSA

Personalised recommendations