Advertisement

Selected Problems of Artificial Neural Networks Development

  • Zenon Waszczyszyn
  • Marek Słoński
Part of the CISM International Centre for Mechanical Sciences book series (CISM, volume 512)

Abstract

The chapter discusses selected problems of applications of Standard (deterministic) Neural Networks (SNN) but the main attention is focused on Bayesian Neural Networks (BNNs). In Sections 2 and 3 the problems of regression analysis, over-fitting and regularization are discussed basing on two types of network, i.e. Feed-forward Layered Neural Network (FLNN) and Radial Basis Function NN (RBFN). Application of Principal Component Analysis (PCA) is discussed as a method for reduction of input space dimensionality. In Section 4 the application of Kalman filtering to learning of SNNs is presented. Section 5 is devoted to discussion of some basics related to Bayesian inference. Then Maximum Likelihood (ML) and Maximum A Posterior (MAP) methods are presented as a basis for formulation of networks SNN-ML and SNN-MAP. A more general Bayesian framework corresponding to formulation of simple, semi-probabilistic network S-BNN, true probabilistic T-BNN and Gaussian Process GP-BNN is discussed. Section 6 is devoted to the analysis of four study cases, related mostly to the analysis of structural engineering and material mechanics problems.

Keywords

Radial Basis Function Neural Network Marginal Likelihood Relevance Vector Machine Bayesian Neural Network Order Markov Chain 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Bibliography

  1. Bailer-Jones, C., Sabin, T., MacKay, D. and Withers, P. (1997). Prediction of deformed and annealed microstructures using Bayesian neural networks and Gaussian processes. In Proc. of the Australia-Pacific Forum on Intelligent Processing and Manufacturing of Materials.Google Scholar
  2. Bishop, C. M. (1995). Neural Networks for Pattern Recognition. Oxford University Press.Google Scholar
  3. Bishop, C. M. (2006). Pattern Recognition and Machine Learning. SpringerGoogle Scholar
  4. Bishop, C. M. and Tipping, M. E. (2003). Bayesian regression and classification. In J. Suykens, S. B. C. M., G. Horvath and Vandewalle, J., editors, Advances in Learning Theory: Methods, Models and Applications, NATO Science Series III: Computer and Systems Sciences, pages 267–285. IOS Press.Google Scholar
  5. Buntine, W. L. and Weigend, A. S. (1991). Bayesian back propagation. Complex Systems, 5(6):603–64.zbMATHGoogle Scholar
  6. Ciesielski, R., Kuźniar, K., Maciag, E. and Tatara, T. (1992). Empirical formulae for fundamental natural periods of building with load bearing walls (in polish). Archives of Civil Engineering, 38:291–199.Google Scholar
  7. Demuth, H. and Beale, M. (1998). Neural Network Toolbox: For use with MATLAB: User’s Guide, Version 3. The Mathworks Inc.Google Scholar
  8. Eurocode8 (2003). Design of Structures for Earthquake Resistance.Google Scholar
  9. Foresee, F. D. and Hagan, M. T. (1997). Gauss-Newton approximation to Bayesian learning. In IEEE International Conference on Neural Networks (IJCNN’97), volume III, pages III-1930–III-1935. IEEE.Google Scholar
  10. Furtak, K. (1984). Strength of the concrete under multiple repeated loads, (in Polish). Arch. of Civil Eng., 30.Google Scholar
  11. Haykin, S. S. (1999). Neural Networks: A Comprehensive Introduction. Prentice Hall.Google Scholar
  12. Haykin, S. S. (2001). Kalman Filtering and Neural Networks. John Wiley & Sons, Inc.Google Scholar
  13. Jakubek, M. (2007). Application of Neural Networks in Experimental Mechanics of Structures and Materials, (in Polish). Ph.D. thesis, Institute for Computational Civil Engineering, Cracow University of Technology.Google Scholar
  14. Jang, J. S. R., Sun, C. T. and Mizutani, E. (1997). Neuro-Fuzzy and Soft Computing. Prentice Hall.Google Scholar
  15. Kaczmarczyk, Ł. (2006). Numerical Analysis of Multiscale Problems of Mechanics of Hetero-Homogeneous Continua (in Polish). Ph.D. thesis, Institute for Computational Civil Engineering, Cracow University of Technology.Google Scholar
  16. Kaczmarczyk, Ł. and Waszczyszyn, Z. (2007). Identification of characteristic length of micro-structure for second order multiscale model by Bayesian neural networks. Computer Assisted Mech. Eng. Sci., 14:183–196.zbMATHGoogle Scholar
  17. Kasperkiewicz, J., Racz, J. and A. Dubrawski (1995). HPC strength prediction using artificial neural network. J. Comp. in Civ. Engrg., 9(4):1–6.CrossRefGoogle Scholar
  18. Korbicz, J., Obuchowicz, A. and Uciński, D. (1994). Artificial Neural Networks: Foundations and Applications (in Polish). Akademicka Oficyna Wydawnicza.Google Scholar
  19. Kouznetsova, V. (2002). Computational Homogenization for the Multi-Scale Analysis of Multi-Phase Materials. Ph.D. thesis, Technische Universiteit Eindhoven, The Netherlands.Google Scholar
  20. Krok, A. (2006). Simulation of hysteresis loops for a superconductor using neural networks with Kalman filtering. Computer Assisted Mech. Eng. Sci., 13:575–582.zbMATHGoogle Scholar
  21. Krok, A. (2007). Analysis of Selected Problems of Mechanics of Structures and Materials by ANN and Kalman Filtering (in Polish). Ph.D. thesis, Institute for Computational Civil Engineering, Cracow University of Technology.Google Scholar
  22. Krok, A. and Waszczyszyn, Z. (2007). Kalman filtering for neural prediction of response spectra from mining tremors. Computers & Structures, 85: 1257–1263.CrossRefGoogle Scholar
  23. Kuźniar, K. (2003). BP Neural network computation of Response Spectra using a subpicture idea. In L., R. and J, K., editors, Proc. Neural Networks and Soft Computing, pages 754–759. T.U. of Częstochowa, Springer, Częstochowa/Zakopane.Google Scholar
  24. Kuźniar, K. (2004). Analysis of vibrations of medium height buildings subjected to mining tremors with application of neural networks (in Polish). Cracow University of TechnologyGoogle Scholar
  25. Kuźniar, K., Maciąg, E. and Waszczyszyn, Z. (2000). Computation of fundamental natural periods of vibrations of medium-hight buildings by neural networks. Archives of Civil Engineering, 46:515–523.Google Scholar
  26. Kuźniar, K. and Waszczyszyn, Z. (2007). Neural networks for the simulation and identification of building subjected to paraseismic excitations. In Lagaros, N. D. and Tsompanakis, Y., editors, Intelligent Computational Paradigms in Earthquake Engineering. Idea Group Publishing.Google Scholar
  27. Lampinen, J. and Vehtari, A. (2001). Bayesian approach for neural networks — review and case studies. Neural Networks, 14(3):7–24. (Invited article).CrossRefGoogle Scholar
  28. Lefik, M. (2005). Application of Artificial Neural Networks in Mechanics and Engineering (in Polish). Łódź University of Technology.Google Scholar
  29. Lefik, M. and Schrefler, B. (2002). One-dimensional model of cable-in-conduit superconductor under cyclic loading using artificial neural networks. Fusion Engineering and Design, 60:105–117.CrossRefGoogle Scholar
  30. Lou, K.-N. and Perez, R. (1996). A new system identification technique using Kalman filtering and multilayer neural networks. Artificial Intelligence in Engineering, 10:1–8.CrossRefGoogle Scholar
  31. MacKay, D. J. C. (1992). Bayesian interpolation. Neural Computation, 4(3):415–447.CrossRefGoogle Scholar
  32. MacKay, D.J.C. (1998). Introduction to Gaussian processes. In Bishop, C. M., editor, Neural Networks and Machine Learning, NATO ASI Series, pages 133–166. Springer.Google Scholar
  33. MacKay, D. J. C. (2003). Information Theory, Inference and Learning Algorithms. Cambridge University Press.Google Scholar
  34. Masters, T. (1993). Practical Neural Network Recipes in C++. Academic Press.Google Scholar
  35. Nabney, I. T. (2004). Netlab: Algorithms for Pattern Recognition. Springer-Verlag, London.Google Scholar
  36. Neal, R. M. (1992). Bayesian training of backpropagation networks by the hybrid Monte Carlo method. Technical Report CRG-TR-92-1.Google Scholar
  37. Neal, R. M. (2004). Software for Flexible Bayesian Modeling and Markov Chain Sampling. Technical report, University of Toronto.Google Scholar
  38. Nijihuis, A., Noordman, N. and ten Kate, H. (1998). Mechanical and Electrical testing of an ITER CS1 Model Coil Conductor under Tramsferse Loading in a Cryogenic Press, Preliminary report. Technical report, University of Twente.Google Scholar
  39. Pham, D. and Liu, X. (1995). Neural Networks for Identification, Prediction and Control. Springer.Google Scholar
  40. Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. The MIT Press, Cambridge, Massachusetts.zbMATHGoogle Scholar
  41. Rojas, R. (1996). Neural Networks — A Systematic Introduction. Springer.Google Scholar
  42. Sato, T. and Sato, M. (1997). Structural identification using neural networks and kalman filtering. JSCE, 14:23–32.Google Scholar
  43. Słoński, M. (2005). Prediction of concrete fatigue durability using Bayesian neural networks. Computer Assisted Mech. Eng. Sci., 12:259–265.Google Scholar
  44. Słoński, M. (2006). Bayesian regression approaches on example of concrete fatigue failure prediction. Computer Assisted Mech. Eng. Sci., 13:655–668.zbMATHGoogle Scholar
  45. Słoński, M. (2007). HPC strength prediction using Bayesian neural networks. Computer Assisted Mech. Eng. Sci., 14:345–352.zbMATHGoogle Scholar
  46. Słoński, M. (2009). A comparison between Bayesian neural networks and other machine learning methods for predicting properties of concrete. Proc. of 18th International Conference on Computer Methods in Mechanics, CMM-2009, Zielona-Góra, Poland.Google Scholar
  47. Tipping, M. E. (2001). Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research, 1:211–244.zbMATHCrossRefMathSciNetGoogle Scholar
  48. Tipping, M. E. (2004). Bayesian Inference: An Introduction to Principles and Practice in Machine Learning. In O. Bousquet, U. v. L. and Rätsch, G., editors, Advanced Lectures on Machine Learning, volume 3176 of Lecture Notes in Computer Science, pages 41–62. Springer.Google Scholar
  49. Twomey, J. M. and Smith, A. E. (1997). Validation and verification. In Kartam, N., Flood, I. and Garrett, J. H., editors. Neural Networks for Civil Engineers: Fundamentals and Applications. ASCE, New York.Google Scholar
  50. A. Vehtari and J. Vanhatalo. MCMC Methods for MLP and GP and Stuff (for Matlab) V2.1. (A User Manual Laboratory of Computational Engineering, Helsinki University of TechnologyGoogle Scholar
  51. Waszczyszyn, Z. (1999). Neural Networks in the Analysis and Design of Structures. CISM Courses and Lectures No. 404. Springer, Wien-New York.Google Scholar
  52. Waszczyszyn, Z. (2006). Artificial neural networks in civil and structural engineering: Ten years of research in Poland. Computer Assisted Mech. Eng. Sci., 13:489–512.zbMATHGoogle Scholar
  53. Waszczyszyn, Z. and Słoński, M. (2006). Bayesian neural networks for prediction of response spectra. Foundations of Civil and Environmental Engineering. 7:343–361.Google Scholar
  54. Waszczyszyn, Z. and Ziemiański, L. (2005). Neural networks in the identification analysis of structural mechanics problems, Ch. 7. In Mróz, Z. and Stavroulakis, G., editors, Parameter Identification of Materials and Structures, CISM Lecture Notes No. 469, pages 265–340. Springer, Wien-New York.CrossRefGoogle Scholar
  55. Zell, A., Mache, N., Sommer, T. and et al. (1994). Stuttgart Neural Network Simulator. User manual, ver. 3.2. Technical report, University of Stuttgart, Germany.Google Scholar

Copyright information

© CISM, Udine 2010

Authors and Affiliations

  • Zenon Waszczyszyn
    • 1
    • 2
  • Marek Słoński
    • 2
  1. 1.Rzeszów University of TechnologyRzeszówPoland
  2. 2.Institute for Computational Civil EngineeringCracow University of TechnologyKrakówPoland

Personalised recommendations