Advances of Soft Computing in Engineering pp 237-316 | Cite as

# Selected Problems of Artificial Neural Networks Development

## Abstract

The chapter discusses selected problems of applications of Standard (deterministic) Neural Networks (SNN) but the main attention is focused on Bayesian Neural Networks (BNNs). In Sections 2 and 3 the problems of regression analysis, over-fitting and regularization are discussed basing on two types of network, i.e. Feed-forward Layered Neural Network (FLNN) and Radial Basis Function NN (RBFN). Application of Principal Component Analysis (PCA) is discussed as a method for reduction of input space dimensionality. In Section 4 the application of Kalman filtering to learning of SNNs is presented. Section 5 is devoted to discussion of some basics related to Bayesian inference. Then Maximum Likelihood (ML) and Maximum A Posterior (MAP) methods are presented as a basis for formulation of networks SNN-ML and SNN-MAP. A more general Bayesian framework corresponding to formulation of simple, semi-probabilistic network S-BNN, true probabilistic T-BNN and Gaussian Process GP-BNN is discussed. Section 6 is devoted to the analysis of four study cases, related mostly to the analysis of structural engineering and material mechanics problems.

## Keywords

Radial Basis Function Neural Network Marginal Likelihood Relevance Vector Machine Bayesian Neural Network Order Markov Chain## Preview

Unable to display preview. Download preview PDF.

## Bibliography

- Bailer-Jones, C., Sabin, T., MacKay, D. and Withers, P. (1997). Prediction of deformed and annealed microstructures using Bayesian neural networks and Gaussian processes. In
*Proc. of the Australia-Pacific Forum on Intelligent Processing and Manufacturing of Materials*.Google Scholar - Bishop, C. M. (1995).
*Neural Networks for Pattern Recognition*. Oxford University Press.Google Scholar - Bishop, C. M. (2006).
*Pattern Recognition and Machine Learning*. SpringerGoogle Scholar - Bishop, C. M. and Tipping, M. E. (2003). Bayesian regression and classification. In J. Suykens, S. B. C. M., G. Horvath and Vandewalle, J., editors,
*Advances in Learning Theory: Methods, Models and Applications*, NATO Science Series III: Computer and Systems Sciences, pages 267–285. IOS Press.Google Scholar - Buntine, W. L. and Weigend, A. S. (1991). Bayesian back propagation.
*Complex Systems*, 5(6):603–64.zbMATHGoogle Scholar - Ciesielski, R., Kuźniar, K., Maciag, E. and Tatara, T. (1992). Empirical formulae for fundamental natural periods of building with load bearing walls (in polish).
*Archives of Civil Engineering*, 38:291–199.Google Scholar - Demuth, H. and Beale, M. (1998).
*Neural Network Toolbox: For use with MATLAB: User’s Guide, Version 3*. The Mathworks Inc.Google Scholar - Eurocode8 (2003). Design of Structures for Earthquake Resistance.Google Scholar
- Foresee, F. D. and Hagan, M. T. (1997). Gauss-Newton approximation to Bayesian learning. In
*IEEE International Conference on Neural Networks (IJCNN’97)*, volume III, pages III-1930–III-1935. IEEE.Google Scholar - Furtak, K. (1984). Strength of the concrete under multiple repeated loads, (in Polish).
*Arch. of Civil Eng.*, 30.Google Scholar - Haykin, S. S. (1999).
*Neural Networks: A Comprehensive Introduction*. Prentice Hall.Google Scholar - Haykin, S. S. (2001).
*Kalman Filtering and Neural Networks*. John Wiley & Sons, Inc.Google Scholar - Jakubek, M. (2007).
*Application of Neural Networks in Experimental Mechanics of Structures and Materials, (in Polish)*. Ph.D. thesis, Institute for Computational Civil Engineering, Cracow University of Technology.Google Scholar - Jang, J. S. R., Sun, C. T. and Mizutani, E. (1997).
*Neuro-Fuzzy and Soft Computing*. Prentice Hall.Google Scholar - Kaczmarczyk, Ł. (2006).
*Numerical Analysis of Multiscale Problems of Mechanics of Hetero-Homogeneous Continua (in Polish)*. Ph.D. thesis, Institute for Computational Civil Engineering, Cracow University of Technology.Google Scholar - Kaczmarczyk, Ł. and Waszczyszyn, Z. (2007). Identification of characteristic length of micro-structure for second order multiscale model by Bayesian neural networks.
*Computer Assisted Mech. Eng. Sci.*, 14:183–196.zbMATHGoogle Scholar - Kasperkiewicz, J., Racz, J. and A. Dubrawski (1995). HPC strength prediction using artificial neural network.
*J. Comp. in Civ. Engrg.*, 9(4):1–6.CrossRefGoogle Scholar - Korbicz, J., Obuchowicz, A. and Uciński, D. (1994).
*Artificial Neural Networks: Foundations and Applications (in Polish)*. Akademicka Oficyna Wydawnicza.Google Scholar - Kouznetsova, V. (2002).
*Computational Homogenization for the Multi-Scale Analysis of Multi-Phase Materials*. Ph.D. thesis, Technische Universiteit Eindhoven, The Netherlands.Google Scholar - Krok, A. (2006). Simulation of hysteresis loops for a superconductor using neural networks with Kalman filtering.
*Computer Assisted Mech. Eng. Sci.*, 13:575–582.zbMATHGoogle Scholar - Krok, A. (2007).
*Analysis of Selected Problems of Mechanics of Structures and Materials by ANN and Kalman Filtering (in Polish)*. Ph.D. thesis, Institute for Computational Civil Engineering, Cracow University of Technology.Google Scholar - Krok, A. and Waszczyszyn, Z. (2007). Kalman filtering for neural prediction of response spectra from mining tremors.
*Computers & Structures*, 85: 1257–1263.CrossRefGoogle Scholar - Kuźniar, K. (2003). BP Neural network computation of Response Spectra using a subpicture idea. In L., R. and J, K., editors,
*Proc. Neural Networks and Soft Computing*, pages 754–759. T.U. of Częstochowa, Springer, Częstochowa/Zakopane.Google Scholar - Kuźniar, K. (2004).
*Analysis of vibrations of medium height buildings subjected to mining tremors with application of neural networks (in Polish)*. Cracow University of TechnologyGoogle Scholar - Kuźniar, K., Maciąg, E. and Waszczyszyn, Z. (2000). Computation of fundamental natural periods of vibrations of medium-hight buildings by neural networks.
*Archives of Civil Engineering*, 46:515–523.Google Scholar - Kuźniar, K. and Waszczyszyn, Z. (2007). Neural networks for the simulation and identification of building subjected to paraseismic excitations. In Lagaros, N. D. and Tsompanakis, Y., editors,
*Intelligent Computational Paradigms in Earthquake Engineering*. Idea Group Publishing.Google Scholar - Lampinen, J. and Vehtari, A. (2001). Bayesian approach for neural networks — review and case studies.
*Neural Networks*, 14(3):7–24. (Invited article).CrossRefGoogle Scholar - Lefik, M. (2005).
*Application of Artificial Neural Networks in Mechanics and Engineering (in Polish)*. Łódź University of Technology.Google Scholar - Lefik, M. and Schrefler, B. (2002). One-dimensional model of cable-in-conduit superconductor under cyclic loading using artificial neural networks.
*Fusion Engineering and Design*, 60:105–117.CrossRefGoogle Scholar - Lou, K.-N. and Perez, R. (1996). A new system identification technique using Kalman filtering and multilayer neural networks.
*Artificial Intelligence in Engineering*, 10:1–8.CrossRefGoogle Scholar - MacKay, D. J. C. (1992). Bayesian interpolation.
*Neural Computation*, 4(3):415–447.CrossRefGoogle Scholar - MacKay, D.J.C. (1998). Introduction to Gaussian processes. In Bishop, C. M., editor,
*Neural Networks and Machine Learning*, NATO ASI Series, pages 133–166. Springer.Google Scholar - MacKay, D. J. C. (2003).
*Information Theory, Inference and Learning Algorithms*. Cambridge University Press.Google Scholar - Masters, T. (1993).
*Practical Neural Network Recipes in C++*. Academic Press.Google Scholar - Nabney, I. T. (2004).
*Netlab: Algorithms for Pattern Recognition*. Springer-Verlag, London.Google Scholar - Neal, R. M. (1992). Bayesian training of backpropagation networks by the hybrid Monte Carlo method. Technical Report CRG-TR-92-1.Google Scholar
- Neal, R. M. (2004). Software for Flexible Bayesian Modeling and Markov Chain Sampling. Technical report, University of Toronto.Google Scholar
- Nijihuis, A., Noordman, N. and ten Kate, H. (1998). Mechanical and Electrical testing of an ITER CS1 Model Coil Conductor under Tramsferse Loading in a Cryogenic Press, Preliminary report. Technical report, University of Twente.Google Scholar
- Pham, D. and Liu, X. (1995).
*Neural Networks for Identification, Prediction and Control*. Springer.Google Scholar - Rasmussen, C. E. and Williams, C. K. I. (2006).
*Gaussian Processes for Machine Learning*. The MIT Press, Cambridge, Massachusetts.zbMATHGoogle Scholar - Rojas, R. (1996).
*Neural Networks — A Systematic Introduction*. Springer.Google Scholar - Sato, T. and Sato, M. (1997). Structural identification using neural networks and kalman filtering.
*JSCE*, 14:23–32.Google Scholar - Słoński, M. (2005). Prediction of concrete fatigue durability using Bayesian neural networks.
*Computer Assisted Mech. Eng. Sci.*, 12:259–265.Google Scholar - Słoński, M. (2006). Bayesian regression approaches on example of concrete fatigue failure prediction.
*Computer Assisted Mech. Eng. Sci.*, 13:655–668.zbMATHGoogle Scholar - Słoński, M. (2007). HPC strength prediction using Bayesian neural networks.
*Computer Assisted Mech. Eng. Sci.*, 14:345–352.zbMATHGoogle Scholar - Słoński, M. (2009). A comparison between Bayesian neural networks and other machine learning methods for predicting properties of concrete. Proc. of
*18th International Conference on Computer Methods in Mechanics*, CMM-2009, Zielona-Góra, Poland.Google Scholar - Tipping, M. E. (2001). Sparse Bayesian learning and the relevance vector machine.
*Journal of Machine Learning Research*, 1:211–244.zbMATHCrossRefMathSciNetGoogle Scholar - Tipping, M. E. (2004). Bayesian Inference: An Introduction to Principles and Practice in Machine Learning. In O. Bousquet, U. v. L. and Rätsch, G., editors,
*Advanced Lectures on Machine Learning*, volume 3176 of*Lecture Notes in Computer Science*, pages 41–62. Springer.Google Scholar - Twomey, J. M. and Smith, A. E. (1997). Validation and verification. In Kartam, N., Flood, I. and Garrett, J. H., editors.
*Neural Networks for Civil Engineers: Fundamentals and Applications*. ASCE, New York.Google Scholar - A. Vehtari and J. Vanhatalo. MCMC Methods for MLP and GP and Stuff (for Matlab) V2.1. (A User Manual Laboratory of Computational Engineering, Helsinki University of TechnologyGoogle Scholar
- Waszczyszyn, Z. (1999).
*Neural Networks in the Analysis and Design of Structures*. CISM Courses and Lectures No. 404. Springer, Wien-New York.Google Scholar - Waszczyszyn, Z. (2006). Artificial neural networks in civil and structural engineering: Ten years of research in Poland.
*Computer Assisted Mech. Eng. Sci.*, 13:489–512.zbMATHGoogle Scholar - Waszczyszyn, Z. and Słoński, M. (2006). Bayesian neural networks for prediction of response spectra.
*Foundations of Civil and Environmental Engineering*. 7:343–361.Google Scholar - Waszczyszyn, Z. and Ziemiański, L. (2005). Neural networks in the identification analysis of structural mechanics problems, Ch. 7. In Mróz, Z. and Stavroulakis, G., editors,
*Parameter Identification of Materials and Structures*, CISM Lecture Notes No. 469, pages 265–340. Springer, Wien-New York.CrossRefGoogle Scholar - Zell, A., Mache, N., Sommer, T. and et al. (1994). Stuttgart Neural Network Simulator. User manual, ver. 3.2. Technical report, University of Stuttgart, Germany.Google Scholar