Utilizing artificial neural network approach for solving two-dimensional integral equations

This paper surveys the artificial neural networks approach. Researchers believe that these networks have the wide range of applicability, they can treat complicated problems as well. The work described here discusses an efficient computational method that can treat complicated problems. The paper intends to introduce an efficient computational method which can be applied to approximate solution of the linear two-dimensional Fredholm integral equation of the second kind. For this aim, a perceptron model based on artificial neural networks is introduced. At first, the unknown bivariate function is replaced by a multilayer perceptron neural net and also a cost function to be minimized is defined. Then a famous learning technique, namely, the steepest descent method, is employed to adjust the parameters (the weights and biases) to optimize their behavior. The article also examines application of the method which turns to be so accurate and efficient. It concludes with a survey of an example in order to investigate the accuracy of the proposed method.


Introduction
Recently, integral equations have been extensively investigated theoretically and numerically. Note that they occur in a wide variety of physical applications, various fields of neural sciences and numerous applications such as electrical engineering, economics, elastically, plasticity, etc. Since these equations usually cannot be solved explicitly, it is going to be obtained in approximate solutions. There are several numerical methods for approximating solution of Fredholm and Volterra integral equations in one-and twodimensions. For example, Tricomi in his book [25] introduced the classical method of successive approximations for integral equations. Variational iteration method [15] was effective and convenient for solving integral equations. The Homotopy analysis method (HAM) was proposed by Liao [16] and then has been applied in [1]. The Taylor expansion approach was presented for solving integral equations by Kanwal and Liu [14] and then has been extended in [17]. In addition, Jafari et al. [12] applied Legendre wavelets method to find numerical solution of linear integral equations. In [13] an architecture of artificial neural networks (NNs) was suggested to approximate solution of linear Fredholm integral equations systems. For this aim, first the truncation of the Taylor expansions for unknown functions was substituted in the origin system. Then the purposed neural network has been applied for adjusting the real coefficients of given expansions in resulting system. In [9], a numerical method based on feedforward neural networks has been presented for solving Fredholm integral equations of the second kind. The Bernstein polynomials have frequently been applied in the solution of integral equations and approximation theory [5-7, 19, 20]. Also, there are many articles which deal with the solution and analysis of two-dimensional Fredholm and Volterra integral equations. Mirzaei and Dehghan [22] described a numerical scheme based on the moving least squares (MLS) method for solving integral equations in one-and two-dimensional spaces. The method was a meshless method, since it did not require any background interpolation or approximation cells and it did not depend on the geometry of domain. Hadizadeh and Asgary [11] using the bivariate Chebyshev collocation method solved the linear Volterra-Fredholm integral equations of the second kind. Alipanah and Esmaeili [2] approximated the solution of the two-dimensional Fredholm integral equation using Gaussian radial basis function based on Legendre-Gauss-Lobatto nodes and weights. Two-dimensional orthogonal triangular functions are used in [3,18] as a new set of basis functions to approximate solutions of nonlinear two-dimensional integral equations. Babolian et al. [4] applied two-dimensional rationalized Haar functions for finding the numerical solution of nonlinear second kind two-dimensional integral equations. They reduced the present problem to solve a nonlinear system of algebraic equations using bivariate collocation method and Newton-Cotes nodes. Moreover, some different valid methods for solving these kind of equations have been developed. This paper focuses on constructing a new algorithm with the use of feed-forward neural networks to reach an approximate solution of the linear two-dimensional Fredholm integral equation. For this purpose, first unknown twovariable function in the problem is replaced by a three-layer perceptron neural network. Supposedly, the limits of integrations are partitioned into set points, this architecture of neural networks can calculate the output corresponding to input vector. Now a cost function to be minimized is defined on the set points. Consequently, the suggested neural net using a learning algorithm that is based on the gradient descent method adjusts parameters (the weights and biases) to any desired degree of accuracy. Here is an outline of the paper. In ''Preliminaries'', the basic notations and definitions of the integral equations and the artificial neural networks are briefly presented. ''The general method'' describes how to find approximate solution of the given two-dimensional integral equations using proposed approach. Finally in ''An example'', an numerical example is provided and results are compared with the analytical solutions to demonstrate the validity and applicability of the method.

Preliminaries
In this section we will focus on the basic definitions and introductory concepts in integral equations. In addition the basic principles of artificial neural network (ANN) approach are presented and reviewed for solving linear second kind two-dimensional integral equations (2D-IEs).

Integral equations
Integral equations appear in many scientific and engineering applications, especially when initial value problems for boundary value problems are converted to integral equations. As stated before, we will review some integral equations and linear two-dimensional integral equations of the second kind as well.
Definition 2.1 Let f : ½a; b ! R: For each partition P ¼ ft 0 ; t 1 ; . . .; t n g of ½a; b and for arbitrary n i ½t iÀ1 ; t i ð1 i n), suppose The definite integral of f ðtÞ over ½a; b is provided that this limit exists in the metric D [25].

Definition 2.2 The linear two-dimensional Fredholm integral equation (2D-FIE) of the second kind is presented by the form [2]
Fðx; where k is a constant parameter, the kernel k and f are given analytic functions on L 2 ð½a; b Â ½c; dÞ. The two-variable unknown function F that must be determined appears inside and outside the integral signs. This is a characteristic feature of a second kind integral equation. It is important to point out that if the unknown function appears only inside the integral signs, the resulting equation is of first kind.
If the kernel function satisfies kðx; y; s; tÞ ¼ 0; s [ x; t [ y in Eq. (1), we obtain the linear twodimensional Volterra integral equation (2D-VIE) [24] Fðx; yÞ ¼ f ðx; yÞ þ k Z y c Z x a kðx; y; s; tÞFðs; tÞdsdt; It should be noted that, if one of the limits of integration varies, the integral equation is called a Volterra-Fredholm integral equation. It is clear that, two-dimensional integral equations appear in many forms. Three distinct ways that depend on the limits of integration are used to characterize these equations which have been are briefly introduced. Notice that, if the function f ðx; yÞ in the present integral equations is identically zero, the equation is called homogeneous. Otherwise it is called inhomogeneous. These three concepts play a major role in the structure of the solution.

Artificial neural networks
Artificial neural networks (ANNs) can be considered as simplified computational structures that are inspired by observed process in natural networks of biological neurons in the brain. They are nonlinear mapping architectures based on the function of the human brain, therefore can be considered as powerful tools for modeling, especially when the underlying data relationship is unknown. A very important feature of these networks is their adaptive nature, where ''learning by example'' replaces ''programming'' in solving problems. In other words, in contrast to conventional methods, which are used to perform specific task, most neural networks are more versatile. This feature raises a very appealing computational model which can be applied to solve variety of problems.
The multilayer feed-forward neural network or multilayer perceptron (MLP) that had been proposed by Rosenblatt [23] is very popular and is used more than other neural network type for a wide variety of tasks. The present network learned by back-propagation algorithm is based on supervised procedure. In other words, the network constructs a model based on examples of data with known output.
In this subsection, an architecture of MLP model is discussed here briefly. We intend to give a short review on learning of the given neural network. First consider a threelayer ANN with two input units, N neurons in hidden layer and one output unit. Mathematical representation of the present neural network is given in Fig. 1. Using the figure, input-output relation of each unit and calculated output u N ðx; yÞ can be written as follows:

Input units:
The input neurons make no change in their inputs, so: Hidden units: Input into a node in hidden layer is a weighted sum of outputs from nodes connected to it. Each unit takes its net input and applies an activation function to it. The input/ output relation is normally given as follows: where netðpÞ describes the result of the net outputs o i impacting on unit p. Also, w pi are weights connecting neuron i to neuron p and b p is a bias for neuron p. Bias term is baseline input to a node in absence of any other inputs. Output unit: The general method In this section, we intend to use the MLP method to get a new numerical approach for solving the linear twodimensional Fredholm integral equation of the second kind.
In other words, how to apply this method to make a series approximation for the solution Fðx; yÞ in (1) will be Fig. 1 Schematic diagram of the proposed MLP described. The output of two-layer MLP network that is defined in Eq. (1) can be rewritten as follows: In order to approximate function u, first the intervals ½a b and ½c d are partitioned into set points x i and y j ; respectively. Thus, the following set of equations will be obtained: Cost function First suppose that u N ðx; yÞ is the approximate solution with the adjustable parameters (wights and biases) for the unknown Fðx; yÞ: After substituting this solution instead of the unknown function in the given 2D-FIE, the Eq. (1) can be transformed to a sum squared error minimization problem corresponding to the proposed neural network. So, the error function is regarded as a function on the weights and biases space of the net for x ¼ x i and y ¼ y j as follows: where a kðx i ; y j ; s; tÞu N ðs; tÞdsdt: Now the total error of the network is defined as: The goal then is to minimize this function; therefore, we must deduce a back-propagation learning algorithm using the present cost function.

Proposed learning algorithm
Multilayer feed-forward neural network is learned by backpropagation algorithm that is based on supervised procedure. In other words, the MLP network is trained using a supervised learning algorithm which uses the training data to adjust the network weights and biases. Now let w p;q ; W p and b p ðfor p ¼ 1; . . .; N; q ¼ 1; 2Þ are initialized at small random values for input signals. For parameter w p;q adjustment rule can be written as follows: w p;q ðr þ 1Þ ¼ w p;q ðrÞ þ Dw p;q ðrÞ; p ¼ 1; . . .; N; q ¼ 1; 2; ð10Þ Dw p;q ðrÞ ¼ Àg: oE i;j ow p;q þ a:Dw p;q ðr À 1Þ; where r is the number of adjustments, g is the learning rate and a is the momentum term constant. Similarly this adjustment rule can be written for other weight parameters. Thus, our problem is to calculate the derivative oE i;j ow p;q in (11). The derivative can be calculated as follows: Using a similar procedure as mentioned above, we have the correspondingly corollary for parameters W p and b p ; in which we are refrained from going through proof details. So, we have: The MLP neural nets are the sample of regular networks, therefore they can approximate any continuous function on a compact set to arbitrary accuracy [10]. Now the learning algorithm can be summarized as follows: Learning process Step 1: g [ 0; a [ 0 and Emax [ 0 are chosen. Then quantities w p;q ; W p and b p ðp ¼ 1; . . .; N; q ¼ 1; 2Þ are initialized at small random values.
Step 2: Let r :¼ 0 where r is the number of iterations of the learning algorithm. Then the running error E is set to 0.
Step 3: Let r :¼ r þ 1: Repeat below procedure for different values of i and j: i Forward calculation: Calculate the output vector u N ðx i ; y j Þ by presenting the input vectors x i and y j : ii Back propagation: Adjust the parameters w p;q ; W p and b p using the cost function (8).
Step 4: Cumulative cycle error is computed by adding the present error to E.
Step 5: The training cycle is completed. For E\Emax terminate the training session. If E [ Emax then E is set to 0 and we initiate a new training cycle by going back to Step 3.

An example
In this section, in order to investigate the accuracy of the proposed method, we have chosen an example of linear two-dimensional integral equations of the second kind. For the example, the computed values of the approximate solution are calculated over a number of iterations and the cost function is plotted. Also, to show the efficiency of the present method for our problem, results will be compared with the exact solution. with the exact solution Fðx; yÞ ¼ x:cosðyÞ: In this example, we illustrate the use of the FNN technique to approximate the solution of this integral equation. In the following simulations, we use the specifications as follows: 1. The number of hidden units: N ¼ 3; 2. Learning rate g ¼ 0:5; 3. Momentum constant a ¼ 0:05: Numerical result can be found in Table 1, and Fig. 2 shows the cost function in the 20 iterations. Figures 3, 4, 5, 6 show the convergence behaviors for computed values of the weight parameters w p;q and W p ; bias b p for different number of iterations.
There is no magic formula for selecting the optimum number of hidden neurons. However, some thumb rules are available for calculating number of hidden neurons. A rough approximation can be obtained by the geometric pyramid rule proposed by Masters [21]. For a three-layer network with n input and m output neurons, the hidden layer would have at least ½ ffiffiffiffiffiffi nm p þ 1 neurons. To show convergence of the proposed method we solve Example 4.1 using shifted Legandre collocation method.
The reason for choosing shifted Legandre collocation method is its simplicity. The details of shifted Legandre collocation method are as follows.
Shifted Legandre collocation method The Legendre polynomials, P n ðxÞ; n ¼ 0; 1; . . .; are the eigenfunctions of the singular Sturm-Liouville problem ð1 À x 2 ÞP 0 n ðxÞ À Á 0 þnðn þ 1ÞP n ðxÞ ¼ 0: Also, they are orthogonal with respect to L 2 inner product on the interval ½À1; 1 with the weight function wðxÞ ¼ 1; that is where d nm is the Kronecker delta. The Legendre polynomials satisfy the recursion relation P nþ1 ðxÞ ¼ 2n þ 1 n þ 1 xP n ðxÞ À n n þ 1 P nÀ1 ðxÞ; where P 0 ðxÞ ¼ 1 and P 1 ðxÞ ¼ x: If P n ðxÞ is normalized so that P n ð1Þ ¼ 1; then for any n, the Legendre polynomials in terms of power of x are where n 2 Â Ã denotes the integer part of n 2 : The Legendre-Gauss-Lobatto (LGL) collocation points À1 ¼ x 0 \x 1 \ Á Á Á \x N ¼ 1 are the roots of P 0 N ðxÞ together with the points -1 and 1. Explicit formulas for the LGL points are not known. The LGL points have the property that which is exact for polynomials of degree at most 2N À 1; where w i ; 0 6 i 6 N, are LGL quadrature weights. For more details about Legendre polynomials, see [8].  The shifted Legendre polynomials (ShLP) on the interval t 2 ½0; 1 are defined by b P n ðtÞ ¼ P n 2t À 1 ð Þ; n ¼ 0; 1; . . .; which are obtained by an affine transformation from the Legendre polynomials. The set of ShLP is a complete L 2 ½0; 1-orthogonal system with the weight function wðtÞ ¼ 1: Thus, any function f 2 L 2 ½0; 1 can be expanded in terms of ShLP.
The ShLGL (Shifted Legendre-Gauss-Lobatto) collocation points 0 ¼ t 0 \t 1 \ Á Á Á \t N ¼ 1 on the interval ½0; 1 are obtained by shifting the LGL points, x i ; using the transformation Thanks to the property of the standard LGL quadrature, it follows that for any polynomial p of degree at most 2N À 1 on ð0; 1Þ; whereŵ i ¼ 1 2 w i ; 0 6 i 6 N, are ShLGL quadrature weights. The results stated above are also satisfied for Legendre-Gauss and Legendre-Gauss-Radau quadrature rules.
The function Fðx; yÞ is approximated by a ShLP of degree at most N as Now, by substituting (18) and collocation points (17) in (16), we have By solving this linear system we can find a ij ; i; j ¼ 0; 1; . . .; N; and then approximate the solution Fðx; yÞ: We solved the Example 4.1 using the method described for N ¼ 2 and N ¼ 3: Results are shown in Table 2. By comparing Tables 1 and 2 we find that obtained results in Table 1 are in concordance.

Conclusions
This paper suggested a new computational method to solve a two-dimensional Fredholm integral equations. So, a feedforward artificial neural network has been proposed. This network is able of estimating approximate solution of assumed equation using the learning algorithm which is based on steepest descent rule. Clearly, in order to obtain accurate solution, many learning procedure should be considered. The analyzed examples illustrated the ability and reliability of the present approach. The obtained solutions, in comparison with exact solutions admit a remarkable accuracy. Extensions to the case of more general of integral equations are left for future studies.
Open Access This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.