Computational Intelligence in Remote Sensing Image Registration: A survey

In recent years, computational intelligence has been widely used in many fields and achieved remarkable performance. Evolutionary computing and deep learning are important branches of computational intelligence. Many methods based on evolutionary computation and deep learning have achieved good performance in remote sensing image registration. This paper introduces the application of computational intelligence in remote sensing image registration from the two directions of evolutionary computing and deep learning. In the part of remote sensing image registration based on evolutionary calculation, the principles of evolutionary algorithms and swarm intelligence algorithms are elaborated and their application in remote sensing image registration is discussed. The application of deep learning in remote sensing image registration is also discussed. At the same time, the development status and future of remote sensing image registration are summarized and their prospects are examined.


Introduction
Remote sensing image registration has important applications in remote sensing image processing, and is a basic problem in many remote sensing information extraction and processing technologies. In the practical application of remote sensing images, registration is an indispensable step in the process of image fusion [1] , change detection [2] , feature recognition [3] , image mosaics [4] , image segmentation [5] and image classification [6,7] . Many later theories and applications are carried out on the assumption that the registration problem has been solved. The accuracy of remote sensing image registration will directly affect the accuracy and application effects of the final application results. In recent years, with the continuous improvement of human ability in observing the earth [8] , the registration of remote sensing images has made great progress [9] . The purpose of image registration is trying to match two or more images which are about the same object but taken in different situations, such as with different time, different viewpoints and different sensors [10,11] . Compared with natural image registration, remote sensing image registration is more challenging because remote sensing images have more complex features [12] . The procedure of remote sensing image registration can be divided into five steps [10] , which are preprocessing, feature selection, feature correspondence, determining transformation functions, and resampling. In this paper, we mainly discuss the applications of computational intelligence in remote sensing image registration, which mainly includes evolutionary algorithms [13,14] and deep learning [15,16] .
Evolutionary algorithms have been widely used in remote sensing image registration, and have achieved great performance [17,18] . An evolutionary algorithm is a heuristic global optimization probability search algorithm, of which the principle is to simulate the evolution of the survival of the fittest in the ecosystem [19,20] . Compared with traditional optimization algorithms, evolutionary algorithms have the following advantages: 1) Evolutionary algorithms do not require the strict definitions of mathematical models to solve problems [21] . When the actual application requirements are abstracted as an optimization problem, the evolutionary algorithm could make full use of the corresponding fitness function to find the approximate optimal solution without relying on other information, and the actual application requirements could be satisfied.
2) In the optimization process, the traditional optimization strategy is to start from an initial value and then iterate step by step to find the optimal solution, but the evolutionary algorithm starts from a population that contains many points, and finds the optimal or suboptimal solution through the evolution iteration of the population [22] .
3) The robustness and embedded parallelism of the evolutionary algorithm makes the algorithm very effective in finding the global optimal solution, and it is not easy to fall into the dilemma of the local optimal solution, and realize the global optimization in the probability sense [23] .
Because the task of remote sensing image registration could be regarded as an optimization problem by establishing a reasonable model and evolutionary algorithms have great performance in optimization problems, evolutionary algorithm has important applications in solving the task of remote sensing image registration [24] .
Due to the powerful learning ability of deep learning, it has been widely used in computer vision tasks [25] , such as image registration [26] , object detection [27] , change detection [28] and image fusion [29] . Especially with the introduction of a series of outstanding feature extraction networks, such as AlexNet [30] , VGGNet [31] , and GoogleNet [32] , remote sensing image registration based on deep learning has achieved satisfactory performance. The convolutional neural networks (CNNs) simulate the connection pattern between neurons in animal visual cortical tissue, which are a deep learning structure of a multilayer feed-forward artificial neural network [33] . CNNs are generally composed of multiple convolutional layers, pooling layers and fully connected layers connected to each other [34] . The convolution layer uses various convolution kernels to perform convolution operations on the input to extract various features [35] . The pooling layer performs dimensionality reduction on the input through pooling operations, thereby reducing the number of network parameters [36] . The fully connected layer is usually the last part of CNNs and is a traditional multilayer perceptron network [37] . Each neuron is connected to each neuron of the previous layer. The powerful feature extraction and representation capabilities of CNNs can overcome the instability of lowlevel features and improve the reliability of registration [38] .
Compared with natural images, remote sensing images could be obtained in various ways, the image content is difficult to understand, and their features are not obvious. The method of manually designed features has a narrow scope of application in remote sensing images. It is impossible to predict uncontrollable changes in remote sensing images, and it is difficult to extract discriminative features. For the traditional method, there is no feedback of information between feature extraction and feature matching. Deep neural networks have powerful learning capabilities, which can extract higher-dimension-al features for registration. Deep learning further improves the accuracy and robustness of remote sensing image registration [38] .
Computational intelligence is intelligent [39] , parallel [40] and robust [24] , and does not depend on the characteristics of the problem itself. The computational intelligencebased algorithms are basically optimized for solving problems in a group collaboration manner, which is very suitable for large-scale parallel processing. Algorithms based on computational intelligence have good fault tolerance and are insensitive to initial conditions. It can find optimal solutions under different conditions. The remainder of this paper is organized as follows. In Table 1, we show the abbreviations that appear in the text. The remote sensing image registration based on the evolutionary algorithm is described in detail in Section 2. The remote sensing image registration based on deep learning is reported in Section 3, and Section 4 draws the conclusions.

Remote sensing image registration based on evolutionary computation
Evolutionary computing [41] is a kind of adaptive artificial intelligence technology that can simulate the evolution processes and mechanisms of biology to solve problems. There are many methods based on evolutionary computation that have been surveyed in the field of natural image registration [42−46] and medical image registration [47−51] . Although not all the algorithms are effective in remote sensing image registration, it has been proved that some algorithms can achieve better performance in remote sensing image registration. According to the frequency of various algorithms used in remote sensing image registration, this paper mainly divides them into two  [52−63] and remote sensing image registration based on swarm intelligence algorithms [64−70] .

Method based on evolutionary algorithm
Evolutionary algorithms [71] are inspired by biological evolution and genetic theory. An evolutionary algorithm is a general problem-solving algorithm based on natural selection and population genetics. It uses the selection mechanism of survival of the fittest to find the most suitable solution. Among many evolutionary algorithms, genetic algorithms [72,73] are more widely used in remote sensing image registration than genetic programming, evolution strategies, and evolutionary planning because of their relatively mature development. At the same time, as a new and efficient heuristic parallel search technology, the differential evolution algorithm [74−76] has the advantages of fast convergence, fewer control parameters and robust optimization results, so its application in remote sensing image registration has also been studied. Next, we give a brief overview of genetic algorithms and differential evolution algorithms, and discuss the applications of genetic algorithms [52−57] and differential evolution algorithms [58−63] in remote sensing image registration. Table 2 shows a summary of the application of evolutionary algorithms in remote sensing image registration.

Overview of genetic algorithm
A genetic algorithm was first proposed by Holland, a professor of the University of Michigan [72] . It is a computational model simulating the natural selection and genetic mechanisms of Darwinian biological evolution, and a method of searching the optimal solution by simulating the natural evolution [77] . Genetic algorithms are mainly used in feature matching [52,54,55,57] and parameter estimation of transformation models [53,56] in remote sensing image registration. We combine the basic flow of genetic algorithms and take [55,57] for example. The basic flow chart of the algorithms for applying genetic algorithm to the feature matching step in the image registration process is shown in Fig. 1. First of all, we use the existing methods to roughly match the information of the reference image and the sensed image. Secondly, we encode the information after the coarse matching to construct the initial population. And then, the population was randomly selected according to the fitness function value, and the population position was updated by crossover and mutation population coding. The fitness function is calculated to update the optimal solution and judge the convergence condition. If the convergence condition is satisfied, the optimal solution is output, otherwise the crossover and mutation operation is performed again.
Next, the transformation model is estimated, and finally the registration image is obtained. When using genetic algorithms to solve specific problems, the coding rules of chromosomes, the selection of fitness function, the design of genetic operators and the determination of algorithm parameters need to be analyzed for specific problems. This is also the difficulty of Table 2 Application research of evolutionary algorithm

Author
Year Method improvement Advantages Inglada and Adragna [52] 2001 Using genetic algorithm to find the optimal set of control points.
Avoid local minima.
Makrogiannis et al. [53] 2004 Combining coarse registration with genetic algorithm and fine registration with scale optical flow method.
Achieve higher accuracy registration while avoiding excessive calculation.
De Falco et al. [58] 2007 Using differential evolution for remote sensing image registration.
Innovative introduction of differential evolution algorithm to avoid local optimization.
De Falco et al. [61] 2009 Applying distributed differential evolution to remote sensing image registration.
Reduce the risk of population particle stagnation.

Hu et al. [62] 2012
An adaptive differential evolution strategy is proposed, which combines probability matching and multiresolution techniques to register remote sensing images.
Probability matching technology is used in DE to independently choose the most suitable strategy when solving problems.
Ma et al. [63] 2014 Orthogonal learning strategy applied to remote sensing image registration.
Use fewer experimental samples to find the best combination between levels of factors.
Zhang et al. [54] 2014 Applying genetic algorithm to the combination of coarse matching with grid matching and fine matching with triangle mapping.
Reduce the amount of calculation and reduce the error of manual decision.
Gou and Ma [55] 2016 Construct a graph by delaunay triangulation, and solve the graph similarity by genetic algorithm to find the best matching point.
In the case of large gray scale differences, the accuracy is still high.
Li et al. [56] 2016 The genetic algorithm mainly corrects the image rotation to preprocess the image.
More matching control points can be obtained to improve the accuracy of registration.
Yavari et al. [57] 2018 Use genetic algorithm to get the best quantity and best combination of well-distributed ground control information.
Avoid the influence of some manual operations on the experiment in the registration process and improve the accuracy of registration.
using genetic algorithms to solve problems. Therefore, the rational application of genetic algorithms is the key to solving the problem.

Application research of genetic algorithm
Genetic algorithms are mainly used in feature matching and parameter estimation of transformation models in remote sensing image registration. Next, we will discuss these two parts. In [52,54,55,57], a genetic algorithm is applied to feature matching in image registration. The genetic algorithms are applied to the parameter estimation of transformation models in image registration. Next, we discuss them separately [53−56] .
Using the genetic algorithm for matching. Inglada and Adragna [52] proposed automatic multisensor image registration by edge matching using genetic algorithms. A genetic algorithm is used to find the optimal set of control points by minimizing the distance on the basis of the proposed minimization measurement method. This method does not require too high extraction accuracy to achieve high registration accuracy, and at the same time, the use of genetic algorithms avoids the problem of local optimal solutions. In order to register aerial images as satellite images or map images, Zhang et al. [54] proposed an image registration with position and similarity constraints based on a genetic algorithm. The genetic algorithm is mainly to minimize the mean square error of the two grids and minimize the total deformation of the grid points. Compared with similar methods in the past, the main advantage of this method is to avoid errors in the manual operation and manual decision-making process. Gou and Ma [55] presented a method based on delaunay triangulation and a genetic algorithm for multisource remote sensing registration. The genetic algorithm mainly solves the optimal solution of graph similarity, so as to obtain the optimal set of control points for solving the transformation model. A prominent point of this method is to use the genetic algorithm to eliminate mismatches and find the best set of corresponding points, so as to improve the accuracy of registration. Yavari et al. [57] proposed a method for selecting the best distribution and the best amount of ground control information based on genetic algorithms to improve the accuracy of registration. Compared with the past methods, this method is a significant achievement in reducing the errors caused by the extraction process and achieving sub-pixel precision registration.
Using genetic algorithms to estimate the transformation parameters. Makrogiannis et al. [53] proposed an affine mapping scheme based on random optimization in order to solve the distortion caused by changes in time, viewpoint and terrain in remote sensing images. The genetic algorithm is used to search the searching space defined by the affine transformation parameters in the coarse registration stage. In order to achieve sub-pixel registration and avoid the high cost of genetic algorithms, the registration scheme is completed by a hierarchical optical flow approach. The combination of coarse registration and fine registration reduces the computational complexity while ensuring registration accuracy. Li et al. [56] proposed a multi-angle normalized cross-correlation method based on genetic optimization. This method can determine the maximum value of the correlation coefficient between two images through the genetic algorithm, so as to detect the angle offset between the target image and the reference image, and complete the rotation offset correction. The advantage of this newly added preprocessing operation is that more matching points can be obtained, thereby improving the accuracy of registration. The authors used a genetic algorithm to preprocess the registration image, which provided a new way for us to solve the problem of image registration.

Overview of differential evolution algorithm
A differential evolution algorithm is a heuristic random search algorithm based on population difference. The algorithm is proposed to solve Chebyshev polynomials [22,78,79] . A differential evolution algorithm is successfully applied to remote sensing image registration due to its simple principle, few controlled parameters and strong robustness. According to the existing references, a  Fig. 1 Remote sensing image registration-genetic algorithm flow chart differential evolution algorithm is only used for parameter estimation of a transformation model.
Combined with the basic flow of the differential evolution algorithm, the flow chart of the algorithm applied to remote sensing image registration is shown Fig. 2. Firstly, the reference image and the image to be registered are input, and the parameters of the transformation model are randomly initialized in the search space of the transformation parameters. Then, according to the fitness function, the operations of mutation, crossover and selection are carried out until the output conditions are met. Finally, according to the optimal transformation parameters, the image to be registered is matched with the reference image to obtain the final image registration result. 2.1.4 Application research of differential evolution algorithm De Falco et al. [58,59] proposed a differential evolution for the registration of remotely sensed images. The authors innovatively introduce differential evolution into remote sensing image registration to effectively avoid the problem of local optimization. In order to increase the chance of finding the optimal value, De Falco et al. [60,61] applied the distributed difference algorithm to remote sensing image registration. This algorithm reduces the risk of subpopulation stagnation of the differential evolution algorithm. The distributed difference algorithm is used to solve the global optimal parameters of the affine transformation to maximize mutual information. Hu et al. [62] proposed a remote sensing image registration algorithm based on the differential evolution algorithm and adaptive strategy selection. The method is mainly used to solve the traditional differential evolution algorithm used for remote sensing image registration when the calculation is large and the convergence speed is slow. Compared with traditional differential evolution algorithms, this algorithm can achieve sub-pixel accuracy registration. Ma et al. [63] proposed an algorithm based on orthogonal learning differential evolution for remote sensing image registration. The salient point of this method is that orthogonal learning strategies are used to optimize the crossover process in differential evolution, so that people use fewer experimental samples to find the best combination between various factor levels. This method has faster convergence speed and more robust result than the previous meta-heuristic algorithms in remote sensing images.
Some problems in remote sensing image registration can be solved by using evolutionary algorithms. Through reasonable chromosome coding and an adaptive cross mutation strategy, the convergence speed of the algorithm can be significantly improved under the premise of ensuring the diversity of solution sets. The application of evolutionary algorithm in remote sensing image registration provides more possibilities for remote sensing image registration. How to use the improved evolutionary algorithm to solve the optimization problem of remote sensing image registration parameters is still an important research question in the future.

Method based on swarm intelligence
Swarm intelligence algorithms [80−82] are widely used in remote sensing image registration, and particle swarm optimization algorithms play an important role because of their outstanding performance. Next, the principles and applications of particle swarm optimization algorithms are introduced [64−69] . In addition to particle swarm optimization algorithms, ant colony algorithms have been applied to remote sensing image registrations [70] , so their applications will be briefly introduced. According to the existing references, particle swarm algorithms and ant colony algorithms are all applied to the parameter estimation of transformation models. Table 3 shows a summary of the applications of swarm intelligence algorithms in remote sensing image registration.

Overview of particle swarm optimization
PSO algorithm was proposed by Kennedy and Eberhart in 1995 [83] . Particle swarm optimization algorithms are a kind of swarm optimization algorithms that simulate the foraging activity of birds [83−85] . Particle swarm optimization is used to find the optimal transformation model parameters in remote sensing image registration [64−69] . Combined with the basic process of particle swarm optimization algorithms, the basic process applied to remote sensing image registration is shown in Fig. 3. First, randomly sample and generate particle swarms within the transform parameter range and initialize them. Second, the optimal position found by the group (Gbest) and the optimal position found by the particles (Pbest) are calculated. Finally, the fitness value of each particle is calculated, and if the stopping criterion is not met, the velocity of the particle is updated by (1), the position of the particle is updated by (2), and the Gbest of the particle swarm and the Pbest of the particle are updated. Recycling updates the fitness value of each particle. If the stopping criterion is met, the op-timal model conversion parameters are output. The classical velocity and position updating rules are as follows: (1) where (t) and (t) are the position and velocity of the i-th particle on iteration t, c and c are the two learning rate weights, r and r are the two random parameters uniformly distributed in [0, 1], w is the inertia weight.

Application research of particle swarm optimization
In swarm intelligence algorithms, the particle swarm optimization algorithm has been widely used in remote sensing image registration because of its fast convergence speed, simplicity of the algorithm itself, and few adjustment parameters. Lu et al. [64] adopted a registration framework from coarse to fine. In the coarse matching stage, the particle swarm optimization algorithm is used to search for the best rigid parameters to obtain the globally optimal mutual information. In the stage of fine registration, the Harris detector is used for fine registration of the coarse registration results. This method has a strong automatic registration ability and can deal with large displacements between remote sensing images quickly and steadily. Zhang et al. [65] proposed to combine the PSO algorithm with the Powell direction set algorithm, in order to avoid the problem of falling into local maxima during the registration process of multisensor remote sensing images. PSO is used to compute a good initial guess for following the Powell's direction set method (PDSM). At the same time, the combination of the PSO algorithm and the Powell algorithm also effectively avoids the calculation of the objective function gradient and reduces the computational complexity. An et al. [66] Table 3 Research on the application of particle swarm optimization

Author
Year Method improvement Advantages Lu et al. [64] 2007 Combination of coarse registration using particle swarm and fine registration using Harris detector.
Large displacement between remote sensing images can be processed quickly.
Zhang et al. [65] 2009 Combine the PSO algorithm with the Powell direction set algorithm.
Avoid local optimization and realize accurate registration.
An et al. [66] 2009 Innovative application of improved particle swarm optimization algorithm in remote sensing image registration.
Control the amount of calculation and ensure the diversity of population.
Gharbia et al. [67] 2015 Combination of mutual information and thin plate spline method.
This method has better performance for large-scale variation, rotation and intensity variation.
Yavari et al. [68] 2017 Remote sensing image registration using line-based rational function module with particle swarm optimization.
Improve registration accuracy and reduce system errors and the amount of control information required.
Wu et al. [69] 2018 Relative to RANSAC, the model transformation parameters are directly sampled, rather than randomly selected for trial matching.
Significantly improved registration accuracy compared to the existing methods.
Wu et al. [70] 2019 Combination of ant colony algorithm and local search algorithm.
Avoiding local optimization and reducing computation.

No
Reference image Sensed image Pretreatment Generate initial particles swarm and initialization

Calculate fitness function
Stopping criteria

Yes
Obtain the optimal parameters of the model

Image registration
Update the best position of particle swarm history Update particle speed and position Update the best position of each particle history Fig. 3 Remote sensing image registration-particle swarm optimization algorithm flow chart proposed an improved particle swarm algorithm based on mutual information for registration. The algorithm abandons the traditional situation of fixed population size, and uses an adaptive variable population size to optimize the function. Therefore, the algorithm cannot only control the amount of calculation, but also ensure the diversity of the population and improve the search ability of the particles. But compared with the traditional exhaustion based on mutual information, the accuracy is greatly reduced. Gharbia et al. [67] proposed the mutual information and thin plate spline method to register remote sensing images. After the thin plate spline provides the geometric representation of the corresponding landmark's relative position, the particle swarm optimization algorithm is mainly used to improve the corresponding relationship between landmarks and realize the maximum mutual information function. Compared with the previous methods, this method has better performance for large-scale variation, rotation and intensity variation. Yavari et al. [68] proposed an optimized rational function model based on linear features for high-resolution satellite image registration. The rational function model is suitable for registration due to its versatility and independence from the sensor, but the combination of over-parameterization in the rational function and the incorrect number and order of terms in the rational function is the key problem to be solved. Particle swarm optimization algorithm is used to find the optimal combination of unrelated items due to its excellent global search ability. Wu et al. [69] proposed a particle swarm optimization sample consensus algorithm. The salient point of this algorithm is to abandon the traditional random sample consensus method of tentative matching, but to directly use the particle swarm algorithm to sample the transformation model. The algorithm has remarkable performance in dealing with low accuracy and more matches.
Particle swarm optimization has the characteristic that individuals can exchange information with each other. Particle swarm optimization has the advantage of fast convergence speed, but it is easy to fall into local search and lose the diversity of the particle swarm, and the local search ability of the particle swarm algorithm is weak. Therefore, it will be a good research direction in the future to increase the population diversity with the particle swarm algorithm to enhance its search ability, and to combine the particle swarm algorithm with some local search strategies to enhance its local search ability.

Application research of ant colony algorithm
The ant colony algorithm was first proposed by Italian researchers Dorigo and Maniezzo in the 1990s, which was inspired by ants' foraging behavior [86] . The ant colony algorithm has been widely used in natural image registration [87−89] and image detection [90−92] , but in remote sensing image registration, it is still in the exploratory stage. Wu et al. [70] have made remarkable achievements in their research.
In order to avoid falling into local optimums when solving complex multimodal problems, Wu et al. [70] proposed a multimodal continuous ant colony optimization algorithm for multisensor remote sensing image registration. The introduction of the ant colony algorithm mainly avoids falling into the local optimal situation and improves the global search ability. At the same time, in order to reduce the computational complexity, local search is introduced to significantly improve the search efficiency rate. Compared with the traditional methods, the proposed method is more effective and robust.

Introduction to the application of other algorithms
Other evolutionary calculation methods are also used in image registration, e.g., the simulated annealing algorithm [93] for remote sensing image registration. However, due to the limitations of the simulated annealing algorithm, there is no application of the simulated annealing algorithm in remote sensing image registration in recent years. There are more evolutionary calculation methods for medical image registration, such as the artificial immune system [51] , firefly algorithm [94] , coevolutionary algorithm [95] , etc. We also hope that these algorithms can also be applied to remote sensing image registration. Of course, there are still some popular methods of evolutionary computation that have not been found in image registration, such as the bat algorithm [96] , brain storm optimization algorithm [97] , etc. Whether these methods can be applied to remote sensing image registration is also a problem worthy of discussion.

Application of deep learning in remote sensing image registration
The representative of feature-based image registration method is SIFT [98] , a representative of feature-based image registration methods, has good performance in natural image processing tasks, because the generated feature descriptors have rotation, scaling, and translation invariance. However, because the imaging mechanism of remote sensing images is complex, its appearance is determined by the radiation charateristics, the geometric charateristics of the objects, and sensor's sending or receiving configuration. Therefore, the SIFT originally designed to process natural images may not continue to maintain good performance in remote sensing images task. In recent years, with the rapid development of deep learning and its outstanding performance in the field of computer vision, such as semantic segmentation [99,100] , object detection [101] , etc., this reflects its superiority compared to traditional methods. The representative framework is the convolutional neural network (CNN) [33] . The deeper the layer of the neural network is, the more information contained in the final extracted features is. Deep learning is a learning algorithm based on neural networks. The deep learning model consists of many network layers that convert input into specific outputs. The layer between the input layer and the output layer is usually called the hidden layer. Neural networks that contain multiple hidden layers are called deep neural networks. At present, deep learning is gradually developing. The CNN, GAN and AE are the most popular branches. CNN is the most widely used deep neural network framework. It is suitable for processing image data with neatly arranged pixels. Specifically, it is mainly composed of a convolutional layer, a pooling layer, and a fully connected layer. Every time the input image passes through a convolutional layer, it will convolute with a set of convolution kernels, each generates a new feature map. Compared with traditional multilayer perceptrons, CNN usually uses permutation invariant functions (e.g., maximum pooling, average pooling) to aggregate local information, currently popular frameworks in CNN include VGG network [31] , residual network [102] , etc. AE is a distributed data set representation used to learn compression. AE can achieve data compression and dimensionality reduction through a hidden layer. It is usually used for feature level processing. In the field of remote sensing image processing, AE is mainly used for feature representation [103,104] . The purpose of transfer learning is to improve the learning performance of target learners in the target domain by transferring knowledge contained in different but related domains, or to minimize the number of examples that need to be labeled in the target domain. Transfer learning is mainly used to solve the problems of fewer datasets in remote sensing image registration tasks. GANs [105] are a very popular category of unsupervised models in deep learning. They include a generative network and a discriminant network, and the two networks compete with each other. The generative network learns to map from the latent space to the specific data distribution of interest, and the discriminant network is used to distinguish real data and generated data. The goal of the entire network training is to deceive the discriminating network by generating generated data that looks real and has a real data distribution, prompting it to make a judgment that the generated data is the same as the real data. GAN has been successfully used in image processing tasks [106] . This section summarizes the application of deep learning in remote sensing image registration tasks. According to the difference of the loss function and the format of the network output, it can be divided into methods based on representation learning and metric learning.

Application research of representation learning in remote sensing image registration
The essence of representation learning is to transform raw data into a form that can be used by neural networks, i.e., to convert raw data into corresponding highlevel features. Table 4 summarizes the methods of deep learning applied to representation learning.

Using the network to extract features
Due to the characteristics of remote sensing images, the hand-craft features such as SIFT, HOG [107] used in Table 4 Analysis of deep learning method for representation learning

Author
Year Method improvement Advantages Ye et al. [38] 2018 Integrating the depth features extracted by CNN and the local features, and fusing the obtained features into the PSO-SIFT algorithm.
The accuracy is higher than PSO-SIFT algorithm.
Wang et al. [119] 2018 An end-to-end registration deep network is proposed, which mainly includes two stages: 1) learning of mapping function, 2) image registration.
Applying transfer learning to reduce training costs, results are more accurate than traditional methods.
Quan et al. [120] 2018 GAN was used to automatically create more training data without manually standardizing data, and they used a dual-channel deep network as a matching network.
The introduction of GAN as a data set generator makes the registration result more accurate.
Ma et al. [109] 2019 Using CNN to obtain approximate spatial relationship, and then apply the matching strategy considering spatial relationship to the local feature-based method.
Since the spatial relationship is considered, the matching result is accurate and robust.
Dong et al. [111] 2019 A DescNet network model is designed to extract the depth features of the image, and replaced the maxpooling operation by increasing the stride size of the convolution filter.
The number of correct matching points it obtains is higher than others′, and is robust to large gray difference.
Kim et al. [121] 2019 An end-to-end deep neural network for estimating the transformation matrix between aerial images is proposed, which in turn estimates the rotation and affine transformation parameters between the images.
Orderly estimated network hierarchy reduces estimation errors and its accuracy is higher than A2net and CNNGeo.
Yang et al. [116] 2019 The VGG16 network was selected as the feature extractor, and chosen to gradually increase the inliers for registration.
Improved accuracy of results.
Park et al. [123] 2019 A three-input and bidirectional architecture is proposed to estimate the transformation matrix between the source image and the target image.
Compared to traditional methods such as SIFT and SURF, it has a higher PCK.
natural images are not suitable for the requirements of remote sensing image registration tasks, but neural networks can obtain more robust and high-level features. The CNN can automatically acquire more expressive features than statistical methods through learning [108] . Therefore, many literatures choose to directly transfer the picture to the neural network, obtain the image representation features through the network, and then use the obtained features in combination with traditional methods for registration tasks. Compared with traditional hand-craft features, it can effectively improve the accuracy of the registration task. Ma et al. [109] proposed a new two-step registration method which is based on deep and local features regions, this method includes a matching method based on CNN features and a spatial relationship remote sensing image registration method based on a point matching strategy. They selected the VGG16 [31] network model as the depth feature extractor, in order to eliminate the loss of location information which was caused by maximum pooling, they proposed the location adjustment strategy, and used a local feature-based method matching strategy to obtain the final correspondences. This method not only has a good effect on SAR map registration, but also can be applied to multimodal images. Ye et al. [38] chose VGG16 as the depth feature extractor, they merged the obtained depth feature and SIFT feature into a feature vector, and then integrated it into the PSO-SIFT [110] algorithm for remote sensing image registration. Compared with PSO-SIFT, this method improves the accuracy of registration. Dong et al. [111] proposed a network (DescNet) which can effectively process images with large gray difference. Compared with other networks, this network can obtain more corresponding feature points, this can solve the problem that the number of matching points is not large enough to cause deviations or even errors in the estimated transformation model [112,113] . They adopted strategy of aggressively mining the hardest negatives to solve the problem of imbalance in the number of training samples, and because the maxpooling operation reduces the performance of the final output feature descriptor [114] , they replaced the maxpooling operation by increasing the stride size of the convolution filter. Experimental results prove that Descnet has a higher inlier ratio than MatchNet [115] . Yang et al. [116] used VGG16 as the feature extractor, they didn't use the fixed outliers, but chose to gradually increase the inliers for registration. In the initial stage of registration, the most reliable feature points were used to quickly determine the rough matching, then the registration details were optimized by gradually increasing the number of feature points, and the wrong matching points were constrained according to the geometric local information.
The final experimental results were significantly improved compared to GLMDTPS (global and local mixture distance and thin plate spline) [117] , GLCATE (global-local correspondence and transformation estimation) [118] .

Using the network for matching
In the image registration task, the accuracy of matching features has a great impact on the entire registration task. Wang et al. [119] proposed a novel network structure to select the matching points, they used a traditional local feature extraction method to extract feature points, constructed the corresponding local regions around these features, and used DNN networks to match these patch pairs. The entire network framework is divided into two stages: mapping function learning and image registration. The mapping function learning phase mainly learns the mapping function by using unregistered images and their affine transformed images, and the registration phase uses the above-mentioned trained DNN to predict matching labels of patch pairs from the perceived image and the reference image. If the label is 1, the center point of the matched patch pair is used as the initial keypoint of image registration. They used local constraints and global constraints to weaken the matching error, and self-learning was introduced to solve the small data problem of remote sensing images used for training. The results show this method has high accuracy. Quan et al. [120] used GAN to automatically create more training data without manually standardizing data, this method can increase the accuracy of registration. They used a dual-channel deep network as a matching network. The two channels have the same structure but do not share parameters. This kind of network is helpful to extract the features of different models. They used CNN as a feature extractor to preserve the spatial information of the image. They used multiple constraints (correlation constraints, geometric constraints) when removing mismatches. Experimental results show that this method had a good performance in processing multimodal image registration tasks. 3.1.3 Using the network to estimate the transformation parameters Kim et al. [121] proposed an end-to-end transformation parameter estimation network to gradually estimate the rotation and affine transformation parameters between aerial image pairs and each parameter estimation stage includes: feature extraction, feature matching, gradual masking, and transformation estimation. In order to accurately estimate the transformation parameters, the gradual masking method is used to reduce the influence of irrelevant feature points. Experimental results show that this method has a high accuracy. Vakalopoulou et al. [122] proposed a CNN-based network for estimating the rigid and deformable parameters between high-resolution satellite image pairs, and then used a 2D space converter layer to align the source image with the target image. Park et al. [123] proposed an end-to-end scalable network with a two-stream architecture and a bidirectional training architecture for estimating the transformation parameters for aerial image pairs. They added a set of internally enhanced target image data, this can make the trained model robust to environmental changes. The bidirectional training architecture can make the result more accurate. This network can achieve better estimation results, but still have some weakness in detailed matching.

Application research of metric learning in remote sensing image registration
Metric learning refers to the use of neural networks to directly obtain similarity measures between the input pictures. Remote sensing image matching is the most important link in the process of remote sensing image registration. It often uses metric learning to convert image matching problems into similarity measurement problems between images. Most literatures used Siamese convolutional networks [124] and their variants for metric learning. Table 5 summarizes the methods of deep learning method applied to metric learning.
Hughes et al. [125] proposed a pseudo-Siamese network to solve the matching problem of optical remote sensing images and synthetic aperture radar images. The fully connected layer is used to fuse two features learned by the network branch. There is no parameter sharing between the two network streams, and only feature information was fused in the final decision stage. The final network output the 0, 1 as a match prediction for input image pairs. Zhang et al. [126] adopted the Siamese full convolutional network structure to match image pairs captured by multiple sensors. The structure of the network branch is similar to HardNet [127] and in order to ensure the spatial accuracy, they set the stride value as 1 on each convolutional layer. The parameters were shared between the two branches to solve the problem of few training data sets, the two network branches were combined through convolution operations. They used Harris' corner detector [128] to select local candidate image patches, then these patches are fed into the network, the response of the final score map indicates the similarity between the master patch and the slave sub-patch at each position of the large search patch, the point with the highest similarity can be regarded as a matched point. This model is simple, highly accurate, and suitable for processing matching images captured by multiple sensors. Merkle et al. [129] used the Siamese structure to solve the problem of matching optical-SAR remote sensing images, the feature vector corresponding to the optical image and the feature matrix corresponding to the SAR image are obtained through CNN. Then the dot product layer [130] calculates the similarity between the feature vector and each element in the feature matrix, and finally obtains a score map. The point with the highest value in the search space is used to obtain matching pairs. The experimental results show that this method has a good effect on the image matching which is obtained by multiple sensors. He et al. [131] designed a new framework for matching remote sensing images with complex backgrounds based on the Siamese network, which includes three steps: S-Harris corner detection, network training and patch matching. In this method, the grid S-Harris algorithm was used to determine the coordinates of the point matches, and a search strategy based on Gaussian pyramid coupling quadtree was used to narrow down the search space and compare multiscale conjugate pairs, the Siamese network was used to find similarities for the patch pairs and obtain the initial matches, finally the whole to local quadratic polynomial constraints and the random sample consensus algorithm [132] were used to remove false matches. This method can effectively avoid the impact of complex background changes on the matching results, but the accuracy of the matching model is affected by the unknown spatial resolution of the image.
Due to the unique characteristics of remote sensing images, such as few data sets and complex imaging mechanisms, traditional methods (such as SIFT, SURF [133] , ASIFT [134] ) have difficulty meeting the requirements of current remote sensing image registration tasks, deep learning has great advantages over traditional machine Merkle et al. [129] 2017 Based on the Siamese architecture, a network is proposed to match multisensor images of different sizes, and the dot product is used to calculate the similarity.
The network model is suitable for multisensor and multi-resolution images.
Hughes et al. [125] 2018 A pseudo-Siamese convolution network is designed to match SAR and optical remote sensing images.
The network model can accurately predict the matching image blocks.
He et al. [131] 2018 It proposed a matching network for complex background factors, and used Gaussian pyramid coupling quadtree to optimize the matching point search space, used RANSAC and the whole-to-local quadratic polynomial constraints to remove the wrong matching points.
It has a high matching accuracy and can effectively solve the problem of complex background factors which will affect the matching results.
Zhang et al. [126] 2019 Designing a matching neural network based on the Siamese network architecture for matching SAR and optical remote sensing images, and using the similarity score to remove the wrong matching points.
It has accurate matching results and a simple strategy for removing mismatched points.
learning methods. However, there are still some problems in how to use the deep learning method in remote sensing image registration. Therefore, this is still a valuable research direction in the future.

Conclusions
After decades of development, remote sensing image registration has been widely used in image fusion, change feature detection and other fields. Although many results have been achieved, it still faces many problems, such as registration of multi-source remote sensing images, image nonlinear registration, real-time registration of remote sensing images, etc. This article reviews the application of computational intelligence in remote sensing image registration, which is mainly divided into two sections: evolutionary algorithms and neural networks. In recent years, a large number of improved algorithms based on evolutionary algorithms and some novel network models have been proposed, and applied to remote sensing image registration. These methods have achieved great success, and many algorithms have been verified to be effective. With the continuous development of computational intelligence, there will be more novel remote sensing image registration methods.

Open Access
This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.