1 Introduction

Technologies ensuring the avoidance of potential hazards during landing are currently in development. Such technologies are proposed in the literature using camera [1–3], and Lidar [4–9]. In December 2013, the Chinese Chang’e 3 mission team claimed of having operated and demonstrated with success for the first time camera-based HDA from images taken in real time during the descent towards the Moon. HDA techniques aim at imaging the surface and extract local slope, local roughness and local illumination information. This information is then used to autonomously select landing locations that satisfy these constraints by minimizing some cost function combining these constraints. The safest location identified is then selected and the spacecraft trajectory is guided and controlled towards this updated target location. Upcoming planetary exploration missions to the Moon, Mars and small bodies will necessarily benefit from this capability.

The paper presents two different HDA configurations, one using passive (camera) sensors and another relying on an active (scanning Lidar) sensor. The objectives of this paper are to propose an HDA architecture which can deal with both passive and active HDA approaches, quantify the respective performance of passive and active HDA systems for a Moon landing application and conclude on the advantages and drawbacks of both approaches.

The paper presents the latest developments performed in the context of the ESA Lunar Lander Phase B1 program combined with internal development activities performed at NGC Aerospace Ltd.

2 HDA system architecture

The HDA system architecture aims at being generic by design such that it can work with the availability of either Camera images and/or Lidar scans during the landing trajectory. As shown in Fig. 1, raw Lidar scans need to be compensated to remove any distortions caused by the motion of the sensor during the scanning process (which can take up to 5 s). Each range measurement is then referenced on the ground based on the navigation states provided by the Lander navigation subsystem during the scan. The ground referenced Lidar data is then resampled and interpolated to obtain a uniform and complete topographical map of the surface at the desired spatial resolution. This Lidar topographical map is then fed to the hazard map computation functions, discussed in detail later in this section.

Fig. 1
figure 1

HDA system architecture

Full size image

A similar process is also applied to the raw camera images to the exception of a priori and a posteriori motion compensation that is not needed for the camera since the image is taken within a very short duration. The effects of motion on the camera image are considered negligible enough to avoid the need for compensation. The camera image is then ground-referenced. For that matter, the desired area at the desired spatial resolution is projected in the camera image. This process results in a ground-referenced image sampled at the desired spatial resolution. This image is fed to the hazard map computation functions discussed next.

The hazard map computation functions are regrouped together in the HDA system architecture and each function is enabled or not depending on the availability of the Lidar and/or camera images during the landing trajectory. The functions included are the following:

  1. 1.

    Shadow hazard map computation (camera only)

  2. 2.

    Topographical hazard map computation.

  3. 3.

    Fuel and manoeuvrability hazard map computation.

  4. 4.

    Precision hazard map computation.

  5. 5.

    Hazard maps fusion.

  6. 6.

    Hazard maps neighboring.

  7. 7.

    Safe site selection.

Each of these functions is described in detail in Table 1.

Table 1 Hazard maps computation functions
Full size table

The reference mission scenario is based on the ESA Lunar Lander phase B1 activities. The HDA scenario includes two scan sequences of the surface, one at high altitude covering a large area (400 × 400 m) at a spatial resolution of 2.5 m for slope and boulders detection purpose and guiding a first trajectory divert toward the safest area. Then, a second scan is performed after the completion of the first divert (covering an area of 100 × 100 m) at a finer spatial resolution of 0.25 m for slope and roughness detection purpose, guiding a second trajectory divert. The overall landing scenario is illustrated in Fig. 2 while the HDA scenario is shown in detail in Fig. 3.

Fig. 2
figure 2

Lunar landing trajectory scenario from lunar orbit to touchdown

Full size image
Fig. 3
figure 3

Hazard detection and avoidance trajectory scenario

Full size image

The time allocated to acquire the sensor data during the scan sequence and to process the measurements is 10 s per scan. For the Lidar, 5 s is allocated for the scanning of the surface (limited by the sensor scanning mechanism capability) and 5 s for the processing. For the camera, the image acquisition is nearly instantaneous and therefore 10 s is allocated for the image processing. At the end of this 10 s sequence, the coordinates of the safest landing site is provided by the HDA function and fed to the guidance and control functions commanding the divert manoeuvre toward that site. The same process is repeated for the second scan sequence. Each scan sequence involves one Lidar scan of the surface and one camera image.

2.1 Passive HDA option

The passive HDA option represents the minimum cost and weight option since in this case no Lidar is implemented on the spacecraft and the camera with a field of view of 60° is responsible for the detection of all the topographical hazards in combination with the fuel/manoeuvrability and precision hazard maps. Such passive HDA technique is dependent on the lighting conditions, therefore the landing mission must be planned to ensure that the landing manoeuvres will be achieved under favorable lighting conditions. It is desired to have Sun elevation between 5° and 25° to create significant shadows and textures in the images.

The passive HDA option can also be subdivided into two options: (1) the simplest one in terms of processing load based on shadow and texture detection and (2) a more complex one using as well an algorithm to extract 3D information from the camera image to compute slope and roughness information. This second option aims at providing additional information from the camera that is not detectable by using only texture detection. Texture is defined as local variations in intensity of neighboring pixels in the image (derived from 2D information of the surface) while roughness is defined as the difference in height of surface elements relative to the surface mean plane (derived from 3D information of the surface). Actually, surface slopes do not necessarily create textures in camera image and therefore more advanced algorithms are needed to extract this information. The remaining part of this section summarises the passive HDA algorithms that were developed.

2.1.1 Shadow detection

The shadow map computation process identifies dark portions of the ground-referenced image and marks them as unsafe for landing. A value of 1 is assigned to dark areas while a value of 0 is assigned to other areas. The output is binary (i.e., 0 or 1). This way, dark areas are excluded from the safe landing site selection process, but the remaining areas, safe areas, are not favoured one compared to another. It is assumed that a brighter pixel (which could be due to surface orientation with respect to Sun angle or higher surface reflectance) is not safer for landing. This process is a straightforward thresholding process which sets to 1 pixels below the pre-set brightness threshold, i.e., if the pixel intensity I (p, q) is below the parameter I min. The value I min is selected such that it is just above the noise level of the sensor (i.e., perceived brightness noise when exposed to dark area).

2.1.2 Texture detection

The texture detection algorithm computes a hazard map based on the variance of the pixel intensity computed on a subsection of the ground-referenced image. Therefore, the algorithm needs to extract a smaller subsection from the ground-referenced image, to compute the median (which is necessary to compute the variance) and to compute the variance of the ground-referenced image subsection. The actual classification between hazardous (value of 1) and safe (value of 0) is achieved by comparing the variance of the pixel against a user-defined threshold. If the pixel variance is more than the selected threshold, the site is labelled unsafe based on its texture.

2.1.3 Camera-based slope and roughness detection (option 2 only)

Camera-based slope and roughness detection is performed using two steps: (1) the computation of the digital elevation model (DEM) of the surface extracted from the camera image contents and (2) the computation of the slope and roughness information from that DEM.

The DEM is computed from the image using a shape from shading line integration algorithm [10]. The line integration approach only works if the direction of integration coincides with the direction of incoming sunlight. Thus, the image has to be rotated by the magnitude of the azimuth. To maintain a square image, which size is independent of the magnitude of the rotation, the rotated image will be saved in an image of √2 times the image size. As a result, the final, new, image will appear as shown in Fig. 4b, where the red part is the camera image and the blue part contains only empty pixels.

Fig. 4
figure 4

Line integration algorithm. a Line integration algorithm flowchart. b Rotated image, arrow indicating direction of incoming sunlight, black lines are smoothing lines

Full size image

Two line integrations are performed on the image, one integrating from the bottom up and one integrating from the top down. The principle assumes that the gray level gradient between two pixels is proportional to slope. The numerical integration of slope estimates the height. This approach was selected since the algorithm performs better at the beginning of an integration line. In addition, the integration is not performed row wise (thus line by line) but column wise. By choosing this technique the results per column can be filtered along the column after every integration step. This reduces the influence of flawed values (e.g., at very steep slopes and rims) as they are not propagated in their full extend but already slightly smoothened before further usage. The smoothing is done by an averaging filter. This process is depicted in Fig. 5. After the entire DEM is computed (also called herein the Z-map), the results are rescaled and the two resulting Z-maps are mixed. The input for this block is the ground-referenced image and the output is a DEM in the camera frame. The flowchart of the line integration algorithm is given in Fig. 4a.

Fig. 5
figure 5

Line integration algorithm

Full size image

To compute the slope at a given image position (a given pixel), a surface mean plane fitting is performed on a subset of the DEM around that pixel. This subset is typically of the size of the landing area to be assessed by the algorithm. The same process is necessary to compute the roughness. Thus, to save computation time, both algorithms are combined and are using the outcome of the same surface plane fitting. The slope is obtained from the angle between the normal vector of that plane and the local gravity direction. The roughness is computed by calculating the standard deviation of the distance from all points in the subset to the plane. Afterwards, the values for slope and roughness are compared against a tolerance and in case the threshold is exceeded the pixel is marked as unsafe. The outputs of this function are the resulting camera-based slope and roughness hazard maps.

Example of results obtained with a prototype algorithm is shown in Fig. 6 with a Sun elevation angle of 16°. These results compare the true DEM (meters), slope (degrees) and roughness (meters) against the ones extracted from the algorithm. One can observe that the reconstructed DEM (meters), slope (degrees) and roughness (meters) have many similarities with the truth without, however, providing the same level of accuracy. Note that only shapes casting shadows on the surface can be reconstructed with a satisfying level of accuracy.

Fig. 6
figure 6

True DEM, slope and roughness versus computed DEM, slope and roughness

Full size image

The reader shall note that the roughness detection from an estimated DEM resulting from line integration is not the best approach to detect high-frequency features such as boulders and craters. This is due to the numerous smoothing filters applied during the line integration algorithm. From analyses not reported in this paper, it has been shown that the texture detection from the ground referenced image proved to provide a more reliable assessment of surface roughness (even though it is an indirect observation). Therefore, the computed roughness hazard map is not used in the fusion algorithm. This is later referred to as giving the computed roughness hazard map a weight of 0.

2.2 Active HDA option

The active HDA option aims by design at augmenting the robustness and the accuracy of the HDA system by using an active sensor (a Lidar) in addition to the passive sensor (the camera). The Lidar sensor considered is a scanning Lidar providing range measurements at 100 kHz and covering a maximum field of view of 40° × 40°. It takes around 5 s for the mirror mechanism to cover the whole field of view using a raster scan pattern. This active HDA option augments the hazard maps computation functions with: (1) Lidar-based slope detection and (2) Lidar-based roughness detection. Both functions are summarised below.

2.2.1 Lidar-based Slope and Roughness Detection

The process involved to compute the slope hazard map from the Lidar topographical map is similar to the one used on the DEM extracted from the camera image and presented earlier. From the Lidar topographical map, the surface mean plane (SMP) is locally computed using sub-windows of calculation moved systematically over the whole Lidar frame. The local slope is obtained by computing the angle between the normal unit vector of the sub-window SMP and the estimated local gravity vector. Local roughness is computed by subtracting the sub-window SMP from the height of the elements of surface located in the sub-window. At the end of this process, one slope hazard map and one roughness hazard map are obtained where values around 0 mean safe areas and values around 1 are unsafe for landing.

3 Simulations

As discussed in the previous sections, three system-level options are considered for the implementation of the HDA system. These three options are:

  1. 1.

    Passive HDA with shadow and texture detection.

  2. 2.

    Passive HDA with shadow, texture and slope detection.

  3. 3.

    Active HDA with shadow, slope and roughness detection.

Each of the three options has been validated by software simulations on two different surface models, a first model that is moderately hazardous and a second model that is more hazardous and therefore, more challenging in terms of safe landing. The objectives of validating the system over these two models are to evaluate the performance and limitations of the algorithms under different types of topography that can be encountered during real missions. The simulation environment is based on MATLAB/SIMULINK. Further details about the simulation environment are provided next.

3.1 Simulation environment

The MATLAB/SIMULINK simulation environment contains the following components as described in Table 2.

Table 2 Simulation environment components description
Full size table

This simulation environment is in closed loop and can be used to validate the performance of all the functions of the GNC and HDA system. The validation was achieved against mission performance requirements and the main requirements related to HDA performance are summarised in Table 3.

Table 3 HDA system performance requirements
Full size table

The validation plan is presented in Table 4 and, for each test case, 500 Monte Carlo simulations were executed.

Table 4 HDA system validation plan
Full size table

The dispersed parameters considered during the Monte Carlo tests included realistic spacecraft mass, centre-of-mass and inertia, actuators thrust magnitude uncertainties, actuator misalignments, sensor noise, sensor misalignments, initial conditions uncertainties and initial navigation state uncertainties. To minimise simulation time, the landing trajectories were executed from the end of the descent orbit phase, shortly before the braking phase, until touchdown representing a true duration of approximately 1,000 s. The initial conditions and navigation state uncertainties defined at the end of the descent orbit were defined based on dedicated analyses performed by executing several descent orbit trajectories and by deriving the dispersion at the end of the descent orbit.

To validate the performance of the HDA system, it is required to verify the touchdown landing positions against the topographical ground truth of the targeted landing area in terms of slope, roughness and shadow. Figures 7 and 8 present the surface ground truths for each of the surface model.

As shown, the surface model #1 is smoother than surface model #2 with significantly less boulders as shown by comparing the slope and roughness ground truths between the two models. The surface model #1 is a large boulder casting a large shadow on one of its side while surface model #2 is more or less an inclined plane with smaller rocks and craters on it. Concerning the shadow, the surface model #1 has a large fraction of its surface that is shadowed and the shadow map is smooth. For surface model #2 the shadow map is composed of small shadowed regions distributed on the surface similarly to the roughness and slope ground truths. The surface model #2 is essentially composed of surface elements at higher frequencies compared to surface model #1.

Fig. 7
figure 7

Surface model #1

Full size image
Fig. 8
figure 8

Surface model #2

Full size image

3.2 Nominal results: passive HDA

The nominal results when the passive HDA (option 2) is executed over the surface model #1 are shown next. Figure 9 shows the passive HDA fused hazard map result. Dark red areas (cost values of 1) indicate landing sites that are unsafe for landing while dark blue areas (cost values of 0) indicate landing sites that are very safe. The contribution of each hazard (cost) map is then shown in Fig. 10. Based on experiments, the reader shall note that roughness is detected using only the texture hazard map which explains the weight of 0 allocated to the roughness hazard map (wRoughness = 0 on figures). Figures 11 and 12 present similar information but this time during the scan #2 sequence. Strips can be seen in the fused hazard map of scan #2. These are artefacts caused by the line integration algorithm used for slope detection.

Fig. 9
figure 9

Fused hazard map over surface model #1—scan #1

Full size image
Fig. 10
figure 10

Passive HDA hazard maps over surface model #1—scan#1

Full size image
Fig. 11
figure 11

Fused hazard map over surface model #1—scan #2

Full size image
Fig. 12
figure 12

Passive HDA hazard maps over surface model #1—scan #2

Full size image

The nominal results when the passive HDA is executed over the surface model #2 are shown next (Figs. 13, 14, 15, 16).

Fig. 13
figure 13

Fused hazard map over surface model #2—scan #1

Full size image
Fig. 14
figure 14

Passive HDA hazard maps over surface model #2—scan #1

Full size image
Fig. 15
figure 15

Fused hazard map over surface model #2—Scan #2

Full size image
Fig. 16
figure 16

Passive HDA hazard maps over surface model #2—scan #2

Full size image

The execution time profiling of the passive HDA functions is presented in Fig. 17. The profiling results are normalised over the execution time of the most demanding function. The results are used to identify the functions that are the most demanding.

Fig. 17
figure 17

Passive HDA execution time profiling

Full size image

The computer used for the Windows based profiling activity has the following properties (as measured using Dhrystone benchmark version 1.1 via C/C++):

  • Intel Core i5-2400 CPU @ 3.10 GHz

  • 8.00 GB RAM

  • Windows 7 Professional SP1 64 bits

  • 13063 MIPS (Dhrystone benchmark)

As shown in Fig. 12, the most demanding algorithms of the passive HDA option 1 and option 2 are the slope and roughness detection and the texture detection algorithms. Therefore, optimisation efforts should be focused on these functions. The profiling activities have also demonstrated that the passive HDA option 1 and option 2 meet the requirement of 10,000 ms when scaled to the target processor (refer to Table 3).

3.3 Nominal results: active HDA

The nominal results when the active HDA is executed over the surface model #1 are shown next. Figure 18 shows the active HDA resulting fused hazard map. Dark red areas (cost values of 1) indicate landing sites that are unsafe for landing while dark blue areas (cost values of 0) indicate landing sites that are very safe. The contribution of each hazard (cost) map is then shown in Fig. 19. In comparison to the passive option, the camera texture is no longer used (wTexture = 0) while the Lidar slope and roughness are included in the fusion process. Lidar roughness hazard maps are more reliable than camera texture hazard maps for the detection of local roughness hazards. One possible option is to also include camera texture hazard map in the fusion process and attribute it a lower weight compared to Lidar roughness hazard map The camera texture hazard maps remain available as a redundant option in case of Lidar failure. Figures 20 and 21 present similar information but this time during the scan #2 sequence. It shall be noted that passive and active solutions focus on different parts of the region of interest for scan #2, such that Figs. 10 and 17 cannot be directly compared with Figs. 15 and 18.

Fig. 18
figure 18

Fused hazard map over surface model #1—scan #1

Full size image
Fig. 19
figure 19

Active HDA hazard Maps over surface model #1—scan #1

Full size image
Fig. 20
figure 20

Fused hazard map over surface model #1—scan #2

Full size image
Fig. 21
figure 21

Active HDA hazard maps over surface model #1—scan #2

Full size image

The nominal results when the active HDA is executed over the surface model #2 are shown next (Figs. 22, 23, 24, 25).

Fig. 22
figure 22

Fused hazard map over surface model #2—scan #1

Full size image
Fig. 23
figure 23

Active HDA Hazard Maps over Surface Model #2—scan #1

Full size image
Fig. 24
figure 24

Fused Hazard Map over Surface Model #2—scan #2

Full size image
Fig. 25
figure 25

Active HDA hazard maps over surface model #2—scan #2

Full size image

The execution time profiling of the active HDA functions is presented in Fig. 26. Similarly as presented before for the passive HDA (end of section 3.2), the profiling results are normalised over the execution time of the most demanding function. As shown, the slope and roughness detection and the ground referencing algorithms are the most demanding algorithms for the active HDA and therefore optimisation efforts shall focus on them. The profiling activities have also demonstrated that the active HDA option meet the requirement of 10,000 ms when scaled to the target processor (refer to Table 3).

Fig. 26
figure 26

Active HDA execution time profiling

Full size image

3.4 Monte Carlo simulation results

To assess the robustness and performance of each HDA design option, Monte Carlo simulations were performed using the 6 test cases presented earlier in Table 4. The overall results are shown in Table 5 and then for each test case, the landing site locations over the ground truths and the landing site characteristics are shown next (Figs. 27, 28, 29, 30, 31, 32).

Table 5 Monte Carlo overall results
Full size table
Fig. 27
figure 27

Landing sites positions and characteristics of passive HDA option 1 over surface model #1

Full size image
Fig. 28
figure 28

Landing sites positions and characteristics of passive HDA option 1 over surface model #2

Full size image
Fig. 29
figure 29

Landing sites positions and characteristics of passive HDA option 2 over surface model #1

Full size image
Fig. 30
figure 30

Landing sites positions and characteristics of passive HDA Option 2 over surface model #2

Full size image
Fig. 31
figure 31

Landing sites positions and characteristics of active HDA over surface model #1

Full size image
Fig. 32
figure 32

Landing sites positions and characteristics of active HDA over surface model #2

Full size image

4 Analysis

The analysis of the Monte Carlo results demonstrates that:

  1. 1.

    Passive HDA slope detection does not improve the performance for the surfaces under test.

  2. 2.

    Passive HDA texture detection and shadow detection provide good performance on surfaces with limited slope and roughness.

  3. 3.

    Active HDA is necessary to reach robust performance over rough and challenging surfaces.

Passive HDA slope detection does not improve the performance for the surfaces under test Table 5 shows that the percentage of unsafe landings is higher using the passive HDA option 2 compared to option 1. As mentioned in section 2.1, this can be explained by the incapability of the passive HDA option 2 algorithm to reconstruct with accuracy the DEM from the ground referenced images. Moreover, in Section 2.1.3, Fig. 6 shows an example of results on a relatively easy-to-reconstruct surface. The surface is smooth with mostly big craters causing most of the shadows and very little small boulders. The surfaces used in the Monte Carlo simulations (Figs. 7 and 8), are harder to reconstruct, especially surface model #2 which has a significantly higher density of boulders and rocks.

Passive HDA texture detection and shadow detection provide good performance on surfaces with limited slope and roughness As seen in Table 5, the passive HDA option 1 with shadow and texture detection has 0 % of unsafe landings with the moderately hazardous model (surface model #1) and only 4.42 % with the more hazardous model (surface model #2). From that 4.42 % of unsafe landings, the majority (3.82 % out of 4.42 %) are caused by slope detection inaccuracy. As expected from the theory, slopes do not create necessarily texture in camera images and therefore their detection through texture cannot be guaranteed. The Monte Carlo results consolidate this expectation while at the same time demonstrate that texture detection represents a simple and efficient way to identify many and the majority of the topographical hazards on a surface.

Active HDA is necessary to reach robust performance over rough and challenging surfaces While Table 5 shows that all the HDA systems has less than 1 % of unsafe landings for the surface model #1, only the active HDA option reaches less than 1 % of unsafe landings for both surface models. Therefore, the active HDA is the only HDA system meeting the requirements of Table 3 in terms of safe landing probability. Moreover, the active HDA has the lowest average of slope, roughness and the highest average distance to hazard at landing.

5 Conclusion

This paper focused on the comparison of passive and active HDA systems for planetary landing missions. It has presented a unique HDA system architecture enabling the integration of passive (camera) and active (Lidar) sensors, the processing of the measurements provided by these sensors, the detection of hazards and the fusion of the so-called hazard maps to identify the safest landing locations. The comparison of the system performance was performed by software simulations; therefore this paper has presented the simulation environment that was used and has described in details the simulation results. The robustness of each system option has been assessed and compared using Monte Carlo simulations performed over two types of surface containing different hazards density and characteristics.

Based on the analysis of the results, the paper has demonstrated that passive HDA slope detection does not improve the performance for the surfaces under test. Other passive HDA slope detection techniques, different from shape from shading using line integration, may provide better performance and such techniques would need to be investigated during follow-on activities. This paper also demonstrated that passive HDA texture detection and shadow detection provide adequate performance, but only for surfaces with limited slope. Finally, this paper has proven that active HDA is the more robust option and is necessary to meet safe landing requirements over rough and challenging surfaces.

Regarding the applicability of HDA systems to planetary exploration missions, the use of active HDA system is highly recommended for any missions including the ones targeting hazardous or unknown landing areas. Passive HDA system can be envisaged as a potential solution for missions targeting known and moderately hazardous landing areas or as a backup solution in case of active HDA sensor failure.

Potential follow-on activities to this paper will be to consolidate the results using laboratory and full-scale hardware-in-the-loop experiments involving sensor hardware, embedded processing units and representative surface models. Such activities would build on past experience in this area presented in previous papers [7–9].