Spacecraft Formation Keeping and Reconfiguration Using Optimal Visual Servoing

Abstract

This paper proposes a direct visual servoing system for spacecraft guidance in formation flying scenarios. The proposed image-based visual servoing system uses image information for planning and executing formation acquisition, reconfiguration, and maintenance maneuvers. The system assumes that LEDs are located at specific points on the satellites, enabling the visual servoing controller to rely on continuous tracking of these features in the camera’s image plane. Analytical developments demonstrate that the proposed optimal visual control system is stable and optimal, and it acts on both the orbital and attitude dynamics of the spacecraft, considering circular and elliptical reference orbits. The distributed image-based controller defines a cost function that minimizes control efforts, and the paper proses an optimal framework for developing controllers that address the issue. A ROS-based simulation tool was used to test the proposed visual servoing controller in a realistic small-sat formation flying scenario. Results indicate that the proposed distributed control strategy is viable and robust against environmental perturbations and disturbances in sensing and actuation.

1 Introduction

The recent growth of the New Space economy, with thousands of new start-ups and companies proposing new space applications and services, is laying the business foundations for innovative typologies of high-risk-high-reward missions [1]. Some of these applications require collaboration and coordination among multiple spacecraft, often involving complex tasks that necessitate high levels of autonomy, reliability, efficiency, and safety. Payloads and mission specifications often require that spacecraft flight in a given formation maintaining a fixed distance between them [2]. To achieve this, the relative position and attitude between each spacecraft must be maintained with a high degree of accuracy over extended periods.

An overview of spacecraft formation flight control strategies can be found in [3]. Several formation control techniques have been proposed, including virtual structure methods [4] and leader–follower strategies [5]. Different model-based control strategies have been used to achieve the satellite formation flying task. The conventional approach for formation flying scenarios involves the use of Global Navigation Satellite System (GNSS) receivers or radio communications [6]. However, using onboard cameras and visual guidance systems is an attractive option as it is low cost and provides visual information of the observed scenario [7], and in some cases, the entire relative pose of the observed spacecraft [8]. The Technology Readiness Level (TRL) of such cameras onboard space systems [9,10,11] allows for the development and implementation of visual guidance strategies using zooming cameras or even real-time computation of the pose of an observed spacecraft [12, 13]. On the other hand, the still limited computing capabilities of onboard computers require the simplification of the implemented controllers and methods [14]. A leader–follower vision-based architecture was proposed in [15], addressing the problem of having visibility limitations. Image information is also used in [16] in the control of a formation of multiple spacecraft throughout a visual servoing approach that also includes coordination of the attitude of the platforms. In [17], an integrated navigation strategy is described, which makes it possible to reliably and accurately determine the orientation of the main aircraft while operating in conjunction with other unmanned aerial vehicle, forming a synchronized formation. The work presented in [18] describes relationships between the motion of a spacecraft formation, visualization and image processing problems due to the relative motion of objects within the camera field of view. In such a work, the use of cameras is the primary sensor for controlling the relative position between spacecraft, which is also the case under analysis in this paper. However, this paper proposes an optimal direct image-based visual servoing system for spacecraft guidance in formation flying scenarios. This approach presents several innovative contributions: first, the proposed image-based approach allows for performing formation keeping and reconfiguration manoeuvres without the need of any pose estimation algorithm within the pipeline. Moreover, this is done by implementing a direct visual servoing optimal framework with the aim of minimizing energy and fuel consumption during spacecraft guidance and increasing the guidance robustness with respect to other indirect approaches.

The visual servoing systems are commonly classified into image-based controllers and position-based controllers, as described in [19]. While position-based algorithms reconstruct the position and orientation of the spacecraft through computer vision techniques and then control them to achieve desired configurations in the cartesian space, image-based controllers track trajectories and target points directly in the image plane. The presented research introduces a novel image-based approach to guide spacecraft during formation keeping and reconfiguration maneuvers. This methodology eliminates the need for intermediate steps involving 3D reconstruction of relative positions and attitudes of spacecraft, allowing for the direct design of guidance laws in the image plane. This approach doesn’t require a geometric model of the target, which in turn makes it less sensitive to calibration errors and noise [19, 20]. However, it is a prerequisite that the detection and tracking of spacecraft-specific features are always possible in the image plane.

Standard image processing algorithms face numerous difficulties in the orbital environment as a result of substantial changes in various characteristics within a particular image scene. These changes encompass modifications in lighting conditions, leading to diverse levels of luminosity and distinct shadows, which arise when capturing images of the identical scene at different moments. An analysis of vision-based approaches for the specific case of proximity operations can be found in [21]. In [22] a methodology for successfully perform rendezvous maneuvers using purely visual information for feedback is presented. Several studies have focused on spacecraft visual navigation through the utilization of 3D model-based tracking techniques, employing either real space imagery [23, 24] or simulated space laboratory replicas [25]. The findings from these studies demonstrate the effectiveness of computer vision methods in the space environment. In this paper, to simplify the detection process we assume that LEDs are located at specific points on the satellites, enabling the visual servoing controller to rely on continuous tracking of these features in the camera’s image plane. Although this paper does not deal with image processing issues, it is worth mentioning that numerous research endeavours in computer vision have investigated eliminating shadows from images [26], [27]. These approaches would hold significant value in the space environment. Additionally, the operational range of visual servoing can be greatly constrained by a camera equipped with a non-adjustable focal length. This limitation arises as certain feature points of the target object may fall outside the field of view when the camera is in close proximity, and the accuracy of feature extraction may be compromised when the camera is positioned far away from the target. A viable solution to address this issue involves employing a zooming camera. In previous works such as the one in [28], a concurrent IBVS controller that is invariant to camera-intrinsic parameters, i.e., a zooming camera, is utilized for spacecraft uncooperative rendezvous maneuvers. These approaches can increase the robustness of the image features extraction process.

The use of visual servoing controllers can be also categorized as direct or indirect. Indirect visual servoing controllers specify the control action in terms of velocities to be applied at the spacecraft and do not take any dynamics into account. In this case, an internal spacecraft controller translates these inputs into torques and forces to be applied to the spacecraft performing the maneuvers. In contrast, direct controllers generate the forces and torques required for the maneuvers without the need for an internal controller. This approach leverages the relative dynamics between spacecraft to create a more resilient visual servoing system Hardware in the loop [29] and ground verification systems [30, 31] are proposed to experimentally verify and test the reliability of visual servoing control systems and path planning of spacecraft. These investigations indicate that a direct approach should be implemented in visual servoing schemes to account for the system dynamics.

The use of a direct image-based visual servoing approach to guide a free-floating manipulator employing an eye-in-hand camera system is described in [32]. The authors previously presented the application of direct visual servoing to solve problems involving robot manipulators [28], spacecraft rendezvous [33], and guidance of multiple spacecraft [34]. However, the visual servoing approach used in this paper is distinct from the prior works. A direct image-based visual servoing strategy is proposed in this study as a feasible solution to simplify and increase the robustness of the Guidance, Navigation and Control (GNC) of visual servoing algorithms for spacecraft formation flying. To accomplish formation keeping and reconfiguration maneuvers, visual feedback is incorporated in the control action to estimate the forces and torques necessary, taking into consideration the orbital and attitude dynamics.

Additionally, the proposed distributed visual servoing system relies on optimal image-based control. Several approaches in the literature rely on datasets obtained from optimal control applied to the formation scenarios. For example, an approach that includes reinforcement learning was proposed in [35] where a neural network was trained based on a dataset of optimal maneuvres for the specific scenario of formation control with collision avoidance. A policy-based reinforcement learning approach was applied to a decentralized formation flying controller in [36]. Reinforcement learning theory is also proposed in [37] to implement an adaptive optimal controller to achieve the desired satellite formation flying. Other approaches apply directly optimal control to the spacecraft formation flying scenario: a convex optimal control problem is proposed in [38] for multi-satellite formation flying. In [39], an optimal control approach is applied to a tethered space system. LQ optimal control theory was proposed in [40] for the orbital control of multiple satellites. The optimal framework, which was initially proposed for guiding robot manipulators in [41], is adopted in this paper. Such a framework is extended to develop new direct and distributed controllers for spacecraft guidance. Visual feedback is used to determine the necessary forces and torques for spacecraft guidance while taking into account the orbital and attitude dynamics. The proposed visual controller is defined within the optimal control framework, which minimizes energy and fuel consumption during spacecraft guidance. As a result, formation keeping and reconfiguration are conducted in the image space, rendering the reconstruction of the spacecraft’s 3D pose in the formation unnecessary. The control action is directly defined in the image space, which enhances the system’s robustness in contrast to position-based methods. The presented results show the viability of the proposed approach and the robustness against orbital environmental perturbations and disturbances in the actuation and sensing.

Different research groups within the space community have developed multiple spacecraft simulation resources. Ad-hoc libraries and simulation tools are accessible openly on the internet, like SpaceDyn [42], and others have been transformed into commercial products, such as DCAP [43]. A great number of research works have developed ad-hoc simulation tools to analyse spacecraft maneuvers performed in orbit. Among all, it is worth mentioning models that are computationally efficient and accurate, such as the one proposed [44], which is used for simulating the relative motion between two spacecraft in arbitrarily eccentric orbits subjected to orbital perturbations. An estimation architecture for autonomous relative multisatellite navigation is proposed in [45]. New approaches to identifying state transition matrices for spacecraft relative motion are presented in [46] and [47].

On the other hand, numerous robotic testbeds have been proposed to validate navigation and control algorithms. A relatively new example of such facilities is represented by the TRON testbed at Stanford University [48], which is specifically dedicated to simulating rendezvous and relative optical navigation scenarios. Other facilities can simulate formation flying scenarios. This is the case of the formation flying testbed at NASA [49], the hardware-in-the-loop testbed proposed in [50], that integrates orbit and attitude for autonomous satellite formation flying in South Korea, the GNSS navigation testbed for distributed space systems developed by Stanford [51], the GPS/GNSS hardware-in-the-loop testbed proposed for spacecraft formation flight simulation at Virginia Institute of Technology [52], or the software-in-the-loop formation flying testbed proposed in [53]. Nevertheless, these research works use simulation tools developed to perform a specific simulation, lacking standardization, open-source availability, and rigorous validation. This absence of standardization and accessibility obstructs the progress and application of adaptable methodologies and algorithms intended for spacecraft. Keeping in mind this panorama, a simulation framework based on the Robotic Operating System (ROS) is proposed in this paper. ROS is an open-source meta-operative system utilized for developing robot applications, which can be combined with Gazebo, a tool for simulating populations of robots in customized environments. Such tools are modified to include and replicate the primary environmental conditions that spacecraft may encounter in orbit. The proposed simulation system is available for download in.¹ This solution allows for the simulation of complex space robotic systems and, at the same time, takes advantage of the number of packages already developed in ROS for control, vision, teleoperation, and modelling tools. Although Matlab/Simulink could be the main simulation tool in previous spacecraft formation simulation systems, ROS is increasingly considered a suitable option for space projects (e.g. [54, 55]). In addition, Matlab and ROS could be used in conjunction to achieve better results.

The remaining part of the paper is divided into the following sections. Section 2 describes the spacecraft formation flying scenario, the considered coordinate frames and the main equations of motion. Section 3 describes the ROS-based simulation system proposed for the simulation of the spacecraft formation. Section 4 describes the image processing issues, and equations that define the relationship between the time derivatives of the extracted image features and the corresponding motion of the spacecraft. The optimal visual servoing approach is defined in Sect. 5. Section 6 presents the simulation results obtained in the formation flying scenario. Formation keeping and reconfiguration maneuvers are considered in the results section. Finally, concluding remarks and future areas of development are presented in Sect. 7.

2 Spacecraft Formation Equations of Motion

The spacecraft formation flying scenario is represented in Fig. 1. The spacecraft orbiting formation is composed by $N$ orbiting spacecraft ($N=4$ in Fig. 1). Two main coordinate frames are indicated in Fig. 1: Local Vertical Local Horizontal (LVLH) reference frame and Earth Centred Inertial (ECI) reference frame. The absolute positions and velocities of the spacecraft are denoted by ${{}^{I}{\varvec{r}}}_{i}$ and ${{}^{I}{\varvec{v}}}_{i}$, with $i=1\dots N$. Both positions and velocities are defined in the ECI reference frame that is represented as $\left\{I\right\}$ in Fig. 1. Each spacecraft is equipped with a camera which extracts a set of η visual features from another satellite. LEDs located on each satellite are used to facilitate the visual features extraction as it will be described in Sect. 4. Each spacecraft will track an image trajectory defined with respect to a target spacecraft from which a set η visual features points can be extracted. The presented controller can be easily extended to employ other kinds of visual primitives to perform the guidance (only the interaction matrix employed throughout the paper depends on the considered primitive [19]). Therefore, we assume that the target visual features are observable in the initial configuration of the spacecraft formation.

Fig. 1

Formation flying coordinate frames. ECI and LVLH reference frames

This paper assumes that the inter-distances among the spacecrafts, ${d}_{ij}=\left|{{\varvec{r}}}_{i}-{{\varvec{r}}}_{j}\right|$, are reasonably small if compared to the distances of each spacecraft from the Earth’s centre (${\text{max}}\left({d}_{ij}\right)<{\text{min}}({r}_{i})$, with $i,j=1\dots N$). The origin of the LVLH frame, $\left\{L\right\}$, with respect the Inertial frame is initially in the centre of the formation given by:

$${{}^{I}{\varvec{r}}}_{L}=\frac{\sum_{i=1}^{N}{{}^{I}{\varvec{r}}}_{i}}{N}$$

(1)

with the unit vector ${{}^{I}\widehat{{\varvec{x}}}}_{L}$ aligned along the radial direction, ${{}^{I}\widehat{{\varvec{z}}}}_{L}$ along the orbit normal and ${{}^{I}\widehat{{\varvec{y}}}}_{L}$ along the in-track direction.

The position of the i-th spacecraft with respect to the LVLH coordinate frame {L}, can be represented as:

$${{}^{L}{\varvec{d}}}_{i}={{}^{L}{\varvec{R}}}_{I}\left({{}^{I}{\varvec{r}}}_{i}-{{}^{I}{\varvec{r}}}_{L}\right)$$

(2)

where ${{}^{L}{\varvec{R}}}_{I}$ is the rotation matrix between the frames $\left\{I\right\}$ and {L}, defined as in [33], and ${{}^{I}{\varvec{r}}}_{L}$ the relative position between $\left\{I\right\}$ and {L} indicated in Eq. (1).

The relative velocity of the i-th spacecraft, also with respect the frame {L} can be defined as:

$${{}^{L}\dot{{\varvec{d}}}}_{i}={{}^{L}{\varvec{R}}}_{I}\left({{}^{I}{\varvec{v}}}_{i}-{{}^{I}{\varvec{v}}}_{L}\right)-{{}^{L}{\varvec{n}}}_{L}\times {{}^{L}{\varvec{d}}}_{i}$$

(3)

where ${{}^{L}{\varvec{n}}}_{L}={n}_{L}{\widehat{{\varvec{z}}}}_{L}$, with ${n}_{L}=\sqrt{{\mu }_{E}/{a}_{L}^{3}}$ defined as the mean motion of the reference orbit, ${\mu }_{E}=398600\,{\text{km}}^{3}/{\text{s}^{2}}$ Earth’s planetary constant and ${a}_{L}$ the semi-major axis of reference orbit.

Considering the main gravitational effects, the acceleration of the spacecraft i with respect to the LVLH read as [56]:

$${{}^{L}\ddot{{\varvec{d}}}}_{i}={{\varvec{A}}}_{1}{}^{L}{{\varvec{d}}}_{i}+{{\varvec{A}}}_{2}{{}^{L}\dot{{\varvec{d}}}}_{i}+{{\varvec{B}}}_{i}{{}^{L}{\varvec{t}}}_{i}$$

(4)

where ${{\varvec{A}}}_{1}$, ${{\varvec{A}}}_{2}$ and ${{\varvec{B}}}_{i}$ are:

$${{\varvec{A}}}_{1}=\left[\begin{array}{ccc}\frac{2{\mu }_{E}}{{r}^{3}}+\frac{{h}^{2}}{{r}^{4}} & -\frac{2\left({{}^{I}{\varvec{r}}}_{i}.{{}^{I}{\varvec{v}}}_{i}\right)h}{{r}^{4}}& 0\\ \frac{2\left({{}^{I}{\varvec{r}}}_{i}.{{}^{I}{\varvec{v}}}_{i}\right)h}{{r}^{4}}& \frac{{\mu }_{E}}{{r}^{3}}+\frac{{h}^{2}}{{r}^{4}} & 0\\ 0& 0& -\frac{{\mu }_{E}}{{r}^{3}}\end{array}\right]$$

(5)

$${{\varvec{A}}}_{2}=\left[\begin{array}{ccc}0& \frac{2h}{{r}^{2}}& 0\\ -\frac{2h}{{r}^{2}}& 0& 0\\ 0& 0& 0\end{array}\right]$$

(6)

$${{\varvec{B}}}_{i}=\frac{1}{{m}_{i}}\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$$

(7)

where $r=\left|{{}^{I}{\varvec{r}}}_{i}\right|$, $h=\left|{{}^{I}{\varvec{r}}}_{i}\times {{}^{I}{\varvec{v}}}_{i}\right|$ is the magnitude of the angular momentum, and ${{}^{L}{\varvec{t}}}_{i}$ is the thrust applied to the i-th spacecraft with respect the LVLH frame, and ${m}_{i}$ is the mass of each spacecraft. In this paper, it is assumed that thrust ${{}^{L}{\varvec{t}}}_{i}$ can be applied to the system in a continuous way, leaving to future studies a more detailed characterization of the thruster models. The overall $\Delta v$ to perform the maneuvres can be computed through $\Delta v=\int (\left|{{}^{L}{\varvec{t}}}_{i}\right|/{m}_{i}) dt.$ In Eq. (4) we assume that the reference orbit is circular or nearly-circular and the magnitude of the thrust applied to perform the formation maneuver is small enough to not compromise the validity of such assumption.

As shown in Fig. 1, a set of reference frames are also used to represent the attitude of each spacecraft. These reference frames are moving rigidly with each member of the formation and are defined as $\left\{{S}_{i}\right\}$, with $i=\mathrm{1,2},..N$. Considering these previous coordinate frames, it is possible to define the spacecraft thrusts with respect the spacecraft body reference frames. The rotation matrix that defines the orientation of a given spacecraft i with respect to the LVLH reference frame can be expressed as:

$${{}^{L}{\varvec{R}}}_{i}={{}^{L}{\varvec{R}}}_{I}{{}^{I}{\varvec{R}}}_{i}={{}^{L}{\varvec{R}}}_{I}{{}^{i}{\varvec{R}}}_{I}^{T}$$

(8)

where ${{}^{i}{\varvec{R}}}_{I}$ is the rotation matrix that defines the attitude of each spacecraft with respect to the Inertial reference frame. This rotation matrix can be defined in terms of quaternions ${Q}_{i}={\left[\begin{array}{cc}{{\varvec{q}}}_{i}& {q}_{i}\end{array}\right]}^{T}$ as follows ([57]):

$${{}^{i}{\varvec{R}}}_{I}=\left[\left({q}_{i}^{2}-{{\varvec{q}}}_{i}^{T}{{\varvec{q}}}_{i}\right){\varvec{E}}+2{{\varvec{q}}}_{i}{{\varvec{q}}}_{i}^{T}-2{q}_{i}{\widetilde{{\varvec{q}}}}_{i}\right]$$

(9)

where ${\widetilde{{\varvec{q}}}}_{i}$ is defined as the skew-symmetric matrix of ${{\varvec{q}}}_{i}$ and ${\varvec{E}}$ is the identity matrix. Considering ${{}^{i}{\varvec{\omega}}}_{i}$ as the angular velocity of the spacecraft i, the time derivative of ${Q}_{i}={\left[\begin{array}{cc}{{\varvec{q}}}_{i}& {q}_{i}\end{array}\right]}^{T}$ can be obtained as:

$$\dot{Q}_{i} = \frac{1}{2}{{\varvec{\Omega}}}\left( {{\varvec{\omega}}_{i} } \right)Q_{i} = \frac{1}{2}\left[ {\begin{array}{*{20}c} {{}^{i}\widetilde{{{\varvec{\omega}}_{i} }}} & {{}^{i}{\varvec{\omega}}_{i} } \\ { - {}^{i}{\varvec{\omega}}_{i}^{T} } & 0 \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {{\varvec{q}}_{i} } \\ {q_{i} } \\ \end{array} } \right]$$

(10)

Each spacecraft, i, is supposed to be guided by applying a control torque, ${{}^{i}{\varvec{\tau}}}_{i}$, with respect to the corresponding body coordinate frame, $\left\{{S}_{i}\right\}$. Each spacecraft is modelled under the assumption that the spacecraft are subjected not only to control actions of the proposed controller but also to environmental torques. In this last case, ${{}^{i}{\varvec{\tau}}}_{e}$, are defined as the disturbing external torque applied to the spacecraft. Considering the previous mentioned torque, the equations that define the attitude dynamics can be defined as:

$${{}^{i}\dot{{\varvec{\omega}}}}_{i}={{}^{i}{\varvec{J}}}_{i}^{-1}\left(-{}^{i}\widetilde{{{\varvec{\omega}}}_{i}}{{}^{i}{\varvec{J}}}_{i}{{}^{i}{\varvec{\omega}}}_{i}+{{}^{i}{\varvec{\tau}}}_{e}+{{}^{i}{\varvec{\tau}}}_{i}\right)$$

(11)

where ${{}^{i}{\varvec{J}}}_{i}$ is the matrix that defines the moment of inertia matrix of each spacecraft, i, with respect to and expressed in the spacecraft body frame. In this simplified model, each satellite is assumed being equipped with actuators to have full control of the orbit and attitude dynamics, e.g., thrusters and reaction wheels, respectively.

The system composed by Eqs. (4) and (11) represents the core of the equations of motion that characterizes each spacecraft in the model scenario, that can be rewritten as follows:

$$\varvec{F}_{{c,i}} + \varvec{F}_{{e,i}} = \varvec{I}_{i} {\ddot{\varvec{x}}}_{i} + \varvec{C}_{i}$$

(12)

where ${\ddot{\varvec x}}_{i} = \left[ {\begin{array}{*{20}c} {{\ddot{\varvec r}}_{i}^{T} } & {\dot{\varvec{\omega }}_{i}^{T} } \\ \end{array} } \right]^{T} \in \Re^{6}$ contains the absolute linear and angular accelerations of the i-th spacecraft, ${\varvec{I}}_{i} \in \Re^{6 \times 6}$ is the mass matrix containing the mass and moments of the inertia of the spacecraft, ${\varvec{C}}_{i} \in \Re^{6}$ contains the non-linear velocity/displacement-dependent terms, ${\varvec{F}}_{{{c,i}}} \in \Re^{6}$ contains the force and moment exerted by the spacecraft actuators, and ${\varvec{F}}_{{{e,i}}} \in \Re^{6}$ contains the external/disturbing forces and torques applied to the spacecraft.

3 Formation Flying Simulation

This section describes the main properties and architecture of the ROS-based simulation system proposed for the spacecraft formation flying scenario. Figure 2 represents the main modules and architecture of the proposed simulation system. This ROS setup allows the simulation of the orbital dynamics of the spacecraft proximity scenarios, including main relative orbit effects and perturbations, but it can be used also for simulating free-floating conditions only or with used defined trajectories.

Fig. 2

ROS based simulation system

Section 3.1 shows the configuration setup required to simulate the different kinds of trajectories and Sect. 3.2 describes the plugins implemented to perform the simulation by using Gazebo. The simulation system is implemented as a ROS publisher node that generates the trajectories described by the spacecraft that compose the formation. The publisher node provides the timing, ${t}_{a}$, of the corresponding spacecraft, and the values of the matrices ${{\varvec{R}}}_{L}$ and ${{\varvec{t}}}_{L}$ that represent the rotation and translation of the LVLH frame with respect the inertial frame (and the corresponding velocity, ${{\varvec{v}}}_{L}\left(t\right)$), respectively.

3.1 Simulation Setup

For the simulation of simple orbits, orbital parameters such as semi-major axis, eccentricity, argument of periapsis, right ascension of ascending node, inclination, and instant of time of perigee passage, should be defined. This module obtains the frame position and orientation of frame {L} with respect to the inertial frame, ${{\varvec{t}}}_{L}$ and ${{\varvec{R}}}_{L}$, and its linear velocity, ${{\varvec{\nu}}}_{L}$, as a function of the previously indicated Kepler parameters that define an orbit.

To solve the orbital reference position problem, the approach described in [58] is adopted, where an iterative method for obtaining the true anomaly is proposed. The model also considers the effects the Earth’s second harmonic of the gravity field (J2-effect) to obtain the position and orientation of the LVLH reference frame.

To calculate the forces generated by the relative motion of each spacecraft relative to the LVLH reference frame, a linearisation of the equations of motion relative to the orbit described in [59] is used. The perturbations concerning the gravity gradient have also been considered. More information about the models used to propagate the true orbital dynamics of the spacecraft and the system validation can be found in [60].

The simulation of specific trajectories (custom path) is performed by setting a series of points that are introduced by the user in the ROS parameter server (a data structure composed of a set timeseries of positions, velocities and accelerations to achieve each pose), and the Custom Path module generates the interpolated trajectory.

All the previous information is provided by the user in the Ain’t Markup Language (YAML) configuration file and stored in the ROS Parameter Server. The ROS Parameter server is common to all nodes, and it offers great freedom in sharing parameters between different nodes. The user defines in the launch file the orbit or trajectory that best fits the trajectory to be modelled, and the Publisher Node instantiates the corresponding object.

3.2 Gazebo Plugins

Gazebo is a simulation tool for fast and efficient simulation development. Originally, Gazebo was conceived as a tool for 3D simulation of multi-robot environments with the ability to recreate complex and customizable environments. Like ROS, Gazebo is open-source and freely available. However, Gazebo lacks a simulation environment for on-orbit robotics applications such as the spacecraft formation flying scenario presented in this paper. The architecture described in Fig. 2 presents the implemented packages and setting required to simulate the formation flying scenario by using ROS and Gazebo.

Two of the main features of Gazebo are used in the generation of the simulation environment. The first is that the description and parameterization of the simulations is done by using the Simulation Description Format (SDF) specification language in an XML file that the simulator loads at start up (Model Spacecraft 1 … Model Spacecraft N in Fig. 2). This description allows for defining the kinematic and dynamic parameters of each spacecraft in the formation. The second Gazebo feature used to simulate spacecraft formation flying is the possibility of extending the Gazebo functionality through plugins. These plugins have access to all the simulation elements, being able to consult the state of these and, at the same time, to act on and interact with them. One of the implemented plugins (OORplugin World in Fig. 2) modifies the parameters of the physics engine used by Gazebo to eliminate effects such as gravity, wind, and magnetism since they were originally applied under the assumption that the simulation takes place on the Earth’s surface. The same module substitutes such environmental effects with the torques and forces experienced in orbit during the formation flying. Additionally, all the links of the simulated spacecraft will be subject to forces and torques, such as perturbation from gravity gradient and forces generated by the relative motion to an elliptical orbital. The different plugins apply these forces for each simulated spacecraft (OORplugin Spacecraft 1 … OORplugin Spacecraft N in Fig. 2). Other perturbations, such as aerodynamic drag, solar radiation pressure and effects of third-body effects will be included in the next versions of the simulation system.

4 Image Processing and Visual Control Properties

Figure 3 represents the camera configuration of a given spacecraft in the formation. The camera coordinate frame, $\left\{{S}_{i}\right\}$, presents the ${\widehat{{\varvec{z}}}}_{c}$ axis aligned along the outward direction of the camera optical axis and pointing towards the next spacecraft in the formation. Additionally, the LEDs pattern represented in Fig. 3 is composed of four LEDs that form a square pattern centred in the body reference frame with a distance of 0.4 m between them.

Fig. 3

Visual features and camera coordinate frame

The controller proposed in Sect. 5 is an image-based controller, i.e., this controller eliminates the need for 3D reconstruction of relative positions and attitudes of spacecraft, allowing for the direct design of guidance control laws in the image plane. Therefore, previous positions won’t be used for the controller definition. This section describes visual control properties and relationships between visual and 3D information that will be used in the definition of the proposed controller. Let ${}^{c}{\varvec{p}}={\left[\begin{array}{ccc}{}^{c}x& {}^{c}y& {}^{c}z\end{array}\right]}^{T}$ be the position of an observed point in the camera coordinate frame. The corresponding projection is defined as an image feature point and can be computed by considering a pin-hole camera model as:

$${\varvec{s}}=\left[\begin{array}{c}X\\ Y\end{array}\right]=\frac{1}{{}^{c}z}\left[\begin{array}{c}{}^{c}x\\ {}^{c}y\end{array}\right]$$

(13)

where $X$ and $Y$ are the image coordinates of ${\varvec{s}}$, measured in pixels.

Let the observed spacecraft be endowed with a set of $\eta$ visible feature points whose position in the ECI frame are ${{}^{I}{\varvec{p}}}_{t}={{\varvec{p}}}_{t}$, with $t=1,\dots ,\eta$. Some LEDs located on the observed spacecraft are used to detect the $\eta$ characteristic points. Let ${{\varvec{s}}}_{t}={\left[\begin{array}{cc}{X}_{t}& {Y}_{t}\end{array}\right]}^{T}$ be the image feature vector corresponding to the t-th target point (LEDs’ position). The image variation of ${{\varvec{s}}}_{t}$ can be computed as:

$${\dot{{\varvec{s}}}}_{t}=\left[\begin{array}{c}{\dot{X}}_{t}\\ {\dot{Y}}_{t}\end{array}\right]={{\varvec{L}}}_{t}{\overline{{\varvec{R}}}}_{c}^{T}{\dot{{\varvec{x}}}}_{{\text{c}}}={{\varvec{J}}}_{t}{\dot{{\varvec{x}}}}_{{\text{c}}}$$

(14)

where ${\dot{{\varvec{x}}}}_{{\text{c}}}$ are the camera angular and linear velocities, ${\overline{{\varvec{R}}} }_{c}=diag\left({{\varvec{R}}}_{c},{{\varvec{R}}}_{c}\right)$, being ${{\varvec{R}}}_{c}$ the camera attitude or absolute camera rotation with respect to the inertia coordinate frame, and the interaction matrix ${{\varvec{L}}}_{t}$ is equal to [19]:

$${{\varvec{L}}}_{t}=\left[\begin{array}{ccc}-\frac{1}{{}^{c}{z}_{t}}& 0& \begin{array}{ccc}\frac{{X}_{t}}{{}^{c}{z}_{t}}& {X}_{t}{Y}_{t}& \begin{array}{cc}-1-{X}_{t}^{2}& {Y}_{t}\end{array}\end{array}\\ 0& -\frac{1}{{}^{c}{z}_{t}}& \begin{array}{ccc}\frac{{Y}_{t}}{{}^{c}{z}_{t}}& -1-{X}_{t}^{2}& \begin{array}{cc}{X}_{t}{Y}_{t}& -{X}_{t}\end{array}\end{array}\end{array}\right]$$

(15)

with ${}^{c}{z}_{t}$ the third component of ${{}^{c}{\varvec{p}}}_{t}={{}^{c}{\varvec{R}}}_{I}{{\varvec{p}}}_{t}$, with ${{}^{c}{\varvec{R}}}_{I}$ the rotation between the inertial and camera frames. In Eq. (14), the time variation of ${{\varvec{s}}}_{t}$ is only affected by the camera motion. Therefore, only fixed targets are considered. However, in spacecraft formation keeping and reconfiguration tasks the spacecraft are in motion and, consequently, the observed visual feature points are also in motion. In this case, Eq. (14) can be reformulated as:

$${\dot{{\varvec{s}}}}_{t}=\left[\begin{array}{c}{\dot{X}}_{t}\\ {\dot{Y}}_{t}\end{array}\right]={{\varvec{J}}}_{t}{\dot{{\varvec{x}}}}_{{\text{c}}}+\frac{\widehat{\partial {{\varvec{s}}}_{t}}}{\partial t}$$

(16)

where the term $\frac{\widehat{\partial {{\varvec{s}}}_{t}}}{\partial t}$ represents the time variation of ${{\varvec{s}}}_{t}$ due to the generally unknown target motion. Kalman filtering or more elaborate filtering methods [61, 62] can then be used to compute the estimation of this term. The Kalman filter estimates the time variation of the target image feature points in order to compensate for the possible errors introduced by unmodeled movements of the target. This estimation is used in Eq. (16) to obtain the relationship between the time derivative of the extracted image feature points and the camera motion.

As it will be described in Sect. 5, an optimal image-based visual servoing approach is proposed for the spacecraft guidance. The main advantages of the proposed image-based approach over a position-based system are: a 3D model of the target is not needed, and the performance is robust with respect to perturbations of the robot and camera models. As it is described in Sect. 5, the computation of control signals involves the interaction matrix (15) that depends on the depths of each visual feature, ${}^{c}{z}_{t}$. As described in [19] different approaches can be used to compute these depths. The first option is to compute the current depth of each point at each iteration using classical approaches like the ones described in [63]. In [64] a method to on-line estimate the value of depth for point features while the camera is moving through the scene is presented. This last approach uses tools from nonlinear observer theory. As indicated in [19], another popular approach is to choose an estimation of this interaction matrix ${{\varvec{L}}}_{t}={{\varvec{L}}}_{t}^{*}$, where ${{\varvec{L}}}_{t}^{*}$ is the value of the interaction matrix considering the depths obtained at the final position, which is generally not difficult to obtain in practice. In this case, the depths in ${{\varvec{L}}}_{t}^{*}$ are constant, and only the desired depth of each point must be set, which means no varying 3-D parameters have to be estimated during the visual servoing. This approach is the one considered in the paper.

Some LEDs located on the spacecraft are used to detect the $\eta$ characteristic points from the spacecraft camera that is observing them to perform the formation-keeping and reconfiguration tasks ($\eta =4$ in Fig. 3). To avoid ambiguities and to make easier the visual features extraction, only the LEDs located in the observed face are turned on. The computer vision algorithm that allows obtaining the visual features is based on the separation of the colour channels to obtain, on the one hand, the first visual feature (through the green channel), as well as the rest of the features (through the red channel). Figure 3 shows the LEDs on one of the faces of the observed spacecraft. Each spacecraft has one green LED and three red ones on four of the faces of the cube that the satellite forms. Once the green led is segmented, the first visual feature is obtained. In order to obtain the correct sequence for the rest of the visual features, a computer vision algorithm has been implemented. The third visual feature is the one that has a greater distance in the image plane with respect to the first one. The second and fourth features are then obtained by comparing the x component with the one of the first (${X}_{1}$) and third (${X}_{3}$) features. If ${X}_{3}$ is greater than ${X}_{1}$, then feature 2 is the one with less y component of the two unlabelled features. Otherwise, feature 2 has a greater y component than the two unlabelled features. Thus, all the visual features are correctly labelled regardless of the relative orientation between the satellites.

5 Optimal Visual Servoing

This section proposes an optimal and direct image-based visual servoing strategy to perform the spacecraft guidance. The presented visual servoing system is based on the approach proposed in [65] where an optimal control framework is defined and applied to nonlinear mechanical systems. In our case, the control reference for each spacecraft controller, i, is a desired image trajectory to be tracked specified in the image space, ${{\varvec{s}}}_{i}^{*}\left(t\right)$. Therefore, the distributed controller can perform the spacecraft guidance without the need to estimate relative positions of the observed spacecraft. Visual feedback is used in the distributed control action to determine the forces and torques required for the spacecraft guidance considering the orbital and attitude dynamics. Specifically, the proposed visual controller is defined within the optimal control framework that allows performing the spacecraft guidance while the energy and fuel is minimized.

A cost function that takes into account and weights the control efforts applied to the system is defined in such way that it can be minimized by the optimal framework. Specifically, by considering that each spacecraft would be subjected to forces and torques generically calculated in the vector ${\varvec{F}}_{c,i}$, it is straightforward to consider a cost function defined as:

$$\begin{array}{c}{\varvec{\Omega }}_{i}\end{array}\text{ = }{{\varvec{F}}_{c,i}}^{\text{T}}{{\varvec{W}}}_{i} \, {\varvec{F}}_{c,i}$$

(17)

where ${{\varvec{W}}}_{i}$ is a weight matrix to be defined. This last matrix allows to indicate how the control action is distributed over the different motion directions.

The next step to define the optimal control action is to determine the system constraints that specify the visual servoing task. A set of $m$ constraints are used to define the visual servoing tasks that need to be executed during the maneuvers. These constraints can be generically defined as:

$$\begin{array}{*{20}c} {\varvec{A}}_{i} \\ \end{array} \left( {t} \right)\ddot{\varvec{x}}_{{ci}} = \varvec{b}_{i} \left( {t} \right)$$

(18)

where $\begin{array}{*{20}c} {\varvec{A}}_{i} \\ \end{array} \left( {t} \right) \in \Re^{m \times 6}$ and ${\varvec{b}}_{i} \left( t \right) \in \Re^{m \times 1}$ defines the system constraints. Additionally, ${\ddot{{\varvec{x}}}}_{ci}$ is the absolute linear and angular accelerations of the camera of the i-th spacecraft in the Cartesian LVLH frame. However, using a visual servoing approach, the image-based visual servoing task needs to be defined in the image plane. This is done by indicating the desired trajectories of the η visual features to be tracked in the image plane as ${{\varvec{s}}}_{i}^{*}\left(t\right)$ and imposing the following conditions to be satisfied:

$$\left({\ddot{{\varvec{s}}}}_{i}^{*}-{\ddot{{\varvec{s}}}}_{ti}\right)\text{ } + { \, {\varvec{K}}}_{di}\left({\dot{{\varvec{s}}}}_{i}^{*}-{\dot{{\varvec{s}}}}_{i}\right)\text{ } + { \, {\varvec{K}}}_{\text{pi}}\left({{\varvec{s}}}_{i}^{*}-{{\varvec{s}}}_{i}\right)= \text{ 0}$$

(19)

where ${\ddot{{\varvec{s}}}}_{i}^{*}$, ${\dot{{\varvec{s}}}}_{i}^{*}$ and ${{\varvec{s}}}_{i}^{*}$ are the desired accelerations, velocities, and positions of the η captured visual features in the image plane for each spacecraft, respectively. ${\varvec{K}}_{\text{pi}}$ and ${\varvec{K}}_{di}$ are proportional and derivative gain matrices, respectively. Different approaches can be used to define these image trajectories [66]. A frequent approach is to perform the path planning in the 3D space and then project the visual features in the image. Classical approaches [67] and more recent ones [68] propose the definition of the trajectories of the visual features directly in the image space. Given start and final locations over the scene, the planners output a short and navigable visual path to be tracked. We should also mention other methods that use convex optimization techniques to define optimal features paths on the image plane [69].

Equation (19) can be expressed in terms of image errors as follows:

$${\ddot{{\varvec{s}}}}_{i}^{*}+{ \, {\varvec{K}}}_{di}{\dot{{\varvec{e}}}}_{si}+{ \, {\varvec{K}}}_{pi}{{\varvec{e}}}_{si}={\ddot{{\varvec{s}}}}_{ti}$$

(20)

where ${{\varvec{e}}}_{si}$ and ${\dot{{\varvec{e}}}}_{si}$ are the image errors and time derivatives of the image error between the observed and desired image features extracted by the camera of the spacecraft i, respectively.

The image acceleration or second derivative of ${{\varvec{s}}}_{ti}$ can be obtained by differentiating Eq. (16) with respect to time:

$${\ddot{{\varvec{s}}}}_{ti}={{\varvec{J}}}_{ti}{\ddot{{\varvec{x}}}}_{ci}+{\dot{{\varvec{J}}}}_{ti}{\dot{{\varvec{x}}}}_{ci}+\frac{\widehat{\partial {{\varvec{e}}}_{si}}}{\partial t}$$

(21)

This last equation can be expressed in the form indicated in Eq. (17):

$${{\varvec{J}}}_{ti}{\ddot{{\varvec{x}}}}_{{\text{ci}}}={\ddot{{\varvec{s}}}}_{ti}-{\dot{{\varvec{J}}}}_{ti}{\dot{{\varvec{x}}}}_{ci}-\frac{\widehat{\partial {{\varvec{e}}}_{si}}}{\partial t}$$

(22)

Therefore, the value of the matrices ${\varvec{A}}_{i}$ and ${{\varvec{b}}}_{i}$ that define the visual servoing task can be easily obtained as:

$${\varvec{A}}_{i}={{\varvec{J}}}_{ti}$$

(23)

$$\varvec{b}_{i} = {\ddot{\varvec{s}}}_{ti} - {\dot{\varvec{J}}}_{{ti}} {\dot{\varvec{x}}}_{{ci}} - \frac{{\widehat{{\partial \varvec{e}_{{si}} }}}}{{\partial t}}$$

(24)

Under such assumptions and definitions, the control action that performs the visual servoing task defined by the constraints indicated in Eqs. (23) and (24), considering the system dynamics defined by Eq. (12), and that simultaneously optimizes the function (18) depending on the value of the weight matrix ${{\varvec{W}}}_{i}$, can be obtained as:

$${\varvec{F}}_{c,i} = {\varvec{W}}_{i}^{ - 1/2} \left( {\varvec{A}}_{i} {\varvec{I}}_{i}^{ - 1} \varvec{W}_{i}^{ - 1/2} \right)^{ + } \cdot\left( {{\varvec{b}}_{i} + {\varvec{A}}_{i} {\varvec{I}}_{i}^{ - 1} {\varvec{C}}_{i} - {\varvec{A}}_{i} {\varvec{I}}_{i}^{ - 1} {\varvec{F}}_{e,i} } \right)$$

(25)

where the symbol + denotes the pseudo-inverse matrix.

This is proved by defining an auxiliary variable as $\begin{array}{*{20}c} {\varvec{\sigma}} \\ \end{array} = {\varvec{W}}_{i}^{1/2} \varvec{F}_{c,i} = {\varvec{W}}_{i}^{1/2} \left( {{\varvec{I}}_{i} \ddot{\varvec{x}}_{i} + {\varvec{C}}_{i} - {\varvec{F}}_{e,i} } \right)$, and then deriving the linear and angular accelerations as $\varvec{\ddot{x}}_{i} {\text{ = }}\varvec{I}_{i} ^{{ - {\text{1}}}} \left( {\mathbf{W}_{i}^{{ - {\text{1/2}}}} \varvec{\sigma} - \mathbf{C}_{i} + {\varvec{F}}_{{e,i}} } \right)$.

If we then consider that the camera and body coordinate frames are coinciding and not moving each other, and from the system constraints defined by Eq. (18), we can obtain:

$${\varvec{A}}_{i} {\varvec{I}}_{i}^{ - 1} {\varvec{W}}_{i}^{{ - {1/2}}} { }\begin{array}{*{20}c} {\varvec{\sigma}} \\ \end{array} { = }{\varvec{b}}_{i} { + } {\varvec{A}} _{i} {\varvec{I}}_{i}^{{ - {1}}} {\varvec{C}}_{i} {- } {\varvec{A}} _{i} {\varvec{I}}_{i}^{{ - {1}}} {\varvec{F}}_{e,i}$$

(26)

Thus, the vector σ which minimizes

${\varvec{\varOmega}}= {\varvec{\sigma}}^{T} {\varvec{\sigma}}$ while fulfilling Eq. (26) is $\begin{array}{*{20}c} {\varvec{\sigma}} \\ \end{array} { = }\left( {{\varvec{A}}_{i} {\varvec{I}}_{i}^{{ - {1}}} {\varvec{W}}_{i}^{{ - {1/2}}} } \right)^{ + } \left( {{\varvec{b}}_{i} { + }{\varvec{A}}_{i} {\varvec{I}}_{i}^{{ - {1}}} {\varvec{C}}_{i} - {\varvec{A}}_{i} {\varvec{I}}_{i}^{{ - {1}}} {\varvec{F}}_{e,i} } \right)$, and consequently it is possible to notice that the control actions of a given spacecraft, i, can be defined as ${\varvec{F}}_{c,i} { = }{\varvec{W}}_{i}^{{ - {1/2}}} \begin{array}{*{20}c} {\varvec{\sigma}} \\ \end{array}$, so that Eq. (25) is consequently demonstrated.

Considering the visual servoing constraints indicated by Eqs. (23) and (24), the control action that defines the force and moment to be applied to each spacecraft can be obtained from Eq. (25):

$${\varvec{F}}_{{c,i}} {\text{ = }}\varvec{W}_{i}^{{ - {\text{1/2}}}} \left( {\varvec{J}_{{ti}} \varvec{I}_{i} ^{{ - {\text{1}}}} \varvec{W}_{i}^{{ - {\text{1/2}}}} } \right)^{{\text{ + }}} \left( {\varvec{\ddot{s}}_{{ti}} - \varvec{\dot{J}}_{{ti}} \varvec{\dot{x}}_{{\text{i}}} - \frac{{\widehat{{\partial \varvec{e}_{{si}} }}}}{{\partial t}} + \varvec{J}_{{ti}} \varvec{I}_{i} ^{{ - {\text{1}}}} \varvec{C}_{i} - \varvec{J}_{{ti}} \varvec{I}_{i} ^{{ - {\text{1}}}} {\varvec{F}}_{{e,i}} } \right)$$

(27)

Therefore, Eq. (27) defines the control action that minimizes the cost function indicated in Eq. (17) while the visual servoing task defined by Eqs. (23) and (24) is performed. Equation (17) defines the cost function to be minimized by applying the control action obtained in (27). This control action depends on the weight matrix ${{\varvec{W}}}_{i}$. A wide range of visual servoing control laws with different dynamic properties can be obtained depending on the value of ${{\varvec{W}}}_{i}$. For example, by defining this matrix as ${{\varvec{W}}}_{i}$ = ${{{\varvec{I}}}_{i}}^{-{2}}$ in Eq. (27):

$$\varvec{F}_{{c,i}} = ~\varvec{I}_{i} \varvec{J}_{{ti}}^{ + } \cdot\left( {\varvec{\ddot{s}}_{{ti}} - \varvec{\dot{J}}_{{ti}} \varvec{\dot{x}}_{{\text{i}}} - \frac{{\widehat{{\partial \varvec{e}_{{si}} }}}}{{\partial t}} + \varvec{J}_{{ti}} \varvec{I}_{i} ^{{ - {\text{1}}}} \varvec{C}_{i} - \varvec{J}_{{ti}} \varvec{I}_{i} ^{{ - {\text{1}}}} {\varvec{F}}_{{e,i}} } \right)$$

(28)

Additionally, by defining the weight matrix as the inverse of the inertia matrix, ${{\varvec{W}}}_{i}={{{\varvec{I}}}_{i}}^{-{1}}$:

$${\varvec{F}}_{c,i}\text{=}{{\varvec{I}}}_{i}^{1/2} \, {\left({{\varvec{J}}}_{ti}{{\varvec{I}}}_{i}^{-\text{1/2}}\right)}^{+}\left({\ddot{{\varvec{s}}}}_{ti}-{\dot{{\varvec{J}}}}_{ti}{\dot{{\varvec{x}}}}_{{\text{i}}}-\frac{\widehat{\partial {{\varvec{e}}}_{si}}}{\partial t}+{{\varvec{J}}}_{ti}{{{\varvec{I}}}_{i}}^{-{1}}{{\varvec{C}}}_{i}- \, {{\varvec{J}}}_{ti}{{{\varvec{I}}}_{i}}^{-{1}}{\varvec{F}}_{e,i}\right)$$

(29)

Another value for the choice of ${{\varvec{W}}}_{i}$ is ${\varvec{D}},$ where ${\varvec{D}}$ is a positive diagonal matrix. This matrix allows distributing the actuated forces so that small forces and torques can be obtained in correspondence of large weights and vice-versa:

$${\varvec{F}}_{c,i}= \text{ } {{\varvec{D}}}^{-\text{1/2}}{\left({{\varvec{J}}}_{ti}{{{\varvec{I}}}_{i}}^{-{1}}{{\varvec{D}}}^{-\text{1/2}}\right)}^{+}\left({\ddot{{\varvec{s}}}}_{ti}-{\dot{{\varvec{J}}}}_{ti}{\dot{{\varvec{x}}}}_{{\text{i}}}-\frac{\widehat{\partial {{\varvec{e}}}_{si}}}{\partial t}+{{\varvec{J}}}_{ti}{{{\varvec{I}}}_{i}}^{-{1}}{{\varvec{C}}}_{i}- \, {{\varvec{J}}}_{ti}{{{\varvec{I}}}_{i}}^{-{1}}{\varvec{F}}_{e,i}\right)$$

(30)

Another value for the choice of ${{\varvec{W}}}_{i}$ is ${{{\varvec{D}}{\varvec{I}}}_{i}}^{-2}$, where ${\varvec{D}}$ is a diagonal positive matrix:

$${\varvec{F}}_{c,i}\text{=}{{\varvec{D}}}^{-\text{1/2}}{{\varvec{I}}}_{i} \, {\left({{\varvec{J}}}_{ti}{{{\varvec{I}}}_{i}}^{-{1}}{{\varvec{D}}}^{-\text{1/2}}{{\varvec{I}}}_{i}\right)}^{+}\left({\ddot{{\varvec{s}}}}_{ti}-{\dot{{\varvec{J}}}}_{ti}{\dot{{\varvec{x}}}}_{{\text{i}}}-\frac{\widehat{\partial {{\varvec{e}}}_{si}}}{\partial t}+{{\varvec{J}}}_{ti}{{{\varvec{I}}}_{i}}^{-{1}}{{\varvec{C}}}_{i}- \, {{\varvec{J}}}_{ti}{{{\varvec{I}}}_{i}}^{-{1}}{\varvec{F}}_{e,i}\right)$$

(31)

The control laws (28)-(31) will be evaluated in the results section in formation keeping and formation reconfiguration maneuvers.

6 Results

This section presents the simulation results obtained with the proposed strategy to control the relative position between spacecraft and to maintain and reconfigure a specific spacecraft formation. The proposed optimal visual servoing approach is employed in a close formation flying scenario, as shown in Fig. 4. This figure displays the initial configuration of the spacecraft formation comprising four spacecraft. The camera allows a resolution of 1024 × 1024 px, with a fixed focal length of 400 mm and a pixel size of $10\times 10$ (μm × μm). The camera field of view of each spacecraft is also depicted, and we assume that the target visual features (LEDs) are observable in the initial configuration. Table 1 shows the initial spacecraft configuration of the formation, indicating the position of each spacecraft with respect to the LVLH frame. Furthermore, Table 2 specifies the moments of inertia of the spacecraft in the formation. The orbital parameters of the spacecraft are defined by the following parameters: semi-major axis $=12000$ km, eccentricity $=1.0000{\text{e}}-05$, inclination $=1.0472$ rad, initial right ascension $=0$, initial argument of perigee $=0$, initial mean anomaly $=1.0472$ rad, rate of right ascension $=-1.1010{\text{e}}-07$, and rate of argument of perigee $=0$. In such orbit, the gravity gradient is the main perturbing action and other effects, such as the differential drag effects are neglected.

Fig. 4

Initial configuration of the spacecraft formation scenario

Table 1 Initial spacecraft configuration of the formation w.r.t. the LVLH reference frame

Table 2 Moments of inertia of the spacecraft

6.1 Formation Keeping Maneuver

This section describes the simulation results obtained during the application of the proposed visual servoing system to maintain the initial spacecraft configuration against orbital environmental perturbations and disturbances in the actuation and sensing.

As described in Sect. 4, four visual features are extracted from each observed spacecraft at the initial positions indicated in Table 1. Figure 5 represents the positions of these four visual features in the image space. In this simulation, the controller is required to maintain the formation of spacecraft throughout the entire 3-h duration of the experiment by compensating for any environmental perturbations that may occur. The value of the proportional and derivative matrices of the controller (see Eq. (19)) are set as ${\text{K}}_{pi}=0.001{\varvec{E}}$ and ${\text{K}}_{di}=0.05{\varvec{E}}$, respectively. The gains in the aforementioned matrices have been selected by using a trial-and-error approach, but always complying with the hypothesis of the adopted visual servoing controller.

Fig. 5

Extracted image features at the initial configuration of the spacecraft formation keeping scenario

The four controllers defined in Eqs. (28)–(31) with different values for the matrix ${{\varvec{W}}}_{i}$ have been tested. In these equations the value of the diagonal matrix ${\varvec{D}}$ is defined so the weight associated to the first three components are tenth of the last three components. Therefore, the translation error is rapidly compensated. Table 3 shows the mean image error during the formation-keeping maneuver. This table also shows the value of $\Delta {\varvec{v}}$ obtained during the tracking. In this case, the best performance (lowest image error and $\Delta {\varvec{v}}$) has been obtained with Eq. (29) where the gain matrix is equal to the inverse of the inertia matrix (${{\varvec{W}}}_{i}$ = ${{{\varvec{I}}}_{i}}^{-1}$). The behaviour of this last controller is shown in detail in Figs. 6, 7, 8, 9.

Table 3 Performance of the formation-keeping maneuver for different values of ${{\varvec{W}}}_{i}$

Fig. 6

Spacecraft trajectories with respect to the LVLH coordinate frame (left) and relative position error among spacecraft (right) for the spacecraft formation keeping scenario

Fig. 7

Mean image error. Formation keeping maneuver

Fig. 8

Relative distance between the spacecraft. Formation keeping maneuver

Fig. 9

Forces during the formation-keeping maneuver

Figure 6 shows the spacecraft trajectories (left) and the relative position error among spacecraft (right) during the formation keeping maneuvre in the LVLH frame. Figure 7 illustrates the mean image error, denoted as $\left|{{\varvec{e}}}_{si}\right|$, which is used to compute the control law. This error, defined in the image space, remains consistently low throughout the entire maneuver duration. Table 1 indicates that the desired relative distance between the spacecraft is 400 m. Figure 8 depicts the relative distance between each spacecraft, which is maintained around the desired one with minor adjustments to compensate for perturbations, such as the gravity gradient. These effects are also visible in the thrusts applied during the formation-keeping maneuver, as shown in Fig. 9. Additionally, the generated torques remain low (with maximum values of 0.005 Nm) throughout the entire 3-h trajectory duration.

6.2 Formation Reconfiguration Maneuver of a Coplanar Formation

This section shows the results of a formation reconfiguration maneuver obtained by applying the proposed visual servoing approach. The initial spacecraft configuration is the same as in the previous case, with the initial position of the spacecraft as in Table 1. In this case, the proposed controllers are utilized to decrease the mutual distances between the spacecraft, ultimately bringing them to a final range of $180\,{\text{m}}$. Initially, the spacecraft are $400\,{\text{m}}$ apart, and the visual features extracted are illustrated in Fig. 5. The desired image trajectory, which enables spacecraft reconfiguration, is defined as a linear trajectory between the positions depicted in Fig. 5 and the corresponding image feature positions presented in Table 4.

Table 4 Final position for the image features

The distributed visual servoing algorithm performs the maneuver reconfiguration by tracking the desired image trajectory for the maneuver duration (0.3 h) while compensating for perturbations and the effects of the relative orbital motion. The proportional and derivative gain matrices are ${\text{K}}_{pi}=0.004{\varvec{E}}$ and ${\text{K}}_{di}=0.01{\varvec{E}}$, respectively. Table 5 illustrates the mean image error and ∆v acquired during the tracking of the desired trajectory, which reflects the performance of the controllers presented in Eqs. (28)–(31). These equations define the value of the diagonal matrix ${\varvec{D}}$, which allocates a weight to the first three components as one-tenth that of the weights in the last three components. This diagonal matrix can be utilized to distribute the torques and reduce the effort required in certain degrees of freedom.

Table 5 Performance of the formation reconfiguration maneuver for different values of ${{\varvec{W}}}_{i}$

Similar to the formation-keeping maneuver, the optimal performance is achieved when ${{\varvec{W}}}_{i}$ = ${{{\varvec{I}}}_{i}}^{-1}$ (as shown in Eq. (29)). However, the tracking image error, $\left|{{\varvec{e}}}_{si}\right|$ is greater in magnitude than the values presented in Table 3 due to the distinct nature of the two experiments. Figure 10 shows the spacecraft trajectories (left) and the relative position error (right) in the LVLH coordinate frame. Figure 11 provides a more detailed representation of the image error, which is the average error between the extracted visual features and the desired ones obtained from the desired trajectory. This image error remains consistently low throughout the entire duration of the trajectory, which spans 0.3 h. Notably, the controller not only facilitates tracking the desired trajectory in the image space but also compensates for gravity gradient perturbations. The implementation of the controller defined in Eq. (29) results in the reduction of relative distances between the spacecraft from 400 to 180 m, as demonstrated by Fig. 12. It is worth noting that this is obtained without the computation of 3D poses of the spacecraft but only tracking trajectories in the image plane. Figure 13 displays the applied thrusts during the formation reconfiguration maneuver, revealing that the forces in the z-direction are significantly smaller than those in the x and y directions. This is attributable to all four spacecraft being located within the same plane.

Fig. 10

Spacecraft trajectories with respect to the LVLH coordinate frame (left) and relative position error among spacecraft (right) for the formation reconfiguration maneuver

Fig. 11

Image error during the formation reconfiguration maneuver

Fig. 12

Relative distance between the spacecrafts during the formation reconfiguration maneuver

Fig. 13

Forces during the formation reconfiguration maneuver

6.3 Formation Reconfiguration Maneuver of a Non-coplanar Formation

This section presents a formation reconfiguration maneuver where the initial formation includes out-of-plane spacecraft. The spacecraft’s initial configuration is the previous maneuvers, except for spacecraft 1, which is positioned $5\,{\text{m}}$ outside the plane defined by the other spacecraft. Table 6 lists the initial positions of each spacecraft with respect to the LVLH frame. Similar to Sect. 6.2, the primary objective of this reconfiguration maneuver is to decrease the relative distance between the spacecraft from $400\,{\text{m}}$ to $180\,{\text{m}}$. To achieve this, a linear trajectory for the desired image is defined between the initial features extracted by the cameras and the final image positions for these features indicated in Table 4. The control matrices for the distributed visual servoing algorithm are set as ${{\varvec{W}}}_{{i}}$ = ${{{\varvec{I}}}_{i}}^{-1}$, ${\text{K}}_{pi}=0.004{\varvec{E}}$, ${\text{K}}_{di}=0.01{\varvec{E}}$ for the weight, proportional, and derivative gains, respectively. The distributed visual servoing algorithm tracks the desired image trajectory throughout the maneuver’s duration while compensating for the effects of relative orbital motion and perturbations.

Table 6 Initial spacecraft configuration of the formation w.r.t. the LVLH reference frame

Figure 14 shows the trajectory of the spacecraft (left) and the corresponding relative position error in the LVLH frame. The image error, ${{\varvec{e}}}_{si}$, obtained during the trajectory with the new initial conditions is illustrated in Fig. 15. The desired image features are successfully achieved, and the image error remains low once the initial perturbation is compensated for. The spacecraft reconfiguration maneuver is performed correctly, as evidenced by the reduction of relative distance to 180 m, shown in Fig. 16. Additionally, Fig. 17 displays the forces applied to each spacecraft during the experiment, highlighting the small forces in the z direction that compensate for the out-of-plane deviation.

Fig. 14

Spacecraft trajectories with respect to the LVLH coordinate frame (left) and relative position error among spacecraft (right) for the formation reconfiguration maneuver (out-of-plane experiment)

Fig. 15

Mean image error. Formation reconfiguration maneuver (out-of-plane experiment)

Fig. 16

Relative distance between the spacecraft. Formation reconfiguration maneuver (out-of-plane experiment)

Fig. 17

Forces during the formation reconfiguration maneuver (out-of-plane experiment)

6.4 Formation Keeping Maneuver in LEO Orbits with Force Perturbations

Two additional test cases are detailed in this section to show the system behaviour in LEO orbits. In these experiments, the considered orbit semi-major axis is 6790 km, and two different spacecraft formations are considered: 6 spacecraft in a hexagonal formation with a fixed distance of 450 m between them and 3 spacecraft in a triangular formation with a fixed distance of 650 m between them. Taking into account the considered LEO orbit and the spacecraft properties, a maximum drag perturbation of $2966\times {10}^{-4}$ N is obtained. In order to evaluate the controller with environmental effects and unmodeled perturbations, an additional external perturbation is added. This perturbation is implemented as a sinusoidal function with a module of ${10}^{-3}$ N opposite to the spacecraft motion (for the hexagonal formation), and a sinusoidal function with a module of ${10}^{-3}$ N in random directions (for the triangular formation).

Figure 18 shows a 3D representation of both formations jointly with the view obtained from a given spacecraft. The direction of the z-axis for each camera is represented in blue. As in the previous experiments, four visual features are extracted from each observed spacecraft (corresponding with the leds on the observed spacecraft). Figure 18 also represents the camera view for the spacecraft 1 of each formation. The controller is required to maintain the formation of spacecraft throughout the entire 12-h duration of the experiment by compensating for any environmental perturbations that may occur. The value of the proportional and derivative matrices of the controller (see Eq. (19)) are set as ${\text{K}}_{pi}=0.001{\varvec{E}}$ and ${\text{K}}_{di}=0.05{\varvec{E}}$, respectively. Figure 19 shows the trajectories of the spacecraft in both the hexagonal formation (left) and triangular formation (right) cases, with respect to the LVLH frame. Figure 20 illustrates the image error, denoted as $\left|{{\varvec{e}}}_{si}\right|$, which is used to compute the control law for the spacecraft 1 of each formation. This error, defined in the image space, remains consistently low due the controller compensation of the perturbation effects. Figure 21 shows the relative distance between spacecraft 1 and the observed spacecraft for each formation. This distance is maintained around the desired one with minor adjustments to compensate for perturbations. This desired distance is 450 m for the hexagonal formation and 650 m for the triangular formation. Figure 22 shows the corresponding relative position error among the spacecraft for both the hexagonal and triangular formation cases in the LVLH coordinate frame, respectively. Finally, as an example, Fig. 23 displays the forces applied to the first spacecraft of each formation during the formation-keeping maneuver. Greater peaks in the triangular formation are obtained that can be observed more clearly in the z-direction.

Fig. 18

a) 3D representation of the hexagonal formation, and view from one spacecraft, b) 3D representation of the triangular formation, and view from one spacecraft

Fig. 19

Spacecraft trajectories with respect to the LVLH coordinate frame for the hexagonal formation (left) and triangular formation (right)

Fig. 20

Image error during the tracking for the hexagonal formation (left) and the triangular formation (right)

Fig. 21

Relative distance between two spacecraft in the hexagonal formation (left) and triangular formation (right)

Fig. 22

Relative position error among spacecraft in the hexagonal formation (left) and triangular formation (right)

Fig. 23

Forces during the formation keeping maneuver for one of the spacecraft in the hexagonal formation (left) and triangular formation (right)

7 Conclusions

This paper introduced an image-based method for spacecraft formation-keeping and reconfiguration maneuvers. The proposed algorithm is suitable for scenarios involving multiple spacecraft equipped with onboard cameras that observe a set of image features extracted from another spacecraft. To achieve computational efficiency and distributed control, an optimal control-based approach was utilized. The developed controller utilizes the input images from the onboard cameras to directly command specific maneuvers without the need for reconstructing the 3D position of the spacecraft, resulting in improved efficiency.

In addition, the paper presented the architecture of the various software modules proposed for the simulation of spacecraft formation based on ROS and Gazebo. This architecture enabled the simulation of complex space robotic systems, leveraging the numerous packages already developed in ROS for control, vision, and modelling.

The proposed optimal framework yielded four different controllers, as represented by Eqs. (28)–(31), each with distinct dynamic properties. These controllers enable tracking of a specified trajectory in the image space without requiring the computation of the spacecraft’s 3D position in the formation. Their effectiveness was assessed through three different scenarios, involving formation keeping and reconfiguration maneuvers. While all image-based controllers proved suitable for spacecraft guidance, the optimal controller’s weight matrix was found to be most effective when set equal to the inverse of the inertia matrix.

We are currently enhancing OnOrbitROS by introducing new elements that make the simulations more realistic. One noteworthy addition is incorporating additional disturbances, like implementing differential drag effects in the simulation.