1 Introduction

Martian dust storms have had a major impact on Mars exploration efforts. Planet-wide storms can bring exploration operations to a halt by reducing available power or obscuring view. This has serious implications for both the explore-ability and habitability of Mars. An understanding of the Martian dust cycle, with its connections between regional and global storms, thermal exchange between the surface and atmosphere, and the circulation that can eventually lead to atmospheric escape, remains a top scientific priority [1].

As dynamic, widespread phenomena, these processes are not only challenging to capture within their global context from the lower orbits that have dominated Mars orbiter missions, but also nearly impossible to capture in the required detail if higher orbits are to be assumed (e.g., Areostationary). Our goal in this paper is to find a middle ground using a constellation of small orbiters collaborating in low Mars orbit. We present initial feasibility studies of a constellation of small orbiters using medium-resolution cameras to track dust devils on the surface of Mars in real-time.

By using a combination of medium-resolution imagers, onboard processing, and cross-platform cueing, the constellation will enable short and long time-scale studies of dust devil formation, propagation, and global distribution over seasonal weather variations. We take, as inspiration, the recent NASA study on a multi-satellite network (MOSAIC) [1] for Mars.

We proceed as follows. We make use of a five satellite network where we assume that the five satellites are sufficiently small enough to be delivered as a secondary payload, perhaps on an ESPA ring accompanying a main mission[2]. The satellites are assumed to have basic telemetry and commanding interfaces direct to Earth, but otherwise relay their data through the main mission platform. We do not model the main satellite or nearby telecommunications relays, and simply assume there is “sufficient” capacity for this role. Note that MOSAIC [1] includes such relays for their mission concept, and significant literature points towards to the need and high likelihood of such relays being deployed after the retirement of the Mars Reconnaissance Orbiter[3, 4].

However, we do assume that the five satellites are equipped with telecom payloads to communicate among members of the network. The main use of this limited communication ability is to exchange information about the surface conditions between each observer and “cue” other members. There is a precedent for this concept of operation. For instance, it resembles the “A-train” configuration near Earth, but includes significantly more autonomy to cope with the long propagation delays between Earth and Mars [5]. Similarly, it also closely resembles the “sensor network” configuration, in that cues are sent among members of the network to schedule follow-up observations [6]. The total data volume exchanged over the network will be kept small so that communications is not the limiting factor of the design.

The members of the constellation are assumed to have medium-resolution framing imagers. The ground sample distance is assumed to be similar to the Mars Context Camera [7] (CTX). This setup (approximately 6 meters / pixel) allows the largest dust devils to be resolved [8]. For example, Fig. 1 shows a famous, large dust devil at CTX resolution.

Global coverage is achieved by “being vigilant” that is, taking significantly more imagery than what is usual. Details are preserved since the images are taken in low Mars orbit and with higher temporal frequency than usual. However, this introduces a data bottleneck: the constellation is gathering significantly more imagery than can be downlinked. As such, significant onboard processing is required to prepare downlink reports of surface activity, and onboard autonomy is required to schedule follow-up observations of the most promising dust devil candidates, as discussed in [9].

One way of automating the detection of dust devils is by using convolutional neural networks (CNNs). Testing of a basic, ResNet 50-based CNN trained with a few hundred labels shows that one can achieve Average Precision (AP) values of around 0.7 in CTX data (a detailed description is out of scope of this paper but forthcoming). The AP could be further optimized by applying a stricter score threshold value (optimize precision), which would reduce the recall, however. Initial testing with CTX data indicates that a CNN-based detector is fairly robust against varying geomorphic backgrounds (dunes, craters, etc.), image quality, and illumination conditions, while being able to detect small, large, faint, and distinct dust devils alike (see Fig. 2). Experience has shown that dust devils as small as \(\sim\)60 meters across can be detected and mapped in CTX images. In this work, we assume that a well trained classifier is available for use on-board the cubesats.

With onboard processing and modest cross-platform communication, we now present our core problem statement: how to position the constellation and schedule the observations to achieve maximum science return? The remainder of the paper focuses on designing an autonomous onboard scheduling system that trades exploration (i.e., look in areas that have not been photographed recently) versus exploitation (i.e., take another image of a known dust devil).

Fig. 1
figure 1

A CTX image and HiRISE zoom-in (inset) of a large dust devil in Amazonis Planitia. A framing version of CTX, likely well within today’s photo-imaging capabilities, may be able to record near-real-time imagery and track dust devil size, distribution, and shape over time when combined with onboard processing. Credit: NASA JPL, University of Arizona

Full size image
Fig. 2
figure 2

Performance illustration of an exemplary, CNN-based dust devil detector, operating on rectangular tiles of various sizes cropped from full-frame CTX images. A) & B) correct detections of small and large dust devils (white, TP = true positive); C) a dust devil missed by the detector (yellow, FN = false negative); D) false detections (red, FP = false positive). The network confidence score is displayed with its respective detection. The smallest (and faint) dust devil in this example is \(\sim\)100 meters across

Full size image

2 State of the Art

Automated tasking, as part of a ground-orbiting system, has a long history in space missions[10, 11], including successful demonstrations for earth observing spacecraft [12]. A more detailed review of autonomous agents for space exploration can be found in [13] and [14]. The state of the art is timeline-based, meaning exact subsystem sequences are derived for a single agent using constraints and resource-based cost functions [15] in a framework known as ASPEN. ASPEN is based on its previous generation, Continuous Activity Scheduling Planning Execution and Replanning (CASPER), and along with Eagle Eye, a specialized version of the software, the ASPEN scheduler generates a baseline mission operations plan from observation requests. The scheduler “greedily” schedules commands in a priority-first manner. A number of heuristics have been considered for scheduling, see e.g. [16,17,18,19], and [20] (for communication scheduling). In addition to generating an initial plan, ASPEN/Eagle Eye, steps through a plan execution phase, an on board image analysis phase, an onboard replanning phase, and a target reimaging phase. When rescheduling, the algorithm again follows a greedy approach and searches the current plan within the desired time window and replaces the earliest available observation that can be replaced with an updated observation request. Recent work (demonstrations of which are two decades in the making) has moved toward expressing mission goals and decomposing them into activities onboard the spacecraft [21].

An alternative approach to agile satellite tasking relies on mixed-integer programming; this approach has been proposed both for single satellites [22] and for satellite constellations [23, 24], including deployment on Planet’s Earth-observing constellation (albeit with ground-based planning) [24].

There are relatively few precedents for multi-agent variants of these planning and scheduling technologies, and even fewer examples of demonstrations, beyond the work in [24]. See [25] for an example of multi-rover autonomous operations and [26] for multi-aircraft / multi-spacecraft networks for earth observations, and [27] for recent work on constellation communication networks and tasking. A particularly close example is work by Nag et al. [28], which discusses the science benefit of a constellation for multi-angle simultaneous observations of the Earth atmosphere. In that work, Nag et al. optimize the constellation’s configuration to maximize science gain. However, onboard autonomy was not considered. In [29] an optimization framework is presented that can schedule multi-satellite constellations. However, the framework in [29] (like all related work mentioned) is optimized for quick convergence; in contrast, the work in this paper produces provably optimal schedules. We focus on analyzing the benefit of communications and onboard planning. We take as input, models of dust devil activity and a simplified model of the spacecraft dynamics and controls to allow a comprehensive, long-term study of the efficacy of the network.

It is worth noting that automated detection of dust devils on Mars is not a new concept. The work from Castano et al. [30] describes a dust-devil detection system that has been proposed in [30] for the Opportunity rover on Mars. The technique for autonomous targeting relevant surface science phenomena (including dust devils) is presented in [31]. In this work we focus on the multi-asset aspect of dust-devil detection and more specifically on the planning and coordination of observations.

Given the many decade history of automated planning and scheduling for space operations, our study takes as granted a framework for turning high-level commands into low-level control sequences, such as ASPEN [32], and the network design considerations discussed in [27].

3 Problem Statement

We formulate the constellation follow-up observation scheduling as follows.

We assume that dust devils occur across the Martian surface according to a stochastic process with known spatial distribution. Once a dust devil forms, its temporal duration and spatial evolution are also stochastic, with known distribution.

A constellation of spacecraft observes the Martian surface from orbit with cameras as described in the Sect. 1. Each spacecraft is equipped with a dust devil detector (with imperfect precision and recall) that can detect dust devils present in images and infer their location on the surface. The orbits are fixed; however, the spacecraft can decide where to point their cameras, subject to slewing constraints that enforce a maximum slewing rate.

Spacecraft can share their observations (specifically, the location of detected dust devils) through communication links when they are in line-of-sight contact of each other.

The goal of the problem is to maximize the number of follow-on observations of dust devils; specifically, we assume that it is more valuable to collect multiple observations of the same dust devil than to gather single observations of distinct dust devils, up to a maximum number of observations per event.

We formalize the scheduling problem as follows.

We define:

  • \(\mathcal {R}\), a set of regions of interest \(\mathcal {R} = \{r_{1},...,r_{R}\}\) on the surface of Mars where dust devils may occur; spacecraft observe these regions to look for previously-undetected dust devils. Each region should be imaged at most once over a given time window.

  • \(\mathcal {E}\), a set of observed transient events \({\mathcal {E} = e_{1}, \ldots , e_{E}}\) on the surface of Mars, modeling dust devils that have been detected and for which follow-up observations are desirable. Each event is associated with a given time window, which captures the expected duration of the dust devil. Within that window, the event should be observed as many times as possible.

  • \(H = (H_{start}, H_{end})\), a planning time horizon with start and end time points.

  • \(\mathcal {S}\), a set of spacecraft agents \(\mathcal {S} = \{a_{1},...,a_{N}\}\) orbiting Mars.

  • \(\mathcal {A}\), a set of attitudes that each spacecraft \(a \in \mathcal {S}\) can assume at every time step.

  • \(\mathcal {O}\), a set of observation opportunities \(\{o_{i,a,t}\}\) for agent \(i\in \mathcal {S}\) at time t in attitude a within the time \((H_{start},H_{end})\). For each observation opportunity, an observation opportunity function \((o_{i,a,t}) \mapsto (\mathcal {O} \cup \mathcal {E} \cup \emptyset )\) reports what regions and what transient events can be observed by agent \(a \in \mathcal {S}\) in attitude \(a \in \mathcal {A}\) at time \(t\in (H_{start},H_{end})\).

  • \(\mathcal {U} (r, o): {\mathcal {O} \cup \mathcal {E} , \mathcal {N}} \mapsto \mathbb {R}\), a scoring function that maps regions of interest \(r \in \mathcal {R}\) and transient events \(e\in \mathcal {E}\), and the number o of previous observations of the region or event of interest, to a score value. For regions \(\mathcal {R}\), the priority function encodes the expected number of new dust devils that will be observed there, multiplied by the value of a new observation; for events \(\mathcal {E}\), the scoring function denotes the scientific interest of collecting the o-th follow-on observation of the event.

Our objective is to produce an observation schedule by selecting \(\hat{O} \subseteq O\) to maximize \(\sum _{r \in R} U(r)\) subject to agents’ attitude constraints.

We are now in a position to formalize the constellation follow-up observation scheduling problem.

Problem 1

Find the set of observations \(\hat{O} \subseteq O\) that maximize \(\sum _{r \in R} U(r, o)\) subject to instrument constraints.

4 Autonomous Pointing for Science Planning

4.1 A Centralized ILP Algorithm

We start by presenting a centralized integer programming formulation to solve Problem 1.

We discretize the time horizon of the problem in equally-spaced intervals \(\mathcal {T} = [t_1, \ldots , t_T]\). We also discretize the set of attitudes that each spacecraft can point to into set \(\mathcal {A} = [k_1, \ldots , k_K]\) of discrete attitudes.

Pointing function

We assume that a pointing function \(p: (\mathcal {S}, \mathcal {A}, \mathcal {T})\mapsto \{\mathcal {R} \cup \mathcal {E} \cup \emptyset \}\) is known. The set p(i, a, t) represents the regions \(r\in \mathcal {R}\) and the known, ongoing transient events \(e\in \mathcal {E}\) that can be observed by spacecraft \(i\in \mathcal {S}\) in attitude \(a\in \mathcal {A}\) at time \(t\in \mathcal {T}\).

Slewing constraints

We also assume that the spacecraft cannot transition between arbitrary pairs of attitudes in one time step due to slewing constraints. For a given attitude \(k\in \mathcal {A}\), we denote the set of feasible prior attitudes (i.e., the set of attitudes from which it is possible to transition to P(k) in one time step) as P(k).

Variables

We define the following variables:

  • X(s, a, t), \({s}\in \mathcal {S}, a\in \mathcal {A}, t\in \mathcal {T}\), a set of Boolean variables. X(s, a, t) is 1 iff spacecraft s assumes attitude a at time t.

  • Y(s, a, t), \({s}\in \mathcal {S}, a\in \mathcal {A}, t\in \mathcal {T}\), a set of Boolean variables. X(s, a, t) is 1 iff spacecraft s captures data in attitude a at time t.

  • O(e, o), \(e\in \mathcal {R} \cup \mathcal {E}, o\in [1, \ldots , \overline{O}(e)]\), where \(\overline{O}(e)\) is the maximum number of observations of region or event e. A set of Boolean variables. O(e, o) is 1 iff region or event e is observed at least o times.

Problem Formulation

We are now in a position to formalize Problem 1 as a ILP.

$$\begin{aligned}&\underset{X, Y, O}{\max } \sum \limits _{e\in \mathcal {R}\cup \mathcal {E}} \sum \limits _{o\in [1, \ldots , \overline{O}(e)]} {U}(e,o) O(e,o) \end{aligned}$$
(1a)
$$\begin{aligned}&\text {subject to} \nonumber \\&O(e,o) \le O(e,o-1) \quad \quad \quad \quad \quad \quad \quad \quad \quad \forall e\in \mathcal {R} \cup \mathcal {E}, o\in [2, \ldots , \overline{O}(e)] \end{aligned}$$
(1b)
$$\begin{aligned}&\sum \limits _{o=1}^{\overline{O}(e)} O(e,o) = \sum \limits _{t\in \mathcal {T}} \sum \limits _{s\in \mathcal {S}} \sum \limits _{a\in \mathcal {A}} Y(s,a,t) \cdot \mathbf {1}_{e\in p(s,a,t)} \quad \forall e \in (\mathcal {R}\cup \mathcal {E}) \end{aligned}$$
(1c)
$$\begin{aligned}&Y(i,a,t) \le X(i,a,t) \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \forall i \in \mathcal {S}, a \in \mathcal {A}, t\in \mathcal {T} \end{aligned}$$
(1d)
$$\begin{aligned}&\sum \limits _{a\in \mathcal {A}} X(i,a,t) = 1 \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \forall i \in \mathcal {S}, t \in \mathcal {T} \end{aligned}$$
(1e)
$$\begin{aligned}&X(i,a,t+1) \le \sum \limits _{l\in P{a}} X(i,l,t) \quad \quad \quad \quad \quad \quad \forall i \in \mathcal {S}, \forall t \in \mathcal {T}\setminus t_T, \forall a \in \mathcal {A} \end{aligned}$$
(1f)

Equation 1a captures the reward for observing a given event or region for the o-th time. Equation 1b ensures that the o-th observation of a region or event if only performed after the \((o-1)\)-th observation. Equation 1c enforces that the overall number of observations of a region matches the number of times a spacecraft has captured an image pointing at the said region. Equation 1d guarantees that a spacecraft only captures an observation in a given attitude if it is indeed pointed in that attitude. Equation 1e ensures that each spacecraft assumes only one attitude at every time step. Finally, Eq. 1f ensures that the sequence of scheduled observations is compatible with the spacecrafts’ attitude slewing constraints.

4.2 Distributed Implementation

Solving Eqs. 1a-1f requires knowledge of all observed events, and the solution prescribes a set of attitudes for all spacecraft—therefore, the problem cannot directly be used to control a constellation where spacecraft are not in constant communication with each other and must make independent decisions based on their own observations. To overcome this, we propose a shared-world implementation of the ILP, shown in Fig. 3. Each spacecraft independently detects dust devils in its own observations and adds its detections to a local copy of a global event map. When two spacecraft are in contact, they reconcile their copies of the global event map by sharing their observations; if a conflict arises (e.g., if two spacecraft have observed the same region but only one has detected a dust devil), we optimistically assume that the detection was correct. In the following section, we discuss our selection of a Sun synchronous orbit in order to ensure that the lighting conditions (shadow angles and shadow lengths) are the same for every pass, thus reducing false positive detections. Each spacecraft individually solves Eqs. 1a-1f based on its local copy of the global event map, obtaining a set of prescribed attitudes for the entire constellation. Each spacecraft then executes the pointing commands prescribed to itself by the ILP, and disregards the pointing commands for other spacecraft.

With this approach, if the spacecrafts’ global events map are perfectly synchronized, all spacecraft act in accordance with the same solution to Eqs. 1a-1f. If the global events maps are not synchronized (because, e.g., information about a detected event has not yet been propagated to all spacecraft), each spacecraft is guaranteed to have access to a solution, even though the spacecraft may act in an inconsistent manner (e.g., a region may be imaged twice, or it may not be imaged because each spacecraft thinks that another one will capture it). In the Numerical Results, we show that, for the orbits and network topologies considered in this paper, information propagates readily through the constellation, and the impact of imperfect synchronization between the spacecrafts’ global event maps is negligible.

Fig. 3
figure 3

Architecture of the proposed constellation scheduler. (A) Each spacecraft captures images of the Martian surface. A dust devil detector searches for dust devils in captured imagery. (B) When a dust devil is detected, its location is added to a map database which tracks possible active dust devils that should be targeted for revisit. Each spacecraft executes an instance of the constellation planner (described in the Sect. 4), which uses the locations of the detected dust devils to prescribe pointing locations for every spacecraft. (C) The spacecraft then executes the pointing and image capture commands assigned to itself by the constellation planner, and discards the assignments for other spacecraft. The map database of detected dust devils is opportunistically synchronized across spacecraft when a communication link is available

Full size image

4.3 Receding-horizon Approach

Obtaining a solution to Problem 1 requires knowledge of observed dust devils: therefore, the problem should be re-solved whenever a new dust devil is observed. To address this, we implement the algorithm in a receding-horizon fashion. Each spacecraft solves Eqs. 1a-1f within a fixed time horizon – in the simulation results, we use a fifteen-minute time horizon. The selection of the time horizon represents a trade-off between computational performance and optimality of the resulting schedule; specifically, the fifteen-minute time horizon is sufficient to plan slewing motions from any pointing attitude to any other pointing attitude, and allows spacecraft to plan revisits of dust devils detected by multiple spacecraft ahead of them in a train; at the same time, the optimization problem can consistently be solved in under 100 ms, as discussed in Sect. 5.5. As soon as new information is received (either through a direct observation, or through an update to the global event map provided by another spacecraft), the spacecraft re-solves the problem with the same horizon (that is, for the fifteen minutes following the update). Also, before the horizon is reached, the problem is solved again, irrespective of whether new information is available. The receding horizon approach ensures that the computation cost is highly contained (as shown in the Sect. 5), while incorporating new information in the solution as soon as it becomes available.

5 Numerical Simulation & Results

5.1 Dust Devil Model

Dust devils form around low-pressure air pockets, where the air is drawn into a narrow rising column through the surrounding cooler air. A study that combined data from the High Resolution Stereo Camera (HRSC) and the Mars Orbiter Camera (MOC) found that some large dust devils on Mars have a diameter of 700 m and last for at least 26 minutes [8]. The largest dust devils can reach a height of 8 km. A recent study found that on average, during any given day, one 13 m wide dust devil appears in every square kilometer on Mars [33].

As a dust devil moves across the surface of Mars, it lifts the top layer of dust into the atmosphere and exposes the dark underlying surface. These dark tracks can last a few weeks, at which point they are either covered up as a result of wind action or they become oxidized from exposure to sunlight and the Martian atmosphere, and thus return to same red color as the rest of the planet. Dust devil tracks can be more than 30 m wide and extend for more than 4 km.

In our simulations, we use a simple random model for dust devil generation. We assume that no dust devils exist at latitudes greater than \(70^{\circ }\) and no dust devils exist before 11 am or after 4 pm local solar time. We center the active dust devil region at 1:30 pm local solar time. Over this region, we simulate dust devils with an average density of one per ten square kilometers. Though this is a more conservative estimate than those found in the literature [33], it is adequate to test our onboard scheduler. In our simulations, the mean dust devil lifetime is 26 minutes and start and end times are computed for each dust devil. Furthermore, we have not taken into account the motion of the dust devils across the surface and their associated ground track. This will be left as future work, along with a more sophisticated dust devil model.

5.2 Orbits

In this paper we propagate two-body orbits for a train of five cubesats in a low Mars (200 km altitude), Sun synchronous orbit. Two different train constellations are considered. The first has satellites in the train separated by one minute time intervals, and the second uses six minute intervals. We have selected a low Mars orbit such that the cubesat camera (for example, HiREV [34]) is capable of imaging the surface with a resolution of less than 10 m. A Sun synchronous orbit is chosen to ensure that the lighting conditions (shadow angles and shadow lengths) are the same for every pass. Although we do not simulate dust devil event detection through image processing in this paper, we assume that using the same lighting conditions for each pass will aid the eventual dust devil event detection algorithm—thus reducing the potential for false positives. Each cubesat camera has a field of view with \(2.5^\circ\) boresight angle [34]. For these simulations, we have selected twenty-one camera positions (Fig. 4), spaced by approximately two to three degrees apart in the along-track direction (\(0, \pm 3, \pm 5,\) and \(\pm 7.5^\circ\)) and five degrees apart in the across-track direction (\(0, \pm 5^\circ\)). We assume that the slew/settle time to move between adjacent camera positions is one minute [35].

Figure 5 shows a snapshot taken during the simulation where the cubesats in the train are separated by one minute intervals. The top panel shows a spherical Mars that is experiencing a southern hemisphere summer. Each yellow dot on the surface represents an active dust devil. The low-Mars orbit is also shown in cyan/red, along with the ground track. Note that the five satellites are colored red, blue, green, black and cyan respectively. The magenta circles represent currently detected dust devils. The bottom panel shows a 2D projection of Mars, along with the same information plotted as in the top panel.

Figure 6 shows an enlarged view for six snapshots taken at one minute time intervals. To prevent clutter in these figures, the ground coverage bounding box is shown for just 5 of the 21 possible camera positions. Of course, in reality, the camera can only image one of these regions at a time. Looking at the top-left panel, if the lead satellite (red) was pointing \(5^{\circ }\) ahead of nadir (i.e. camera position 5), it would detect one dust devil event (yellow dot with magenta circle), and if it was pointing \(5^{\circ }\) behind nadir (i.e. camera position 17) it would detect two dust devil events. The next snapshot (top-right) shows that one minute later the second satellite (blue) is capable of imaging the dust devils that were previously detected by the lead satellite one minute earlier. The remaining panels show the satellite train moving further along in its orbit as each minute passes.

5.3 Field of View

The camera field of view is modelled as a square and the corresponding observable surface of Mars is computed as the region enclosed within the four corners of this square projected onto the surface of a spherical Mars. A dust devil is detected if it falls within this visible surface element on Mars. In our simulations, all the orbits are sampled at one minute time intervals. Thus, if the lead spacecraft detects an event in, say, camera position 1, it can communicate the detection to the second spacecraft, whose onboard scheduler can decide to slew from its current camera position to camera position 1 in order to re-image the dust devil event one minute later — thus enabling the dust devil time history to be recorded. Similarly, the second spacecraft can communicate to the following one so its scheduler can, in turn, command a slew to the same position, and so on. If the lead spacecraft detects a dust devil event and the second spacecraft also detects an event, then the onboard scheduler, running in real-time, will determine which positions the following spacecraft should slew to in order to maximize the total number of follow-on dust devil observations.

5.4 Communication

In our simulations, we assume that each cubesat is equipped with a patch antenna. For this orbit geometry (200 km altitude), with the one minute separation between satellites, all elements of the chain are constantly within line-of-sight communication, thus enabling information to be transferred between the satellites at any time. In the case where satellites are separated by six minutes, the line-of-sight, and hence the communication link, only extends to the satellite immediately in front and behind of it. We assume that communication between satellites within line-of-sight of each other takes 60 s (one simulation step), a highly conservative estimate for the data volumes considered in this work. We assess the performance of the proposed approach in both scenarios.

Fig. 4
figure 4

Schematic showing the 21 possible field of view regions that the camera could point to in order to observe dust devils. Position 11 is nadir pointing

Full size image
Fig. 5
figure 5

A snapshot taken during the simulation that shows the five sun synchronous, low-Mars orbiting satellites, flying over a region of dust devil activity (yellow dots) during a southern hemisphere summer. The top panel shows a spherical Mars and the bottom panel shows a two-dimensional projection. The respective orbits and ground tracks are also shown

Full size image
Fig. 6
figure 6

A series of six snapshots taken at one minute intervals that show the instantaneous ground coverage for each of the five cubesats in the chain. Note that all five camera slew positions are shown simultaneously in this figure. Yellow dots represent active dust devils and magenta circles represent possible detections

Full size image

5.5 Results

Comparison with upper and lower bounds

First, we compare the performance of the proposed constellation scheduling approach with an “oracle” upper bound and a naive lower bound.

The oracle solves Eqs. 1a-1f once with full knowledge of the location of all present and future dust devils. The problem is solved in a centralized manner. The solution provided by the oracle represents an (unattainable) upper bound on the performance of the constellation. The lower bound is obtained by simply setting the attitude of all spacecraft to be in a nadir-pointing direction at all times.

The two scenarios are compared with the outcome of the simulation where each spacecraft solves Eqs. 1a-1f in a receding-horizon fashion, as described in the Autonomous Pointing section, and spacecraft exchange observations along available communication links as discussed in the Communication section. To ensure a fair comparison, all spacecraft are assumed to have access to an ideal event detector with perfect precision and recall (that, is, it never produces false positives or negatives); the effect on performance of a non-ideal event detector is discussed in the next section.

The results are reported in Table 1 and Fig. 7 for a set of orbits with one minute spacing between spacecraft, and in Table 2 and Fig. 8 for orbits with six minute spacing.

In the widely-spaced case, the proposed approach results in a 230% increase in the number of events that receive at least one follow-up observation compared to the nadir-pointing approach. The average number of follow-up observations per event increases from 0.21 to 0.77, an almost four-fold increase. The overall number of observations is 20% higher compared to the nadir-pointing case. In contrast, the number of unique events detected decreases by 18% as expected, the autonomous pointing approach privileges follow-on observations as opposed to new event detections, in line with the proposed reward function, which privileges revisits over new detections.

The impact of on-board autonomy is smaller, but still significant, in the closely-spaced case: the approach is able to increase the number of events observed three or more times from zero to 53, and the average number of follow-up observations increases by 15%, at the price of a 9.3% reduction in the number of unique events observed.

Fig. 7
figure 7

Closely-spaced orbits: distribution of the number of follow-on observations for the proposed approach, compared to an upper and lower bound. The effect of a non-ideal event detector are also shown

Full size image
Table 1 Performance of the proposed distributed pointing approach compared to an “oracle” upper bound and a fixed-attitude lower bound
Full size table
Fig. 8
figure 8

Widely-spaced orbits: Distribution of the number of follow-on observations for the proposed approach, compared to an upper and lower bound

Full size image
Fig. 9
figure 9

Widely-spaced orbits: Effect of communications on distribution of the number of follow-on observations

Full size image
Fig. 10
figure 10

Widely-spaced orbits: Effect of non-ideal detector on distribution of the number of follow-on observations

Full size image
Table 2 Performance of the proposed distributed pointing approach compared to an “oracle” upper bound and a fixed-attitude lower bound
Full size table

Effect of inter-spacecraft communication

As discussed in the Sect. 4.2, the distributed implementation relies on inter-spacecraft communication to maintain a common picture of observed events; this can result in sub-optimal, and potentially inconsistent, behavior at the constellation level if inter-spacecraft links are sporadic or if communication between spacecrafs requires multiple hops.

To explore the impact of inter-spacecraft communications on system performance, we compare the proposed approach, where spacecraft communicate with their neighbors when a line-of-sight link is available, with two scenarios: an idealized scenario where all spacecraft can exchange information instantly, and a no-communications scenario where each spacecraft only relies on information it has collected itself. We focus our attention on the widely-spaced scenario, where each spacecraft can only communicate with its immediate neighbors in the constellation.

Results are shown in Table 2 and Fig. 9.

There is no difference in performance between the proposed approach and the upper bound where all spacecraft can communicate with each other, showing that the proposed approach is able to effectively spread information about event detections across the constellation. In contrast, the case where satellites do not communicate with each other results in performance comparable to the nadir-pointing case, with a small reduction in the average number of follow-up observations (vs. the almost four-fold improvement obtained with communications). Collectively, these results show that the proposed distributed optimization approach is able to approach the performance of a centralized controller, and that constellation-level optimization through inter-satellite communication is critical to achieve this performance.

Non-ideal event detector

We also assess the impact of using a non-ideal event detector with non-perfect precision and recall. Specifically, we simulate the performance of a detector with 70% recall, i.e., a 30% false negative rate. Due to limitations of the simulation environment, the study of the effect of non-ideal precision is left as future work. Results are shown in Tables 1 and 2 and in Figs. 7 and 10.

Predictably, the imperfect recall results in a roughly 30% reduction in the overall number of observations collected. Specifically, we observe 15% and 19% reductions in the number of unique detected events in the closely-spaced and widely-spaced cases respectively; the reduction in the average number of follow-up images collected for each event is 31% in both the closely-spaced case and the widely-spaced case. Compared to a nadir-pointing case with imperfect recall, the proposed approach performs similarly to the perfect case, with a two-fold to fourfold increase in the number of events with at least two observations, and a threefold to five-fold increase in the average number of follow-up observations per event, in the widely-spaced case; and only slightly better performance compared to the nadir-pointing case in the closely-spaced case.

Computational performance

The ILP was solved with the CPLEX solver. The time required to formulate the problem (i.e., encode the cost function and constraints) and solve it on a commodity desktop workstation is shown in Fig. 11. The time required to formulate the problem remains below 500 ms except for rare outliers, and the time required to solve it is consistently always below 100 ms - results which suggest that the proposed approach could be readily implemented on more modest CPUs well-suited for spaceflight.

Fig. 11
figure 11

Computation times to solve the proposed ILP on a Xeon E5-2687W

Full size image

6 Visualization System

To visualize and interact with the simulation, we designed a visualization system that adheres to the same mission design methodology as discussed in the previous sections. Past work shows interfaces [36,37,38] are critical to (i) foster trust in the data and tool from the operator’s perspective, and (ii) understand the details of what the scheduler is doing onboard. Interactions provide a dialog between the user and the system [39], and this allows operators to gain confidence in the system as they explores datasets to uncover insights and be part of the decision making process. To allow the operator to explore the onboarding scheduler, the system is composed of three interactive views: overview, coverage map, and time scrubber (Fig. 12). All views are interlinked: the overview and coverage map visualize data corresponding to the particular timestamp from the time scrubber.

Fig. 12
figure 12

A visualization system (a) Side-bar menu with options to change the map projection and orientation (b) Coverage map depicts the ground track of the constellation (c) Overview provides the state of the constellation and dust devils (d) Counters for the total number of dust devil events and dust devil observations (e) Time scrubber depicts the current time position within the simulation

Full size image

Overview

The overview (Fig. 12b) provides a quick assessment of the constellation performance in terms of dust devil observations. Dust devils are represented as brown triangles, which appear and disappear throughout the simulation run. A counter (Fig. 12d) helps count the total number of dust devil events. In contrast, spacecraft are represented by their bounding boxes and spacecraft number. As mentioned in the Sect. 5.3, these bounding boxes are determined from the 21 slew positions each spacecraft is capable of achieving (only five are shown in the visualization in Fig. 12). All bounding boxes are mapped with two visual encodings. First, for each spacecraft, a bounding box has solid edges if it represents the spacecraft’s current camera position, while the rest are represented with thin dashed edges. Second, a bounding box is colored red if a spacecraft has observed one or more dust devils in its field of view at a particular time point (e.g., Spacecraft 4 has observed a dust devil at \(t = 32580\)). Otherwise, the color remains transparent. The visual encoding for the dust devil follows a similar logic, where the dust devil is colored red if it has been observed and remains brown otherwise. To help operators keep track of the simulation, a second counter (Fig. 12d) counts the total number of dust devil events the constellation has observed.

The combination of these visual encodings allows the operator to not only visually see when and where spacecraft have observed dust devils, but also highlights where the vehicles could have observed a dust devil event (i.e., a dust devil is within the field of view of one of the bounding boxes that are not in use).

The overview also visualizes the terminator to showcase the daylit side (white illuminated region) and the night side (solid dark grey filled-in region) of Mars throughout the simulation run. This corresponds to the timer (unit in hours) at the top of the overview. The default map projection is Kavrayskiy VII, but operators can change different map projections as well as interactively zoom-in, rotate, and pan using the side-bar menu (Fig. 12a). Operators can also see details on-demand (i.e., via an interactive tooltip).

Coverage Map

The coverage map (Fig. 12b) dynamically visualizes the ground track of the constellation throughout the simulation (\(t = 0\) to \(t = 88320\)). The ground track is colored dark steel-blue, and the path corresponds to the movement of the spacecraft in the overview (Fig. 12b). The coverage map provides a visual insight as to which regions the constellation has frequently visited and vice versa. This can help operators compare different constellation formations, and, in particular, different inter-spacecraft spacing.

Time Scrubber

The time scrubber (Fig. 12e) is a horizontal slider control that lets the operator dynamically seek any time position within the currently running simulation. The scrubber ‘knob’ (circle) moves to indicate the current time position. The position of the scrubber knob determines the information being displayed in the overview and coverage map. Operators can also pause and play when analyzing the simulation.

7 Conclusions

Several million dust devil events occur on Mars every day. These events last, on average, about 30 minutes and range in size from meters to hundreds of meters in diameter. Designing low-cost missions that will improve our knowledge of dust devil formation and evolution, and their connection to atmospheric dynamics and the dust cycle, is fundamental to informing future crewed Mars lander missions about surface conditions. In this paper we presented a mission for a constellation of low orbiting Mars cubesats, each carrying imagers with agile pointing capabilities. The goal was to maximize the number of dust devil follow-up observations through real-time, onboard scheduling. Realistic inter-agent communication link opportunities were computed and included in our simulation, which allowed for real-time event detection information to be shared within the constellation. In particular, we found that the communications are especially important for acquiring follow-up observations, and that the proposed distributed scheduling mechanism is able to capture a large fraction of all dust devil observations that are possible for a given orbit configuration, significantly outperforming a nadir-pointing heuristic. This is a significant result that will aid in the study of dust devil science on the surface of Mars, in preparation for future crewed missions.