Maximizing Dust Devil Follow-Up Observations on Mars Using Cubesats and On-Board Scheduling

Several million dust devil events occur on Mars every day. These events last, on average, about 30 minutes and range in size from meters to hundreds of meters in diameter. Designing low-cost missions that will improve our knowledge of dust devil formation and evolution, and their connection to atmospheric dynamics and the dust cycle, is fundamental to informing future crewed Mars lander missions about surface conditions. In this paper we present a mission for a constellation of low orbiting Mars cubesats, each carrying imagers with agile pointing capabilities. The goal is to maximize the number of dust devil follow-up observations through real-time, on-board scheduling. We study scenarios where cubesats are equipped with a 2.5 degree boresight angle camera that accommodates twenty-one slew positions (including nadir). We assume a concept of operations where the cubesats autonomously survey the surface of Mars and can autonomously detect dust devils from their surface imagery. When a dust devil is detected, the constellation is autonomously re-tasked through an onboard distributed scheduler to capture as many follow-on images of the event as possible, so as to study its evolution. The cubesat orbits are propagated assuming two-body dynamics and the ground tracks and camera field of view are computed assuming a spherical Mars. Realistic inter-agent communication link opportunities are computed and included in our optimization, which allow for real-time event detection information to be shared within the constellation. We compare against a powerful “omniscient” oracle which has a priori knowledge of all dust devil activity to show the gap between predicted performance and the best possible outcome. In particular, we show that the communications are especially important for acquiring follow-up observations, and that a realistic distributed scheduling mechanism is able to capture a large fraction of all dust devil observations that are possible for a given orbit configuration, significantly outperforming a nadir-pointing heuristic.

Mars Context Camera [7] (CTX). This setup (approximately 6 meters / pixel) allows the largest dust devils to be resolved [8]. For example, Fig. 1 shows a famous, large dust devil at CTX resolution. Global coverage is achieved by "being vigilant" that is, taking significantly more imagery than what is usual. Details are preserved since the images are taken in low Mars orbit and with higher temporal frequency than usual. However, this introduces a data bottleneck: the constellation is gathering significantly more imagery than can be downlinked. As such, significant onboard processing is required to prepare downlink reports of surface activity, and onboard autonomy is required to schedule follow-up observations of the most promising dust devil candidates, as discussed in [9].
One way of automating the detection of dust devils is by using convolutional neural networks (CNNs). Testing of a basic, ResNet 50-based CNN trained with a few hundred labels shows that one can achieve Average Precision (AP) values of around 0.7 in CTX data (a detailed description is out of scope of this paper but forthcoming). The AP could be further optimized by applying a stricter score threshold value (optimize precision), which would reduce the recall, however. Initial testing with CTX data indicates that a CNN-based detector is fairly robust against varying geomorphic backgrounds (dunes, craters, etc.), image quality, and illumination conditions, while being able to detect small, large, faint, and distinct dust devils alike (see Fig. 2). Experience has shown that dust devils as small as ∼ 60 meters across can be detected and mapped in CTX images. In this work, we assume that a well trained classifier is available for use on-board the cubesats.
With onboard processing and modest cross-platform communication, we now present our core problem statement: how to position the constellation and schedule the observations to achieve maximum science return? The remainder of the paper focuses on designing an autonomous onboard scheduling system that trades Fig. 1 A CTX image and HiRISE zoom-in (inset) of a large dust devil in Amazonis Planitia. A framing version of CTX, likely well within today's photo-imaging capabilities, may be able to record near-realtime imagery and track dust devil size, distribution, and shape over time when combined with onboard processing. Credit: NASA JPL, University of Arizona exploration (i.e., look in areas that have not been photographed recently) versus exploitation (i.e., take another image of a known dust devil).

State of the Art
Automated tasking, as part of a ground-orbiting system, has a long history in space missions [10,11], including successful demonstrations for earth observing spacecraft [12]. A more detailed review of autonomous agents for space exploration can be found in [13] and [14]. The state of the art is timeline-based, meaning exact subsystem sequences are derived for a single agent using constraints and resourcebased cost functions [15] in a framework known as ASPEN. ASPEN is based on its previous generation, Continuous Activity Scheduling Planning Execution and Replanning (CASPER), and along with Eagle Eye, a specialized version of the software, the ASPEN scheduler generates a baseline mission operations plan from observation requests. The scheduler "greedily" schedules commands in a priorityfirst manner. A number of heuristics have been considered for scheduling, see e.g. [16][17][18][19], and [20] (for communication scheduling). In addition to generating an initial plan, ASPEN/Eagle Eye, steps through a plan execution phase, an on board image analysis phase, an onboard replanning phase, and a target reimaging phase. . The network confidence score is displayed with its respective detection. The smallest (and faint) dust devil in this example is ∼100 meters across When rescheduling, the algorithm again follows a greedy approach and searches the current plan within the desired time window and replaces the earliest available observation that can be replaced with an updated observation request. Recent work (demonstrations of which are two decades in the making) has moved toward expressing mission goals and decomposing them into activities onboard the spacecraft [21].
An alternative approach to agile satellite tasking relies on mixed-integer programming; this approach has been proposed both for single satellites [22] and for satellite constellations [23,24], including deployment on Planet's Earth-observing constellation (albeit with ground-based planning) [24].
There are relatively few precedents for multi-agent variants of these planning and scheduling technologies, and even fewer examples of demonstrations, beyond the work in [24]. See [25] for an example of multi-rover autonomous operations and [26] for multi-aircraft / multi-spacecraft networks for earth observations, and [27] for recent work on constellation communication networks and tasking. A particularly close example is work by Nag et al. [28], which discusses the science benefit of a constellation for multi-angle simultaneous observations of the Earth atmosphere. In that work, Nag et al. optimize the constellation's configuration to maximize science gain. However, onboard autonomy was not considered. In [29] an optimization framework is presented that can schedule multi-satellite constellations. However, the framework in [29] (like all related work mentioned) is optimized for quick convergence; in contrast, the work in this paper produces provably optimal schedules. We focus on analyzing the benefit of communications and onboard planning. We take as input, models of dust devil activity and a simplified model of the spacecraft dynamics and controls to allow a comprehensive, long-term study of the efficacy of the network.
It is worth noting that automated detection of dust devils on Mars is not a new concept. The work from Castano et al. [30] describes a dust-devil detection system that has been proposed in [30] for the Opportunity rover on Mars. The technique for autonomous targeting relevant surface science phenomena (including dust devils) is presented in [31]. In this work we focus on the multi-asset aspect of dust-devil detection and more specifically on the planning and coordination of observations. Given the many decade history of automated planning and scheduling for space operations, our study takes as granted a framework for turning high-level commands into low-level control sequences, such as ASPEN [32], and the network design considerations discussed in [27].

Problem Statement
We formulate the constellation follow-up observation scheduling as follows.
We assume that dust devils occur across the Martian surface according to a stochastic process with known spatial distribution. Once a dust devil forms, its temporal duration and spatial evolution are also stochastic, with known distribution.
A constellation of spacecraft observes the Martian surface from orbit with cameras as described in the Sect. 1. Each spacecraft is equipped with a dust devil detector (with imperfect precision and recall) that can detect dust devils present in images and infer their location on the surface. The orbits are fixed; however, the spacecraft can decide where to point their cameras, subject to slewing constraints that enforce a maximum slewing rate.
Spacecraft can share their observations (specifically, the location of detected dust devils) through communication links when they are in line-of-sight contact of each other.
The goal of the problem is to maximize the number of follow-on observations of dust devils; specifically, we assume that it is more valuable to collect multiple observations of the same dust devil than to gather single observations of distinct dust devils, up to a maximum number of observations per event.
We formalize the scheduling problem as follows.
We define: Our objective is to produce an observation schedule by selecting Ô ⊆ O to maximize ∑ r∈R U(r) subject to agents' attitude constraints. We are now in a position to formalize the constellation follow-up observation scheduling problem.

Problem 1 Find the set of observations Ô ⊆ O that maximize
∑ r∈R U(r, o) subject to instrument constraints.

A Centralized ILP Algorithm
We start by presenting a centralized integer programming formulation to solve Problem 1.
We discretize the time horizon of the problem in equally-spaced intervals T = [t 1 , … , t T ] . We also discretize the set of attitudes that each spacecraft can point to into set A = [k 1 , … , k K ] of discrete attitudes.

Pointing function
We assume that a pointing function p ∶ (S, A, T) ↦ {R ∪ E ∪ �} is known. The set p(i, a, t) represents the regions r ∈ R and the known, ongoing transient events e ∈ E that can be observed by spacecraft i ∈ S in attitude a ∈ A at time t ∈ T .

Slewing constraints
We also assume that the spacecraft cannot transition between arbitrary pairs of attitudes in one time step due to slewing constraints. For a given attitude k ∈ A , we denote the set of feasible prior attitudes (i.e., the set of attitudes from which it is possible to transition to P(k) in one time step) as P(k).

Variables
We define the following variables:

Problem Formulation
We are now in a position to formalize Problem 1 as a ILP.

Distributed Implementation
Solving Eqs. 1a-1f requires knowledge of all observed events, and the solution prescribes a set of attitudes for all spacecraft-therefore, the problem cannot directly be used to control a constellation where spacecraft are not in constant communication with each other and must make independent decisions based on their own observations. To overcome this, we propose a shared-world implementation of the ILP, shown in Fig. 3. Each spacecraft independently detects dust devils in its own observations and adds its detections to a local copy of a global event map. When two spacecraft are in contact, they reconcile their copies of the global event map by sharing their observations; if a conflict arises (e.g., if two spacecraft have observed the same region but only one has detected a dust devil), we optimistically assume that the detection was correct. In the following section, we discuss our selection of a Sun synchronous orbit in order to ensure that the lighting conditions (shadow angles and shadow lengths) are the same for every pass, thus reducing false positive detections. Each spacecraft individually solves Eqs. 1a-1f based on its local copy of the global With this approach, if the spacecrafts' global events map are perfectly synchronized, all spacecraft act in accordance with the same solution to Eqs. 1a-1f. If the global events maps are not synchronized (because, e.g., information about a detected event has not yet been propagated to all spacecraft), each spacecraft is guaranteed to have access to a solution, even though the spacecraft may act in an inconsistent manner (e.g., a region may be imaged twice, or it may not be imaged because each spacecraft thinks that another one will capture it). In the Numerical Results, we show that, for the orbits and network topologies considered in this paper, information propagates readily through the constellation, and the impact of imperfect synchronization between the spacecrafts' global event maps is negligible.

Receding-horizon Approach
Obtaining a solution to Problem 1 requires knowledge of observed dust devils: therefore, the problem should be re-solved whenever a new dust devil is observed. To address this, we implement the algorithm in a receding-horizon fashion. Each spacecraft solves Eqs. 1a-1f within a fixed time horizon -in the simulation results, we use a fifteen-minute time horizon. The selection of the time horizon represents a trade-off between computational performance and optimality of the resulting schedule; specifically, the fifteen-minute time horizon is sufficient to plan slewing motions from any pointing attitude to any other pointing attitude, and allows spacecraft to plan revisits of dust devils detected by multiple spacecraft ahead of them in a train; at the same time, the optimization problem can consistently be solved in under 100 ms, as discussed in Sect. 5.5. As soon as new information is received (either through a direct observation, or through an update to the global event map provided by another spacecraft), the spacecraft re-solves the problem with the same horizon (that is, for the fifteen minutes following the update). Also, before the horizon is reached, the problem is solved again, irrespective of whether new information is available. The receding horizon approach ensures that the computation cost is highly contained (as shown in the Sect. 5), while incorporating new information in the solution as soon as it becomes available.

Dust Devil Model
Dust devils form around low-pressure air pockets, where the air is drawn into a narrow rising column through the surrounding cooler air. A study that combined data from the High Resolution Stereo Camera (HRSC) and the Mars Orbiter Camera (MOC) found that some large dust devils on Mars have a diameter of 700 m and last for at least 26 minutes [8]. The largest dust devils can reach a height of 8 km. A recent study found that on average, during any given day, one 13 m wide dust devil appears in every square kilometer on Mars [33].
As a dust devil moves across the surface of Mars, it lifts the top layer of dust into the atmosphere and exposes the dark underlying surface. These dark tracks can last a few weeks, at which point they are either covered up as a result of wind action or they become oxidized from exposure to sunlight and the Martian atmosphere, and thus return to same red color as the rest of the planet. Dust devil tracks can be more than 30 m wide and extend for more than 4 km.
In our simulations, we use a simple random model for dust devil generation. We assume that no dust devils exist at latitudes greater than 70 • and no dust devils exist before 11 am or after 4 pm local solar time. We center the active dust devil region at 1:30 pm local solar time. Over this region, we simulate dust devils with an average density of one per ten square kilometers. Though this is a more conservative estimate than those found in the literature [33], it is adequate to test our onboard scheduler. In our simulations, the mean dust devil lifetime is 26 minutes and start and end times are computed for each dust devil. Furthermore, we have not taken into account the motion of the dust devils across the surface and their associated ground track. This will be left as future work, along with a more sophisticated dust devil model.

Orbits
In this paper we propagate two-body orbits for a train of five cubesats in a low Mars (200 km altitude), Sun synchronous orbit. Two different train constellations are considered. The first has satellites in the train separated by one minute time intervals, and the second uses six minute intervals. We have selected a low Mars orbit such that the cubesat camera (for example, HiREV [34]) is capable of imaging the surface with a resolution of less than 10 m. A Sun synchronous orbit is chosen to ensure that the lighting conditions (shadow angles and shadow lengths) are the same for every pass. Although we do not simulate dust devil event detection through image processing in this paper, we assume that using the same lighting conditions for each pass will aid the eventual dust devil event detection algorithm-thus reducing the potential for false positives. Each cubesat camera has a field of view with 2.5 • boresight angle [34]. For these simulations, we have selected twenty-one camera positions (Fig. 4), spaced by approximately two to three degrees apart in the along-track direction ( 0, ±3, ±5, and ±7.5 • ) and five degrees apart in the across-track direction ( 0, ±5 • ). We assume that the slew/settle time to move between adjacent camera positions is one minute [35]. Figure 5 shows a snapshot taken during the simulation where the cubesats in the train are separated by one minute intervals. The top panel shows a spherical Mars that is experiencing a southern hemisphere summer. Each yellow dot on the surface represents an active dust devil. The low-Mars orbit is also shown in cyan/red, along with the ground track. Note that the five satellites are colored red, blue, green, black and cyan respectively. The magenta circles represent currently detected dust devils. The bottom panel shows a 2D projection of Mars, along with the same information plotted as in the top panel.

Field of View
The camera field of view is modelled as a square and the corresponding observable surface of Mars is computed as the region enclosed within the four corners of this square projected onto the surface of a spherical Mars. A dust devil is detected if it falls within this visible surface element on Mars. In our simulations, all the orbits are  sampled at one minute time intervals. Thus, if the lead spacecraft detects an event in, say, camera position 1, it can communicate the detection to the second spacecraft, whose onboard scheduler can decide to slew from its current camera position to camera position 1 in order to re-image the dust devil event one minute later -thus enabling the dust devil time history to be recorded. Similarly, the second spacecraft can communicate to the following one so its scheduler can, in turn, command a slew to the same position, and so on. If the lead spacecraft detects a dust devil event and the second spacecraft also detects an event, then the onboard scheduler, running in realtime, will determine which positions the following spacecraft should slew to in order to maximize the total number of follow-on dust devil observations.

Communication
In our simulations, we assume that each cubesat is equipped with a patch antenna. For this orbit geometry (200 km altitude), with the one minute separation between satellites, all elements of the chain are constantly within line-of-sight communication, thus enabling information to be transferred between the satellites at any time.
In the case where satellites are separated by six minutes, the line-of-sight, and hence the communication link, only extends to the satellite immediately in front and behind of it. We assume that communication between satellites within line-of-sight of each other takes 60 s (one simulation step), a highly conservative estimate for the data volumes considered in this work. We assess the performance of the proposed approach in both scenarios.

Results
Comparison with upper and lower bounds First, we compare the performance of the proposed constellation scheduling approach with an "oracle" upper bound and a naive lower bound.
The oracle solves Eqs. 1a-1f once with full knowledge of the location of all present and future dust devils. The problem is solved in a centralized manner. The solution provided by the oracle represents an (unattainable) upper bound on the performance of the constellation. The lower bound is obtained by simply setting the attitude of all spacecraft to be in a nadir-pointing direction at all times. The two scenarios are compared with the outcome of the simulation where each spacecraft solves Eqs. 1a-1f in a receding-horizon fashion, as described in the Autonomous Pointing section, and spacecraft exchange observations along available communication links as discussed in the Communication section. To ensure a fair comparison, all spacecraft are assumed to have access to an ideal event detector with perfect precision and recall (that, is, it never produces false positives or negatives); the effect on performance of a non-ideal event detector is discussed in the next section.
The results are reported in Table 1 and Fig. 7 for a set of orbits with one minute spacing between spacecraft, and in Table 2 and Fig. 8 for orbits with six minute spacing.
In the widely-spaced case, the proposed approach results in a 230% increase in the number of events that receive at least one follow-up observation compared to   the nadir-pointing approach. The average number of follow-up observations per event increases from 0.21 to 0.77, an almost four-fold increase. The overall number of observations is 20% higher compared to the nadir-pointing case. In contrast, the number of unique events detected decreases by 18% as expected, the autonomous pointing approach privileges follow-on observations as opposed to new event detections, in line with the proposed reward function, which privileges revisits over new detections.
The impact of on-board autonomy is smaller, but still significant, in the closelyspaced case: the approach is able to increase the number of events observed three or more times from zero to 53, and the average number of follow-up observations increases by 15%, at the price of a 9.3% reduction in the number of unique events observed.
Effect of inter-spacecraft communication As discussed in the Sect. 4.2, the distributed implementation relies on inter-spacecraft communication to maintain a common picture of observed events; this can result in sub-optimal, and potentially inconsistent, behavior at the constellation level if inter-spacecraft links are sporadic or if communication between spacecrafs requires multiple hops.
To explore the impact of inter-spacecraft communications on system performance, we compare the proposed approach, where spacecraft communicate with their neighbors when a line-of-sight link is available, with two scenarios: an idealized scenario where all spacecraft can exchange information instantly, and a nocommunications scenario where each spacecraft only relies on information it has collected itself. We focus our attention on the widely-spaced scenario, where each spacecraft can only communicate with its immediate neighbors in the constellation.
Results are shown in Table 2 and Fig. 9. There is no difference in performance between the proposed approach and the upper bound where all spacecraft can communicate with each other, showing that the proposed approach is able to effectively spread information about event detections across the constellation. In contrast, the case where satellites do not communicate with each other results in performance comparable to the nadir-pointing case, with a small reduction in the average number of follow-up observations (vs. the Fig. 9 Widely-spaced orbits: Effect of communications on distribution of the number of follow-on observations almost four-fold improvement obtained with communications). Collectively, these results show that the proposed distributed optimization approach is able to approach the performance of a centralized controller, and that constellation-level optimization through inter-satellite communication is critical to achieve this performance.

Non-ideal event detector
We also assess the impact of using a non-ideal event detector with non-perfect precision and recall. Specifically, we simulate the performance of a detector with 70% recall, i.e., a 30% false negative rate. Due to limitations of the simulation environment, the study of the effect of non-ideal precision is left as future work. Results are shown in Tables 1 and 2 and in Figs. 7 and 10.
Predictably, the imperfect recall results in a roughly 30% reduction in the overall number of observations collected. Specifically, we observe 15% and 19% reductions in the number of unique detected events in the closely-spaced and widely-spaced cases respectively; the reduction in the average number of follow-up images collected for each event is 31% in both the closely-spaced case and the widely-spaced case. Compared to a nadir-pointing case with imperfect recall, the proposed approach performs  similarly to the perfect case, with a two-fold to fourfold increase in the number of events with at least two observations, and a threefold to five-fold increase in the average number of follow-up observations per event, in the widely-spaced case; and only slightly better performance compared to the nadir-pointing case in the closely-spaced case.

Computational performance
The ILP was solved with the CPLEX solver. The time required to formulate the problem (i.e., encode the cost function and constraints) and solve it on a commodity desktop workstation is shown in Fig. 11. The time required to formulate the problem remains below 500 ms except for rare outliers, and the time required to solve it is consistently always below 100 ms -results which suggest that the proposed approach could be readily implemented on more modest CPUs wellsuited for spaceflight.

Visualization System
To visualize and interact with the simulation, we designed a visualization system that adheres to the same mission design methodology as discussed in the previous sections. Past work shows interfaces [36][37][38] are critical to (i) foster trust in the data and tool from the operator's perspective, and (ii) understand the details of what the scheduler is doing onboard. Interactions provide a dialog between the user and the system [39], and this allows operators to gain confidence in the system as they explores datasets to uncover insights and be part of the decision making process. To allow the operator to explore the onboarding scheduler, the system is composed of three interactive views: overview, coverage map, and time scrubber (Fig. 12). All views are interlinked: the overview and coverage map visualize data corresponding to the particular timestamp from the time scrubber.
Overview The overview (Fig. 12b) provides a quick assessment of the constellation performance in terms of dust devil observations. Dust devils are represented as Fig. 11 Computation times to solve the proposed ILP on a Xeon E5-2687W brown triangles, which appear and disappear throughout the simulation run. A counter (Fig. 12d) helps count the total number of dust devil events. In contrast, spacecraft are represented by their bounding boxes and spacecraft number. As mentioned in the Sect. 5.3, these bounding boxes are determined from the 21 slew positions each spacecraft is capable of achieving (only five are shown in the visualization in Fig. 12). All bounding boxes are mapped with two visual encodings. First, for each spacecraft, a bounding box has solid edges if it represents the spacecraft's current camera position, while the rest are represented with thin dashed edges. Second, a bounding box is colored red if a spacecraft has observed one or more dust devils in its field of view at a particular time point (e.g., Spacecraft 4 has observed a dust devil at t = 32580 ). Otherwise, the color remains transparent. The visual encoding for the dust devil follows a similar logic, where the dust devil is colored red if it has been observed and remains brown otherwise. To help operators keep track of the simulation, a second counter (Fig. 12d) counts the total number of dust devil events the constellation has observed.
The combination of these visual encodings allows the operator to not only visually see when and where spacecraft have observed dust devils, but also highlights where the vehicles could have observed a dust devil event (i.e., a dust devil is within the field of view of one of the bounding boxes that are not in use).
The overview also visualizes the terminator to showcase the daylit side (white illuminated region) and the night side (solid dark grey filled-in region) of Mars throughout the simulation run. This corresponds to the timer (unit in hours) at the top of the overview. The default map projection is Kavrayskiy VII, but operators can change different map projections as well as interactively zoom-in, rotate, and pan using the side-bar menu (Fig. 12a). Operators can also see details on-demand (i.e., via an interactive tooltip).

Coverage Map
The coverage map (Fig. 12b) dynamically visualizes the ground track of the constellation throughout the simulation ( t = 0 to t = 88320 ). The ground track is colored dark steel-blue, and the path corresponds to the movement of the spacecraft in the overview (Fig. 12b). The coverage map provides a visual insight as to which regions the constellation has frequently visited and vice versa. This can help operators compare different constellation formations, and, in particular, different inter-spacecraft spacing.

Time Scrubber
The time scrubber (Fig. 12e) is a horizontal slider control that lets the operator dynamically seek any time position within the currently running simulation. The scrubber 'knob' (circle) moves to indicate the current time position. The position of the scrubber knob determines the information being displayed in the overview and coverage map. Operators can also pause and play when analyzing the simulation.

Conclusions
Several million dust devil events occur on Mars every day. These events last, on average, about 30 minutes and range in size from meters to hundreds of meters in diameter. Designing low-cost missions that will improve our knowledge of dust devil formation and evolution, and their connection to atmospheric dynamics and the dust cycle, is fundamental to informing future crewed Mars lander missions about surface conditions. In this paper we presented a mission for a constellation of low orbiting Mars cubesats, each carrying imagers with agile pointing capabilities. The goal was to maximize the number of dust devil follow-up observations through real-time, onboard scheduling. Realistic inter-agent communication link opportunities were computed and included in our simulation, which allowed for real-time event detection information to be shared within the constellation. In particular, we found that the communications are especially important for acquiring follow-up observations, and that the proposed distributed scheduling mechanism is able to capture a large fraction of all dust devil observations that are possible for a given orbit configuration, significantly outperforming a nadir-pointing heuristic. This is a significant result that will aid in the study of dust devil science on the surface of Mars, in preparation for future crewed missions.