Abstract
While novel artificial intelligence and machine learning techniques are evolving and disrupting established terrestrial technologies at an unprecedented speed, their adaptation onboard satellites is seemingly lagging. A major hindrance in this regard is the need for high-quality annotated data for training such systems, which makes the development process of machine learning solutions costly, time-consuming, and inefficient. This paper presents “the OPS-SAT case”, a novel data-centric competition that seeks to address these challenges. The powerful computational capabilities of the European Space Agency’s OPS-SAT satellite are utilized to showcase the design of machine learning systems for space by using only the small amount of available labeled data, relying on the widely adopted and freely available open-source software. The generation of a suitable dataset, design and evaluation of a public data-centric competition, and results of an onboard experimental campaign by using the competition winners’ machine learning model directly on OPS-SAT are detailed. The results indicate that adoption of open standards and deployment of advanced data augmentation techniques can retrieve meaningful onboard results comparatively quickly, simplifying and expediting an otherwise prolonged development period.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Data availability
The training and test datasets used for “the OPS-SAT case” Kelvins competition are publicly available on Zenodo.
Abbreviations
- AI:
-
artificial intelligence
- EO:
-
Earth observation
- ESA:
-
European Space Agency
- FPGA:
-
field programmable gate array
- ML:
-
machine learning
- ODSET:
-
original dataset
- PDSET:
-
private dataset
References
Furano, G., Meoni, G., Dunne, A., Moloney, D., Ferlet-Cavrois, V., Tavoularis, A., Byrne, J., Buckley, L., Psarakis, M., Voss, K. O., et al. Towards the use of artificial intelligence on the edge in space systems: Challenges and opportunities. IEEE Aerospace and Electronic Systems Magazine, 2020, 35(12): 44–56.
Giuffrida, G., Fanucci, L., Meoni, G., Batic, M., Buckley, L., Dunne, A., van Dijk, C., Esposito, M., Hefele, J., Vercruyssen, N., et al. The Φ-sat-l mission: The first on-board deep neural network demonstrator for satellite earth observation. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1–14.
Mateo-Garcia, G., Veitch-Michaelis, J., Purcell, C., Longepe, N., Reid, S., Anlind, A., Bruhn, F., Parr, J., Mathieu, P. P. In-orbit demonstration of a re-trainable machine learning payload for processing optical imagery. Scientific Reports, 2023, 13(1): 10391.
Růžička, V., Vaughan, A., De Martini, D., Fulton, J., Salvatelli, V., Bridges, C., Mateo-Garcia, G., Zantedeschi, V. RaV Æn: Unsupervised change detection of extreme events using ML on-board satellites. Scientific Reports, 2022, 12(1): 16939.
Guerrisi, G., Del Frate, F., Schiavon, G. Artificial intelligence based on-board image compression for the Φ-Sat-2 mission. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2023, 16: 8063–8075.
Del Rosso, M. P., Sebastianelli, A., Spiller, D., Mathieu, P. P., Ullo, S. L. On-board volcanic eruption detection through CNNs and satellite multispectral imagery. Remote Sensing, 2021, 13(17): 3479.
Izzo, D., Martens, M., Pan, B. F. A survey on artificial intelligence trends in spacecraft guidance dynamics and control. Astrodynamics, 2019, 3(4): 287–299.
Galli, A., Giardina, P., Guta, M., Lossi, L., Mancina, A., Moscato, V., Patrone, F., Roseti, C., Romano, S. P., Sperli, G., et al. AI for zero-touch management of satellite networks in B5G and 6G infrastructures. In: Proceedings of the International Workshop on Artificial Intelligence in beyond 5G and 6G Wireless Networks, 2022.
Ferreira, P. V. R., Paffenroth, R., Wyglinski, A. M., Hackett, T. M., Bilen, S. G., Reinhart, R. C., Mortensen, D. J. Reinforcement learning for satellite communications: From LEO to deep space operations. IEEE Communications Magazine, 2019, 57(5): 70–75.
Derksen, D., Meoni, G., Lecuyer, G., Mergy, A., Martens, M., Izzo, D. Few-shot image classification challenge onboard OPS-SAT. In: Proceedings of the 35th Conference on Neural Information Processing Systems, 2021.
Sebastianelli, A., Del Rosso, M. P., Ullo, S. L. Automatic dataset builder for Machine Learning applications to satellite imagery. SoftwareX, 2021, 15: 100739.
Kikaki, K., Kakogeorgiou, I., Mikeli, P., Raitsos, D. E., Karantzalos, K. MARIDA: A benchmark for Marine Debris detection from Sentinel-2 remote sensing data. PLoS One, 2022, 17(1): e0262247.
Gomez, P., Meoni, G. MSMatch: Semisupervised multispectral scene classification with few labels. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 11643–11654.
Richards, J. A. Remote Sensing Digital Image Analysis. Springer Cham, 2022.
Izzo, D., Meoni, G., Gómez, P., Dold, D., Zoechbauer, A. Selected trends in artificial intelligence for space applications. arXiv preprint, 2022, arXiv:2212.06662.
Kisantal, M., Sharma, S., Park, T. H., Izzo, D., Martens, M., D’Amico, S. Satellite pose estimation challenge: Dataset, competition design, and results. IEEE Transactions on Aerospace and Electronic Systems, 2020, 56(5): 4083–4098.
Park, T. H., Martens, M., Lecuyer, G., Izzo, D., D’Amico, S. SPEED+: Next-generation dataset for spacecraft pose estimation across domain gap. In: Proceedings of the IEEE Aerospace Conference, 2022: 1–15.
Martens, M., Izzo, D., Krzic, A., Cox, D. Super-resolution of PROBA-V images using convolutional neural networks. Astrodynamics, 2019, 3(4): 387–402.
Chen, B., Liu, D. Q., Chin, T. J., Rutten, M., Derksenv, D., Martens, M., von Looz, M., Lecuyer, G., Izzo, D. Spot the GEO satellites: From dataset to Kelvins SpotGEO challenge. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2021: 2086–2094.
Evans, D., Labrèche, G., Mladenov, T., Marszk, D., Zelenevskiy, V., Shiradhonkar, V. OPS-SAT LEOP and commissioning: Running a nanosatellite project in a space agency context. In: Proceedings of the 36th Annual Small Satellite Conference, 2022: SSC22-IX-06.
Abderrahmane, N., Miramond, B., Kervennic, E., Girard, A. SPLEAT: SPiking Low-power Event-based ArchiTecture for in-orbit processing of satellite imagery. In: Proceedings of the International Joint Conference on Neural Networks, 2022: 1–10.
Labrèche, G., Evans, D., Marszk, D., Mladenov, T., Shiradhonkar, V., Zelenevskiy, V. Artificial intelligence for autonomous planning and scheduling of image acquisition with the SmartCam app on-board the OPS-SAT spacecraft. In: Proceedings of the AIAA SCITECH Forum, 2022: AIAA 2022–2508.
Labrèche, G., Evans, D., Marszk, D., Mladenov, T., Shiradhonkar, V., Soto, T., Zelenevskiy, V. OPS-SAT spacecraft autonomy with TensorFlow lite, unsupervised learning, and online machine learning. In: Proceedings of the IEEE Aerospace Conference, 2022: 1–17.
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. TensorFlow: Large-scale machine learning on heterogeneous systems. 2015. Software available from tensorflow.org
Park, T. H., Martens, M., Jawaid, M., Wang, Z., Chen, B., Chin, T. J., Izzo, D., D’Amico, S. Satellite pose estimation competition 2021: Results and analyses. Acta Astronautica, 2023, 204: 640–665.
Uriot, T., Izzo, D., Simões, L. F., Abay, R., Einecke, N., Rebhan, S., Martinez-Heras, J., Letizia, F., Siminski, J., Merz, K. Spacecraft collision avoidance challenge: Design and results of a machine learning competition. Astrodynamics, 2022, 6(2): 121–140.
Derksen, D., Meoni, G., Lecuyer, G., Mergy, A., Martens, M., Izzo, D. The OPS-SAT case dataset (1.0.0). Zenodo, 2022, https://doi.org/10.5281/zenodo.6524750
Derksen, D., Meoni, G., Lecuyer, G., Mergy, A., Martens, M., Izzo, D. The OPS-SAT case: Test dataset (1.0.0). Zenodo, 2023, https://doi.org/10.5281/zenodo.10301862
Grizonnet, M., Michel, J., Poughon, V., Inglada, J., Savinaud, M., Cresson, R. Orfeo ToolBox: Open source processing of remote sensing images. Open Geospatial Data, Software and Standards, 2017, 2(1): 15.
Tan, M. X., Le, Q. V. EfficientNet: Rethinking model scaling for convolutional neural networks. In: Proceedings of the 36th International Conference on Machine Learning, 2019: 6105–6114.
Liu, R. Higher accuracy on vision models with EfficientNet-Lite. 2020. Available at https://blog.tensorflow.org/2020/03/higher-accuracy-on-vision-models-with-efficientnet-lite.html
Musial, J. P., Bojanowski, J. S. Comparison of the novel probabilistic self-optimizing vectorized earth observation retrieval classifier with common machine learning algorithms. Remote Sensing, 2022, 14(2): 378.
Deshpande, P., Belwalkar, A., Dikshit, O., Tripathi, S. Historical land cover classification from CORONA imagery using convolutional neural networks and geometric moments. International Journal of Remote Sensing, 2021, 42(13): 5144–5171.
Navarro, A., Silva, I., Catalão, J., Falcão, J. An operational Sentinel-2 based monitoring system for the management and control of direct aids to the farmers in the context of the Common Agricultural Policy (CAP): A case study in mainland Portugal. International Journal of Applied Earth Observation and Geoinformation, 2021, 103: 102469.
Sohn, K., Berthelot, D., Li, C. L., Zhang, Z. Z., Carlini, N., Cubuk, E. D., Kurakin, A., Zhang, H., Raffel, C. FixMatch: Simplifying semi-supervised learning with consistency and confidence. In: Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020: 596–608.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. PyTorch: An imperative style, high-performance deep learning library. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019: 8026–8037.
European Space Agency. the_opssat_case_starter_kit. 2022. Available at https://gitlab.com/EuropeanSpaceAgency/the_opssat_case_starter_kit
Keras. 2015. Available at https://keras.io
Cheng, G., Han, J. W., Lu, X. Q. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 2017, 105(10): 1865–1883.
Chollet, F. Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1800–1807.
Pitié, F., Kokaram, A. C., Dahyot, R. Automated colour grading using colour distribution transfer. Computer Vision and Image Understanding, 2007, 107(1–2): 123–137.
Papadakis, N., Provenzi, E., Caselles, V. A variational model for histogram transfer of color images. IEEE Transactions on Image Processing, 2011, 20(6): 1682–1695.
Rabin, J., Ferradans, S., Papadakis, N. Adaptive color transfer with relaxed optimal transport. In: Proceedings of the IEEE International Conference on Image Processing, 2014: 4852–4856.
Ferradans, S., Papadakis, N., Peyré, G., Aujol, J. F. Regularized discrete optimal transport. SIAM Journal on Imaging Sciences, 2014, 7(3): 1853–1882.
Buslaev, A., Iglovikov, V. I., Khvedchenya, E., Parinov, A., Druzhinin, M., Kalinin, A. A. Albumentations: Fast and flexible image augmentations. Information, 2020, 11(2): 125.
Lin, T. Y., Goyal, P., Girshick, R., He, K. M., Dollar, P. Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(2): 318–327.
Yeung, M., Sala, E., Schönlieb, C. B., Rundo, L. Unified focal loss: Generalising Dice and cross entropy-based losses to handle class imbalanced medical image segmentation. Computerized Medical Imaging and Graphics, 2022, 95: 102026.
Muller, R., Kornblith, S., Hinton, G. When does label smoothing help? In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019: 4694–4703.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z. Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 2818–2826.
Barz, B., Denzler, J. Deep learning on small datasets without pre-training using cosine loss. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 2020: 1360–1369.
Bradski, G. The openCV library. Dr. Dobb’s Journal: Software Tools for the Professional Programmer, 2000, 25(11): 120–123.
Acknowledgements
The authors would like to thank David Evans, Georges Laebrèche, Sam Bammens, and Vladimir Zelenevskiy for providing data and support in the preparation of “the OPS-SAT case” competition and in the running of the onboard satellite experiments.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
The authors have no competing interests to declare that are relevant to the content of this article. The author Dario Izzo is the Editor-in-Chief of this journal.
Additional information
Gabriele Meoni, Ph.D., is an assistant professor in the Faculty of Aerospace Engineering of Delft University of Technology. From September 2021 until April 2023 he was an internal research fellow the Φ-lab division of the European Space Agency (ESA). During October 2022 until March 2023 he was visiting researcher in AI Sweden. He was former research fellow of the ESA Advanced Concepts Team, which he joined in the 2020 after receiving his Ph.D. degree from University of Pisa (supervisor Prof. Luca Fanucci) in information engineering. His research topics of interest include satellite onboard processing, embedded computing systems, and edge computing.
Marcus Märtens has been working as scientific crowd-sourcing engineer and internal research fellow at the Advanced Concepts Team of the European Space Agency (ESA) from February 2018 to 2022. He holds an HUMIES gold medal for developing algorithms achieving human competitive results in trajectory optimization. He designed and conducted multiple open scientific competitions centered around artificial intelligence application for space, including satellite super-resolution and pose estimation at ESA. In 2018, he received his Ph.D. degree from the Delft University of Technology for his work on information propagation in complex networks. Marcus has worked together with experts from different fields and authored works related to space, neuroscience, cyber-security and gaming.
Dawa Derksen received his master degree in aerospace engineering from the Institut Supérieur de l’Aéronautique et de l’Espace (ISAE-Supaéro), Toulouse, France, in 2016, and his Ph.D. degree from the the Centre d’Etudes Spatiales de la Biosphère (CESBIO) Laboratory, Toulouse, France, in 2019. His Ph.D. topic was the operational production of image processing algorithms applied to the large scale classification of the Earth observation images for land cover mapping.
Kenneth See received his bachelor degree with honours in aerospace engineering and his bachelor degree in computer science in 2020 from the University of Adelaide. From 2020 to 2022, he worked as a modelling and simulation engineer at Inovor Technologies. Currently, Kenneth is a research engineer at Lockheed Martin Australia, STELaRLab. His research interests include orbit determination, state estimation, tracking, and fusion.
Toby Lightheart received his B.Eng. degree from the University of Tasmania, Australia, in 2008. He completed his Ph.D. degree at the University of Adelaide, Australia, in 2018 on constructive algorithms for artificial neural networks and approximations of neuroplasticity. He worked at Inovor Technologies on nanosatellites and simulation and modelling. He currently works at the Australian Government Department of Defence.
Anthony Sécher received his Ph.D. degree in prehistory and archaeological sciences from the University of Bordeaux in 2017. He, then, joined in 2021 Capgemini Engineering’s R&I Department in Blagnac, in the Hybrid Intelligence team. His work is part of the ND2I research project and focuses on new applications of computer vision to soil recognition.
Arnaud Martin received his Ph.D. degree in artificial intelligence. Since 2012, he is a senior data scientist and was Tech Lead IA/Deep Learning France in the team Hybrid Intelligence of Capgemini Engineering. He is the leader for the whole of France in the field of AI and more specifically deep learning, design/adaptation of intelligent systems, using and creating new techniques from the field of artificial intelligence, mainly deep learning, on both GPU and edge servers.
David Rijlaarsdam has his Master of Science degree in aerospace engineering from the Delft University of Technology with a specialization in space system engineering. He currently is a senior space system engineer for Ubotica Technologies, where he manages the space system research group. He has previously been part of the automation and robotics section of the European Space Agency and part of the advanced architecture team of Intel Movidius.
Vincenzo Fanizza graduated with his master degree in aerospace engineering at Delft University of Technology. He worked as an intern at Ubotica Technologies, where he learned to develop systems based on artificial intelligence and apply machine learning to space imagery. His interests are related to the general application of AI and ML to space missions, ranging from the Earth observation to spacecraft relative navigation.
Dario Izzo received his doctoral degree in aeronautical engineering from the University Sapienza of Rome, Rome, Italy, in 1999, his second master degree in satellite platforms from the University of Cranfield, Bedford, UK, in 2002, and his Ph.D. degree in mathematical modeling from the University Sapienza of Rome, in 2003. He lectured classical mechanics and space flight mechanics with the University Sapienza of Rome. He then joined the European Space Agency, Noordwijk, the Netherlands, where he later became the scientific coordinator with the Advanced Concepts Team. He devised and managed the Global Trajectory Optimization Competitions events, the ESA Summer of Code in Space, and the Kelvins innovation and competition platform. He authored or coauthored more than 170 papers in international journals and conferences making key contributions to the understanding of flight mechanics and spacecraft control and pioneering techniques based on evolutionary and machine learning approaches. Dr. Izzo received the Humies Gold Medal and led the team winning the eighth edition of the Global Trajectory Optimization Competition.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Meoni, G., Märtens, M., Derksen, D. et al. The OPS-SAT case: A data-centric competition for onboard satellite image classification. Astrodyn (2024). https://doi.org/10.1007/s42064-023-0196-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s42064-023-0196-y