Evaluation of Maritime Event Detection Against Missing Data
Detecting and preventing maritime events like collisions or unusual behaviour of vessels are of high importance for maritime safety and security. As the trust of human operators in automated maritime event detection and prediction depends on the quality of the corresponding algorithms, the evaluation methodology becomes a driving force for the future development of maritime event detection and forecasting methods. The main contribution of this article consists in the development of an evaluation methodology and its application to a selected set of maritime event detectors. The approach links a reference dataset, controlled data variations, maritime event detection algorithms with internal parameters, and performance criteria. Among pre-established possible input data variations applied to a reference Automatic Identification System (AIS) dataset, the article focuses on the evaluation of detection accuracy of maritime event detectors implemented with the Event Calculus logical language against variable amounts of missing data, as a frequently observable type of AIS data degradation. Twelve maritime event pattern detectors are evaluated and most of them are found to vary very little in performance while only one detector shows an unexpected strong performance drop giving insights into how to improve the detection method. Results are provided on a real AIS data enriched with specific simulated events.
KeywordsEvaluation methodology Maritime event detection Event Calculus Datasets creation Missing data Data veracity
This work was supported by project datAcron, which has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 687591. The authors wish to thank the NATO Allied Command Transformation (NATO-ACT) for supporting the CMRE project on Data Knowledge and Operational Effectiveness (DKOE).
- 1.Andler, S., Fredin, M., Gustafsson, F., van Laere, J., Nilsson, M., Svenson, P.: SMARTracIn - a concept for spoof resistant tracking of vessel and detection of adverse intentions. In: SPIE Defense, Security, and Sensing, Orlando, FL (2009)Google Scholar
- 4.Auslander, B., Gupta, K.M., Aha, D.W.: A comparative evaluation of anomaly detection algorithms for maritime video surveillance. In: Sensors, and Command, Control, Communications, and Intelligence (C3I) Technologies for Homeland Security and Homeland Defense X, vol. 8019, p. 801907. SPIE (2011)Google Scholar
- 5.Iphar, C., Jousselme, A.L., Ray, C.: Pseudo-synthetic datasets in support to maritime surveillance algorithms assessment. In: Proceedings of the VERITA Workshop, 19ieme Journées Francophones Extraction et Gestion des Connaissances (EGC 2019), January 2019Google Scholar
- 6.ITU: Technical characteristics for an automatic identification system using time-division multiple access in the VHF maritime mobile band (2010)Google Scholar
- 7.Jousselme, A.L., Maupin, P.: Comparison of uncertainty representations for missing data in information retrieval. In: Proceedings of the 16th International Conference on Information Fusion, pp. 1902–1909. IEEE (2013)Google Scholar
- 9.Lavesson, N., Davidsson, P.: Evaluating learning algorithms and classifiers. Int. J. Intell. Inf. Database Syst. 1(1), 37–52 (2007)Google Scholar
- 10.Margineantu, D.D., Dietterich, T.G., et al.: Bootstrap methods for the cost-sensitive evaluation of classifiers (2000)Google Scholar
- 12.Przymusinski, T.: On the declarative semantics of stratified deductive databases and logic programs. In: Foundations of Deductive Databases and Logic Programming. Morgan (1987)Google Scholar
- 14.Riveiro, M., Falkman, G.: Supporting the analytical reasoning process in maritime anomaly detection: evaluation and experimental design. In: 2010 14th International Conference Information Visualisation, pp. 170–178. IEEE (2010)Google Scholar
- 15.Roy, J., Davenport, M.: Exploitation of maritime domain ontologies for anomaly detection and threat analysis. In: Proceedings of the IEEE international Waterside Security Conference (WSS) (2010)Google Scholar