Advertisement

VeReMi: A Dataset for Comparable Evaluation of Misbehavior Detection in VANETs

  • Rens W. van der HeijdenEmail author
  • Thomas Lukaseder
  • Frank Kargl
Conference paper
Part of the Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering book series (LNICST, volume 254)

Abstract

Vehicular networks are networks of communicating vehicles, a major enabling technology for future cooperative and autonomous driving technologies. The most important messages in these networks are broadcast-authenticated periodic one-hop beacons, used for safety and traffic efficiency applications such as collision avoidance and traffic jam detection. However, broadcast authenticity is not sufficient to guarantee message correctness. The goal of misbehavior detection is to analyze application data and knowledge about physical processes in these cyber-physical systems to detect incorrect messages, enabling local revocation of vehicles transmitting malicious messages. Comparative studies between detection mechanisms are rare due to the lack of a reference dataset. We take the first steps to address this challenge by introducing the Vehicular Reference Misbehavior Dataset (VeReMi) and a discussion of valid metrics for such an assessment. VeReMi is the first public extensible dataset, allowing anyone to reproduce the generation process, as well as contribute attacks and use the data to compare new detection mechanisms against existing ones. The result of our analysis shows that the acceptance range threshold and the simple speed check are complementary mechanisms that detect different attacks. This supports the intuitive notion that fusion can lead to better results with data, and we suggest that future work should focus on effective fusion with VeReMi as an evaluation baseline.

Keywords

Misbehavior detection Vehicular networks Intrusion detection 

Notes

Acknowledgement

The authors thank Florian Diemer and Leo Hnatek for the contribution of several detector implementations in Maat, and Henning Kopp for discussions regarding the Gini index. Experiments for this work were performed on the computational resource bwUniCluster funded by the Ministry of Science, Research and the Arts Baden-Württemberg and the Universities of the State of Baden-Württemberg, Germany, within the framework program bwHPC. This work was supported in part by the Baden-Württemberg Stiftung gGmbH Stuttgart as part of the project IKT-05 AutoDetect of its IT security research programme.

References

  1. 1.
    Cárdenas, A.A., Baras, J.S., Seamon, K.: A framework for the evaluation of intrusion detection systems. In IEEE Symposium on Security and Privacy, p. 15–pp. IEEE (2006)Google Scholar
  2. 2.
    Codeca, L., Frank, R., Faye, S., Engel, T.: Luxembourg SUMO traffic (LuST) scenario: traffic demand evaluation. IEEE Intell. Transp. Syst. Mag. 9(2), 52–63 (2017)CrossRefGoogle Scholar
  3. 3.
    Damgaard, C.: Gini coefficient. From MathWord - A Wolfram Web Resource, Created by Eric W. Weisstein. http://mathworld.wolfram.com/GiniCoefficient.html. Accessed 9 Feb 2018
  4. 4.
    Davis, J., Goadrich, M.: The relationship between precision-recall and ROC curves. In: Proceedings of the 23rd International Conference on Machine Learning, ICML 2006, pp. 233–240. ACM, New York (2006)Google Scholar
  5. 5.
    Dietzel, S., van der Heijden, R.W., Decke, H., Kargl, F.: A flexible, subjective logic-based framework for misbehavior detection in V2V networks. Proceeding of IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks 2014, pp. 1–6 (2014)Google Scholar
  6. 6.
    Eberz, S., Rasmussen, K.B., Lenders, V., Martinovic, I.: Evaluating behavioral biometrics for continuous authentication: challenges and metrics. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, ASIA CCS 2017, pp. 386–399. ACM, New York (2017)Google Scholar
  7. 7.
    Jimenez, I., Sevilla, M., Watkins, N., Maltzahn, C., Lofstead, J., Mohror, K., Arpaci-Dusseau, A., Arpaci-Dussea, R.: The popper convention: making reproducible systems evaluation practical. In: 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 1561–1570, May 2017Google Scholar
  8. 8.
    Joerer, S., Sommer, C., Dressler, F.: Toward reproducibility and comparability of IVC simulation studies: a literature survey. IEEE Commun. Mag. 50(10), 82–88 (2012)CrossRefGoogle Scholar
  9. 9.
    Kumar, V., Petit, J., Whyte, W.: Binary hash tree based certificate access management for connected vehicles. In: Proceedings of the 10th ACM Conference on Security and Privacy in Wireless and Mobile Networks, WiSec 2017, pp. 145–155. ACM, New York (2017)Google Scholar
  10. 10.
    Leinmüller, T., Schoch, E., Kargl, F., Maihöfer, C.: Decentralized position verification in geographic ad hoc routing. Security Commun. Netw. 3, 289–302 (2008)CrossRefGoogle Scholar
  11. 11.
    Lo, N.-W., Tsai, H.-C.: Illusion attack on VANET applications-a message plausibility problem. In: 2007 IEEE Globecom Workshops, pp. 1–8. IEEE (2007)Google Scholar
  12. 12.
    Mitchell, R., Chen, I.-R.: A survey of intrusion detection techniques for cyber-physical systems. ACM Comput. Surv. 46(4), 55:1–55:29 (2014)CrossRefGoogle Scholar
  13. 13.
    Raya, M., Papadimitratos, P., Gligor, V.D., Hubaux, J.P.: On data-centric trust establishment in ephemeral ad hoc networks. In: IEEE INFOCOM 2008 - The 27th Conference on Computer Communications, April 2008Google Scholar
  14. 14.
    Saini, M., Alelaiwi, A., Saddik, A.E.: How close are we to realizing a pragmatic VANET solution? a meta-survey. ACM Comput. Surv. 48(2), 29:1–29:40 (2015)CrossRefGoogle Scholar
  15. 15.
    Saito, T., Rehmsmeier, M.: The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE 10(3), 1–21 (2015)CrossRefGoogle Scholar
  16. 16.
    Schmidt, R.K., Leinmueller, T., Schoch, E., Held, A., Schaefer, G.: Vehicle behavior analysis to enhance security in VANETs. In: Proceedings of the 4th IEEE Vehicle-to-Vehicle Communications Workshop (V2VCOM2008) (2008)Google Scholar
  17. 17.
    Sommer, C., German, R., Dressler, F.: Bidirectionally coupled network and road traffic simulation for improved IVC analysis. IEEE Trans. Mob. Comput. 10(1), 3–15 (2011)CrossRefGoogle Scholar
  18. 18.
    Stübing, H., Firl, J., Huss, S.A.: A two-stage verification process for Car-to-X mobility data based on path prediction and probabilistic maneuver recognition. In: 2011 IEEE Vehicular Networking Conference (VNC), pp. 17–24. IEEE (2011)Google Scholar
  19. 19.
    van der Heijden, R.W., Al-Momani, A., Kargl, F., Abu-Sharkh, O.M.F.: Enhanced position verification for VANETs using subjective logic. In: 2016 IEEE 84th Vehicular Technology Conference (VTC-Fall), pp. 1–7, September 2016Google Scholar
  20. 20.
    van der Heijden, R.W., Dietzel, S., Leinmüller, T., Kargl, T.: Survey on misbehavior detection in cooperative intelligent transportation systems (2016). Arxiv Pre-Print http://arxiv.org/abs/1610.06810. Accessed 9 Feb 2018
  21. 21.
    van der Heijden, R.W., Kargl, F.: Evaluating misbehavior detection for vehicular networks. In: 5th GI/ITG KuVS FG Inter-vehicle Communication, p. 5 (2017)Google Scholar
  22. 22.
    van der Heijden, R.W., Lukaseder, T., Kargl, F.: Analyzing attacks on cooperative adaptive cruise control (CACC). In: 2017 IEEE Vehicular Networking Conference (VNC), pp. 45–52, November 2017Google Scholar

Copyright information

© ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering 2018

Authors and Affiliations

  • Rens W. van der Heijden
    • 1
    Email author
  • Thomas Lukaseder
    • 1
  • Frank Kargl
    • 1
  1. 1.Ulm UniversityUlmGermany

Personalised recommendations