Evaluation in the absence of absolute ground truth: toward reliable evaluation methodology for scan detectors

  • Mansour Alsaleh
  • P. C. van Oorschot
Regular Contribution


Although network reconnaissance through scanning has been well explored in the literature, new scan detection proposals with various detection features and capabilities continue to appear. To our knowledge, however, there is little discussion of reliable methodologies to evaluate network scanning detectors. In this paper, we show that establishing ground truth labels of scanning activity on non-synthetic network traces is a more difficult problem relative to labeling conventional intrusions. The main problem stems from lack of absolute ground truth (AGT). We identify the specific types of errors this admits. For real-world network traffic, typically many events can be equally interpreted as legitimate or intrusions, and therefore, establishing AGT is infeasible since it depends on unknowable intent. We explore how an estimated ground truth based on discrete classification criteria can be misleading since typical detection accuracy measures are strongly dependent on the chosen criteria. We also present a methodology for evaluating and comparing scan detection algorithms. The methodology classifies remote addresses based on continuous scores designed to provide a more accurate reference for evaluation. The challenge of conducting a reliable evaluation in the absence of AGT applies to other areas in network intrusion detection, and corresponding requirements and guidelines apply.


Evaluation Absolute ground truth  Ground truth reference Scan detection 


  1. 1.
    Allman, M., Paxson, V., Terrell, J.: A brief history of scanning. In: Proceedings the 7th ACM SIGCOMM Conference on Internet Measurement (2007)Google Scholar
  2. 2.
    Axelsson, S.: The base-rate fallacy and the difficulty of intrusion detection. ACM Trans. Inf. Syst. Secur. (TISSEC). 3(3), 186–205 (2000)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Bro intrusion detection system. Accessed May 2010
  4. 4.
    Casado, M., Freedman, M.J.: Peering through the shroud: the effect of edge opacity on IP-based client identification. In: 4th USENIX Symposium on Networked Systems Design and Implementation (NDSS’07) (2007)Google Scholar
  5. 5.
    Coull, S.E., Wright, C.V., Monrose, F., Collins, M.P., Reiter, M.K.: Playing devil’s advocate: inferring sensitive information from anonymized network traces. In: NDSS (2007)Google Scholar
  6. 6.
    Floyd, S., Paxson, V.: Difficulties in simulating the internet. IEEE/ACM Trans. Netw. 9, 392–403 (2001)CrossRefGoogle Scholar
  7. 7.
    Gates, C.: Co-ordinated port scans: a model, a detector and an evaluation methodology. PhD thesis, Dalhousie University (2006)Google Scholar
  8. 8.
    Gates, C., McNutt, J.J., Kadane, J.B., Kellner, M.: Scan detection on very large networks using logistic regression modeling. In: Proceedings of the 11th IEEE Symposium on Computers and Communications (ISCC’06) (2006)Google Scholar
  9. 9.
    Heberlein, L.T., Dias, G.V., Levitt, K.N., Mukherjee, B., Wood, J., Wolber, D.: A network security monitor. In: IEEE Symposium on Security and Privacy, p. 296 (1990)Google Scholar
  10. 10.
    Jin, R., Ghahramani, Z.: Learning with multiple labels. Adv. Neural Inf. Process. Syst. 15, 897–904 (2002)Google Scholar
  11. 11.
    Jung, J.: Real-time detection of malicious network activity using stochastic models. PhD thesis, Massachusetts Institute of Technology (2006)Google Scholar
  12. 12.
    Jung, J., Paxson, V., Berger, A.W., Balakrishnan, H.: Fast portscan detection using sequential hypothesis testing. In: IEEE Symposium on Security and Privacy (2004)Google Scholar
  13. 13.
    Kang, M.G., Caballero, J., Song, D.: Distributed evasive scan techniques and countermeasures. In: Proceedings of the Conference on Detection of Intrusions and Malware and Vulnerability Assessment (2007)Google Scholar
  14. 14.
    Kato, N., Nitou, H., Ohta, K., Mansfield, G., Nemoto, Y.: A real-time intrusion detection system (IDS) for large scale networks and its evaluations. IEICE Trans. Commun. E82–B(11), 1817–1825 (1999)Google Scholar
  15. 15.
  16. 16.
    Kim, H., Kim, S., Kouritzin, M.A., Sun, W.: Detecting network portscans through anomaly detection. In: Proceedings of SPIE: Signal Processing, Sensor Fusion, and Target Recognition XIII, vol. 5429, p. 254 (2004)Google Scholar
  17. 17.
    Lam, L., Suen, S.: Application of majority voting to pattern recognition: an analysis of its behavior and performance. IEEE Trans. Syst. Man Cybern. A Syst. Hum. 27(5), 553–568 (1997)CrossRefGoogle Scholar
  18. 18.
    Leckie, C., Kotagiri, R.: A probabilistic approach to detecting network scans. In: Proceedings of the Eighth IEEE Network Operations and Management, Symposium (NOMS’02) (2002)Google Scholar
  19. 19.
    Li, Z., Goyal, A., Chen, Y.: Honeynet-based botnet scan traffic analysis. In: Botnet detection: countering the largest security threat. Advances in information security, vol. 36, pp. 25–44 (2008)Google Scholar
  20. 20.
    Li, Z., Goyal, A., Chen, Y., Paxson, V.: Automating analysis of large-scale botnet probing events. In: ASIACCS (2009)Google Scholar
  21. 21.
    Lippmann, R., Haines, J., Fried, D., Korba, J., Das, K.: The 1999 DARPA off-line intrusion detection evaluation. Comput. Netw. 34(4), 579–595 (2000)CrossRefGoogle Scholar
  22. 22.
    Lippmann, R.P., Cunningham, R.K., Fried, D.J., Graf, I., Kendall, K.R., Webster, S.E., Zissman, M.A.: Results of the DARPA 1998 offline intrusion detection evaluation. In: Proceedings of the Symposium on Recent Advances in Intrusion Detection (RAID’99) (1999)Google Scholar
  23. 23.
    Mahoney, M.V., Chan, P.K.: An analysis of the 1999 DARPA /Lincoln Laboratory evaluation data for network anomaly detection. In: Proceedings of the Sixth International Symposium on Recent Advances in Intrusion Detection (RAID’03) (2003)Google Scholar
  24. 24.
    Mason, J., Small, S., Monrose, F., MacManus, G.: English shellcode. In: Proceedings of the 16th ACM Conference on Computer and Communications Security (2009)Google Scholar
  25. 25.
    McHugh, J.: Testing intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln laboratory. ACM Trans. Inf. Syst. Secur. (TISSEC) 3, 262–294 (2000)CrossRefGoogle Scholar
  26. 26.
    Ptacek, T., Newsham, T., Simpson, H.J.: Insertion, evasion, and denial of service: eluding network intrusion detection. Technical report, Secure Networks, Inc., January (1998)Google Scholar
  27. 27.
    Ringberg, H., Roughan, M., Rexford, J.: The need for simulation in evaluating anomaly detectors. SIGCOMM Comput. Commun. Rev. 38, 55–59 (2008)CrossRefGoogle Scholar
  28. 28.
    Ringberg, H., Soule, A., Rexford, J.: WebClass: adding rigor to manual labeling of traffic anomalies. SIGCOMM Comput. Commun. Rev. 38, 35–38 (2008)CrossRefGoogle Scholar
  29. 29.
    Roelker, D., Norton, M., Hewlett, J.: sfPortscan. Accessed Jan 2010
  30. 30.
    Roesch, M.: Snort: lightweight intrusion detection for networks. In: Proceedings of the 13th Systems Administration Conference (LISA’99) (1999)Google Scholar
  31. 31.
    Sheng, V., Provost, F., Ipeirotis, P.: Get another label? Improving data quality and data mining using multiple, noisy labelers. In: Proceedings of the Conference on Knowledge Discovery and Data Mining (2008)Google Scholar
  32. 32.
    Simon, G., Xiong, H., Eilertson, E., Kumar, V.: Scan detection: a data mining approach. In: Proceedings of the International Conference on Data Mining (SIAM’06) (2006)Google Scholar
  33. 33.
    Designer, Solar., Magazine, Phrack.: Designing and attacking port scan detection tools. 8(53), July 8, 1998, article 13.
  34. 34.
    Sommer, R., Paxson, V.: Outside the closed world: on using machine learning for network intrusion detection. In: IEEE Symposium on Security and Privacy, May (2010) Google Scholar
  35. 35.
    Staniford, S., Hoagland, J.A., McAlerney, J.M.: Practical automated detection of stealthy portscans. J. Comput. Secur. 10(1/2), 105–136 (2002)Google Scholar
  36. 36.
  37. 37.
    Thabtah, F., Cowling, P., Peng, Y.: Multiple labels associative classification. Knowl. Inf. Syst. 9(1), 109–129 (2006)Google Scholar
  38. 38.
    Vigna, G.: Network intrusion detection: dead or alive? In: Proceedings of the 26th Annual Computer Security Applications Conference (ACSAC’10) (2010)Google Scholar
  39. 39.
    Weaver, N., Staniford, S., Paxson, V.: Very fast containment of scanning worms, revisited. In: Christodorescu, M., Jha, S., Maughan, D., Song, D., Wang, C. (eds.) Malware Detection. Advances in information security vol. 27, chap. 6. Springer, pp. 113–145 (2007)Google Scholar
  40. 40.
    Zhang, Y., Fang, B.: A novel approach to scan detection on the backbone. In: Sixth International Conference on Information Technology: New, Generations (ITNG’09), April (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  1. 1.School of Computer ScienceCarleton University OttawaCanada

Personalised recommendations