Knowledge Discovery from Honeypot Data for Monitoring Malicious Attacks

  • Huidong Jin
  • Olivier de Vel
  • Ke Zhang
  • Nianjun Liu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5360)


Owing to the spread of worms and botnets, cyber attacks have significantly increased in volume, coordination and sophistication. Cheap rentable botnet services, e.g., have resulted in sophisticated botnets becoming an effective and popular tool for committing online crime these days. Honeypots, as information system traps, are monitoring or deflecting malicious attacks on the Internet. To understand the attack patterns generated by botnets by virtue of the analysis of the data collected by honeypots, we propose an approach that integrates a clustering structure visualisation technique with outlier detection techniques. These techniques complement each other and provide end users both a big-picture view and actionable knowledge of high-dimensional data. We introduce KNOF (K-nearest Neighbours Outlier Factor) as the outlier definition technique to reach a trade-off between global and local outlier definitions, i.e., K th -Nearest Neighbour (KNN) and Local Outlier Factor (LOF) respectively. We propose an algorithm to discover the most significant KNOF outliers. We implement these techniques in our hpdAnalyzer tool. The tool is successfully used to comprehend honeypot data. A series of experiments show that our proposed KNOF technique substantially outperforms LOF and, to a lesser degree, KNN for real-world honeypot data.


Knowledge discovery outlier detection density-based cluster visualisation botnet honeypot data Internet security 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barford, P., Yegneswaran, V.: An inside look at botnets. In: Book Series Advances in Information Security: Malware Detection, Part III, vol. 27, pp. 171–191 (2007)Google Scholar
  2. 2.
    The Honeynet Project (ed.): Know Your Enemy: Learning about Security Threats, 2nd edn. Addison Wesley Professional, Reading (May 2004)Google Scholar
  3. 3.
    Provos, N., Holz, T.: Virtual Honeypots: From Botnet Tracking to Intrusion Detection. Addison-Wesley Professional, Reading (2007)Google Scholar
  4. 4.
    Pouget, F., Dacier, M., Zimmerman, J., Clark, A., Mohay, G.: Internet attack knowledge discovery via clusters and cliques of attack traces. Journal of Information Assurance and Security 1, 21–32 (2006)Google Scholar
  5. 5.
    Grizzard, J.B., Sharma, V., Nunnery, C., Kang, B.B., Dagon, D.: Peer-to-peer botnets: Overview and case study. In: HotBots 2007 (April 2007) Paper No. 1 Google Scholar
  6. 6.
    Nazario, J.: Botnet tracking: Tools, techniques, and lessons learned (2007) (accessed November 14, 2007),
  7. 7.
    Ankerst, M., Breunig, M.M., Kriegel, H.P., Sander, J.: OPTICS: ordering points to identify the clustering structure. In: SIGMOD 1999, pp. 49–60 (1999)Google Scholar
  8. 8.
    Han, J., Kamber, M.: Data Mining: Concepts and Techniques, 2nd edn. Morgan Kaufmann Publishers, San Francisco (March 2006)zbMATHGoogle Scholar
  9. 9.
    Jin, H., Wong, M.L., Leung, K.S.: Scalable model-based clustering for large databases based on data summarization. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(11), 1710–1719 (2005)CrossRefGoogle Scholar
  10. 10.
    Ramaswamy, S., Rastogi, R., Shim, K.: Efficient algorithms for mining outliers from large data sets. In: SIGMOD 2000, pp. 427–438 (2000)Google Scholar
  11. 11.
    Breunig, M.M., Kriegel, H.P., Ng, R.T., Sander, J.: LOF: identifying density-based local outliers. In: SIGMOD 2000, pp. 93–104 (2000)Google Scholar
  12. 12.
    Lazarevic, A., Ertöz, L., Kumar, V., Ozgur, A., Srivastava, J.: A comparative study of anomaly detection schemes in network intrusion detection. In: SDM 2003 (2003)Google Scholar
  13. 13.
    Lee, W., Stolfo, S.J.: Data mining approaches for intrusion detection. In: USENIX SSYM 1998, p. 6 (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Huidong Jin
    • 1
    • 2
  • Olivier de Vel
    • 3
  • Ke Zhang
    • 1
    • 2
  • Nianjun Liu
    • 1
    • 2
  1. 1.NICTA Canberra Lab, Locked Bag 8001Canberra ACTAustralia
  2. 2.RSISE, the Australian National UniversityCanberra ACTAustralia
  3. 3.Command, Control, Communications and Intelligence Division, DSTOEdinburghAustralia

Personalised recommendations