Normalizing Security Events with a Hierarchical Knowledge Base

  • David JaegerEmail author
  • Amir Azodi
  • Feng Cheng
  • Christoph Meinel
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9311)


An important technique for attack detection in complex company networks is the analysis of log data from various network components. As networks are growing, the number of produced log events increases dramatically, sometimes even to multiple billion events per day. The analysis of such big data highly relies on a full normalization of the log data in realtime. Until now, the important issue of full normalization of a large number of log events is only insufficiently handled by many software solutions and not well covered in existing research work. In this paper, we propose and evaluate multiple approaches for handling the normalization of a large number of typical logs better and more efficient. The main idea is to organize the normalization in multiple levels by using a hierarchical knowledge base (KB) of normalization rules. In the end, we achieve a performance gain of about 1000x with our presented approaches, in comparison to a naive approach typically used in existing normalization solutions. Considering this improvement, big log data can now be handled much faster and can be used to find and mitigate attacks in realtime.


Network security Event logs Normalization Knowledge base 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    United States Computer Emergency Readiness Team (US-CERT). USCERT Year in Review CY 2012. Tech. rep. US Department of Homeland Security (2012)Google Scholar
  2. 2.
    US Office of Management and Budget. Fiscal Year 2012 Report to Congress on the Implementation of The Federal Information Security Management Act of 2002 (March 2013)Google Scholar
  3. 3.
    Kent, K., Souppaya, M.: Guide to Computer Security Log Management. In: NIST special publication (September 2006).
  4. 4.
    Gerhards, R.: The Syslog Protocol. RFC 5424 (Proposed Standard). Internet Engineering Task Force (March 2009).
  5. 5.
    Chuvakin, A., Marty, R., et al.: Common Event Expression. White Paper, MITRE (June (2008)Google Scholar
  6. 6.
    Hewlett-Packard. Implementing ArcSight CEF. 20. Hewlett-Packard (June 2013)Google Scholar
  7. 7.
    Barnum, S., Martin, R., et al.: The CybOX Language Specification. Draft 1. The MITRE Corporation (April 2012)Google Scholar
  8. 8.
    Sapegin, A., Jaeger, D., et al.: Hierarchical Object Log Format for Normalisation of Security Events. In: Proceedings of the 9th International Conference on Information Assurance and Security (IAS 2013), Yassmine Hammamet, Tunisia, pp. 25–30 (December 2013)Google Scholar
  9. 9.
    Friedl, J.E.F.: Mastering Regular Expressions. In: Oram, A. (ed.) 3rd edn. O’Reilly Media (August 2006)Google Scholar
  10. 10.
    Sparvieri, L.: SAP HANA Text Analysis. SAP (January 2014).
  11. 11.
    Kobayashi, S., Fukuda, K., Esaki, H.: Towards an NLPbased log template generation algorithm for system log analysis. In: Proceedings of The Ninth International Conference on Future Internet Technologies, p. 11 (2014)Google Scholar
  12. 12.
    Azodi, A., Jaeger, D., et al.: Pushing the limits in event normalisation to improve attack detection in IDS/SIEM systems. In: Proceedings of the First Internation Conference on Advanced Cloud and Big Data (CBD 2013), Nanjing, China (December 2013)Google Scholar
  13. 13.
    Azodi, A., Jaeger, D., et al.: A new approach to building a multi- tier direct access knowledge base for IDS/SIEM systems. In: Proceedings of the 11th IEEE International Conference on Dependable, Autonomic and Secure Computing (DASC 2013), Chengdu, China (December 2013)Google Scholar
  14. 14.
    Real-time Event Analysis and Monitoring System (REAMS). (visited on November 5, 2015)
  15. 15.
    The Honeynet Project. Honeynet Challenges: Scan of the Month 34. Web Site (2005). (visited on May 4, 2013)

Copyright information

© IFIP International Federation for Information Processing 2015

Authors and Affiliations

  • David Jaeger
    • 1
    Email author
  • Amir Azodi
    • 1
  • Feng Cheng
    • 1
  • Christoph Meinel
    • 1
  1. 1.Hasso Plattner InstituteUniversity of PotsdamPotsdamGermany

Personalised recommendations