Anonymization of System Logs for Preserving Privacy and Reducing Storage

  • Siavash GhiasvandEmail author
  • Florina M. Ciorba
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 887)


System logs constitute valuable information for analysis and diagnosis of systems behavior. The analysis is highly time-consuming for large log volumes. For many parallel computing centers, outsourcing the analysis of system logs (syslogs) to third parties is the only option. Therefore, a general analysis and diagnosis solution is needed. Such a solution is possible only through the syslog analysis from multiple computing systems. The data within syslogs can be sensitive, thus obstructing the sharing of syslogs across institutions, third party entities, or in the public domain. This work proposes a new method for the anonymization of syslogs that employs de-identification and encoding to provide fully shareable system logs. In addition to eliminating the sensitive data within the test logs, the proposed anonymization method provides 25% performance improvement in post-processing of the anonymized syslogs, and more than 80% reduction in their required storage space.


Privacy Anonymization Encoding System logs Data quality Size reduction Performance improvement 



This work is in part supported by the German Research Foundation (DFG) in the Cluster of Excellence “Center for Advancing Electronics Dresden” (cfaed). The authors also thank Holger Mickler and the administration team of Technical University of Dresden, Germany for their support in collecting the monitoring information on the Taurus high performance computing cluster.

Disclaimer. References to legal excerpts and regulations in this work are provided only to clarify the proposed approach and to enhance explanation. In no event will authors of this work be liable for any incidental, indirect, consequential, or special damages of any kind, based on the information in these references.


  1. 1.
    Cranor, L., Rabin, T., Shmatikov, V., Vadhan, S., Weitzner, D.: Towards a privacy research roadmap for the computing community. ArXiv e-prints (2016)Google Scholar
  2. 2.
    Redman, T.C.: Data Driven: Profiting from Your Most Important Business Asset. Harvard Business Press (2008)Google Scholar
  3. 3.
    European Commission Decision. Accessed 06 June 2017
  4. 4.
    General data protection regulation. Accessed 06 June 2017
  5. 5.
    Sweeney, L.: Simple demographics often identify people uniquely. Carnegie Mellon University, Data Privacy (2000, working paper)Google Scholar
  6. 6.
    Dahlberg, R., Pulls, T.: Standardized Syslog Processing : Revisiting Secure Reliable Data Transfer and Message Compression, Karlstad, Sweden (2016)Google Scholar
  7. 7.
    New rsyslog 7.4.0. Accessed 06 June 2017
  8. 8.
    Logstash, centralize, transform and stash your data. Accessed 06 June 2017
  9. 9.
    Sanjappa, S., Ahmed, M.: Analysis of logs by using logstash. In: Proceedings of the 5th International Conference on Frontiers in Intelligent Computing: Theory and Applications, pp. 579–585. Springer, Singapore (2017)Google Scholar
  10. 10.
    Loggy, log management. Accessed 06 June 2017
  11. 11.
    Siem, log management, compliance. Accessed 06 June 2017
  12. 12.
    Blazing-fast log management and server monitoring. Accessed 06 June 2017
  13. 13.
    Gholami, A., Laure, E., Somogyi, P., Spjuth, O., Niazi, S., Dowling, J.: Privacy-preservation for publishing sample availability data with personal identifiers. J. Med. Bioeng. 4(2) (2015)CrossRefGoogle Scholar
  14. 14.
    Templ, M., Kowarik, A., Meindl, B.: Statistical disclosure control methods for anonymization of microdata and risk estimation. Accessed 06 June 2017
  15. 15.
    Dai, C., Ghinita, G., Bertino, E., Byun, J.-W., Li, N.: TIAMAT: a tool for interactive analysis of microdata anonymization techniques. Proc. VLDB Endow. 2(2), 1618–1621 (2009)CrossRefGoogle Scholar
  16. 16.
    Ciglic, M., Eder, J., Koncilia, C.: Anonymization of data sets with null values. Trans. Large-Scale Data-Knowl.-Cent.Ed Syst. XXIV: Spec. Issue Database-Expert.-Syst. Appl., 193–220 (2016)Google Scholar
  17. 17.
    UTD anonymization toolbox. Accessed 06 June 2017
  18. 18.
    Xiao, X., Wang, G., Gehrke, J.: Interactive anonymization of sensitive data. In: Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data, SIGMOD 2009, pp. 1051–1054. ACM, New York (2009)Google Scholar
  19. 19.
    Machanavajjhala, A., Gehrke, J., Kifer, D., Venkitasubramaniam, M.: L-diversity: privacy beyond k-anonymity. In: 22nd International Conference on Data Engineering (ICDE 2006), pp. 24–24, April 2006Google Scholar
  20. 20.
    Meyerson, A., Williams, R.: On the complexity of optimal k-anonymity. In: Proceedings of the Twenty-Third ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, PODS 2004, pp. 223–228. ACM, New York (2004)Google Scholar
  21. 21.
    Bayardo, R.J., Agrawal, R.: Data privacy through optimal k-anonymization. In: 21st International Conference on Data Engineering (ICDE 2005), pp. 217–228, April 2005Google Scholar
  22. 22.
    Rath, C.: Usable privacy-aware logging for unstructured log entries. In: 11th International Conference on Availability, Reliability and Security (ARES), pp. 272–277, August 2016Google Scholar
  23. 23.
    Privacy-aware logging made easy. Accessed 06 June 2017
  24. 24.
    The syslog protocol. Accessed 06 June 2017
  25. 25.
    Ghiasvand, S., Ciorba, F.M.: Toward resilience in HPC: a prototype to analyze and predict system behavior. In: Poster at International Supercomputing Conference (ISC), June 2017Google Scholar
  26. 26.
    Demonstration of annonymization and event pattern detection. Accessed 06 June 2017
  27. 27.
    Alakuijala, J., Kliuchnikov, E., Szabadka, Z., Vandevenne, L.: Comparison of brotli, deflate, zopfli, lzma, lzham and bzip2 compression algorithms. Accessed 06 June 2017
  28. 28.
    Collin, L.: A quick benchmark: Gzip vs. Bzip2 vs. LZMA. Accessed 06 June 2017
  29. 29.
    Quick benchmark: Gzip vs Bzip2 vs LZMA vs XZ vs LZ4 vs LZO. Accessed 06 June 2017
  30. 30.
    Mahoney, M.: 10 gb compression benchmark. Accessed 06 June 2017
  31. 31.
    Bertoni, G., Daemen, J., Peeters, M., Van Assche, G.: The KECCAK SHA-3 submission. Accessed 06 June 2017
  32. 32.
  33. 33.
    Yoo, A.B., Jette, M.A., Grondona, M.: SLURM: simple Linux utility for resource management. In: Proceedings of 9th International Workshop on Job Scheduling Strategies for Parallel Processing, pp. 44–60. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  34. 34.
    Terms of use of the HPC systems at the ZIH, Technical University Dresden, Germany. Accessed 06 June 2017
  35. 35.
    Order for the Information Technology Facilities and Services and for the Information Security of the Technical University of Dresden (IT-Regulations), Germany. Accessed 06 June 2017
  36. 36.
    Order for the Establishment and Operation of an Identity Management System at the Technical University of Dresden, Germany. Accessed 06 June 2017
  37. 37.
    Information leaflet on IT resources, Technical University Dresden, Germany. Accessed 06 June 2017
  38. 38.
    Ghiasvand, S., Ciorba, F.M., Tschüter, R., Nagel, W.E.: Analysis of node failures in high performance computers based on system logs. In: Poster at International Conference for High Performance Computing, Networking, Storage and Analysis (SC15) (2015)Google Scholar
  39. 39.
    Fournier-Viger, P., Lin, J. C., Vo, B., Truong, T.C., Zhang, J., Le, H.B.: A survey of itemset mining. Wiley Interdisc. Rew.: Data Mining and Knowledge Discovery 7(4) (2017). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Technical University of DresdenDresdenGermany
  2. 2.University of BaselBaselSwitzerland

Personalised recommendations