Analyzing Logs of the University Data Repository

  • Janusz Sosnowski
  • Piotr Gawkowski
  • Krzysztof Cabaj
  • Marcin Kubacki
Part of the Studies in Computational Intelligence book series (SCI, volume 541)


Identification of execution anomalies is very important for the maintenance and performance refinement of computer systems. For this purpose we can use system logs. These logs contain vast amounts of data, hence there is a great demand for techniques targeted at log analysis. The paper presents our experience with monitoring event and performance logs related to data repository operation. Having collected representative data from the monitored systems we have developed original algorithms of log analysis and problem predictions, they are based on various data mining approaches. These algorithms have been included in the implemented tools: LogMiner, FEETS, ODM. Practical significance of the developed approaches has been illustrated with some examples of exploring data repository logs. To improve the accuracy of problem diagnostics we have developed supplementary log database which can be filled in by system administrators and users.


System monitoring Event and performance logs Dependability 



This work is supported by the National Centre for Research and Development (NCBiR) under Grant No. SP/I/1/77065/10.


  1. 1.
    Chandola, V., Baerjee, A., Kumar, V.: Anomaly detection, a survey. ACM Comput. Surv. 41(3), 1–58 (2009)Google Scholar
  2. 2.
    Chen, C., Singh, N., Yajnik, M.: Log analytics for dependable enterprise telephony. In: Proceedings of 9th European Dependable Computing Conference, IEEE Computer Society, pp. 94–101 (2012)Google Scholar
  3. 3.
    Cinque, M., et al.: A logging approach for effective dependability evaluation of computer systems. In: Proceedings of 2nd IEEE International Conference on Dependability, pp. 105–110 (2009)Google Scholar
  4. 4.
    Naggapan, M., Vouk, M.A.: Abstracting log lines to log event types for mining software system logs. In: Proceedings of Mining Software Repositories, pp. 114–117 (2010)Google Scholar
  5. 5.
    Oliner, A., Stearley, J.: What supercomputers say: A study of five system logs. In: Proceedings of the IEEE/IFIP International Conference on Dependable Systems and Networks (2007)Google Scholar
  6. 6.
    Salfiner, F., Lenk, M., Malek, M.: A survey of failure prediction methods. ACM Comput. Surv. 42(3), 10.1–10.42 (2010)Google Scholar
  7. 7.
    Yu, L., Zheng, Z., Lan, Z.: Practical online failure prediction for blue gene/p: period-based vs. event-driven. In: Proceedings of the IEEE/IFIP International Conference on Dependable Systems and Networks Workshops, pp. 259–264 (2011)Google Scholar
  8. 8.
    Sosnowski, J., Gawkowski, P., Cabaj, K.: Exploring the space of system monitoring. In: Bembenik, R., et al. (eds.) Intelligent Tools for Building a Scientific Information Platform: Advanced Architectures and Solutions. Studies in Computational Intelligence, vol. 467, pp. 501–517 (2013). ISBN 978-3-642-35646-9Google Scholar
  9. 9.
    Sosnowski, J., Gawkowski, P., Cabaj, K.: Event and performance logs in system management and evaluation. In: Jałowiecki, P., Orłowski, A. (eds.) Information Systems in Management XIV, Security and Effectiveness of ICT Systems, pp. 83–93. WULS Press, Warsaw (2011). ISBN 978-83-7583-371-3Google Scholar
  10. 10.
    Sosnowski, J., Kubacki, M., Krawczyk, H.: Monitoring event logs within a cluster system. In: Zamojski, W. et al. (eds.) Complex Systems and Dependability. Advances in Intelligent and Soft Computing, vol. 170, pp. 259–271. Springer, Berlin (2012)Google Scholar
  11. 11.
    Kubacki, M., Sosnowski, J.: Creating a knowledge data base on system dependability and resilience. Control Cybern. 42(1), 287–307 (2013)Google Scholar
  12. 12.
    Berry, M.W., Kogan, J.: Text Mining Applications and Theory. Wiley, Chichester (2010)Google Scholar
  13. 13.
    Han, J., Kamber, M., Pei, J.: Data Mining Concepts and Techniques. Elsevier, Amsterdam (2012)Google Scholar
  14. 14.
    Hershey, P., Silio, C.B.: Systems engineering approach for event monitoring and analysis in high speed enterprise communication systems. In: IEEE International Systems Conference SysCon (2009)Google Scholar
  15. 15.
    Kufel, L.: Security event monitoring in a distributed systems environment. IEEE Secur. Priv. 11(1), 36–42 (2013)Google Scholar
  16. 16.
    Cinque, M., Cotroneo, D., Pecchia, A.: Event logs for the analysis of software failures, a rule based approach. IEEE Trans. Softw. Eng. 39(8), 806–821 (2013)CrossRefGoogle Scholar
  17. 17.
    Fu, X., Rebn, R., Jianfeng, Z., Wei, Z., Zhen, J., Gang, L.: LogMaster: mining event correlations in logs of large-scale cluster systems. In: Proceedings of IEEE Symposium on Reliable Distributed Systems, pp. 71–80 (2012)Google Scholar
  18. 18.
    Vaarandi, R.: A data clustering algorithm for mining patterns from event logs. In: Proceedings of 3rd IEEE Workshop on IP operations and Management, pp. 119–126 (2003)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  1. 1.Institute of Computer ScienceWarsaw University of TechnologyWarsawPoland

Personalised recommendations