Scalable Offline Monitoring

  • David Basin
  • Germano Caronni
  • Sarah Ereth
  • Matúš Harvan
  • Felix Klaedtke
  • Heiko Mantel
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8734)


We propose an approach to monitoring IT systems offline, where system actions are logged in a distributed file system and subsequently checked for compliance against policies formulated in an expressive temporal logic. The novelty of our approach is that monitoring is parallelized so that it scales to large logs. Our technical contributions comprise a formal framework for slicing logs, an algorithmic realization based on MapReduce, and a high-performance implementation. We evaluate our approach analytically and experimentally, proving the soundness and completeness of our slicing techniques and demonstrating its practical feasibility and efficiency on real-world logs with 400 GB of relevant data.


Temporal Structure Predicate Symbol Constant Symbol MapReduce Framework Check Compliance 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abiteboul, S., Hull, R., Vianu, V.: Foundations of Databases: The Logical Level. Addison Wesley (1994)Google Scholar
  2. 2.
    Alur, R., Henzinger, T.A.: Logics and models of real time: A survey. In: Huizing, C., de Bakker, J.W., Rozenberg, G., de Roever, W.-P. (eds.) REX 1991. LNCS, vol. 600, pp. 74–106. Springer, Heidelberg (1992)Google Scholar
  3. 3.
    Baier, C., Katoen, J.-P.: Principles of Model Checking. The MIT Press (2008)Google Scholar
  4. 4.
    Barre, B., Klein, M., Soucy-Boivin, M., Ollivier, P.-A., Hallé, S.: MapReduce for parallel trace validation of LTL properties. In: Qadeer, S., Tasiran, S. (eds.) RV 2012. LNCS, vol. 7687, pp. 184–198. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  5. 5.
    Barringer, H., Goldberg, A., Havelund, K., Sen, K.: Rule-based runtime verification. In: Steffen, B., Levi, G. (eds.) VMCAI 2004. LNCS, vol. 2937, pp. 44–57. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  6. 6.
    Barringer, H., Groce, A., Havelund, K., Smith, M.: Formal analysis of log files. J. Aero. Comput. Inform. Comm. 7, 365–390 (2010)CrossRefGoogle Scholar
  7. 7.
    Basin, D., Harvan, M., Klaedtke, F., Zălinescu, E.: MONPOLY: Monitoring usage-control policies. In: Khurshid, S., Sen, K. (eds.) RV 2011. LNCS, vol. 7186, pp. 360–364. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  8. 8.
    Basin, D., Harvan, M., Klaedtke, F., Zălinescu, E.: Monitoring data usage in distributed systems. IEEE Trans. Software Eng. 39(10), 1403–1426 (2013)CrossRefGoogle Scholar
  9. 9.
    Basin, D., Klaedtke, F., Müller, S., Pfitzmann, B.: Runtime monitoring of metric first-order temporal properties. In: Proceedings of the 28th IARCS Annual Conference on Foundations of Software Technology and Theoretical Computer Science (FSTTCS). Leibniz International Proceedings in Informatics (LIPIcs), vol. 2, pp. 49–60. Schloss Dagstuhl - Leibniz Center for Informatics (2008)Google Scholar
  10. 10.
    Bauer, A., Goré, R., Tiu, A.: A first-order policy language for history-based transaction monitoring. In: Leucker, M., Morgan, C. (eds.) ICTAC 2009. LNCS, vol. 5684, pp. 96–111. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  11. 11.
    Chomicki, J.: Efficient checking of temporal integrity constraints using bounded history encoding. ACM Trans. Database Syst. 20(2), 149–186 (1995)CrossRefGoogle Scholar
  12. 12.
    Dean, J., Ghemawat, S.: MapReduce: Simplified data processing on large clusters. In: Proceedings of the 6th Symposium on Operating System Design and Implementation (OSDI), pp. 137–150. USENIX Association (2004)Google Scholar
  13. 13.
    Dinesh, N., Joshi, A., Lee, I., Sokolsky, O.: Checking traces for regulatory conformance. In: Leucker, M. (ed.) RV 2008. LNCS, vol. 5289, pp. 86–103. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  14. 14.
    Enderton, H.: A Mathematical Introduction to Logic, 2nd edn. Academic Press (2001)Google Scholar
  15. 15.
    Garg, D., Jia, L., Datta, A.: Policy auditing over incomplete logs: theory, implementation and applications. In: Proceedings of the 18th ACM Conference on Computer and Communications Security (CCS), pp. 151–162. ACM Press (2011)Google Scholar
  16. 16.
    Google. Protocol Buffers: Googles Data Interchange Format (2013),
  17. 17.
    Groce, A., Havelund, K., Smith, M.: From scripts to specification: The evaluation of a flight testing effort. In: Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering (ICSE), vol. 2, pp. 129–138. ACM Press (2010)Google Scholar
  18. 18.
    Hallé, S., Villemaire, R.: Runtime enforcement of web service message contracts with data. IEEE Trans. Serv. Comput. 5(2), 192–206 (2012)CrossRefGoogle Scholar
  19. 19.
    Maggi, F.M., Montali, M., Westergaard, M., van der Aalst, W.M.P.: Monitoring business constraints with linear temporal logic: An approach based on colored automata. In: Rinderle-Ma, S., Toumani, F., Wolf, K. (eds.) BPM 2011. LNCS, vol. 6896, pp. 132–147. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  20. 20.
    Marz, N.: STORM: Distributed and fault-tolerant realtime computation,
  21. 21.
    Neumeyer, L., Robbins, B., Nair, A., Kesari, A.: S4: Distributed stream computing. In: Proceedings of the 11th International Conference on Data Mining Workshops (ICDMW), pp. 170–177. IEEE Computer Society (2010)Google Scholar
  22. 22.
    Roşu, G., Chen, F.: Semantics and algorithms for parametric monitoring. Log. Method. Comput. Sci. 8(1), 1–47 (2012)Google Scholar
  23. 23.
    Roger, M., Goubault-Larrecq, J.: Log auditing through model-checking. In: Proceedings of the 14th IEEE Computer Security Foundations Workshop (CSFW), pp. 220–234. IEEE Computer Society (2001)Google Scholar
  24. 24.
    Sistla, A.P., Wolfson, O.: Temporal triggers in active databases. IEEE Trans. Knowl. Data Eng. 7(3), 471–486 (1995)CrossRefGoogle Scholar
  25. 25.
    Wikipedia. MurmurHash — Wikipedia, the free encyclopedia (2013),

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • David Basin
    • 1
  • Germano Caronni
    • 2
  • Sarah Ereth
    • 3
  • Matúš Harvan
    • 4
  • Felix Klaedtke
    • 5
  • Heiko Mantel
    • 3
  1. 1.Institute of Information SecurityETH ZurichSwitzerland
  2. 2.Google Inc.Switzerland
  3. 3.Department of Computer ScienceTU DarmstadtGermany
  4. 4.ABB Corporate ResearchSwitzerland
  5. 5.NEC Europe Ltd.HeidelbergGermany

Personalised recommendations