Abstract
We present a synthetic benchmarking framework that targets the systematic evaluation of RV tools for message-based concurrent systems. Our tool can emulate various load profiles via configuration. It provides a multi-faceted view of measurements that is conducive to a comprehensive assessment of the overhead induced by runtime monitoring. The tool is able to generate significant loads to reveal edge case behaviour that may only emerge when the monitoring system is pushed to its limit. We evaluate our framework in two ways. First, we conduct sanity checks to assess the precision of the measurement mechanisms used, the repeatability of the results obtained, and the veracity of the behaviour emulated by our synthetic benchmark. We then showcase the utility of the features offered by our tool in a two-part RV case study.
Supported by the doctoral student grant (No: 207055-051) and the TheoFoMon project (No: 163406-051) under the Icelandic Research Fund, the BehAPI project funded by the EU H2020 RISE under the Marie Skłodowska-Curie action (No: 778233), the ENDEAVOUR Scholarship Scheme (Group B, national funds), and the MIUR project PRIN 2017FTXR7S IT MATTERS.
Chapter PDF
Similar content being viewed by others
References
Aceto, L., Achilleos, A., Francalanza, A., Ingólfsdóttir, A., Kjartansson, S.Ö.: Determinizing Monitors for HML with Recursion. JLAMP 111, 100515 (2020)
Agha, G., Mason, I.A., Smith, S.F., Talcott, C.L.: A Foundation for Actor Computation. JFP 7(1), 1–72 (1997)
Apache Software Foundtation: Jmeter (2020), https://jmeter.apache.org
Attard, D.P.: detectEr (2020), https://github.com/duncanatt/detecter-inline
Attard, D.P., Cassar, I., Francalanza, A., Aceto, L., Ingólfsdóttir, A.: Introduction to Runtime Verification. In: Behavioural Types: from Theory to Tools, pp. 49–76. Automation, Control and Robotics, River (2017)
Attard, D.P., Francalanza, A.: A Monitoring Tool for a Branching-Time Logic. In: RV. LNCS, vol. 10012, pp. 473–481 (2016)
Attard, D.P., Francalanza, A.: Trace Partitioning and Local Monitoring for Asynchronous Components. In: SEFM. LNCS, vol. 10469, pp. 219–235 (2017)
Bartocci, E., Falcone, Y., Bonakdarpour, B., Colombo, C., Decker, N., Havelund, K., Joshi, Y., Klaedtke, F., Milewicz, R., Reger, G., Rosu, G., Signoles, J., Thoma, D., Zalinescu, E., Zhang, Y.: First International Competition on Runtime Verification: Rules, Benchmarks, Tools, and Final Results of CRV 2014. Int. J. Softw. Tools Technol. Transf. 21(1), 31–70 (2019)
Bartocci, E., Falcone, Y., Francalanza, A., Reger, G.: Introduction to Runtime Verification. In: Lectures on RV, LNCS, vol. 10457, pp. 1–33. Springer (2018)
Berkovich, S., Bonakdarpour, B., Fischmeister, S.: Runtime Verification with Minimal Intrusion through Parallelism. FMSD 46(3), 317–348 (2015)
Blackburn, S.M., Garner, R., Hoffmann, C., Khan, A.M., McKinley, K.S., Bentzur, R., Diwan, A., Feinberg, D., Frampton, D., Guyer, S.Z., Hirzel, M., Hosking, A.L., Jump, M., Lee, H.B., Moss, J.E.B., Phansalkar, A., Stefanovic, D., VanDrunen, T., von Dincklage, D., Wiedermann, B.: The DaCapo Benchmarks: Java Benchmarking Development and Analysis. In: OOPSLA. pp. 169–190 (2006)
Blessing, S., Fernandez-Reyes, K., Yang, A.M., Drossopoulou, S., Wrigstad, T.: Run, Actor, Run: Towards Cross-Actor Language Benchmarking. In: AGERE!@SPLASH. pp. 41–50 (2019)
Bodden, E., Hendren, L.J., Lam, P., Lhoták, O., Naeem, N.A.: Collaborative Runtime Verification with Tracematches. J. Log. Comput. 20(3), 707–723 (2010)
Bonakdarpour, B., Finkbeiner, B.: The Complexity of Monitoring Hyperproperties. In: CSF. pp. 162–174 (2018)
Buyya, R., Broberg, J., Goscinski, A.M.: Cloud Computing: Principles and Paradigms. Wiley-Blackwell (2011)
Cesarini, F., Thompson, S.: Erlang Programming: A Concurrent Approach to Software Development. O’Reilly Media (2009)
Chappell, D.: Enterprise Service Bus: Theory in Practice. O’Reilly Media (2004)
Chen, F., Rosu, G.: Mop: An Efficient and Generic Runtime Verification Framework. In: OOPSLA. pp. 569–588 (2007)
Chen, F., Rosu, G.: Parametric Trace Slicing and Monitoring. In: TACAS. LNCS, vol. 5505, pp. 246–261 (2009)
Ciemiewicz, D.M.: What Do You mean? - Revisiting Statistics for Web Response Time Measurements. In: CMG. pp. 385–396 (2001)
Cornejo, O., Briola, D., Micucci, D., Mariani, L.: In the Field Monitoring of Interactive Application. In: ICSE-NIER. pp. 55–58 (2017)
Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM 51(1), 107–113 (2008)
El-Hokayem, A., Falcone, Y.: Monitoring Decentralized Specifications. In: ISSTA. pp. 125–135 (2017)
Francalanza, A.: A Theory of Monitors (Extended Abstract). In: FoSSaCS. LNCS, vol. 9634, pp. 145–161 (2016)
Francalanza, A., Aceto, L., Achilleos, A., Attard, D.P., Cassar, I., Della Monica, D., Ingólfsdóttir, A.: A Foundation for Runtime Monitoring. In: RV. LNCS, vol. 10548, pp. 8–29 (2017)
Francalanza, A., Aceto, L., Ingólfsdóttir, A.: Monitorability for the Hennessy-Milner Logic with Recursion. FMSD 51(1), 87–116 (2017)
Francalanza, A., Pérez, J.A., Sánchez, C.: Runtime Verification for Decentralised and Distributed Systems. In: Lectures on RV, LNCS, vol. 10457, pp. 176–210. Springer (2018)
Francalanza, A., Xuereb, J.: On Implementing Symbolic Controllability. In: COORDINATION. LNCS, vol. 12134, pp. 350–369 (2020)
Ghosh, S.: Distributed Systems: An Algorithmic Approach. CRC (2014)
Gray, J.: The Benchmark Handbook for Database and Transaction Processing Systems. Morgan Kaufmann (1993)
Grove, D.A., Coddington, P.D.: Analytical Models of Probability Distributions for MPI Point-to-Point Communication Times on Distributed Memory Parallel Computers. In: ICA3PP. LNCS, vol. 3719, pp. 406–415 (2005)
Harman, M., O’Hearn, P.W.: From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis. In: SCAM. pp. 1–23 (2018)
Hoguin, L.: Cowboy (2020), https://ninenines.eu
Hoguin, L.: Ranch (2020), https://ninenines.eu
Imam, S.M., Sarkar, V.: Savina - An Actor Benchmark Suite: Enabling Empirical Evaluation of Actor Libraries. In: AGERE!@SPLASH. pp. 67–80 (2014)
Jin, D., Meredith, P.O., Lee, C., Rosu, G.: JavaMOP: Efficient Parametric Runtime Monitoring Framework. In: ICSE. pp. 1427–1430 (2012)
Kayser, B.: What is the expected distribution of website response times? (2017, last accessed, 19th Jan 2021), https://blog.newrelic.com/engineering/expected-distributions-website-response-times
Kim, M., Viswanathan, M., Kannan, S., Lee, I., Sokolsky, O.: Java-mac: A Run-Time Assurance Approach for Java Programs. FMSD 24(2), 129–155 (2004)
Kshemkalyani, A.D.: Distributed Computing: Principles, Algorithms, and Systems. Cambridge University Press (2011)
Kuhtz, L., Finkbeiner, B.: LTL Path Checking is Efficiently Parallelizable. In: ICALP (2). LNCS, vol. 5556, pp. 235–246 (2009)
Larsen, K.G.: Proof Systems for Satisfiability in Hennessy-Milner Logic with Recursion. TCS 72(2&3), 265–288 (1990)
Lightbend: Play framework (2020), https://www.playframework.com
Liu, Z., Niclausse, N., Jalpa-Villanueva, C.: Traffic Model and Performance Evaluation of Web Servers. Perform. Evaluation 46(2-3), 77–100 (2001)
Matthes, E.: Python Crash Course: A Hands-On, Project-Based Introduction to Programming. No Starch Press (2019)
Meredith, P.O., Jin, D., Griffith, D., Chen, F., Rosu, G.: An Overview of the MOP Runtime Verification Framework. STTT 14(3), 249–289 (2012)
Myers, G.J., Sandler, C., Badgett, T.: The Art of Software Testing. Wiley (2011)
Navabpour, S., Joshi, Y., Wu, C.W.W., Berkovich, S., Medhat, R., Bonakdarpour, B., Fischmeister, S.: RiTHM: A Tool for Enabling Time-Triggered Runtime Verification for C Programs. In: ESEC/SIGSOFT FSE. pp. 603–606. ACM (2013)
Neykova, R., Yoshida, N.: Let it Recover: Multiparty Protocol-Induced Recovery. In: CC. pp. 98–108 (2017)
Niclausse, N.: Tsung (2017), http://tsung.erlang-projects.org
Nielsen, J.: Usability Engineering. Morgan Kaufmann (1993)
Odersky, M., Spoon, L., Venners, B.: Programming in Scala. Artima Inc. (2020)
Prokopec, A., Rosà, A., Leopoldseder, D., Duboscq, G., Tuma, P., Studener, M., Bulej, L., Zheng, Y., Villazón, A., Simon, D., Würthinger, T., Binder, W.: Renaissance: Benchmarking Suite for Parallel Applications on the JVM. In: PLDI. pp. 31–47 (2019)
Quick, K.: Thespian (2020), http://thespianpy.com
Reger, G., Cruz, H.C., Rydeheard, D.E.: MarQ: Monitoring at Runtime with QEA. In: TACAS. LNCS, vol. 9035, pp. 596–610 (2015)
Roestenburg, R., Bakker, R., Williams, R.: Akka in Action. Manning (2015)
Rossi, R.J.: Mathematical Statistics: An Introduction to Likelihood Based Inference. Wiley (2018)
Scheffel, T., Schmitz, M.: Three-Valued Asynchronous Distributed Runtime Verification. In: MEMOCODE. pp. 52–61 (2014)
Seow, S.C.: Designing and Engineering Time: The Psychology of Time Perception in Software. Addison-Wesley (2008)
Sewe, A., Mezini, M., Sarimbekov, A., Binder, W.: DaCapo con Scala: design and analysis of a Scala benchmark suite for the JVM. In: OOPSLA. pp. 657–676 (2011)
SPEC: SPECjvm2008 (2008), https://www.spec.org/jvm2008
Tarkoma, S.: Overlay Networks: Toward Information Networking. Auerbach (2010)
Welford, B.P.: Note on a Method for Calculating Corrected Sums of Squares and Products. Technometrics 4(3), 419–420 (1962)
White, T.: Hadoop: The Definitive Guide: Storage and Analysis at Internet Scale. O’Reilly Media (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2021 The Author(s)
About this paper
Cite this paper
Aceto, L., Attard, D.P., Francalanza, A., Ingólfsdóttir, A. (2021). On Benchmarking for Concurrent Runtime Verification. In: Guerra, E., Stoelinga, M. (eds) Fundamental Approaches to Software Engineering. FASE 2021. Lecture Notes in Computer Science(), vol 12649. Springer, Cham. https://doi.org/10.1007/978-3-030-71500-7_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-71500-7_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-71499-4
Online ISBN: 978-3-030-71500-7
eBook Packages: Computer ScienceComputer Science (R0)