Skip to main content

ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols Under Network Limitations

Part of the Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering book series (LNICST,volume 404)


Efficient and dependable communication is a highly relevant aspect for Internet of Things (IoT) systems in which tiny sensors, actuators, wearables, or other smart devices exchange messages. Various publish/subscribe protocols address the challenges of communication in IoT systems. The selection process of a suitable protocol should consider the communication behavior of the application, the protocol’s performance, the resource requirements on the end device, and the network connection quality, as IoT environments often rely on wireless networks. Benchmarking is a common approach to evaluate and compare systems, considering the performance and aspects like dependability or security. In this paper, we present our IoT communication benchmarking framework ComBench for publish/subscribe protocols focusing on constrained networks with varying quality conditions. The benchmarking framework supports system designers, software engineers, and application developers to select and investigate the behavior of communication protocols. Our benchmarking framework contributes to (i) show the impact of fluctuating network quality on communication, (ii) compare multiple protocols, protocol features, and protocol implementations, and (iii) analyze scalability, robustness, and dependability of clients, networks, and brokers in different scenarios. Our case study demonstrates the applicability of our framework to support the decision for the best-suited protocol in various scenarios.


  • IoT
  • Publish/subscribe
  • Benchmarking
  • Load testing

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions


  1. 1.

  2. 2.

  3. 3.

  4. 4.

  5. 5.


  1. Al-Fuqaha, A., Guizani, M., Mohammadi, M., Aledhari, M., Ayyash, M.: Internet of things: a survey on enabling technologies, protocols, and applications. IEEE Commun. Surv. Tutor. 17(4), 2347–2376 (2015).

    CrossRef  Google Scholar 

  2. Appel, S., Sachs, K., Buchmann, A.: Towards benchmarking of AMQP. In: Proceedings of the Fourth ACM International Conference on Distributed Event-Based Systems, pp. 99–100 (2010)

    Google Scholar 

  3. Bansal, S., Kumar, D.: IoT application layer protocols: performance analysis and significance in smart city. In: 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT), pp. 1–6. IEEE (2019)

    Google Scholar 

  4. Boano, C.A., et al.: IoTBench: towards a benchmark for low-power wireless networking. In: 2018 IEEE Workshop on Benchmarking Cyber-Physical Networks and Systems (CPSBench). IEEE (2018)

    Google Scholar 

  5. Chen, Y., Kunz, T.: Performance evaluation of IoT protocols under a constrained wireless access network. In: 2016 International Conference on Selected Topics in Mobile & Wireless Networking (MoWNeT), pp. 1–7. IEEE (2016)

    Google Scholar 

  6. Collina, M., Bartolucci, M., Vanelli-Coralli, A., Corazza, G.E.: Internet of Things application layer protocol analysis over error and delay prone links. In: 2014 7th Advanced Satellite Multimedia Systems Conference and the 13th Signal Processing for Space Communications Workshop (ASMS/SPSC), pp. 398–404. IEEE (2014)

    Google Scholar 

  7. Community: Locust Website (2021)., Accessed 15 Apr 2021

  8. Community: MZBench Website (2021)., Accessed 13 Apr 2021

  9. Corp, G.: Gatling Website (2021)., Accessed 13 Apr 2021

  10. Dizdarević, J., Carpio, F., Jukan, A., Masip-Bruin, X.: A survey of communication protocols for internet of things and related challenges of fog and cloud computing integration. ACM Comput. Surv. (CSUR) 51, 1–29 (2019)

    CrossRef  Google Scholar 

  11. Eugster, P.T., Felber, P.A., Guerraoui, R., Kermarrec, A.M.: The many faces of publish/subscribe. ACM Comput. Surv. (CSUR) 35(2), 114–131 (2003)

    CrossRef  Google Scholar 

  12. Gündoğran, C., Kietzmann, P., Lenders, M., Petersen, H., Schmidt, T.C., Wählisch, M.: NDN, CoAP, and MQTT: a comparative measurement study in the IoT. In: Proceedings of the 5th ACM Conference on Information-Centric Networking, pp. 159–171 (2018)

    Google Scholar 

  13. Halili, E.: Apache JMeter. Packt Publishing, Birmingham (2008)

    Google Scholar 

  14. Herrnleben, S., et al.: Evaluation results of ComBench as open data. Technical report, University of Wuerzburg (2021)., Accessed 30 Apr 2021

  15. Iglesias-Urkia, M., Orive, A., Barcelo, M., Moran, A., Bilbao, J., Urbieta, A.: Towards a lightweight protocol for industry 4.0: an implementation based benchmark. In: 2017 IEEE International Workshop of Electronics, Control, Measurement, Signals and their Application to Mechatronics (ECMSM) (2017)

    Google Scholar 

  16. Jacob, R., Boano, C.A., Raza, U., Zimmerling, M., Thiele, L.: Towards a methodology for experimental evaluation in low-power wireless networking. In: Proceedings of the 2nd Workshop on Benchmarking Cyber-Physical Systems and Internet of Things, pp. 18–23 (2019)

    Google Scholar 

  17. Kayal, P., Perros, H.: A comparison of IoT application layer protocols through a smart parking implementation. In: 2017 20th Conference on Innovations in Clouds, Internet and Networks (ICIN), pp. 331–336. IEEE (2017)

    Google Scholar 

  18. von Kistowski, J., Arnold, J.A., Huppler, K., Lange, K.D., Henning, J.L., Cao, P.: How to build a benchmark. In: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015), ICPE ’15. ACM, New York (2015)

    Google Scholar 

  19. Kounev, S., Lange, K.D., von Kistowski, J.: Systems Benchmarking. Springer, Heidelberg (2020).

    CrossRef  Google Scholar 

  20. Naik, N.: Choice of effective messaging protocols for IoT systems: MQTT, CoAP, AMQP and HTTP. In: 2017 IEEE International Systems Engineering Symposium (ISSE), pp. 1–7. IEEE (2017)

    Google Scholar 

  21. Neagoe, T., Cristea, V., Banica, L.: NTP versus PTP in computer networks clock synchronization. In: 2006 IEEE International Symposium on Industrial Electronics, vol. 1, pp. 317–362. IEEE (2006)

    Google Scholar 

  22. Pohl, M., Kubela, J., Bosse, S., Turowski, K.: Performance evaluation of application layer protocols for the internet-of-things. In: 2018 Sixth International Conference on Enterprise Systems (ES), pp. 180–187. IEEE (2018)

    Google Scholar 

  23. Profanter, S., Tekat, A., Dorofeev, K., Rickert, M., Knoll, A.: OPC UA versus ROS, DDS, and MQTT: performance evaluation of industry 4.0 protocols. In: Proceedings of the IEEE International Conference on Industrial Technology (ICIT) (2019)

    Google Scholar 

  24. Sachs, K., Appel, S., Kounev, S., Buchmann, A.: Benchmarking publish/subscribe-based messaging systems. In: Yoshikawa, M., Meng, X., Yumoto, T., Ma, Q., Sun, L., Watanabe, C. (eds.) DASFAA 2010. LNCS, vol. 6193, pp. 203–214. Springer, Heidelberg (2010).

    CrossRef  Google Scholar 

  25. Sachs, K., Kounev, S., Bacon, J., Buchmann, A.: Performance evaluation of message-oriented middleware using the SPECjms2007 benchmark. Perf. Eval. 66(8), 410–434 (2009)

    CrossRef  Google Scholar 

  26. Shukla, A., Chaturvedi, S., Simmhan, Y.: RIoTBench: an IoT benchmark for distributed stream processing systems. Concurr. Comput. Pract. Exp. 29(21), e4257 (2017)

    CrossRef  Google Scholar 

  27. Statista, IHS: Internet of Things - number of connected devices worldwide 2015–2025 (2018).

  28. Talaminos-Barroso, A., Estudillo-Valderrama, M.A., Roa, L.M., Reina-Tosina, J., Ortega-Ruiz, F.: A machine-to-machine protocol benchmark for ehealth applications - use case: respiratory rehabilitation. Comput. Methods Prog. Biomed. 129, 1–11 (2016)

    CrossRef  Google Scholar 

  29. University of Wuerzburg, Institute of Computer Science, Germany, Chair of Software Engineering: Git repository of ComBench (2021).

  30. Wirawan, I.M., Wahyono, I.D., Idfi, G., Kusumo, G.R.: IoT communication system using publish-subscribe. In: 2018 International Seminar on Application for Technology of Information and Communication, pp. 61–65. IEEE (2018)

    Google Scholar 

  31. Zhang, H.L., Zhang, S., Li, X.J., Zhang, P., Liu, S.B.: Research of load testing and result application based on LoadRunner. In: 2012 National Conference on Information Technology and Computer Science. Atlantis Press (2012)

    Google Scholar 

  32. Zhang, K., Rabl, T., Sun, Y.P., Kumar, R., Zen, N., Jacobsen, H.A.: PSBench: a benchmark for content-and topic-based publish/subscribe systems. In: Proceedings of the Posters & Demos Session, pp. 17–18. Association for Computing Machinery (2014)

    Google Scholar 

Download references


This project is funded by the Bavarian State Ministry of Science and the Arts and coordinated by the Bavarian Research Institute for Digital Transformation (bidt).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Stefan Herrnleben .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Herrnleben, S. et al. (2021). ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols Under Network Limitations. In: Zhao, Q., Xia, L. (eds) Performance Evaluation Methodologies and Tools. VALUETOOLS 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 404. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-92510-9

  • Online ISBN: 978-3-030-92511-6

  • eBook Packages: Computer ScienceComputer Science (R0)