Benchmarking Methodology

  • Karsten Saller
  • Kamill Panitzek
  • Max Lehn
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7847)

This Chapter introduces a generically applicable benchmarking methodology for peer-to-peer systems that is derived from the requirements stated in the Introduction of this book. Starting with key concepts a more detailed discussion follows. The methodology comprises a set of best practices, which are applied to real-world scenarios in the later Chapters of this book.

Key Concepts

An abstract model of a benchmarking environment is depicted in Figure  3.1. The main components are the workbench in which the system under test is running, the workload, input parameters, and output results. The system under test incorporates the entire system that is being tested. Since a peer-to-peer system consists of a large number of peers, the system under test corresponds to the entirety of the participating peers.


Hash Table Host Mobility Distribute Hash Table Quality Aspect Fairness Index 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Buford, J.F., Yu, H., Lua, E.K.: Managed Overlays. In: P2P Networking and Applications, chapter 15, pp. 341–360. Morgan Kaufmann, San Francisco (2008)Google Scholar
  2. 2.
    Dabek, F., Zhao, B., Druschel, P., Kubiatowicz, J., Stoica, I.: Towards a Common API for Structured Peer-to-Peer Overlays. In: International Workshop on Peer-To-Peer Systems (2003)Google Scholar
  3. 3.
    Jain, R., Chiu, D.M., Hawe, W.R.: A Quantitative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer Systems. Research Rep. TR-301, Eastern Research Lab (1984)Google Scholar
  4. 4.
    Jain, R.: The Art of Computer Systems Performance Analysis. John Wiley & Sons, Inc., Chichester (1991)zbMATHGoogle Scholar
  5. 5.
    Kaune, S., et al.: Modelling the internet delay space based on geographical locations. In: 17th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP 2009), pp. 301–310 (2009)CrossRefGoogle Scholar
  6. 6.
    Lehn, M., Triebel, T., Gross, C., Stingl, D., Saller, K., Effelsberg, W., Kovacevic, A., Steinmetz, R.: Designing benchmarks for P2P systems. In: Sachs, K., Petrov, I., Guerrero, P. (eds.) Buchmann Festschrift. LNCS, vol. 6462, pp. 209–229. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  7. 7.
    Li, J., Stribling, J., Morris, R., Kaashoek, M.F., Gil, T.M.: A Performance vs. Cost Framework for Evaluating DHT Design Tradeoffs Under Churn. In: Proc. of the 24th Annual Joint Conf. of the IEEE Computer and Communications Societies (2005)Google Scholar
  8. 8.
    Maymounkov, P., Mazières, D.: Kademlia: A peer-to-peer information system based on the XOR metric. In: Druschel, P., Kaashoek, M.F., Rowstron, A. (eds.) IPTPS 2002. LNCS, vol. 2429, p. 53. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  9. 9.
    Mukherjee, P.: A Fully Decentralized, Peer-To-Peer Based Version Control System. PhD thesis, Technische Universität Darmstadt (2010)Google Scholar
  10. 10.
    Rowstron, A., Druschel, P.: Pastry: Scalable, decentralized object location, and routing for large-scale peer-to-peer systems. In: Guerraoui, R. (ed.) Middleware 2001. LNCS, vol. 2218, p. 329. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  11. 11.
    Sachs, K., Kounev, S., Bacon, J., Buchmann, A.: Performance Evaluation of Message-oriented Middleware using the SPECjms2007 Benchmark. Performance Evaluation 66(8), 410–434 (2009)CrossRefGoogle Scholar
  12. 12.
    Stoica, I., Morris, R., Karger, D., Kaashoek, M.F., Balakrishnan, H.: Chord: A scalable peer-to-peer lookup service for internet applications. In: Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, pp. 149–160. ACM, New York (2001)Google Scholar
  13. 13.
    Weingärtner, E., Schmidt, F., Vom Lehn, H., Heer, T., Wehrle, K.: Slicetime: a platform for scalable and accurate network emulation. In: Conference on Networked systems design and implementation, pp. 19–19. USENIX Association (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Karsten Saller
    • 1
  • Kamill Panitzek
    • 2
  • Max Lehn
    • 3
  1. 1.Real-Time-Systems LabTechnische Universität DarmstadtDarmstadtGermany
  2. 2.Telecooperation LabTechnische Universtität DarmstadtDarmstadtGermany
  3. 3.Databases and Distributed Systems GroupTechnische Universtität DarmstadtDarmstadtGermany

Personalised recommendations