Where Has My Time Gone?

  • Noa ZilbermanEmail author
  • Matthew Grosvenor
  • Diana Andreea Popescu
  • Neelakandan Manihatty-Bojan
  • Gianni Antichi
  • Marcin Wójcik
  • Andrew W. Moore
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10176)


Time matters. In a networked world, we would like mobile devices to provide a crisp user experience and applications to instantaneously return results. Unfortunately, application performance does not depend solely on processing time, but also on a number of different components that are commonly counted in the overall system latency. Latency is more than just a nuisance to the user, poorly accounted-for, it degrades application performance. In fields such as high frequency trading, as well as in many data centers, latency translates easily to financial losses. Research to date has focused on specific contributions to latency: from improving latency within the network to latency control on the application level. This paper takes an holistic approach to latency, and aims to provide a break-down of end-to-end latency from the application level to the wire. Using a set of crafted experiments, we explore the many contributors to latency. We assert that more attention should be paid to the latency within the host, and show that there is no silver bullet to solve the end-to-end latency challenge in data centers. We believe that a better understanding of the key elements influencing data center latency can trigger a more focused research, improving the user’s quality of experience.


Virtual Machine Round Trip Time User Space High Frequency Trading Spine Switch 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



We would like to thank the many people who contributed to this paper. We would like to thank Salvator Galea and Robert N Watson, who contributed to early work on this paper. This work has received funding from the EPSRC grant EP/K034723/1, Leverhulme Trust Early Career Fellowship ECF-2016-289, European Union’s Horizon 2020 research and innovation programme 2014-2018 under the SSICLOPS (grant agreement No. 644866), ENDEAVOUR (grant agreement No. 644960) and EU FP7 Marie Curie ITN METRICS (grant agreement No. 607728).

Dataset. A reproduction environment of the experiments, and the experimental results, are both available at and


  1. 1.
    Barroso, L.A.: Landheld Computing. In: IEEE International Solid State Circuits Conference (ISSCC) (2014). KeynoteGoogle Scholar
  2. 2.
    Cheshire, S.: It’s the latency, stupid. Accessed July 2016
  3. 3.
    Guo, C., et al.: RDMA over commodity ethernet at scale. In: SIGCOMM 2016 (2016)Google Scholar
  4. 4.
    Hemminger, S.: NetEm - Network Emulator. Accessed July 2016
  5. 5.
    Kalia, A., et al.: Design guidelines for high performance RDMA systems. In: USENIX ATC, vol. 16, pp. 437–450 (2016)Google Scholar
  6. 6.
    Mayer, M.: What Google knows. In: Web 2.0 Summit (2006)Google Scholar
  7. 7.
    Mittal, R., et al.: TIMELY: RTT-based congestion control for the datacenter. SIGCOMM Comput. Commun. Rev. 45, 537–550 (2015). ACMCrossRefGoogle Scholar
  8. 8.
    Nussbaum, L., Richard, O.: A comparative study of network link emulators. In: SpringSim 2009, pp. 85:1–85:8 (2009)Google Scholar
  9. 9.
    Oracle: Oracle VM VirtualBox. Accessed Oct 2016
  10. 10.
    Paoloni, G.: How to benchmark code execution times on Intel IA-32 and IA-64 instruction set architectures. Technical report 324264–001, Intel (2010)Google Scholar
  11. 11.
    Patterson, D.A.: Latency lags bandwidth. Commun. ACM 47(10), 71–75 (2004)CrossRefGoogle Scholar
  12. 12.
    Rumble, S.M., et al.: It’s time for low latency. In: HotOS 2013, p. 11. USENIX Association (2011)Google Scholar
  13. 13.
    SAP: Big data and smart trading (2012)Google Scholar
  14. 14.
    Singh, A., et al.: Jupiter rising: a decade of clos topologies and centralized control in Google’s datacenter network. SIGCOMM Comput. Commun. Rev. 45(4), 183–197 (2015)CrossRefGoogle Scholar
  15. 15.
    Tolly Enterprises: Mellanox spectrum vs. broadcom StrataXGS Tomahawk 25GbE & 100GbE performance evaluation - evaluating consistency & predictability. Technical report 216112 (2016)Google Scholar
  16. 16.
    Zilberman, N., et al.: NetFPGA SUME: toward 100 Gbps as research commodity. IEEE Micro 34(5), 32–41 (2014)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Noa Zilberman
    • 1
    Email author
  • Matthew Grosvenor
    • 1
  • Diana Andreea Popescu
    • 1
  • Neelakandan Manihatty-Bojan
    • 1
  • Gianni Antichi
    • 1
  • Marcin Wójcik
    • 1
  • Andrew W. Moore
    • 1
  1. 1.Computer LaboratoryUniversity of CambridgeCambridgeUK

Personalised recommendations