A First Comparative Characterization of Multi-cloud Connectivity in Today’s Internet

  • Bahador YeganehEmail author
  • Ramakrishnan Durairajan
  • Reza Rejaie
  • Walter Willinger
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12048)


Today’s enterprises are adopting multi-cloud strategies at an unprecedented pace. Here, a multi-cloud strategy specifies end-to-end connectivity between the multiple cloud providers (CPs) that an enterprise relies on to run its business. This adoption is fueled by the rapid build-out of global-scale private backbones by the large CPs, a rich private peering fabric that interconnects them, and the emergence of new third-party private connectivity providers (e.g., DataPipe, HopOne, etc.). However, little is known about the performance aspects, routing issues, and topological features associated with currently available multi-cloud connectivity options. To shed light on the tradeoffs between these available connectivity options, we take a cloud-to-cloud perspective and present in this paper the results of a cloud-centric measurement study of a coast-to-coast multi-cloud deployment that a typical modern enterprise located in the US may adopt. We deploy VMs in two regions (i.e., VA and CA) of each one of three large cloud providers (i.e., AWS, Azure, and GCP) and connect them using three different options: (i) transit provider-based best-effort public Internet (BEP), (ii) third-party provider-based private (TPP) connectivity, and (iii) CP-based private (CPP) connectivity. By performing active measurements in this real-world multi-cloud deployment, we provide new insights into variability in the performance of TPP, the stability in performance and topology of CPP, and the absence of transit providers for CPP.

Supplementary material


  1. 1.
    A first comparative characterization of multi-cloud connectivity in today’s internet (2020).
  2. 2.
    Ager, B., Chatzis, N., Feldmann, A., Sarrar, N., Uhlig, S., Willinger, W.: Anatomy of a large European IXP. In: SIGCOMM. ACM (2012)Google Scholar
  3. 3.
    Alexander, M., Luckie, M., Dhamdhere, A., Huffaker, B., Claffy, K., Jonathan, S.M.: Pushing the boundaries with bdrmapIT: mapping router ownership at internet scale. In: Internet Measurement Conference (IMC). ACM (2018)Google Scholar
  4. 4.
    Amazon: AWS direct connect.
  5. 5.
    Amazon: AWS direct connect partners.
  6. 6.
    Amazon: AWS transit gateway.
  7. 7.
    Amazon: AWS direct connect pricing (2019).
  8. 8.
    Amazon: EC2 instance pricing - Amazon web services (2019).
  9. 9.
    Anwar, R., Niaz, H., Choffnes, D., Cunha, Í., Gill, P., Katz-Bassett, E.: Investigating interdomain routing policies in the wild. In: Internet Measurement Conference (IMC). ACM (2015)Google Scholar
  10. 10.
    Augustin, B., et al.: Avoiding traceroute anomalies with paris traceroute. In: Internet Measurement Conference (IMC). ACM (2006)Google Scholar
  11. 11.
    Augustin, B., Krishnamurthy, B., Willinger, W.: IXPs: mapped? In: Internet Measurement Conference (IMC). ACM (2009)Google Scholar
  12. 12.
    Ausmees, K., John, A., Toor, S.Z., Hellander, A., Nettelblad, C.: BAMSI: a multi-cloud service for scalable distributed filtering of massive genome data. BMC Bioinf. 19, 240 (2018)CrossRefGoogle Scholar
  13. 13.
    CAIDA: the skitter project (2007).
  14. 14.
    Calder, M., Fan, X., Hu, Z., Katz-Bassett, E., Heidemann, J., Govindan, R.: Mapping the expansion of Google’s serving infrastructure. In: Internet Measurement Conference (IMC). ACM (2013)Google Scholar
  15. 15.
    Calder, M., Flavel, A., Katz-Bassett, E., Mahajan, R., Padhye, J.: Analyzing the performance of an anycast CDN. In: Internet Measurement Conference (IMC). ACM (2015)Google Scholar
  16. 16.
    Calder, M., et al.: Odin: Microsoft’s scalable fault-tolerant \(\{\)CDN\(\}\) measurement system. In: NSDI. USENIX (2018)Google Scholar
  17. 17.
    Chiu, Y.C., Schlinker, B., Radhakrishnan, A.B., Katz-Bassett, E., Govindan, R.: Are we one hop away from a better internet? In: Internet Measurement Conference (IMC). ACM (2015)Google Scholar
  18. 18.
    CloudHarmony: Cloudharmony, transparency for the cloud.
  19. 19.
  20. 20.
    Cunha, Í., et al.: Sibyl: a practical internet route oracle. In: NSDI. USENIX (2016)Google Scholar
  21. 21.
    Demchenko, Y., et al.: Open Cloud Exchange (OCX): architecture and functional components. In: International Conference on Cloud Computing Technology and Science. IEEE (2013)Google Scholar
  22. 22.
    Dhamdhere, A., Dovrolis, C.: The Internet is flat: modeling the transition from a transit hierarchy to a peering mesh. In: CoNEXT. ACM (2010)Google Scholar
  23. 23.
    Durairajan, R., Barford, P., Sommers, J., Willinger, W.: InterTubes: a study of the US long-haul fiber-optic infrastructure. In: SIGCOMM. ACM (2015)Google Scholar
  24. 24.
    Durairajan, R., Ghosh, S., Tang, X., Barford, P., Eriksson, B.: Internet Atlas: a geographic database of the Internet. In: HotPlanet. ACM (2013)Google Scholar
  25. 25.
    Elshazly, H., Souilmi, Y., Tonellato, P.J., Wall, D.P., Abouelhoda, M.: MC-GenomeKey: a multicloud system for the detection and annotation of genomic variants. BMC Bioinf. 18, 49 (2017)CrossRefGoogle Scholar
  26. 26.
    Facebook: Building express backbone: Facebook’s new long-haul network (2017).
  27. 27.
    Gill, P., Arlitt, M., Li, Z., Mahanti, A.: The Flattening Internet topology: natural evolution, unsightly barnacles or contrived collapse? In: Claypool, M., Uhlig, S. (eds.) PAM 2008. LNCS, vol. 4979, pp. 1–10. Springer, Heidelberg (2008). Scholar
  28. 28.
  29. 29.
  30. 30.
    Haq, O., Raja, M., Dogar, F.R.: Measuring and improving the reliability of wide-area cloud paths. In: WWW. ACM (2017)Google Scholar
  31. 31.
    Hofmann, H., Kafadar, K., Wickham, H.: Letter-value plots: boxplots for large data. Technical report. (2011)Google Scholar
  32. 32.
    Huffaker, B., Keys, K., Fomenkov, M., Claffy, K.: AS-to-organization dataset (2018).
  33. 33.
    Hung, C.C., Ananthanarayanan, G., Golubchik, L., Yu, M., Zhang, M.: Wide-area analytics with multiple resources. In: EuroSys Conference. ACM (2018)Google Scholar
  34. 34.
    Internet2: One-Way Ping (OWAMP) (2019).
  35. 35.
    Iyer, A.P., Panda, A., Chowdhury, M., Akella, A., Shenker, S., Stoica, I.: Monarch: gaining command on geo-distributed graph analytics. In: Hot Topics in Cloud Computing (HotCloud). USENIX (2018)Google Scholar
  36. 36.
    Khalidi, Y.: How Microsoft builds its fast and reliable global network (2017).
  37. 37.
    Klöti, R., Ager, B., Kotronis, V., Nomikos, G., Dimitropoulos, X.: A comparative look into public IXP datasets. In: SIGCOMM CCR (2016)Google Scholar
  38. 38.
    Knight, S., Nguyen, H.X., Falkner, N., Bowden, R.A., Roughan, M.: The Internet topology zoo. In: JSAC. IEEE (2011)Google Scholar
  39. 39.
    Krishna, A., Cowley, S., Singh, S., Kesterson-Townes, L.: Assembling your cloud orchestra: a field guide to multicloud management.
  40. 40.
    Labovitz, C., Iekel-Johnson, S., McPherson, D., Oberheide, J., Jahanian, F.: Internet inter-domain traffic. In: SIGCOMM. ACM (2010)Google Scholar
  41. 41.
    Li, A., Yang, X., Kandula, S., Zhang, M.: CloudCmp: comparing public cloud providers. In: Internet Measurement Conference (IMC). ACM (2010)Google Scholar
  42. 42.
    Luckie, M.: Scamper: a scalable and extensible packet prober for active measurement of the Internet. In: Internet Measurement Conference (IMC). ACM (2010)Google Scholar
  43. 43.
    Luckie, M., Dhamdhere, A., Huffaker, B., Clark, D., et al.: bdrmap: inference of borders between IP networks. In: Internet Measurement Conference (IMC). ACM (2016)Google Scholar
  44. 44.
    Madhyastha, H.V., et al.: iPlane: an information plane for distributed services. In: OSDI. USENIX (2006)Google Scholar
  45. 45.
    Mao, Z.M., Rexford, J., Wang, J., Katz, R.H.: Towards an accurate AS-level traceroute tool. In: SIGCOMM. ACM (2003)Google Scholar
  46. 46.
    Marder, A., Smith, J.M.: MAP-IT: multipass accurate passive inferences from traceroute. In: Internet Measurement Conference (IMC). ACM (2016)Google Scholar
  47. 47.
    Mathis, M., Semke, J., Mahdavi, J., Ott, T.: The macroscopic behavior of the TCP congestion avoidance algorithm. In: SIGCOMM CCR (1997)Google Scholar
  48. 48.
    Megaport: Megaport pricing (2019).
  49. 49.
  50. 50.
  51. 51.
    Microsoft: Expressroute partners and peering locations.
  52. 52.
    Motamedi, R., Rejaie, R., Willinger, W.: A survey of techniques for Internet topology discovery. Commun. Surv. Tutor. 17, 1044–1065 (2014)CrossRefGoogle Scholar
  53. 53.
    PacketFabric: Cloud Connectivity (2019).
  54. 54.
    Padhye, J., Firoiu, V., Towsley, D., Kurose, J.: Modeling TCP throughput: a simple model and its empirical validation. In: SIGCOMM CCR (1998)Google Scholar
  55. 55.
    PCH: Packet Clearing House (2019).
  56. 56.
    PeeringDB: PeeringDB (2019).
  57. 57.
    Pu, Q., et al.: Low latency geo-distributed data analytics. In: SIGCOMM CCR (2015)Google Scholar
  58. 58.
    Pureport: Pricing - Pureport (2019).
  59. 59.
    RIPE: RIPE RIS (2019)Google Scholar
  60. 60.
    Schlinker, B., et al.: Engineering egress with edge fabric: steering oceans of content to the world. In: SIGCOMM. ACM (2017)Google Scholar
  61. 61.
    Sermpezis, P., Nomikos, G., Dimitropoulos, X.A.: Re-mapping the Internet: bring the IXPs into play. CoRR (2017)Google Scholar
  62. 62.
    Shavitt, Y., Shir, E.: DIMES: let the internet measure itself. In: SIGCOMM CCR. ACM (2005)Google Scholar
  63. 63.
    Sherwood, R., Bender, A., Spring, N.: Discarte: a disjunctive internet cartographer. In: SIGCOMM. ACM (2008)Google Scholar
  64. 64.
    Sherwood, R., Spring, N.: Touring the Internet in a TCP sidecar. In: SIGCOMM Conference on Measurement. ACM (2006)Google Scholar
  65. 65.
    Spring, N., Mahajan, R., Wetherall, D.: Measuring ISP topologies with Rocketfuel. In: SIGCOMM (2002)Google Scholar
  66. 66.
    Tariq, M.M.B., Dhamdhere, A., Dovrolis, C., Ammar, M.: Poisson versus periodic path probing (or, does pasta matter?). In: Internet Measurement Conference (IMC). ACM (2005)Google Scholar
  67. 67.
    University of Oregon: Route views project.
  68. 68.
    Viswanathan, R., Ananthanarayanan, G., Akella, A.: \(\{\)CLARINET\(\}\): WAN-aware optimization for analytics queries. In: Operating Systems Design and Implementation (\(\{\)OSDI\(\}\)). USENIX (2016)Google Scholar
  69. 69.
    Vulimiri, A., et al.: Wanalytics: geo-distributed analytics for a data intensive world. In: SIGMOD. ACM (2015)Google Scholar
  70. 70.
    Wohlfart, F., Chatzis, N., Dabanoglu, C., Carle, G., Willinger, W.: Leveraging interconnections for performance: the serving infrastructure of a large CDN. In: SIGCOMM. ACM (2018)Google Scholar
  71. 71.
    Yeganeh, B., Durairajan, R., Rejaie, R., Willinger, W.: How cloud traffic goes hiding: a study of Amazon’s peering fabric. In: Internet Measurement Conference (IMC). ACM (2019)Google Scholar
  72. 72.
    ZDNet: Top cloud providers (2019).
  73. 73.
    Zhang, B., Liu, R., Massey, D., Zhang, L.: Collecting the Internet AS-level topology. In: SIGCOMM CCR. ACM (2005)Google Scholar
  74. 74.
    Zhang, H., et al.: Guaranteeing deadlines for inter-data center transfers. Trans. Netw. (TON) 25, 579–595 (2017)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Bahador Yeganeh
    • 1
    Email author
  • Ramakrishnan Durairajan
    • 1
  • Reza Rejaie
    • 1
  • Walter Willinger
    • 2
  1. 1.University of OregonEugeneUSA
  2. 2.NIKSUN Inc.BostonUSA

Personalised recommendations