Abstract
Distributed Hash Tables (DHT) are presently used in several large-scale systems in the Internet and envisaged as a key mechanism to provide identifier-locator separation for mobile hosts in Future Internet. Such P2P-based systems become increasingly complex serving popular social networking, resource sharing applications, and Internet-scale infrastructures. Hierarchy is a standard mechanism for coping with heterogeneity and scalability in distributed systems. To address the shortcomings of flat DHT designs, many hierarchical P2P designs have been proposed over recent years. The last generation is hierarchical DHTs (HDHTs) where nodes are organized onto layers and groups. This article discusses hierarchical architectures applied in structured P2P overlay networks, focusing on HDHT designs. We introduce a framework consisting of conceptual models of network hierarchy, multi-layer hierarchical DHT architectures, principles affecting the design choices, and cost models for system tradeoff analysis, performance evaluation, and scalability estimation. Based on the framework we provide a taxonomy and survey more than 20 hierarchical HDHT proposals.
Similar content being viewed by others
Notes
Resource replication, large routing tables, and some other methods exist to improve the performance, but the cost is higher load to nodes, see the discussion in [11] and references therein.
In basic Chord, every node periodically calls the stabilization and fixfingers procedures—a proactive strategy.
References
Stoica I, Morris R, Karger D, Kaashoek MF, Balakrishnan H (2001) Chord: a scalable peer-to-peer lookup service for internet applications. In: Proc. of ACM SIGCOMM’01. ACM Press, San Diego, pp 149–160
Ratnasamy S, Handley PFM, Karp R, Shenker S (2001) A scalable content-addressable network. In: Proc. of ACM SIGCOMM’01. ACM Press, San Diego, pp 161–172
Rowstron A, Druschel P (2001) Pastry: scalable, distributed object location and routing for large-scale peer-to-peer systems. In: Middleware’01: Proc. of IFIP/ACM Int’l Conf. on distributed systems platforms. Volume 2218 of lecture notes in computer science. Springer-Verlag, pp 329–350
Androutsellis-Theotokis S, Spinellis D (2004) A survey of peer-to-peer content distribution technologies. ACM Comput Surv 36(4):335–371
Lua EK, Crowcroft J, Pias M, Sharma R, Lim S (2005) A survey and comparison of peer-to-peer overlay network schemes. IEEE Commun Surv Tutorials 7(2):72–93
Marti S, Garcia-Molina H (2006) Taxonomy of trust: categorizing P2P reputation systems. Comput Netw 50(4):472–484
Risson J, Moors T (2006) Survey of research towards robust peer-to-peer networks: search methods. Comput Netw 50(17):3485–3521
Meshkova E, Riihijärvi J, Petrova M, Mähönen P (2008) A survey on resource discovery mechanisms, peer-to-peer and service discovery frameworks. Comput Netw 52(11):2097–2128
Karrels DR, Peterson GL, Mullins BE (2009) Structured P2P technologies for distributed command and control. Peer-to-Peer Netw Appl 2:311–333
Urdaneta G, Pierre G, Steen MV (2011) A survey of DHT security techniques. ACM Comput Surv 43(2):8:1–8:49
Korzun D, Gurtov A (2011) Survey on hierarchical routing schemes in “flat” distributed hash tables. Peer-to-Peer Netw Appl 4:346–375
Passarella A (2012) A survey on content-centric technologies for the current internet: CDN and P2P solutions. Comput Commun 35(1):1–32
Götz S, Rieche S, Wehrle K (2005) Selected DHT algorithms. In: Steinmetz R, Wehrle K (eds) Peer-to-Peer systems and applications. Volume 3485 of lecture notes in computer science. Springer, Berlin / Heidelberg, pp 95–117
Birman KP (2005) Reliable distributed systems: technologies, web services, and applications. Springer-Verlag, New York
Buford JF, Yu H, Lua EK (2009) P2P networking and applications. Elsevier
Vu QH, Lupu M, Ooi BC (2010) Peer-to-Peer computing: principles and applications. Springer
Korzun D, Gurtov A (2013) Structured peer-to-peer systems: fundamentals of hierarchical organization, routing, scaling, and security. Springer
Kleinrock L, Kamoun F (1977) Hierarchical routing for large networks: performance evaluation and optimization. Comput Netw 1:155–174
Chiang M, Low SH, Calderbank AR, Doyle JC (2007) Layering as optimization decomposition: a mathematical theory of network architectures. Proc IEEE 95:255–312
Lian J, Naik K, Agnew GB (2007) A framework for evaluating the performance of cluster algorithms for hierarchical networks. IEEE/ACM Trans Netw 15:1478–1489
Lloret J, Palau C, Boronat F, Tomas J (2008) Improving networks using group-based topologies. Comput Commun 31:3438–3450
Xu J, Kumar A, Yu X (2004) On the fundamental tradeoffs between routing table size and network diameter in peer-to-peer networks. IEEE J Sel Areas Commun 22(1):151–163
Loguinov D, Kumar A, Rai V, Ganesh S (2005) Graph-theoretic analysis of structured peer-to-peer systems: routing distances and fault resilience. IEEE/ACM Trans Netw 13(5):1107–1120
Gummadi K, Gummadi R, Gribble S, Ratnasamy S, Shenker S, Stoica I (2003) The impact of DHT routing geometry on resilience and proximity. In: Proc. of ACM SIGCOMM’03. ACM Press, pp 381–394
Castro M, Drushel P, Ganesh A, Rowstron A, Wallach DS (2002) Secure routing for structured peer-to-peer overlay networks. In: Proc. 5th USENIX Symp. on operating system design and implementation (OSDI 2002). ACM Press, Boston, pp 299–314
Kleinberg JM (2006) Complex networks and decentralized search algorithms. In: Proc. Int’l congress of mathematicians (ICM 2006). European Mathematical Society
Keshav S (2006) Efficient and decentralized computation of approximate global state. SIGCOMM Comput Commun Rev 36(1):69–74
Li J, Stribling J, Morris R, Kaashoek MF, Gil TM (2005) A performance vs. cost framework for evaluating DHT design tradeoffs under churn. In: Proc. of IEEE INFOCOM’05. Volume 1., IEEE, pp 225–236
Krishnamurthy B, Wang J, Xie Y (2001) Early measurements of a cluster-based architecture for P2P systems. In: Proc. 1st ACM SIGCOMM workshop on internet measurement (IMW ’01), ACM, pp 105–109
Ratnasamy S, Handley M, Karp R, Shenker S (2002) Topologically-aware overlay construction and server selection. In: Proc. of IEEE INFOCOM’02
Xu Z, Mahalingam M, Karlsson M (2003) Turning heterogeneity into an advantage in overlay routing. In: Proc. of IEEE INFOCOM’03, pp 1499–1509
Sánchez-Artigas M, García López P (2010) Echo: a peer-to-peer clustering framework for improving communication in DHTs. J Parallel Distrib Comput 70:126–143
Dabek F, Kaashoek MF, Karger D, Morris R, Stoica I (2001) Wide-area cooperative storage with CFS. In: Proc. 18th ACM Symp. Operating systems principles (SOSP ’01). ACM Press, pp 202–215
Rufino J, Alves A, Exposto J, Pina A (2004) A cluster oriented model for dynamically balanced DHTs. In: IPDPS’04: Proc. 18th Int’l Symp. on parallel and distributed processing. IEEE Computer Society
Surana S, Godfrey B, Lakshminarayanan K, Karp R, Stoica I (2006) Load balancing in dynamic structured peer-to-peer systems. Perform Eval 63(3):217–240
Tang C, Xu Z, Dwarkadas S (2003) Peer-to-peer information retrieval using self-organizing semantic overlay networks. In: Proc. of ACM SIGCOMM’03. ACM Press, pp 175–186
Wan Y, Asaka T, Takahashi T (2008) A hybrid P2P overlay network for non-strictly hierarchically categorized contents. In: Proc. 8th IEEE Int’l Symp. on Cluster Computing and the Grid (CCGRID ’08). IEEE Computer Society, pp 41–48
Xu M, Zhou S, Guan J (2011) A new and effective hierarchical overlay structure for Peer-to-Peer networks. Comput Commun 34(7):862–874
Asiki A, Tsoumakos D, Koziris N (2010) Distributing and searching concept hierarchies: an adaptive DHT-based system. Clust Comput 13(3):257–276
Yang B, Garcia-Molina H (2003) Designing a super-peer network. In: Proc. 19th Int’l Conf. on data engineering (ICDE’03), pp 49–60
Zoels S, Despotovic Z, Kellerer W (2008) On hierarchical DHT systems—an analytical approach for optimal designs. Comput Commun 31(3):576–590
Ou Z, Harjula E, Koskela T, Ylianttila M (2010) GTPP: general truncated pyramid peer-to-peer architecture over structured DHT networks. Mob Netw Appl 15:729–749
Mislove A, Druschel P (2004) Providing administrative control and autonomy in structured peer-to-peer overlays. In: IPTPS ’04: Proc. 3rd Int’l workshop on peer-to-peer systems. Volume 3279 of lecture notes in computer science. Springer, pp 162–172
Karger DR, Ruhl M (2004) Diminished chord: a protocol for heterogeneous subgroup formation in peer-to-peer networks. In: IPTPS ’04: Proc. 3rd Int’l workshop on peer-to-peer systems. Volume 3279 of lecture notes in computer science. Springer, pp 288–297
Zhang Y, Li D, Chen L, Lu X (2008) Flexible routing in grouped DHTs. In: Proc. 8th IEEE Int’l Conf. on peer-to-peer computing (P2P ’08). IEEE Computer Society, pp 109–118
Zhao BY, Huang L, Stribling J, Rhea SC, Joseph AD, Kubiatowicz JD (2004) Tapestry: a resilient global-scale overlay for service deployment. IEEE J Sel Areas Commun 22(1):41–53
Maymounkov P, Mazières D (2002) Kademlia: a peer-to-peer information system based on the XOR metric. In: IPTPS ’02: Proc. 1st Int’l workshop on peer-to-peer systems. Volume 2429 of lecture notes in computer science. Springer, pp 53–65
Malkhi D, Naor M, Ratajczak D (2002) Viceroy: a scalable and dynamic emulation of the butterfly. In: Proc. 21st Annual Symp. on principles of distributed computing (PODC ’02). ACM Press, pp 183–192
Manku GS, Bawa M, Raghavan P (2003) Symphony: distributed hashing in a small world. In: Proc. 4th USENIX Symp. on internet technologies and systems (USITS’03). USENIX Association, pp 127–140
Gai AT, Viennot L (2004) Broose: a practical distributed hashtable based on the De-Bruijn topology. In: Proc. IEEE 4th Int’l Conf. on peer-to-peer computing (P2P ’04). IEEE Computer Society, pp 167–164
Karger D, Lehman E, Leighton T, Panigrahy R, Levine M, Lewin D (1997) Consistent hashing and random trees: distributed caching protocols for relieving hot spots on the world wide web. In: Proc. 29th Annual ACM Symp. on theory of computing (STOC ’97), ACM, pp 654–663
Kleinberg JM (2000) The small-world phenomenon: an algorithm perspective. In: Proc. 32nd Annual ACM Symp. theory of computing (STOC ’00). ACM Press, pp 163–170
Kempe D, Kleinberg J, Demers A (2004) Spatial gossip and resource location protocols. J ACM 51:943–967
Korzun D, Nechaev B, Gurtov A (2009) Cyclic routing: generalizing lookahead in peer-to-peer networks. In: Proc. 7th IEEE/ACS Int’l Conf. on computer systems and applications (AICCSA2009). IEEE Computer Society, pp 697–704
Duchon P, Hanusse N, Lebhar E, Schabanel N (2006) Towards small world emergence. In: Proc. 18th Annual ACM Symp. on parallelism in algorithms and architectures (SPAA ’06), ACM, pp 225–232
Viswanath B, Post A, Gummadi KP, Mislove A (2010) An analysis of social network-based sybil defenses. In: Proc. ACM SIGCOMM 2010 Conf. applications, technologies, architectures, and protocols for computer communication, ACM, pp 363–374
Risson J, Qazi S, Moors T, Harwood A (2006) A dependable global location service using rendezvous on hierarchic distributed hash tables. In: Proc. Int’l Conf. networking, Int’l Conf. systems and Int’l Conf. mobile communications and learning technologies (ICN/ICONS/MCL ’06). IEEE Computer Society
Singh A, Liu L (2004) A hybrid topology architecture for P2P systems. In: Proc. 13th Int’l Conf. on computer communications and networks (ICCCN 2004), pp 475–480
Tian R, Xiong Y, Zhang Q, Li B, Zhao BY, Li X (2005) Hybrid overlay structure based on random walks. In: IPTPS ’05: Proc. 4th Int’l workshop on peer-to-peer systems. Volume 3640 of lecture notes in computer science. Springer, pp 152–162
Artigas MS, Lopez PG, Ahullo JP, Skarmeta AFG (2005) Cyclone: a novel design schema for hierarchical DHTs. In: Proc. 5th IEEE Int’l Conf. on peer-to-peer computing (P2P ’05). IEEE Computer Society, pp 49–56
Hu J, Li M, Zheng W, Wang D, Ning N, Dong H (2004) Smartboa: constructing P2P overlay network in the heterogeneous Internet using irregular routing tables. In: IPTPS ’04: Proc. 3rd Int’l workshop on peer-to-peer systems. Volume 3279 of lecture notes in computer science. Springer, pp 278–287
Leong B, Liskov B, Demaine E (2004) Epichord: parallelizing the Chord lookup algorithm with reactive routing state management. In: Proc. 12th Int’l Conf. on networks (ICON 2004), pp 270–276
Li J, Stribling J, Morris R, Kaashoek MF (2005) Bandwidth-efficient management of DHT routing tables. In: Proc. of the 2nd symposium on networked systems design and implementation (NSDI ’05), pp 99–114
Garcés-Erice L, Biersack E, Felber PA, Ross KW, Urvoy-Keller G (2003) Hierarchical peer-to-peer systems. In: Proc. ACM/IFIP Int’l Conf. on parallel and distributed computing (Euro-Par 2003), pp 643–657
Li X, Wu J (2004) Hierarchical P2P systems in a small world. In: Proc. 2nd Latin American and Caribbean Conf. for engineering and technology (LACCEI’2004)
Lee JW, Schulzrinne H, Kellerer W, Despotovic Z (2009) mDHT: multicast-augmented DHT architecture for high availability and immunity to churn. In: Proc. 6th IEEE Conf. consumer communications and networking conference (CCNC’09), IEEE, pp 760–764
Gupta I, Birman K, Linga P, Demers A, van Renesse R (2003) Kelips: building an efficient and stable P2P DHT through increased memory and background overhead. In: IPTPS ’03: Proc. 2nd Int’l workshop on peer-to-peer systems. Volume 2735 of lecture notes in computer science. Springer, pp 160–169
Mizrak AT, Cheng Y, Kumar V, Savage S (2003) Structured superpeers: leveraging heterogeneity to provide constant-time lookup. In: Proc. 3rd IEEE workshop on internet applications (WIAPP 2003), pp 104–111
Gupta A, Liskov B, Rodrigues R (2004) Efficient routing for peer-to-peer overlays. In: Proc. 1st Symp. on networked systems design and implementation (NSDI ’04)
Fonseca P, Rodrigues R, Gupta A, Liskov B (2009) Full-information lookups for peer-to-peer overlays. IEEE Trans Parallel Distrib Syst 20(9):1339–1351
Monnerat LR, Amorim CL (2009) Peer-to-peer single hop distributed hash tables. In: Proc. of IEEE Globecom’09
Risson J, Harwood A, Moors T (2006) Stable high-capacity one-hop distributed hash tables. In: ISCC ’06: Proc. 11th IEEE Symp. on computers and communications. IEEE Computer Society, pp 687–694
Zoels S, Despotovic Z, Kellerer W (2006) Cost-based analysis of hierarchical DHT design. In: Proc. 6th IEEE Int’l Conf. on peer-to-peer computing (P2P ’06). IEEE Computer Society, pp 233–239
Zöls S, Hofstätter Q, Despotovic Z, Kellerer W (2009) Achieving and maintaining cost-optimal operation of a hierarchical DHT system. In: Proc. 2009 IEEE Int’l Conf. on communications (ICC’09). IEEE Press, pp 2194–2199
Zoels S, Despotovic Z, Kellerer W (2007) Load balancing in a hierarchical DHT-based P2P system. In: Proc. 2007 Int’l Conf. collaborative computing: networking, applications and worksharing (COLCOM ’07). IEEE Computer Society, pp 353–361
Ren XJ, Gu ZM (2007) SA-Chord: a novel P2P system based on self-adaptive joining. In: Proc. 6th Int’l Conf. grid and cooperative computing (GCC 2007). IEEE Computer Society, pp 75–81
Artigas MS, Lopez PG, Skarmeta AFG (2005) A novel methodology for constructing secure multipath overlays. IEEE Internet Comput 9(6):50–57
Zoels S, Eichhorn M, Tarlano A, Kellerer W (2006) Content-based hierarchies in DHT-based peer-to-peer systems. In: Proc. Int’l Symp. applications and the internet workshops (SAINT Workshops 2006). IEEE Computer Society, pp 105–108
Zhang XM, Wang YJ, Li Z (2007) Research of routing algorithm in hierarchy-adaptive P2P systems. In: Proc. 5th Int’l Symp. parallel and distributed processing and applications (ISPA 2007). Volume 4742 of lecture notes in computer science. Springer, pp 728–739
Xu Z, Zhang Z (2002) Building low-maintenance expressways for P2P systems. Techical Report HPL-2002-41, HP Labs, Palo Alto
Zhang Z, Shi SM, Zhu J (2002) Self-balanced P2P expressway: when Marxism meets Confucian. Techical Report MSR-TR-2002-72, Microsoft Research Asia
Joung YJ, Wang JC (2007) Chord2: a two-layer chord for reducing maintenance overhead via heterogeneity. Comput Commun 51(3):712–731
Tanta-ngai H, McAllister M (2006) A peer-to-peer expressway over chord. Math Comput Model 44(7–8):659–677
Martinez-Yelmo I, Cuevas R, Guerrero C, Mauthe A (2008) Routing performance in a hierarchical DHT-based overlay network. In: Proc. 16th Euromicro Conf. parallel, distributed and network-based processing (PDP 2008). IEEE Computer Society, pp 508–515
Martinez-Yelmo I, Bikfalvi A, Guerrero C, Rumin RC, Mauthe A (2008) Enabling global multimedia distributed services based on hierarchical DHT overlay networks. Int J Internet Protocol Technol (IJIPT) 3(4):234–244
Martinez-Yelmo I, Guerrero C, Rumín RC, Mauthe A (2009) A hierarchical P2PSIP architecture to support skype-like services. In: Proc. 17th Euromicro Int’l Conf. parallel, distributed and network-based processing (PDP 2009). IEEE Computer Society, pp 316–322
Min SH, Holliday J, Cho DS (2006) Optimal super-peer selection for large-scale P2P system. In: Proc. 2006 Int’l Conf. hybrid information technology (ICHIT ’06). IEEE Computer Society, pp 588–593
Le L, Kuo GS (2007) Hierarchical and breathing peer-to-peer SIP system. In: Proc. IEEE Int’l Conf. communications (ICC 2007). IEEE, pp 1887–1892
Heristyo A, Masuyama H, Kasahara S, Takahashi Y (2009) User-search time analysis for hierarchical peer-to-peer overlay networks with time-dependent user-population process. In: Proc. 4th Int’l Conf. queueing theory and network applications (QTNA’09). ACM, pp 5:1–5:4
Guisheng Y, Jie S, Xianghui W (2008) Hierarchical small-world P2P networks. In: Proc. Int’l Conf. internet computing in science and engineering (ICICSE ’08). IEEE Computer Society, pp 452–458
Mesaros VA, Carton B, Roy PV (2003) S-Chord: using symmetry to improve lookup efficiency in chord. In: Proc. Int’l Conf. parallel and distributed processing techniques and applications (PDPTA’03)
Ganesan P, Manku GS (2004) Optimal routing in chord. In: Proc. 15th annual ACM-SIAM Symp. Discrete algorithms (SODA ’04). Society for Industrial and Applied Mathematics, pp 176–185
Zhao BY, Duan Y, Huang L, Joseph AD, Kubiatowicz JD (2002) Brocade: landmark routing on overlay networks. In: IPTPS ’02: Proc. 1st Int’l workshop on peer-to-peer systems. Volume 2429 of lecture notes in computer science. Springer, pp 34–44
Xu Z, Tang C, Zhang Z (2003) Building topology-aware overlays using global soft-state. In: Proc. 23rd Int’l Conf. distributed computing systems (ICDCS’03). IEEE Computer Society, pp 500–508
Zhu Y, Wang H, Hu Y (2003) A super-peer based lookup in structured peer-to-peer systems. In: Proc. ISCA 16th Int’l Conf. parallel and distributed computing systems (PDCS 2003), pp 465–470
Freedman MJ, Mazières D (2003) Sloppy hashing and self-organizing clusters. In: IPTPS ’03: Proc. 2nd Int’l workshop on peer-to-peer systems. Volume 2735 of lecture notes in computer science. Springer, pp 45–55
Xu Z, Min R, Hu Y (2003) HIERAS: a DHT based hierarchical P2P routing algorithm. In: Proc. 32nd Int’l Conf. parallel processing (ICPP 2003). IEEE Computer Society, pp 187–194
Park K, Pack S, Kwon T (2008) Proximity based peer-to-peer overlay networks (P3ON) with load distribution. In: Proc. Int’l Conf. information networking (ICOIN 2007). Towards ubiquitous networking and services. Revised selected papers. Springer-Verlag, pp 234–243
Xu J, Jin H (2009) A structured P2P network based on the small world phenomenon. J Supercomput 48:264–285
Garcés-Erice L, Ross KW, Biersack EW, Felber P, Urvoy-Keller G (2003) Topology-centric look-up service. In: Proc. 5th Int’l Conf. group communications and charges (NGC 2003), workshop on networked group communication. Volume 2816 of lecture notes in computer science. Springer, pp 58–69
Ganesan P, Gummadi K, Garcia-Molina H (2004) Canon in G major: designing DHTs with hierarchical structure. In: Proc. 24th Int’l Conf. distributed computing systems (ICDCS ’04). IEEE Computer Society, pp 263–272
Zhang Y, Chen L, Lu X, Li D (2010) Enabling routing control in a DHT. IEEE J Sel Areas Commun 28(1):28–38
Bermond JC, Choplin S, Prennes S (2003) Hierarchical ring network design. Theory Comput Syst 36:663–682
Artigas MS, Lopez PG, Skarmeta AF (2007) A comparative study of hierarchical DHT systems. In: Proc. 32nd IEEE Conf. on local computer networks (LCN ’07). IEEE Computer Society, pp 325–333
D’Ambrosio M, Dannewitz C, Karl H, Vercellone V (2011) MDHT: a hierarchical name resolution service for information-centric networks. In: Proc. ACM SIGCOMM workshop on Information-centric networking (ICN ’11). ACM, pp 7–12
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Korzun, D., Gurtov, A. Hierarchical architectures in structured peer-to-peer overlay networks. Peer-to-Peer Netw. Appl. 7, 359–395 (2014). https://doi.org/10.1007/s12083-013-0200-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12083-013-0200-z