The GENI Book pp 317-343 | Cite as

The InstaGENI Project

  • Rick McGeer
  • Robert Ricci


In this chapter we describe InstaGENI, built in response to the GENI Mesoscale initiative (Berman et al., Comput Netw 61:5–23, 2014). InstaGENI was designed both as a distributed cloud, to permit experimenters to run distributed systems and networking experiments, across the wide area, and as a meta-cloud, to permit systems researchers to build experimental clouds within the underlying InstaGENI cloud. InstaGENI consists of more than 36 sites spread across the GENI infrastructure, interconnected by a nationwide, deeply-programmable layer-2 network. Each site is capable of functioning as an autonomous, standalone cloud, with builtin HaaS, IaaS, and OpenFlow (The Openflow Switch Specification.; McKeown et al., ACM SIGCOMM CCR 38(2):69–74, 2008) native support. Sites are also and by default linked, to offer slices across the entire GENI Mesoscale infrastructure. InstaGENI targeted and has realized its key design goals of expandability, reliability, resistance to partition, ease of maintenance upgrade, high distribution, and affordability. InstaGENI offers a highly-scalable infrastructure with OpenFlow native both between and across sites. It has demonstrated a high degree of autonomy and remote management, and has demonstrated its meta-cloud properties by hosting an IaaS and PaaS service within it, GENI PlanetLab and the GENI Experiment Engine (Bavier et al., The GENI experiment engine. In: Proceedings of Tridentcom, 2015).


Virtual Machine Control Plane Data Plane Control Node Internet Protocol Address 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



InstaGENI is a large, complex project, and there are many people who contributed to its success. This paper is an extension of, and based heavily, on a journal paper signed by the entire InstaGENI team [8], and we would first like to express our heartfelt thanks to them. Of especial note are Joe Mambretti, Fei Yeh, and Jim Chen, who worked closely with us on network design; Narayan Krishnan, who did the original hardware design to match ProtoGENI’s software specifications. It is hard to adequately describe the logistical challenges in working 35 non-standard orders through a manufacturer, ensuring their delivery to 35 separate sites, and maintaining a complex, multi-year, multi-million dollar budget. InstaGENI was an enormous project management challenge, and we were fortunate that Nicki Watts was kind enough to devote a great deal of time to this project; it literally would not have happened without her. When one of us (McGeer) moved on from HP, Jack Brassil took over as Principal Investigator on the project and completed it brilliantly. We had tremendous support from the GENI Project Office, notably Niky Riga, Heidi Dempsey, Vic Thomas, Henry Yeh, and especially Mark Berman. Leigh Stoller of the Flux research group has been extremely generous in offering operational support to InstaGENI users and experimenters. Larry Singer, then of HP Americas, offered his support for commercialization and Michaela Mezo helped enormously in that area. Shannon Champion of Matrix Integration was instrumental in making InstaGENI an HP product. Moreover, the 36+ PIs and system administrators at the InstaGENI sites have been responsive to our requests and to keep InstaGENI going. We thank all of them.

A special note is given to Chip Elliott. InstaGENI and ExoGENI were Chip’s inspiration when he was GENI Project Director. It was Chip who mapped out the deployment strategy for the GENI Racks, and he worked closely with us on the initial strategic decisions that gave the project its focus. He also sharpened for us the role of these racks in the coming Internet. This project is very much his creation.


  1. 1.
    Anderson, T., Peterson, L., Shenker, S., Turner, J.: Overcoming the Internet impasse through virtualization. Computer 38 (4), 34–41 (2005)CrossRefGoogle Scholar
  2. 2.
    Baldine, I., Xin, Y., Mandal, A., Renci, C., Chase, J., Marupadi, V., Yumerefendi, A., Irwin, D.: Networked cloud orchestration: a GENI perspective. In: 2010 IEEE GLOBECOM Workshops (GC Wkshps), pp. 573–578 (2010)Google Scholar
  3. 3.
    Baldine, I., Xin, Y., Mandal, A., Ruth, P., Heerman, C., Chase, J.: ExoGENI: a multi-domain infrastructure-as-a-service testbed. In: Testbeds and Research Infrastructure. Development of Networks and Communities, pp. 97–113. Springer, New York (2012)Google Scholar
  4. 4.
    Baldin, I., Castillo, C., Chase, J., Orlikowski, V., Xin, Y., Heermann, C., Mandal, A., Ruth, P., Mills, J.: ExoGENI: a multi-domain infrastructure-as-a-service testbed. In: GENI: Prototype of the Next Internet. Springer, New York (2016)Google Scholar
  5. 5.
    Bannazadeh, H., Leon-Garcia, A., Redmond, K., Tam, G., Khan, A., Ma, M., Dani, S., Chow, P.: Virtualized application networking infrastructure. In: Testbeds and Research Infrastructures. Development of Networks and Communities - 6th International ICST Conference, TridentCom 2010, Berlin, 18–20 May 2010, Revised Selected Papers, pp. 363–382 (2010)Google Scholar
  6. 6.
    Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., Warfield, A.: Xen and the art of virtualization. ACM SIGOPS Oper. Syst. Rev. 37 (5), 164–177 (2003)CrossRefGoogle Scholar
  7. 7.
    Bastin, N.: Foam: an openflow aggregate manager. (2013)
  8. 8.
    Bastin, N., Bavier, A., Blaine, J., Chen, J., Krishnan, N., Mambretti, J., Mcgeer, R., Ricci, R., Watts, N.: The instaGENI initiative: an architecture for distributed systems and advanced programmable networks. Comput. Netw. 61, 24–38 (2014)CrossRefGoogle Scholar
  9. 9.
    Bavier, A., McGeer, R.: The GENI experiment engine. In: GENI: Prototype of the Next Internet. Springer, New York (2016)Google Scholar
  10. 10.
    Bavier, A.C., Bowman, M., Chun, B.N., Culler, D.E., Karlin, S., Muir, S., Peterson, L.L., Roscoe, T., Spalink, T., Wawrzoniak, M.: Operating systems support for planetary-scale network services. In: NSDI, vol. 4, pp. 19–19 (2004)Google Scholar
  11. 11.
    Bavier, A., Chen, J., Mambretti, J., McGeer, R., McGeer, S., Nelson, J., O’Connell, P., Tredger, S., Coady, Y.: The GENI experiment engine. In: Proceedings of Tridentcom (2015)CrossRefGoogle Scholar
  12. 12.
    Berman, M., Brinn, M.: Progress and challenges in worldwide federation of future internet and distributed cloud testbeds. In: 2014 International Science and Technology Conference (Modern Networking Technologies) (MoNeTeC), pp. 1–6 (2014)Google Scholar
  13. 13.
    Berman, M., Chase, J.S., Landweber, L., Nakao, A., Ott, M., Raychaudhuri, D., Ricci, R., Seskar, I.: GENI: a federated testbed for innovative network experiments. Comput. Netw. 61, 5–23 (2014). Special issue on Future Internet Testbeds - Part IGoogle Scholar
  14. 14.
    Bhanage, G., Seskar, I., Raychaudhuri, D.: A virtualization architecture for mobile WiMAX networks. SIGMOBILE Mob. Comput. Commun. Rev. 15 (4), 26–37 (2012)CrossRefGoogle Scholar
  15. 15.
    Bhojwani, S., Hemmings, M., Ingalls, D., Krahn, R., Lary, D., Lincke, J., McGeer, R., Ricart, G., Roder, M., Coady, Y., Stege, U.: The ignite distributed collaborative scientific visualization system. In: Distributed Cloud Computing Workshop (2015)CrossRefGoogle Scholar
  16. 16.
    Bhojwani, S., Hemmings, M., Ingalls, D., Krahn, R., Lary, D., Lincke, J., McGeer, R., Ricart, G., Roder, M., Coady, Y., Stege, U.: The ignite distributed collaborative scientific visualization system. In: Proceedings of IEEE CloudCom (2015)CrossRefGoogle Scholar
  17. 17.
    Brinn, M.: GENI architecture foundation. In: GENI: Prototype of the Next Internet. Springer, New York (2016)Google Scholar
  18. 18.
    Brinn, M., Bastin, N., Bavier, A., Berman, M., Chase, J., Ricci, R.: Trust as the foundation of resource exchange in GENI. In: Proceedings of the 10th International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities (TRIDENTCOM) (2015)Google Scholar
  19. 19.
    Brown, D., Ascigil, O., Nasir, H., Carpenter, C., Griffioen, J., Calvert, K.: Designing a GENI experimenter tool to support the choice net internet architecture. In: 2014 IEEE 22nd International Conference on Network Protocols (ICNP), pp. 548–554 (2014)Google Scholar
  20. 20.
    Brown, D., Nasir, H., Carpenter, C., Ascigil, O., Griffioen, J., Calvert, K.: Choicenet gaming: changing the gaming experience with economics. In: Computer Games: AI, Animation, Mobile, Multimedia, Educational and Serious Games (CGAMES), 2014, pp. 1–5 (2014)Google Scholar
  21. 21.
    Chakrabortty, A., Xin, Y.: Hardware-in-the-loop simulations and verifications of smart power systems over an exo-GENI testbed. In: 2013 Second GENI Research and Educational Experiment Workshop (GREE), pp. 16–19 (2013)Google Scholar
  22. 22.
    Chu, Y.-h., Rao, S.G., Zhang, H.: A case for end system multicast (keynote address). In: ACM SIGMETRICS Performance Evaluation Review, vol. 28, pp. 1–12. ACM, New York (2000)Google Scholar
  23. 23.
    Chun, B., Culler, D., Roscoe, T., Bavier, A., Peterson, L., Wawrzoniak, M., Bowman, M.: PlanetLab: an overlay testbed for broad-coverage services. ACM SIGCOMM Comput. Commun. Rev. 33 (3), 3–12 (2003)CrossRefGoogle Scholar
  24. 24.
    Dempsey, H.: The GENI mesoscale network. In: GENI: Prototype of the Next Internet. Springer, New York (2016)Google Scholar
  25. 25.
  26. 26.
  27. 27.
    Elliott, C., Falk, A.: An update on the GENI project. SIGCOMM Comput. Commun. Rev. 39 (3), 28–34 (2009)CrossRefGoogle Scholar
  28. 28.
    Faraji, M., Kang, J., Bannazadeh, H., Leon-Garcia, A.: Identity access management for multi-tier cloud infrastructures. In: 2014 IEEE Network Operations and Management Symposium, NOMS 2014, Krakow, 5–9 May 2014, pp. 1–9 (2014)Google Scholar
  29. 29.
    Farinacci, D., Li, T., Hanks, S., Meyer, D., Traina, P.: Generic Routing Encapsulation (GRE). RFC 2784 (Proposed Standard) (2000). Updated by RFC 2890Google Scholar
  30. 30.
    FITS. Future internet testbed with security. (2015)
  31. 31.
    Fragni, C., Moreira, M.D., Mattos, D.M., Costa, L.H.M., Duarte, O.C.M.: Evaluating Xen, VMware, and OpenVZ virtualization platforms for network virtualization. Universidade Federal do Rio de Janeiro’GTA/PEE/COPPE - Rio de Janeiro (2010)Google Scholar
  32. 32.
    Freedman, M.J., Freudenthal, E., Mazieres, D.: Democratizing content publication with coral. In: NSDI, vol. 4, pp. 18–18 (2004)Google Scholar
  33. 33.
    Fund, F., Dong, C., Korakis, T., Panwar, S.: A framework for multidimensional measurements on an experimental wimax testbed. In: Korakis, T., Zink, M., Ott, M. (eds.) Testbeds and Research Infrastructure. Development of Networks and Communities. Lecture Notes of the Institute for Computer Sciences. Social Informatics and Telecommunications Engineering, vol. 44, pp. 369–371. Springer, Berlin/Heidelberg (2012)Google Scholar
  34. 34.
    GENI Planning Group. GENI design principles. Computer 39 (9), 102–105 (2006)Google Scholar
  35. 35.
  36. 36.
    GPO. The omni GENI client. (2011)
  37. 37.
    Gran, E.G., Dreibholz, T., Kvalbein, A.: Nornet core-a multi-homed research testbed. Comput. Netw. 61, 75–87 (2014)CrossRefGoogle Scholar
  38. 38.
    Griffioen, J., Fei, Z., Nasir, H., Carpenter, C., Reed, J., Wu, X., S.R.P.: The GENI desktop. In: GENI: Prototype of the Next Internet. Springer, New York (2016)Google Scholar
  39. 39.
    Hemmings, M., Lary, D., McGeer, R., Ricart, G.: The ignite distributed collaborative scientific visualization system. In: GENI: Prototype of the Next Internet. Springer, New York (2016)Google Scholar
  40. 40.
    Hermenier, F., Ricci, R.: How to build a better testbed: lessons from a decade of network experiments on Emulab. In: Proceedings of the 8th International ICST Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities (Tridentcom) (2012)Google Scholar
  41. 41.
    Jacobson, V., Mosko, M., Smetters, D., Garcia-Luna-Aceves, J.: Content-centric networking. Whitepaper, Palo Alto Research Center, pp. 2–4 (2007)Google Scholar
  42. 42.
    Jofre, J., Velayos, C., Landi, G., Giertych, M., Hume, A.C., Francis, G., Oton, A.V.: Federation of the bonfire multi-cloud infrastructure with networking facilities. Comput. Netw. 61, 184–196 (2014)CrossRefGoogle Scholar
  43. 43.
    Kang, J., Bannazadeh, H., Leon-Garcia, A.: SAVI testbed: control and management of converged virtual ICT resources. In: 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013), Ghent, 27–31 May 2013, pp. 664–667 (2013)Google Scholar
  44. 44.
    Kang, J., Lin, T., Bannazadeh, H., Leon-Garcia, A.: Software-defined infrastructure and the SAVI testbed. In: Testbeds and Research Infrastructure: Development of Networks and Communities - 9th International ICST Conference, TridentCom 2014, Guangzhou, 5–7 May 2014, Revised Selected Papers, pp. 3–13 (2014)Google Scholar
  45. 45.
    Koren Future Network Testbed. (2015)
  46. 46.
    Krishnappa, D., Lyons, E., Irwin, D., Zink, M.: Network capabilities of cloud services for a real time scientific application. In: 2012 IEEE 37th Conference on Local Computer Networks (LCN), pp. 487–495 (2012)Google Scholar
  47. 47.
    Leon-Garcia, A., Bannazadeh, H.: Savi testbed for applications on software-defined infrastructure. In: GENI: Prototype of the Next Internet. Springer, New York (2016)Google Scholar
  48. 48.
  49. 49.
    McKeown, N., Anderson, T., Balakrishnan, H., Parulkar, G., Peterson, L., Rexford, J., Shenker, S., Turner, J.: Openflow: enabling innovation in campus networks. ACM SIGCOMM CCR 38 (2), 69–74 (2008)CrossRefGoogle Scholar
  50. 50.
    Melazzi, N.B., Detti, A., Mazza, G., Morabito, G., Salsano, S., Veltri, L.: An openflow-based testbed for information centric networking. In: Future Network & Mobile Summit (FutureNetw), 2012, pp. 1–9. IEEE, New York (2012)Google Scholar
  51. 51.
    Moraes, I.M., Mattos, D.M., Ferraz, L.H.G., Campista, M.E.M., Rubinstein, M.G., Costa, L.H.M., de Amorim, M.D., Velloso, P.B., Duarte, O.C.M., Pujolle, G.: Fits: a flexible virtual network testbed architecture. Comput. Netw. 63, 221–237 (2014)CrossRefGoogle Scholar
  52. 52.
    Mueller, P.: Europe’s mission in next-generation networking with special emphasis on the German-lab project. In: GENI: Prototype of the Next Internet. Springer, New York (2016)Google Scholar
  53. 53.
    Nakao, A.: Vnode: a deeply programmable network testbed through network virtualization. (2012)
  54. 54.
    Nelson, J., Peterson, L.: Syndicate: democratizing cloud storage and caching through service composition. In: Proceedings of the 4th annual Symposium on Cloud Computing, p. 46. ACM, New York (2013)Google Scholar
  55. 55.
    NorNet. A real-world, large-scale multi-homing testbed. (2015)
  56. 56.
    Ofelia. Openflow in Europe linking infrastructure and applications. (2015)
  57. 57.
    Ozcelik, I., Brooks, R.R.: Security experimentation using operational systems. In: Proceedings of the Seventh Annual Workshop on Cyber Security and Information Intelligence Research, CSIIRW ’11, pp. 79:1–79:1. ACM, New York (2011)Google Scholar
  58. 58.
    Perino, D., Varvello, M.: A reality check for content centric networking. In: Proceedings of the ACM SIGCOMM Workshop on Information-Centric Networking, pp. 44–49. ACM, New York (2011)Google Scholar
  59. 59.
    Peterson, L., Anderson, T., Culler, D., Roscoe, T.: A blueprint for introducing disruptive technology into the Internet. In: Proceedings of HotNets-I, Princeton, NJ (2002)Google Scholar
  60. 60.
    Peterson, L., Bavier, A., Fiuczynski, M.E., Muir, S.: Experiences building planetlab. In: Proceedings of the 7th Symposium on Operating Systems Design and Implementation, pp. 351–366. USENIX Association (2006)Google Scholar
  61. 61.
    Peterson, L.L., Baker, S., Leenheer, M.D., Bavier, A.C., Bhatia, S., Wawrzoniak, M., Nelson, J.C., Hartman, J.H.: XOS: an extensible cloud operating system. In: Proceedings of the 2nd International Workshop on Software-Defined Ecosystems, BigSystem 2015, Portland, OR, 16 June 2015, pp. 23–30 (2015)Google Scholar
  62. 62.
    Ricci, R., Duerig, J., Stoller, L., Wong, G., Chikkulapelly, S., Seok, W.: Designing a federated testbed as a distributed system. In: Proceedings of the 8th International ICST Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities (Tridentcom) (2012)Google Scholar
  63. 63.
    Ricci, R., Eide, E., The CloudLab Team.: Introducing CloudLab: scientific infrastructure for advancing cloud architectures and applications. USENIX;login: 39 (6), 36–38 (2014)Google Scholar
  64. 64.
    Schwerdel, D., Reuther, B., Zinner, T., Müller, P., Tran-Gia, P.: Future Internet research and experimentation: the G-lab approach. Comput. Netw. 61, 102–117 (2014)CrossRefGoogle Scholar
  65. 65.
    Sherwood, R., Gibb, G., Yap, K.-K., Appenzeller, G., Casado, M., McKeown, N., Parulkar, G.: Can the production network be the testbed? In: Operating Systems Design and Implementation (OSDI) (2010)Google Scholar
  66. 66.
    Soltesz, S., Pötzl, H., Fiuczynski, M.E., Bavier, A., Peterson, L.: Container-based operating system virtualization: a scalable, high-performance alternative to hypervisors. SIGOPS Oper. Syst. Rev. 41 (3), 275–287 (2007)CrossRefGoogle Scholar
  67. 67.
    Soroush, H., Banerjee, N., Corner, M., Levine, B., Lynn, B.: A retrospective look at the UMass dome mobile testbed. ACM SIGMOBILE Mobile Comput. Commun. Rev. 15 (4), 2–15 (2012).CrossRefGoogle Scholar
  68. 68.
    The Openflow Switch Specification. (2009)
  69. 69.
    Vercher, J., Hernandez-Munoz, J., Gomez, L., Sepulveda, A.: An experimental platform for large-scale research facing FI-IoT scenarios. In: Future Network Mobile Summit (FutureNetw), pp. 1–8 (2011)Google Scholar
  70. 70.
    Virtual Worlds Framework. (2015)
  71. 71.
    Wang, L., Park, K., Pang, R., Pai, V.S., Peterson, L.L.: Reliability and security in the codeen content distribution network. In: USENIX Annual Technical Conference, General Track, pp. 171–184 (2004)Google Scholar
  72. 72.
    White, B., Lepreau, J., Stoller, L., Ricci, R., Guruprasad, S., Newbold, M., Hibler, M., Barb, C., Joglekar, A.: An integrated experimental environment for distributed systems and networks. In: Proceedings of the Fifth Symposium on Operating Systems Design and Implementation, pp. 255–270. USENIX Association, Boston (2002)Google Scholar
  73. 73.
    Yuan, H., Song, T., Crowley, P.: Scalable NDN forwarding: Concepts, issues and principles. In: 2012 21st International Conference on Computer Communications and Networks (ICCCN), pp. 1–9. IEEE, New York (2012)Google Scholar
  74. 74.
    Zurawski, J., Swany, M., Beck, M., Ding, Y.: Logistical multicast for data distribution. In: IEEE International Symposium on Cluster Computing and the Grid, 2005. CCGrid 2005, vol. 1, pp. 434–441. IEEE, New York (2005)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.US IgniteWashington, DCUSA
  2. 2.Flux Research GroupUniversity of UtahSalt Lake CityUSA

Personalised recommendations