Skip to main content
SpringerLink
Log in

Using Broadcast Networks to Create On-demand Extremely Large Scale High-throughput Computing Infrastructures

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

A growing number of very simple parallel applications can benefit from the availability of very large computing resource pools that are exploited in an opportunistic way to considerably speed up the processing of these applications. In particular, many applications can scale out to take advantage of the availability of thousands, or even millions, of processors that can be simultaneously used. Some large-scale distributed computing infrastructures (DCI) have already been successfully assembled to this end. However, these DCI are normally designed to cater to specific kinds of applications, and their assemblage is not only costly, but also time consuming. We propose a novel architecture, named OddCI (On-demand DCI), for the cheap, fast and on-demand instantiation and dismantle of very large DCI. These characteristics allow application users to dynamically commission DCI that can cost-effectively be customized to their particular needs, and disposed after being used. Our approach leverages on broadcast communication as an efficient mechanism to enable aggregation of geographically distributed computing resources, including millions of non-traditional processing devices such as mobile phones, and digital television (DTV) receivers. We discuss how the proposed architecture can be implemented on top of a network of DTV receivers, providing evidences of both the technical, as well as the operational feasibility of the OddCI architecture.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. ABNT: Digital Terrestrial Television - Data Coding and Transmission Specification for Digital Broadcasting. Part 1: Data Coding. ABNT/CEE-85 Digital Television Committee. NBR 15606-1 (2007)

  2. ABNT: Digital Terrestrial Television - Data Coding and Transmission Specification for Digital Broadcasting. Part 2: Ginga-NCL for Fixed and Mobile Receivers - XML Application Language for Application Coding. ABNT/CEE-85 Digital Television Committee. NBR 15606-2 (2009)

  3. ABNT: Digital Terrestrial Television - Data Coding and Transmission Specification for Digital Broadcasting. Part 4: Ginga-J - The Environment for the Execution of Procedural Applications - ABNT/CEE-85 Digital Television Committee. NBR 15606-4 (2009)

  4. Altschul, S.F., Gish, W., Miller, W., Myers, E.W., Lipman, D.J.: Basic local alignment search tool. J. Mol. Biol. 215(3), 403–410 (1990)

    Google Scholar 

  5. Amazon: Amazon web services. Available in http://aws.amazon.com (2012). Accessed 24 Aug 2012

  6. Anderson, D.P.: BOINC: a system for public-resource computing and storage. In: Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing, GRID ’04, pp. 4–10. IEEE Computer Society, Washington, DC (2004)

  7. Anderson, D.P., Cobb, J., Korpela, E., Lebofsky, M., Werthimer, D.: SETI@home: an experiment in public-resource computing. Commun. ACM 45, 56–61 (2002)

    Article  Google Scholar 

  8. Andrade, N., Brasileiro, F., Cirne, W., Mowbray, M.: Automatic Grid assembly by promoting collaboration in peer-to-peer Grids. J. Parallel Distrib. Comput. 67, 957–966 (2007)

    Article  MATH  Google Scholar 

  9. Andrzejak, A., Kondo, D., Anderson, D.P.: Ensuring collective availability in volatile resource pools via forecasting. In: Proceedings of the 19th IFIP/IEEE International Workshop on Distributed Systems: Operations and Management: Managing Large-Scale Service Deployment. DSOM ’08, pp. 149–161. Springer, Berlin (2008)

    Google Scholar 

  10. Aron, R., Chana, I.: Formal QoS policy based Grid resource provisioning framework. J. Grid Computing 10, 249–264 (2012). doi:10.1007/s10723-012-9202-y

    Article  Google Scholar 

  11. Batista, C.E.C.F., de Araujo, T.M.U., dos Anjos, D.O., de Castro, M., Brasileiro, F., de Souza Filho., G.: TVGrid: a Grid architecture to use the idle resources on a digital tv network. In: Proc. 7th IEEE International Symposium on Cluster Computing and the Grid (The Latin America Grid Workshop), pp. 823–828. Rio de Janeiro, Brazil (2007)

  12. Bell, D.E., LaPadula, L.J.: Secure computer sytems: united exposition and multics interpretation. Technical report, ESD-TR-75-306, MITRE Co. Hanscom AFB, MA (1976)

    Google Scholar 

  13. Bitcurrent: Bitcurrent team. Available in http://www.bitcurrent.com (2012). Accessed 24 Aug 2012

  14. Boesgaard, M., Zenner, E.: Protecting online transactions with unique embedded key generators. In: Proceedings of the The Second International Conference on Availability, Reliability and Security, pp. 663–669. IEEE Computer Society, Washington, DC (2007)

    Chapter  Google Scholar 

  15. Boincoid: Boincoid - an android port of the boinc platform. Available in http://boincoid.sourceforge.net (2012). Accessed 24 Aug 2012

  16. Buyya, R., Murshed, M.: GridSim: a toolkit for the modeling and simulation of distributed resource management and scheduling for Grid computing. Concurr. Comp.-Pract. E. 14(13), 1175–1220 (2002)

    Article  MATH  Google Scholar 

  17. Cirne, W., Brasileiro, F., Andrade, N., Costa, L., Andrade, A., Novaes, R., Mowbray, M.: Labs of the world, unite!!! J. Grid Computing 4(3), 225–246 (2006)

    Article  MATH  Google Scholar 

  18. Cirne, W., Paranhos, D., Costa, L., Santos-Neto, E., Brasileiro, F., Sauve, J., Silva, F.A.B., Barros, C.O., Silveira, C.: Running bag-of-tasks applications on computational Grids: The MyGrid approach. In: 2003 International Conference on Parallel Processing 2003 Proceedings, pp. 407–416 (2003)

  19. Costa, R., Brasileiro, F., de Souza Filho, G.L., Sousa, D.M.: OddCI: on-demand distributed computing infrastructure. In: 2nd Workshop on Many-Task Computing on Grids and Supercomputers, vol. 16, pp. 1–10. ACM, Portland (2009)

    Google Scholar 

  20. Costa, R., Brasileiro, F., Lemos, G., Mariz, D.: Analyzing the impact of elasticity on the profit of cloud computing providers. In: Proceedings of the IEEE/ACM. Proceedings of the 2nd International Workshop on Cloud Computing and Scientific Applications (CCSA’12), in Conjunction with CCGRID’12. IEEE/ACM, Ottawa (2012)

    Google Scholar 

  21. Fedak, G., Gelas, J.-P., Herault, T., Iniesta, V., Kondo, D., Lefevre, L., Malécot, P., Nussbaum, L., Rezmerita, A., Richard, O.: DSL-lab: a low-power lightweight platform to experiment on domestic broadband internet. In: Proceedings of the 2010 Ninth International Symposium on Parallel and Distributed Computing. ISPDC’10, pp. 141–148. IEEE Computer Society, Washington, DC (2010)

    Chapter  Google Scholar 

  22. Folding@home: Folding@home distributed computing. Available in http://folding.stanford.edu (2012). Accessed 24 Aug 2012

  23. Folding@home: Folding@home petaflop barrier crossed. Available in http://blog.us.playstation.com/2007/09/19/foldinghome-petaflop-barrier-crossed (2012). Accessed 24 Aug 2012

  24. Francois, J., State, R., Festor, O.: Botnets for scalable management. In: Proceedings of the Distributed Systems: Operations and Management 18th IFIP/IEEE International Conference on Managing Virtualization of Networks and Services. DSOM’07, pp. 1–12. Springer, Berlin (2007)

    Google Scholar 

  25. Francois, J., State, R., Festor, O.: Malware models for network and service management. In: Proceedings of the 1st International Conference on Autonomous Infrastructure, Management and Security: Inter-Domain Management. AIMS’07, pp. 192–195. Springer, Berlin (2007)

    Google Scholar 

  26. ISO/IEC. Information Technology: Generic Coding of Moving Pictures and Associated Audio Information. Part 6: Extensions for DSM/CC. ISO/IEC TR 13818.6 (1998)

  27. ITU: Recommendation H.222. Information Technology. Generic Coding of Moving Pictures and Associated Audio Information: Systems, ITU-T (2000)

  28. Keahey, K., Doering, K., Foster, I.: From sandbox to playground: dynamic virtual environments in the Grid. In: Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing. GRID ’04, pp. 34–42. IEEE Computer Society, Washington, DC (2004)

    Chapter  Google Scholar 

  29. Kepler Project: MD5 Cryptographic Library for Lua. Available in http://www.keplerproject.org/md5 (2012). Accessed 24 Aug 2012

  30. Kulesza, R., Lima, J.F.A., Guedes, L., Junior, L.L.A., Meira, S.R.L., Filho, G.L.S.: Ginga-J - an open java-based application environment for interactive digital television services. In: Open Source Systems: Grounding Research. IFIP Advances in Information and Communication Technology, vol. 365, pp. 34–49. Springer, Berlin (2011)

    Google Scholar 

  31. Lagar-Cavilla, H.A., Whitney, J.A., Scannell, A.M., Patchin, P., Rumble, S.M., de Lara, E., Brudno, M., Satyanarayanan, M.: SnowFlock: rapid virtual machine cloning for cloud computing. In: Proceedings of the 4th ACM European Conference on Computer Systems. EuroSys’09, pp. 1–12. ACM, New York (2009)

    Chapter  Google Scholar 

  32. Litzkow, M., Livny, M., Mutka, M.: Condor - a hunter of idle workstations. In: Proceedings of the 8th International Conference of Distributed Computing Systems, pp. 104–111 (1988)

  33. Morris, S., Chaigneau, A.S.: Interactive TV Standards: A Guide to MHP, OCAP, and JavaTV. Focal Press (2005)

  34. NCBI: NCBI Blast. Available in http://blast.ncbi.nlm.nih.gov/Blast.cgi (2012). Accessed 24 Aug 2012

  35. Neill, R., Carloni, L.P., Shabarshin, A., Sigaev, V., Tcherepanov, S.: Embedded processor virtualization for broadband Grid computing. In: , pp. 145–156. IEEE Computer Society, Washington, DC (2011)

    Chapter  Google Scholar 

  36. Neustar Inc.: Neustar Webmetrics. Available in http://www.webmetrics.com (2012). Accessed 24 Aug 2012

  37. Raicu, I., Zhang, Z., Wilde, M., Foster, I., Beckman, P., Iskra, K., Clifford, B. Toward loosely coupled programming on petascale systems. In: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing, SC ’08, pp. 22:1–22:12. IEEE Press, Piscataway (2008)

    Google Scholar 

  38. Raicu, I., Zhao, Y., Dumitrescu, C., Foster, I., Wilde, M.: Falkon: a Fast and light-weight tasK executiON framework. In: SC’07: Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, pp. 1–12. ACM, New York (2007)

    Chapter  Google Scholar 

  39. Rochwerger, B., Breitgand, D., Levy, E., Galis, A., Nagin, K., Llorente, I.M., Montero, R., Wolfsthal, Y., Elmroth, E., Cáceres, J., Ben-Yehuda, M., Emmerich, W., Galán, F.: The reservoir model and architecture for open federated cloud computing. IBM J. Res. Develop. 53, 535–545 (2009)

    Article  Google Scholar 

  40. Rood, B., Lewis, M. Grid Resource Availability Prediction-Based Scheduling and Task Replication. J. Grid Computing 7, 479–500 (2009). doi:10.1007/s10723-009-9135-2

    Article  Google Scholar 

  41. Sarmenta, L.F.G.: Sabotage-tolerance mechanisms for volunteer computing systems. Future Gener. Comput. Syst. 18(4), 561–572 (2002)

    Article  MATH  Google Scholar 

  42. Schellhorn, G., Reif, W., Schairer, A., Karger, P., Austel, V., Toll, D.: Verified formal security models for multiapplicative smart cards. Computer Security 10, 339–367 (2002)

    Google Scholar 

  43. de Lima, M.S., Greve, F., Arantes, L., Sens, P.: The time-free approach to Byzantine failure detection in dynamic networks. In: Proceedings of the 2011 IEEE/IFIP 41st International Conference on Dependable Systems and Networks Workshops, DSNW ’11, pp. 3–8. IEEE Computer Society, Washington, DC (2011)

  44. Sevior, M., Fifield, T., Katayama, N.: Belle Monte-Carlo production on the Amazon EC2 cloud. J. Phys. Conf. Ser. 219(1), 012003 (2010)

    Article  Google Scholar 

  45. Soares, L.F.G., Rodrigues, R.F., Moreno, M.F.: Ginga-NCL: the declarative environment of the Brazilian digital TV system. J. Braz. Comput. Soc. 12, 37–46 (2007)

    Article  Google Scholar 

  46. Wang, L., Laszewski, G., Younge, A., He, X., Kunze, M., Tao, J., Fu, C.: Cloud computing: a perspective study. New Gener. Comput. 28(2), 137–146 (2010)

    Article  MATH  Google Scholar 

  47. Warneke, D., Kao, O.: Nephele: efficient parallel data processing in the cloud. In: 2nd Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS ’09), pp. 16–16. ACM, New York (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Systems and Computing Department, Distributed Systems Lab, Federal University of Campina Grande, Av. Aprígio Veloso, 882 - Bloco CO, Bodocongó, Campina Grande, Paraíba, Brazil

    Rostand Costa & Francisco Brasileiro

  2. Informatics Department, Digital Video Applications Lab, Federal University of Paraíba, Campus I - Cidade Universitária, João Pessoa, Paraíba, Brazil

    Rostand Costa & Guido Lemos Filho

  3. Informatics Department, Federal Institute of Technology of Paraíba, João Pessoa, Paraíba, Brazil

    Dênio Sousa

Authors
  1. Rostand Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Francisco Brasileiro
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guido Lemos Filho
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dênio Sousa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rostand Costa.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Costa, R., Brasileiro, F., Lemos Filho, G. et al. Using Broadcast Networks to Create On-demand Extremely Large Scale High-throughput Computing Infrastructures. J Grid Computing 10, 419–445 (2012). https://doi.org/10.1007/s10723-012-9229-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10723-012-9229-0

Keywords

Search

Navigation