An Approach for Processing Large and Non-uniform Media Objects on MapReduce-Based Clusters

  • Rainer Schmidt
  • Matthias Rella
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7008)

Abstract

Cloud computing enables us to create applications that take advantage of large computer infrastructures on demand. Data intensive computing frameworks leverage these technologies in order to generate and process large data sets on clusters of virtualized computers. MapReduce provides an highly scalable programming model in this context that has proven to be widely applicable for processing structured data. In this paper, we present an approach and implementation that utilizes this model for the processing of audiovisual content. The application is capable of analyzing and modifying large audiovisual files using multiple computer nodes in parallel and thereby able to dramatically reduce processing times. The paper discusses the programming model and its application to binary data. Moreover, we summarize key concepts of the implementation and provide a brief evaluation.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51, 107–113 (2008)CrossRefGoogle Scholar
  2. 2.
    Dean, J., Ghemawat, S.: Mapreduce: a flexible data processing tool. Commun. ACM 53(1), 72–77 (2010)CrossRefGoogle Scholar
  3. 3.
    Ekanayake, J., Pallickara, S., Fox, G.: Mapreduce for data intensive scientific analyses. In: IEEE Fourth International Conference on eScience 2008, pp. 277–284 (2008)Google Scholar
  4. 4.
    Gunarathne, T., Wu, T.-L., Qiu, J., Fox, G.: Cloud computing paradigms for pleasingly parallel biomedical applications. In: Proc. of the 19th ACM Int. Symposium on High Performance Distributed Computing, HPDC 2010, pp. 460–469 (2010)Google Scholar
  5. 5.
    Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: distributed data-parallel programs from sequential building blocks. In: Proc. of the 2nd ACM SIGOPS/EuroSys European Conf. on Computer Systems 2007, pp. 59–72 (2007)Google Scholar
  6. 6.
    Moretti, C., Bui, H., Hollingsworth, K., Rich, B., Flynn, P., Thain, D.: All-pairs: An abstraction for data-intensive computing on campus grids. IEEE Transactions on Parallel and Distributed Systems 21, 33–46 (2010)CrossRefGoogle Scholar
  7. 7.
    Pereira, R., Azambuja, M., Breitman, K., Endler, M.: An architecture for distributed high performance video processing in the cloud. In: Proc. of the 2010 IEEE 3rd International Conference on Cloud Computing, CLOUD 2010, pp. 482–489 (2010)Google Scholar
  8. 8.
    Schmidt, R., Rella, M.: Considering data locality for parallel video processing. ERCIM News (83) (2010)Google Scholar
  9. 9.
    Schmidt, R., Sadilek, C., King, R.: A service for data-intensive computations on virtual clusters. In: International Conference on Intensive Applications and Services, pp. 28–33 (2009)Google Scholar
  10. 10.
    Thusoo, A., Sarma, J.S., N., Jain, Z.S., P., Chakka, N.Z., S., Antony, H.L., Murthy, R.: Hive - a petabyte scale data warehouse using hadoop. In: 2010 IEEE 26th International Conference on Data Engineering (ICDE), pp. 996–1005 (2010)Google Scholar
  11. 11.
    Warneke, D., Kao, O.: Nephele: efficient parallel data processing in the cloud. In: Proc. of the 2nd Workshop on Many-Task Computing on Grids and Supercomputers, MTAGS 2009, pp. 8:1–8:10 (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Rainer Schmidt
    • 1
  • Matthias Rella
    • 1
  1. 1.Austrian Institute of TechnologyViennaAustria

Personalised recommendations