Advertisement

A Distributed Architecture for Multimedia File Storage, Analysis and Processing

  • Andrzej DziechEmail author
  • Andrzej Glowacz
  • Jacek Wszołek
  • Sebastian Ernst
  • Michał Pawłowski
Part of the Studies in Computational Intelligence book series (SCI, volume 467)

Abstract

The paper presents a prototype of a distributed storage, analysis and processing system for multimedia (video, audio, image) content based on the Apache Hadoop platform. The core of the system is a distributed file system with advanced topology, replication and balancing capabilities. Metadata for stored media objects is being managed in a separate database in conformance with well-known metadata models and standards. In addition, the system allows implementation of efficient multimedia data analysis (e.g. face recognition) and processing (e.g. transcoding, digital watermarking) algorithms. Interoperability with external systems, such as the currently-developed Infona is provided by means of an OAI-PMH-compliant interface for metadata harvesting. More in-depth integration is possible by making storage, analysis and processing services available to external entities using the SYNAT integration platform. The article evaluates several important aspects of the proposed distributed repository, such as security, performance, extensibility and interoperability.

Keywords

distributed storage distributed processing MapReduce multimedia Hadoop HDFS performance evaluation scalability high availability 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Open CV Open Source Computer Vision (2012), http://opencv.willowgarage.com
  2. 2.
    SYNAT Project (2012), http://www.synat.pl
  3. 3.
    Apache Software foundation: Hadoop 1.0.2 API Documentation (2012), http://hadoop.apache.org/common/docs/r1.0.2/api/
  4. 4.
    Borthakur, D.: HDFS Architecture Guide, Version 1.0.2 (2012), http://hadoop.apache.org/common/docs/r1.0.2/hdfs_design.html
  5. 5.
    Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: A Distributed Storage System for Structured Data. In: OSDI 2006: Seventh Symposium on Operating System Design and Implementation (2006)Google Scholar
  6. 6.
    Dean, J., Ghemawat, S.: Map Reduce: Simplified Data Processing on Large Clusters. Communications of the ACM 51(1), 1–13 (2008), http://www.usenix.org/events/osdi04/tech/full_papers/dean/dean_html/ CrossRefGoogle Scholar
  7. 7.
    Ghemawat, S., Gobioff, H., Leung, S.T.: The Google file system. ACM SIGOPS Operating Systems Review 37(5), 29 (2003), http://portal.acm.org/citation.cfm?doid=1165389.945450 CrossRefGoogle Scholar
  8. 8.
    Kim, M., Lee, H., Cui, Y.: Performance Evaluation of Image Conversion Module Based on MapReduce for Transcoding and Transmoding in SMCCSE. In: 2011 IEEE Ninth International Conference on Dependable, Autonomic and Secure Computing, pp. 396–403 (2011), http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6118756
  9. 9.
    Kiran, G.V.R.: Online Video Conversion Using Map Reduce. Tech. rep., International Institute of Information Technology, HyderabadGoogle Scholar
  10. 10.
    Lin, J., Dyer, C.: Data-Intensive Text Processing with MapReduce, 1st edn. Morgan & Claypool Publishers (January 2010)Google Scholar
  11. 11.
    Sierra, S.: A Million Little Files (2008), http://stuartsierra.com/2008/04/24/a-million-little-files
  12. 12.
    Simon, B., Massart, D., van Assche, F., Ternier, S., Duval, E., Brantner, S., Olmedilla, D., Miklós, Z.: A simple query interface for interoperable learning repositories. In: Proceedings of the 1st Workshop on Interoperability of Web-Based Educational Systems, pp. 11–18 (2005)Google Scholar
  13. 13.
    Sweeney, C., Liu, L., Arietta, S., Lawrence, J.: HIPI: A Hadoop Image Processing Interface for Image-based MapReduce Tasks. Master’s thesis, University of Virginia (2011)Google Scholar
  14. 14.
    White, T.: Hadoop: The Definitive Guide. O’Reilly Series, vol. 54. O’Reilly Media (2009)Google Scholar
  15. 15.
    White, T.: The Small Files Problem (2009), http://www.cloudera.com/blog/2009/02/the-small-files-problem/

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Andrzej Dziech
    • 1
    Email author
  • Andrzej Glowacz
    • 1
  • Jacek Wszołek
    • 1
  • Sebastian Ernst
    • 2
  • Michał Pawłowski
    • 2
  1. 1.Department of TelecommunicationsAGH University of Science and TechnologyKrakówPoland
  2. 2.ACC Cyfronet AGHAGH University of Science and TechnologyKrakówPoland

Personalised recommendations