A Data Processing Framework for Distributed Embedded Systems

  • Ichiro SatohEmail author
Conference paper
Part of the Studies in Computational Intelligence book series (SCI, volume 616)


A MapReduce-based framework for processing data at nodes on the Internet of Things (IoT) is presented in this paper. Although MapReduce processing and its clones have been designed for high-performance server clusters, the processing itself is simple and generalized, so it should be used in non-high-performance computing environments, e.g., IoT and sensor networks. The proposed framework is unique among the other MapReduce-based processing approaches, because it can locally process the data maintained in nodes on the IoT rather than within high-performance server clusters and data centers. It deploys programs for data processing at the nodes that contain the target data as a map step and executes the programs with the local data. Finally, it aggregates the results of the programs to certain nodes as a reduce step. The architecture of the framework, its basic performance, and its application are also described here.


Sensor Node Target Data Runtime System Data Node Distribute File System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Bu, Y., Howe, B., Balazinska, M., Ernst, M.D.: HaLoop: efficient iterative data processing on large clusters. In: Proceedings of the VLDB Endowment, vol. 3, p. 1 (2010)Google Scholar
  2. 2.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: Proceedings of the 6th Conference on Symposium on Opearting Systems Design and Implementation (OSDI’04) (2004)Google Scholar
  3. 3.
    Ekanayake, J., Li, H., Zhang, B., Gunarathne, T., Bae, S.H., Qiu, J., Fox, G.: Twister: a runtime for iterative MapReduce. In: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing(HPDC’10). ACM (2010)Google Scholar
  4. 4.
    Grossman, R., Gu, Y.: Data mining using high performance data clouds: experimental studies using sector and sphere. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’08), pp.920–927. ACM (2008)Google Scholar
  5. 5.
    Jiang, W., Ravi, V.T., Agrawal, G.: A map-reduce system with an alternate API for multi-core environments. In: Proceedings of 10th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing (2010)Google Scholar
  6. 6.
    Satoh, I.: Mobile agents. In: Handbook of Ambient Intelligence and Smart Environments, pp. 771–791. Springer, Berlin (2010)Google Scholar
  7. 7.
    Satoh, I.: A framework for data processing at the edges of networks. In: Proceedings on 24th International Conference on Database and Expert Systems Applications (DEXA’2013), LNCS, vol. 8056, pp. 304–318. Springer, Berlin (2013)Google Scholar
  8. 8.
    Sehrish, S., Mackey, G., Wang, J., Bent, J.: MRAP: a novel MapReduce-based framework to support HPC analytics applications with access patterns. In: Proceedings of High Performance Distribute Computing (HPDC 2010) (2010)Google Scholar
  9. 9.
    Talbot, J., Yoo, R.M., Kozyrakis, C.: Phoenix++: modular MapReduce for shared-memory systems. In: Proceedings of 2nd International Workshop on MapReduce and Its Applications (MapReduce’11). ACM Press (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.National Institute of InformaticsChiyoda-ku, TokyoJapan

Personalised recommendations