Advertisement

Exploiting Scalable Parallelism for Remote Sensing Analysis Models by Data Transformation Graph

  • Zhenchun Huang
  • Guoqing Li
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9530)

Abstract

According to the great hunger in performance capability and scalability for remote sensing analysis models, it is important to exploit scalable parallelism for remote sensing data analysis models. In this paper, a method named data transformation graph (shortly DTG) is introduced, which describes an analysis model by transformations among data items. DTG can be used to study the solvability and performance of analysis models. Taking global drought detection as an example, its execution and optimization are studied carefully by DTG, and some methods are proposed for accelerating remote sensing data analysis models. At last, a distributed data-intensive computing test system is built based on Robinia, and global drought detection application is implemented for performance evaluation. The test result shows that DTG based parallelization and optimization improves the performance with high efficiency evidently, and DTG is valuable to study and optimize remote sensing data analysis models for higher performance in distributed and parallel computing environments.

Keywords

Scalable parallelism Remote sensing analysis models Data transformation graph Global drought detection Data-intensive computing 

References

  1. 1.
    Loveman, D.B.: High performance fortran. IEEE Parallel Distrib. Technol.: Syst. Appl. 1(1), 25–42 (1993)CrossRefGoogle Scholar
  2. 2.
    Dagum, L., Menon, R.: OpenMP: an industry standard API for shared-memory programming. IEEE Comput. Sci. Eng. 5(1), 46–55 (1998)CrossRefGoogle Scholar
  3. 3.
    Geist, A. (ed.): PVM: Parallel Virtual Machine: A Users’ Guide and Tutorial for Networked Parallel Computing. The MIT Press, Cambridge (1994)zbMATHGoogle Scholar
  4. 4.
    Gropp, W., Lusk, E., Skjellum, A.: Using MPI: Portable Parallel Programming with the Message Passing Interface, vol. 1. MIT Press, Cambridge (1999)zbMATHGoogle Scholar
  5. 5.
    Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the condor experience. Concurr. Comput.: Prac. Exp. 17(2–4), 323–356 (2005)CrossRefGoogle Scholar
  6. 6.
    Cossu, R., Bally, P., Colin, O., Fusco, L.: ESA grid processing on demand for fast access to earth observation data and rapid mapping of flood events. European Geosciences Union General Assembly (2008)Google Scholar
  7. 7.
    Sekiguchi, S., Tanaka, Y., Kojima, I., Yamamoto, N., Yokoyama, S., Tanimura, Y., et al.: Design principles and IT overviews of the GEO Grid. IEEE Syst. J. 2(3), 374–389 (2008)CrossRefGoogle Scholar
  8. 8.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI), pp. 137–150, December 2004Google Scholar
  9. 9.
    White, T.: Hadoop: The Definitive Guide. O’Reilly Media Inc., Sebastopol (2012)Google Scholar
  10. 10.
    Lee, C.A., Gasster, S.D., Plaza, A., Chang, C.I., Huang, B.: Recent developments in high performance computing for remote sensing: a review. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 4(3), 508–527 (2011)CrossRefGoogle Scholar
  11. 11.
    Olston, C., Reed, B., Srivastava, U., Kumar, R., Tomkins, A.: Pig latin: a not-so-foreign language for data. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, pp. 1099–1110. ACM (2008)Google Scholar
  12. 12.
    Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: distributed data-parallel programs from sequential building blocks. ACM SIGOPS Oper. Syst. Rev. 41(3), 59–72 (2007)CrossRefGoogle Scholar
  13. 13.
    Chaiken, R., Jenkins, B., Larson, P.Å., Ramsey, B., Shakib, D., Weaver, S., Zhou, J.: SCOPE: easy and efficient parallel processing of massive data sets. Proc. VLDB Endow. 1(2), 1265–1276 (2008)CrossRefGoogle Scholar
  14. 14.
    Ghemawat, S., Gobioff, H., Leung, S.T.: The google file system. ACM SIGOPS Oper. Syst. Rev. 37(5), 29–43 (2003)CrossRefGoogle Scholar
  15. 15.
    Chang, F., et al.: Bigtable: a distributed storage system for structured data. In: OSDI 2006, pp. 205–218 (2006)Google Scholar
  16. 16.
    Melnik, S., Gubarev, A., Long, J.J., Romer, G., Shivakumar, S., Tolton, M., Vassilakis, T.: Dremel: interactive analysis of web-scale datasets. Proc. VLDB Endow. 3(1–2), 330–339 (2010)CrossRefGoogle Scholar
  17. 17.
    Corbett, J.C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J.J., et al.: Spanner: google’s globally-distributed database. In: Proceedings of the 10th USENIX Symposium on Operating System Design and Implementation (OSDI 2012), pp. 251–264 (2012)Google Scholar
  18. 18.
    Mandl, D.: Matsu: an elastic cloud connected to a sensorweb for disaster response. In: Workshop on Cloud Computing for Spacecraft Operations, Ground System Architectures Workshop (GSAW), 2 March 2011Google Scholar
  19. 19.
    Guan, X., Wu, H., Li, L.: A parallel framework for processing massive spatial data with a SplitCandCMerge paradigm. Trans. GIS 16(6), 829–843 (2012)CrossRefGoogle Scholar
  20. 20.
    Thies, W., Karczmarek, M., Amarasinghe, S.: Streamit: a language for streaming applications. In: Nigel Horspool, R. (ed.) CC 2002. LNCS, vol. 2304, pp. 179–196. Springer, Heidelberg (2002) CrossRefGoogle Scholar
  21. 21.
    Halbwachs, N., Caspi, P., Raymond, P., Pilaud, D.: The synchronous data flow programming language LUSTRE. Proc. IEEE 79(9), 1305–1320 (1991)CrossRefGoogle Scholar
  22. 22.
    Gao, B.C.: NDWI-a normalized difference water index for remote sensing of vegetation liquid water from space. Remote Sens. Environ. 58(3), 257–266 (1996)CrossRefGoogle Scholar
  23. 23.
    Wang, Q.S., Zhao, D., Huang, Z.C.: Research on the performance of virtualization-based remote sensing data processing platform. In: 2012 International Conference on Systems and Informatics (ICSAI 2012), Yantai, China, 19–21 May 2012Google Scholar
  24. 24.
    McGrath, R.E., Xinjian, L., Folk, M.: Java (TM) applications using NCSA HDF files. Concurr. Prac. Exp. 9(11), 1113–1125 (1997)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Department of Computer Science and TechnologyTsinghua UniversityBeijingChina
  2. 2.Institute of Remote Sensing and Digital EarthCASBeijingChina

Personalised recommendations