Abstract
With the continuous development of remote sensing technology, the data volume of remote sensing images has increased exponentially, resulting in many difficulties in the storage, management, transmission, calculation, and other processes of remote sensing images. In order to solve the above problems, this paper studies the use of the Hadoop Distributed File System (HDFS) and related technologies to design and implement a browser/server (B/S) architecture for a massive, multisource, remote sensing images distributed storage management system. The image data are stored in the HDFS, and the image metadata are stored in a MySQL database. The distributed parallel construction of the image pyramid is completed based on the Spark computing engine, and the Akka framework is used to construct WMTS (Web Map Tile Service) to realize the release of remote sensing images. Finally, the rapid visual display of remote sensing images is carried out using Leaflet. The system also supports image data management, image target detection, user management, and other functions. After testing, this system can support the storage and management of multisource remote sensing image data, and can solve perfectly the problems of insufficient storage space and insufficient computing power of a single server. It is found that the upload and download speeds of a large amount of remote sensing images can be close to the maximum speed of a gigabit local area network (LAN). In the gigabit LAN environment, the average upload speed of a single remote sensing image is 97.74 MB/s, and the average download speed is 87.62 MB/s. In terms of image pyramid construction, the speed of a multi-node parallel construction based on Spark is two times higher than that of a single-node construction. Additionally, compared to similar systems, this system has better data transmission and retrieval speed, better data computing ability, and higher concurrency processing ability.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
Due to the nature of this research, the participants of this study do not agree for their data to be shared publicly. Hence, supporting data are not available.
References
Blaschke T (2010) Object based image analysis for remote sensing. ISPRS J Photogramm Remote Sens 65(1):2–16
Deren LI, Liangpei Z, Guisong X (2014) Automatic analysis and mining of remote sensing big data. Acta Geodaetica Cartogr Sin 43(12):1211
Chi M, Plaza A, Benediktsson JA et al (2016) Big data for remote sensing: challenges and opportunities. Proc IEEE 104(11):2207–2219
Huang YQ (2019) The concept and development trend of spatial database management system. China Manage Informationization 22(08):165–166
Lü XF, Cheng C, Gong J et al (2011) Review of data storage and management technologies for massive remote sensing data. Sci China Technol Sci 54:3220–3232
Yan J, Liu Y, Wang L et al (2021) An efficient organization method for large-scale and long time-series remote sensing data in a cloud computing environment. IEEE J Sel Top Appl Earth Observ Remote Sens 14:9350–9363
Wang L, Ma Y, Yan J et al (2018) PipsCloud: high performance cloud computing for remote sensing big data management and processing. Futur Gener Comput Syst 78:353–368
Cheng Y, Zhou K, Wang J, Yan J (2020) Big earth observation data integration in remote sensing based on a distributed spatial framework. Remote Sens 12(6):972. https://doi.org/10.3390/rs12060972
Jing W, Tian D (2018) An improved distributed storage and query for remote sensing data. Procedia Comput Sci 129:238–247
Li J, Zhang P, Li Y et al (2017) A data-check based distributed storage model for storing hot temporary data. Futur Gener Comput Syst 73:13–21
Zheng K, Fu Y (2013) Research on vector spatial data storage schema based on Hadoop platform. Int J Database Theory Appl 6(5):85–94
Zhong Y, Sun S, Liao H, Zhao Y, Fang J (2011) A novel method to manage very large raster data on distributed key-value storage system. In: 2011 19th International Conference on Geoinformatics. Shanghai, China, pp 1–6. https://doi.org/10.1109/GeoInformatics.2011.5980711
Rajak R, Raveendran D, Bh MC, Medasani SS (2015) High resolution satellite image processing using Hadoop framework. In: 2015 IEEE International Conference on Cloud Computing in Emerging Markets (CCEM). Bangalore, India, pp 16–21. https://doi.org/10.1109/CCEM.2015.16
Zhu J, Zhang Z, Zhao F et al (2023) Efficient management and scheduling of massive remote sensing image datasets. ISPRS Int J Geo-Information 12(5):199
Zhou X, Wang X, Zhou Y et al (2021) Rsims: large-scale heterogeneous remote sensing images management system[J]. Remote Sens 13(9):1815
Wang C, Hu F, Hu X et al (2015) A Hadoop-based distributed framework for efficient managing and processing big remote sensing images. ISPRS Ann Photogramm Remote Sens Spat Inf Sci 2:63–66
Kong W, Wang T, Liu L et al (2023) A novel design and application of spatial data management platform for natural resources. J Clean Prod 411:137183
Wei H, Yuheng Z (2017) The massive remote sensing data organization and management strategies//MATEC Web of Conferences. EDP Sciences 128:02011
Wang S, Li G, Yao X et al (2019) A distributed storage and access approach for massive remote sensing data in MongoDB. ISPRS Int J Geo-Information 8(12):533
Rathore MM, Ahmad A, Paul A et al (2016) Urban planning and building smart cities based on the internet of things using big data analytics. Comput Netw 101:63–80
Shan TJ, Zhong HW et al (2019) Building of remote sensing images tile pyramid based on Spark. Intell Comput Appl 9(04):226–229
Zaharia M, Xin RS, Wendell P et al (2016) Apache spark a unified engine for big data processing. Commun ACM 59(11):56–65
Kini A, Emanuele R, Geotrellis (2014) Adding geospatial capabilities to Spark. In: Spark Summit 2014, from https://docs.huihoo.com/apache/spark/summit/2014/Geotrellis-Adding-Geospatial-Capabilities-to-Spark-Ameet-Kini-Rob-Emanuele.pdf
Chen X, Zhang C, Ge B et al (2016) Efficient historical query in HBase for spatio-temporal decision support. Int J Comput Commun Control 11(5):613–630
Jonasson M (2014) The Akka-board–performing mobility, disability and innovation. Disabil Soc 29(3):477–490
Farkas G (2017) Applicability of open-source web mapping libraries for building massive web GIS clients. J Geogr Syst 19(3):273–295
Huang F, Chen S, Wang Q et al (2023) Using deep learning in an embedded system for real-time target detection based on images from an unmanned aerial vehicle: vehicle detection as a case study[J]. Int J Digit Earth 16(1):910–936
Zhu Q, Huang F, Lu J et al (2017) Research on the implementation of multi-source remote sensing image management system based on B/S architecture. 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS). IEEE, pp 5233–5236
Acknowledgements
This study was mainly supported by the National Science Foundation of China (Grant No. 42271390) and the Technological Innovation R&D Project of Chengdu Science and Technology Bureau (Grant No. 2022-YF05-00967-SN). This work was partial funded by the Fundamental Research Funds for the Central Universities (Grant Nos. ZYGX2019J069 and ZYGX2019J072) and Hubei Provincial Key Laboratory of Intelligent Geo-information Processing (China University of Geosciences; Grant Nos. KLIGIP-2018A08).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflicts of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Yang, L., He, W., Qiang, X. et al. Research on remote sensing image storage management and a fast visualization system based on cloud computing technology. Multimed Tools Appl 83, 59861–59886 (2024). https://doi.org/10.1007/s11042-023-17858-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-17858-6