An optimized cluster storage method for real-time big data in Internet of Things

  • Li Tu
  • Shuai LiuEmail author
  • Yan Wang
  • Chi Zhang
  • Ping Li


Data storage, especially big data storage, is a research hot spot in Internet of Things (IoT) system today. In traditional data storage methods, the fault-tolerant algorithm for data copies is adjusted with whole data file, which causes huge redundancy because there are less utilization and more cost of data storage when only a part of data blocks in the file are accessed. Therefore, an optimized cluster storage method for big data in IoT is proposed in this paper. First, weights of data blocks in each historical accessing period are calculated by temporal locality of data access, and the access frequencies of the data block in next period are predicted by the weights. Second, the hot spot of a data block is determined with a threshold which is calculated by previous data access. Meantime, in order to improve the data access efficiency and resource utilization, as well as to reduce the copy storage costs, copy of data block is dynamically adjusted and stored in different groups with high-performance and low-load nodes for data balance. Finally, experimental results show that the storage cost of proposed method is 70% less than that of traditional methods, which means that the proposed method effectively improves the data access speed, reduces storage space, and balances the storage load.


Internet of Things Big data Real time Cluster storage Optimization 



This work is supported by Programs of National Natural Science Foundation of China (No: 61502254), Program for Young Talents of Science and Technology in Universities of Inner Mongolia Autonomous Region (No. NJYT-18-B10), Teaching Reform Project Fund of University of Electronic Science and Technology of China Zhongshan Institute (No: 418YKQN02), Scientific Research Fund of Guangdong Provincial Education Department under Grant (No: 416YCQ01), Teaching Reform Project Fund of Hunan province under Grant (No: 2016[400]), Scientific Research Fund of Hunan Provincial Education Department under Grant Nos: 16C0299, 17C0295), Natural Science Foundation of Hunan Province, China (No. 2018JJ2023). We want to thank Prof Zhang from University of Leicester for his careful checking of language to this paper.


  1. 1.
    Chen B, Ting KM, Washio T et al (2015) Half-space mass: a maximally robust and efficient data depth method. Mach Learn 100(2–3):677–699. MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Cherubini G, Jelitto J, Venkatesan V (2016) Cognitive storage for big data. Computer 49(4):43–51. CrossRefGoogle Scholar
  3. 3.
    Cody BM, Bau D, Gonzalez-Nicolas A (2015) Stochastic injection-strategy optimization for the preliminary assessment of candidate geological storage sites. Hydrogeol J 23(6):1229–1245. CrossRefGoogle Scholar
  4. 4.
    El-Rabiaey MA, Areed NFF, Obayya SSA (2016) Novel plasmonic data storage based on nematic liquid crystal layers. J Lightwave Technol 34(16):3726–3732. CrossRefGoogle Scholar
  5. 5.
    Gaocheng L, Shuai L, Khan M et al (2018) Object tracking in vary lighting conditions for fog based intelligent surveillance of public spaces. IEEE Access 6:29283–29296CrossRefGoogle Scholar
  6. 6.
    Hong W, Yang Y, Li Y et al (2015) Systematic optimization of long gradient chromatography mass spectrometry for deep analysis of brain proteome. J Proteome Res 14(2):829–838. CrossRefGoogle Scholar
  7. 7.
    Ishii T, Shimada K, Hoshizawa T et al (2015) Analysis of vibration effects on holographic data storage system. Jpn J Appl Phys 54(9S):09MA04. CrossRefGoogle Scholar
  8. 8.
    Kaneko K, Kawamoto Y, Nishiyama H et al (2015) An efficient utilization of intermittent surface–satellite optical links by using mass storage device embedded in satellites. Perform Eval 87(C):37–46. CrossRefGoogle Scholar
  9. 9.
    Liu S, Fu W, He L et al (2017) Distribution of primary additional errors in fractal encoding method. Multimed Tools Appl 76(4):5787–5802CrossRefGoogle Scholar
  10. 10.
    Liu S, Bai W, Liu C et al (2018) Parallel fractal compression method for big video data. Complexity 2018:2016976. Google Scholar
  11. 11.
    Shuai L, Xiaochun C, Caihe L et al (2013) Fractal property of generalized M-set with rational number exponent. Appl Math Comput 220:668–675MathSciNetzbMATHGoogle Scholar
  12. 12.
    Shuai L, Xiaochun C, Weina F et al (2014) Numeric characteristics of generalized M-set with its asymptote. Appl Math Comput 243:767–774MathSciNetzbMATHGoogle Scholar
  13. 13.
    Wang J, Huang CG, Liu SK et al (2016) Research of distributed storage method for agricultural scientific data. Comput Eng Appl 52(11):248–253CrossRefGoogle Scholar
  14. 14.
    Wijetunge CD, Saeed I, Boughton BA et al (2015) Exims an improved data analysis pipeline based on a new peak picking method for exploring imaging mass spectrometry data. Bioinformatics 31(19):3198–3206. CrossRefGoogle Scholar
  15. 15.
    Xu Z, Schrama E, Wal WVD (2015) Optimization of regional constraints for estimating the Greenland mass balance with grace level-2 data. Geophys J Int 202(1):381–393. CrossRefGoogle Scholar
  16. 16.
    Yang G, Liu S (2014) Distributed cooperative algorithm for k  M set with negative integer k by fractal symmetrical property. Int J Distrib Sensor Netw 12:3. Google Scholar
  17. 17.
    Yang CT, Shih WC, Huang CL et al (2016) On construction of a distributed data storage system in cloud. Computing 98(1–2):93–118. MathSciNetCrossRefGoogle Scholar
  18. 18.
    Zhang Y, Bhamber R, Riba-Garcia I et al (2015) Streaming visualisation of quantitative mass spectrometry data based on a novel raw signal decomposition method. Proteomics 15(8):1419–1427. CrossRefGoogle Scholar
  19. 19.
    Zhao D, Katsouras I, Asadi K et al (2016) Retention of intermediate polarization states in ferroelectric materials enabling memories for multi-bit data storage. Appl Phys Lett 108(23):1040–1090. Google Scholar
  20. 20.
    Zheng P, Shuai L, Weina F (2017) A review of visual moving target tracking. Multimed Tools Appl 76(16):16989–17018CrossRefGoogle Scholar
  21. 21.
    Zheng P, Shuai L, Arun KS et al (2018) Visual attention feature (VAF): a novel strategy for visual tracking based on cloud platform in intelligent surveillance systems. J Parallel Distrib Comput 120:182–194CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.College of Mechanical Electrical EngineeringUniversity of Electronic Science and Technology of China, Zhongshan InstituteZhongshanChina
  2. 2.College of Computer ScienceInner Mongolia UniversityHohhotChina
  3. 3.School of Information and Electronic EngineeringHunan City UniversityYiyangChina
  4. 4.Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of EducationChangchunChina

Personalised recommendations