Skip to main content

Real-Time Distribution Algorithm for Fully Comparison Data Based on Storm

Abstract

Current data allocation algorithms neglect the problems of unsatisfactory allocation results and long execution time caused by the redundancy of full comparative data and the complexity of data types. To solve these problems, a real-time allocation algorithm of full comparison data based on storm is proposed. Firstly, the phase unwrapping algorithm of minimum spanning tree is used to remove redundant data in full comparison data; then, the distributed data clustering algorithm and storm framework are used to realize the full comparison data clustering after redundancy removal. Several main factors affecting the selection of statistical information are summarized according to the clustering results. Then the communication cost of data loading and transaction processing is determined, and the trade-off between read-only transaction and update transaction cost is achieved. By judging whether the total cost of read-only transaction and update transaction is reduced or not, the replica is eliminated, and a full comparison data allocation algorithm with minimum total cost of read-only transaction and update transaction is proposed to realize real-time allocation of full-comparative data. The example analysis shows that the proposed algorithm can meet the user’s needs in terms of execution time, acceleration ratio, storage efficiency and cost. Compared with the reference algorithm, the proposed algorithm has the lowest execution time, the highest acceleration ratio and the closest allocation cost to the ideal overhead.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. 1.

    Wang ZC, Li HX, Hou HH (2020) Sender data allocation scheme for reducing disorder degree in CMT system. Application Research of Computers 37:551–554

    Google Scholar 

  2. 2.

    Ma XX, Lu G, Fu BZ et al (2020) Implementation methods and performance analysis of non-contiguous data communication in network. Chinese Journal of Computers. 43:1123–1138

    Google Scholar 

  3. 3.

    Lin M, He ZF (2019) Big data allocation mechanism for wireless sensor communication based on cooperative Kalman filter. Electronic Measurement Technology 42:123–127

    Google Scholar 

  4. 4.

    Zhang YM, Jiang JB, Lu JW et al (2019) An iterative data partitioning strategy for MapReduce. Chinese Journal of Computers 42:1873–1885

    Google Scholar 

  5. 5.

    L. P. Xiang, H. J. Yang. Dynamic resource allocation algorithm based on big data stream characteristic and improved SOM clustering. Computer Applications and Software. 36 (2019) 262–268+,280

  6. 6.

    Ding ZC, Ge HW (2020) Density peaks clustering with optimized allocation strategy. Journal of Frontiers of Computer Science & Technology 14:792–802

    Google Scholar 

  7. 7.

    Wang X (2020) Research on dynamic traffic data resource allocation algorithm of ship mobile network. Ship Science and Technology 42:184–186

    Google Scholar 

  8. 8.

    Xv J, Wang XY, Cai ZX et al (2020) Imbalance classification based on informative instances selection. Journal of Frontiers of Computer Science & Technology. 14:401–409

    Google Scholar 

  9. 9.

    L Shuai, W Shuai, L Xinyu, et al. (2021) Human memory update strategy: a multi-layer template update mechanism for remote visual monitoring, IEEE Transactions on Multimedia 23:2188–2198

  10. 10.

    Raju S, Chandrasekaran M (2019) Performance analysis of efficient data distribution in P2P environment using hybrid clustering techniques. Soft Comput 23:9253–9263

    Article  Google Scholar 

  11. 11.

    Ahmad SG, Khan HU, Ijaz S et al (2020) Use case-based evaluation of workflow optimization strategy in real-time computation system. J Supercomput 76:708–725

    Article  Google Scholar 

  12. 12.

    S Wang, X Liu, S Liu, et al. Human Short-Long Term Cognitive Memory Mechanism for Visual Monitoring in IoT-Assisted Smart Cities. IEEE Internet of Things Journal, 2021, online first, doi: https://doi.org/10.1109/JIOT.2021.3077600

  13. 13.

    Shui-Hua W, Xiaosheng W, Yu-dong Z et al (2020) Diagnosis of COVID-19 by wavelet Renyi entropy and three-segment biogeography-based optimization. International Journal of Computational Intelligence Systems 13(1):1332–1344

    Article  Google Scholar 

  14. 14.

    Liu S, Liu D, Muhammad K et al (2021) Effective template update mechanism in visual tracking with background clutter. Neurocomputing 458:615–625

  15. 15.

    Yu-Dong Z, Zhengchao D, Shui-HuaW, et al. (2020) Advances in multimodal data fusion in neuroimaging: overview, challenges, and novel orientation. Information Fusion 64:149–187

    Article  Google Scholar 

  16. 16.

    Pourpanah F, Shi YH, Lim CP et al (2019) Feature selection based on brain storm optimization for data classification. Applied Soft ComputingVolume 80:761–775

    Article  Google Scholar 

  17. 17.

    Wang SH, Govindaraj VV, Górriz JM (2021) Covid-19 classification by FGCNet with deep feature fusion from graph convolutional network and convolutional neural network. Information Fusion 67:208–229

    Article  Google Scholar 

  18. 18.

    Liu S, Wang S, Liu X et al (2021) Fuzzy detection aided real-time and robust visual tracking under complex environments. IEEE Trans Fuzzy Syst 29(1):90–102

    Google Scholar 

  19. 19.

    Castro-Medina F, Rodriguez-Mazahua L, Lopez-Chau A et al (2020) FRAGMENT: a web application for database fragmentation. Allocation and Replication over a Cloud Environment IEEE Latin America Transactions 18:1126–1134

    Google Scholar 

  20. 20.

    Rossi ALD, Soares C, Souza BFD, Ponce de Leon Ferreira de Carvalho AC (2021) Micro-MetaStream: algorithm selection for time-changing data. Inf Sci 565:262–277

    MathSciNet  Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Chang-qing Dong.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Dong, Cq., Chen, C., Ren, N. et al. Real-Time Distribution Algorithm for Fully Comparison Data Based on Storm. Mobile Netw Appl (2021). https://doi.org/10.1007/s11036-021-01824-3

Download citation

Keywords

  • Storm framework
  • Full comparison data
  • Data allocation
  • Real time