Advertisement

An Analysis and Research of Network Log Based on Hadoop

  • Wenqing WangEmail author
  • Xiaolong Niu
  • Chunjie Yang
  • Hongbo Kang
  • Zhentong Chen
  • Yuchen Wang
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 891)

Abstract

With the rapid development of the internet technology, we have entered the era of big data, people product massive amount of data on the internet. Through the analysis and data mining of Web logs, we can dig out valuable information such as user’s behavior preferences. But handling massive amounts of data, the traditional single machine can no longer meet the requirements. With the continuous development of big data technology, massive Hadoop log data can be analyzed through the framework of big data. In this paper, the Hadoop large data platform is built, the MapReduce programming model is used to preprocess the network log, and the Hive data warehouse is used to analyze the processed data in multi dimension. The analysis results have good guiding significance for mastering the user browsing behavior, promoting the promotion effect, optimizing the structure and experience of the website.

Keywords

Big data Data mining Hadoop MapReduce Hive data warehouse 

References

  1. 1.
    Edwards, M.F., Rambani, A.S., Zhu, Y.T., et al.: Design of hadoop-based framework for analytics of large synchrophasor datasets. Procedia Comput. Sci. 12(4), 254–258 (2012)CrossRefGoogle Scholar
  2. 2.
    Chansler, R., Kuang, H., Radia, S., Shvachko, K.: The hadoop distributed file system. In: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST 2010) (MSST), Incline Village, NV, pp. 1–10 (2010).  https://doi.org/10.1109/msst.2010.5496972
  3. 3.
    Dean, J.F., Ghemawat, S.S.: MapReduce: simplified data processing on large clusters. ACM 51(1), 107–113 (2008).  https://doi.org/10.1145/1327452.1327492CrossRefGoogle Scholar
  4. 4.
    Kotiyal, B.F., Kumar, A.S., Pant, B.T., et al.: Big data: mining of log file through hadoop. In: International Conference on Human Computer Interactions, pp. 1–7. IEEE (2014).  https://doi.org/10.1109/ich-ci-ieee.2013.6887797
  5. 5.
    Wang, C.H., Tsai, C.T., Fan, C.C., et al.: A hadoop based weblog analysis system (2014).  https://doi.org/10.1109/u-media.2014.9
  6. 6.
    Suguna, S.F., Vithya, M.S., Eunaicy, J.I.C.: Big data analysis in e-commerce system using Hadoop MapReduce. In: International Conference on Inventive Computation Technologies, pp. 1–6 (2017).  https://doi.org/10.1109/inventive.2016.7824798
  7. 7.
    Du, J.F., Zhang, Z.S., Zhao, C.T.: Analysis on the digging of social network based on user search behavior. Int. J. Smart Home 10(5), 297–304 (2016).  https://doi.org/10.14257/ij-sh.2016.10.5.27
  8. 8.
    Dewangan, S K, Pandey, S., Verma, T.: A distributed framework for event log analysis using MapReduce. In: International Conference on Advanced Communication Control and Computing Technologies, pp. 503–506. IEEE (2017).  https://doi.org/10.1109/icaccct.2016.7831690
  9. 9.
    He, G.F., Ren, S.S., Yu, D.T., et al.: Analysis of enterprise user behavior on hadoop. In: Sixth International Conference on Intelligent Human-Machine Systems and Cybernetics, pp. 230–233. IEEE (2014).  https://doi.org/10.1109/ihmsc.2014.158

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Wenqing Wang
    • 1
    Email author
  • Xiaolong Niu
    • 1
  • Chunjie Yang
    • 1
  • Hongbo Kang
    • 1
  • Zhentong Chen
    • 1
  • Yuchen Wang
    • 1
  1. 1.School of AutomationXi’an University of Posts and TelecommunicationsXi’anChina

Personalised recommendations