Abstract
With the rapid development of information technology, information overload has become an important challenge of Internet. In order to alleviate the growing contradiction between users and massive data, the researchers proposed the concept of the cross-harmonic recommender system. By analyzing characteristic of datasets, recommendation algorithms and method for weight calculation, we introduced a fast and general engine for large-scale data processing and implemented the cross-harmonic recommender system based on Spark, aiming at improving accuracy, diversity and efficiency of the recommender system.
Fund Source: Hunan education department scientific research project (No. 17C0009). Huang Jie, Associate professor/Master, main research fields: Information education in higher vocational, Computer applications (Cloud and Big data), etc. Liu Changsheng, professor/Doctor, main research fields: Computer Application, etc.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Shvachko, K., Kuang, H., Radia, S., et al.: The hadoop distributed file system. In: 2014 IEEE 26th Symposium on IEEE Mass Storage Systems and Technologies (MSST), pp. 1–20 (2014)
Zaharia, M., Das, T., Li, H., et al.: Discretized streams: an efficient and fault-tolerant model for stream processing on large cluster. HotCloud 12, 34–56 (2015)
Meng, X., Bradley, J., Yavuz, B., et al.: Mllib: machine learning in apache spark. J. Mach. Learn. Res. 12(34), 23–34 (2017)
Wang, Z., Sun, L., Zhu, W., et al.: Joint social and content recommendation for user-generated videos in online social network. IEEE Trans. Multimed. 15(3), 698–709 (2017)
Koren, Y.: The bellkor solution to the netfix grand prize. Netflix Prize Doc. 34, 32–45 (2016)
Zaharia, M., Chowdbury, M., Franklin, M.J., et al.: Spark: cluster computing with working sets. HotCloud 10(11), 80–98 (2015)
Li, H., Ghodsi, A., Zaharia, M., et al.: Tachyon: reliable, memory speed storage for cluster computing frameworks. In: Proceedings of the ACM Symposium on Cloud Computing, pp. 13–25. ACM (2016)
Engle, C., Lupher, A., Xin, R., et al.: Shark: first data analysis using coarse-grained distributed memory. In: Proceeding of the 2012 ACM SIGMOD International Conference on Management of Data (2012)
Armbrust, M., Xin, R.S., Lian, C., et al.: Spark SQL: relational data processing in spark. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 1383–1394. ACM (2015)
Deshpande, M., Karypis, G.: Item-based top-N recommendation algorithms. ACM Trans. Inf. Syst. 22(1), 143–177 (2014)
Yang, R., Hu, W., Qu, Y.: Using semantic technology to improve recommender systems based on slope one. In: Li, J., Qi, G., Zhao, D., Nejdl, W., Zheng, H.T. (eds.) Semantic Web and Web Science, pp. 11–23. Springer, New York (2013). https://doi.org/10.1007/978-1-4614-6880-6_2
Takács, G., Pilászy, I., Németh, B., Tikk, D.: Major components of the gravity recommendation system. ACM SIGKDD Explor. Newsl. 9(2), 80–83 (2014)
Nightingale, E.B., Chen, P.M., Flinn, J.: Speculative execution in a distributed file system. ACM (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Jie, H., ChangSheng, L., ChengLi, L. (2019). Design and Implementation of the Cross-Harmonic Recommender System Based on Spark. In: Gui, G., Yun, L. (eds) Advanced Hybrid Information Processing. ADHIP 2019. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 302. Springer, Cham. https://doi.org/10.1007/978-3-030-36405-2_47
Download citation
DOI: https://doi.org/10.1007/978-3-030-36405-2_47
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-36404-5
Online ISBN: 978-3-030-36405-2
eBook Packages: Computer ScienceComputer Science (R0)