Generation of a Social Network Graph by Using Apache Spark
It is planned to create a method of clustering a social network graph. To test the method, it is necessary to generate a graph similar in structure to existing social networks. The article presents an algorithm for the graph-distributed generation. We take into account basic properties such as the power-law distribution of the number of user communities, the dense intersections of social networks, and others. This algorithm also considers the problems that are present in similar works of other authors, for example, the multiple edges problem in the generation process. A special feature of the created algorithm is the implementation depending on the number of communities, rather than on the number of connected users, as is done in other works. This is connected with a peculiarity of the development of the existing social network structure. The properties of its graph are described in the paper. We describe a Table 1 containing the variables needed for the algorithm. A step-by-step generation algorithm is compiled. Appropriate mathematical parameters are calculated for it. The generation is performed in a distributed way by the Apache Spark framework. It is described in detail how the division of tasks with the help of this framework operates. The Erdos–Renyi model for random graphs is used in the algorithm. It is the most suitable and easiest one to implement. The main advantages of the created method are the small amount of resources and faster execution speed in comparison with other similar generators. Speed is achieved through distributed work and the fact that at any time, the network users have their own unique numbers and are ordered by these numbers so that there is no need to sort them out. The designed algorithm will not only promote the creation of an efficient clustering method, but can also be useful in other development areas connected, for example, with social network search engines.
Keywordssocial network generation
Unable to display preview. Download preview PDF.
- 1.Chikhradze, K.K., et al., On a model of social network with user communities for distributed generation of random social graphs, Mach. Learn. Data Anal., 2014, vol. 1, no. 8, pp. 1027–1047.Google Scholar
- 2.Yang, J. and Leskovec, J., Community-affiliation graph model for overlapping network community detection, IEEE 12th Conference (International) on Data Mining, 2012.Google Scholar
- 3.Raigorodskii, A., Models of random graphs and their applications to the web-graphs analysis, RUSSIR-2015, Moscow, 2015.Google Scholar
- 5.Aiello, W., Chung, F., and Lu, L., On the evolution of random graphs. A random graph model for power law graphs. http://people.math.sc.edu/lu/papers/power.pdf.Google Scholar
- 6.Karau, K., et al., Learning Spark: Lightning-fast Data Analysis, DMK Press, Moscow, 2015.Google Scholar
- 7.Vovchok, S.I., Creation of the method for clustering the social media graph, Novye informatsionnye tekhnologii v nauke: Sbornik statey Mezhdunarodnoy nauchno-prakticheskoy konferentsii MTsII OMEGA SAYNS (New Information Technologies: Proc. Int. Sci.-Pract. Conf. MTsII OMEGA SAYNS), 2016, part 2, pp. 34–36.Google Scholar