RLINK: Deep reinforcement learning for user identity linkage

User identity linkage is a task of recognizing the identities of the same user across different social networks (SN). Previous works tackle this problem via estimating the pairwise similarity between identities from different SN, predicting the label of identity pairs or selecting the most relevant identity pair based on the similarity scores. However, most of these methods fail to utilize the results of previously matched identities, which could contribute to the subsequent linkages in following matching steps. To address this problem, we transform user identity linkage into a sequence decision problem and propose a reinforcement learning model to optimize the linkage strategy from the global perspective. Our method makes full use of both the social network structure and the history matched identities, meanwhile explores the long-term influence of processing matching on subsequent decisions. We conduct extensive experiments on real-world datasets, the results show that our method outperforms the state-of-the-art methods.


Introduction
User Identity Linkage (UIL), which aims to recognize the identities (accounts) of the same user across different social platforms, is a challenging task in social network analysis.Nowadays, many users participate in multiple on-line social networks to enjoy more services.For example, a user may use Twitter and Facebook at the same time.However, on different social network platforms, the same user may register different accounts, have different social links and deliver different comments.If different social networks could be integrated together, we could create an integrated profile for each user and achieve better performance in many practical applications, such as link prediction and crossdomain recommendation.So, UIL has recently received increasing attention both in academia and industry.
Most of previous works consider UIL as a one-to-one alignment problem [6,12,34,37], i.e., each user has at most one identity in each social network.Matching models [26,25], label propagation algorithms [5,9,15,20,24,40] and ranking algorithms [12,32,34,39] are commonly used to address this task.These methods generally calculate pairwise similarity between identities and select the most relevant identity pairs according to the similarity score.In more recent works, identity similarity is computed based on user embedding [34], which encodes the main structure of social networks or other features into a low-dimensional and density vector.In most of these methods, each identity pair is matched independently, i.e., one predicted linkage of the identity pair would not be effected by any other.
However, we have an intuition that, the predicted linkages are inter-dependent and the previously matching would have a long-term influence on the subsequent one.This influence is two-fold: (i) if the preceding decisions are right, they would bring in positive auxiliary information to guide the following linkage.For example, if two user from two different social networks share the same friends (their friends have been matched), they are more likely to be matched in the subsequent linkage, as shown in Figure 1; (ii) the previously matched user identity could not be chosen in the subsequent matching process according to the one-to-one constraint [6], Although some previous label propagation based works have made primary attempts on using this influence, they just gave the fixed greedy strategy to iteratively select the candidate identity pair [15,20,24,40].Neither of them re-calculate the pairwise similarity or dynamically adjust the linkage strategy after each matching step due to the high complexity.
In order to model this long-term influence effectively, we novelly consider UIL as a Markov Decision Process and propose a deep reinforcement learning (RL) framework RLink to automatically match identities in two different social networks.Figure 2 illustrates the overall RLink process.In each time stamp, the state consists of two social network structures and previously matched identity pairs.According to the current state, the agent perform an action, i.e., generating an identity pair from the candidates.After performing the action, the state would be changed at the next time and a reward would be fed back to the agent to adjust its policy.Because the action space is large and dynamic in the UIL process, we adapt an Actor-Critic framework, in which the Actor network generates a deterministic action based on current state and the Critic network evaluates the quality of this action-state pair.Concretely, for each state, the Actor firstly encodes the network structure and history decisions to a latent vector representation, and then decodes this vector into an action in the identity embedding space.Based on this, our model could generate the matching sequence automatically.Deep RL model, which learns to directly optimize the overall evaluation metrics, works much better than models which learn with loss functions that just evaluate a particular single decision [10,37].However, there has been spots of attempts on applying RL in the social network analysis field [8,29,23].To the best of our knowledge, we are the first to design a deep RL model for user identity linkage.And in this paper, our RL model is able to produce more accurate results by exploring the long-term influence of independent decisions.
The contributions of this paper can be summarized as follows: -We are the first to consider UIL as a sequence decision problem and innovatively propose a deep reinforcement learning based model in this task, which generates the matching sequence automatically.-The proposed model makes full use of the previous matched identity pairs which may impact on the subsequent linkage, and makes decisions from a global perspective.-We conduct extensive experiments on three pairs of real-word datasets to show that our method achieves better performance than other state-of-theart solutions to the problem.
2 The Proposed Model

Preliminary
Let G = (V, E) represents an online social network, where V = {v 1 , v 2 , ..., v N } is the set of user identities and E ∈ V × V is the set of links in the network.Given two online social network G O (original network) and G T (target network), the task of user identity linkage is to identify hidden user identities pairs across G O and G T .Here, we have a set of node pair between G O and G T to represent given alignment information (ground-truth), denoted as B In this work, we consider UIL task as a markov decision process in which the linkage agent interacts with environment over a sequence of steps.We use s ∈ S to denote the current state, which consists of the network structure and matched identity pairs.Action, which is denoted as a ∈ A, is a pair of identities.The action space of UIL is comprised of all potential identity pairs, the size of which is |V O | × |V T |.At each step, the agent generates an action a based on s, and would receive a reward r(s, a) according to the given alignment information.The goal of RL is to find a linkage policy π : S ← A, which can maximize the cumulative reward for linkage.
Due to the large and dynamic action space, the policy-based RL [4,7] which compute probability distribution of every action and the Q-learning [17,31] which evaluate the value of each potential action-state pair are timeconsuming.To reduce the computational cost, we adapt the Actor-Critic framework, where the actor inputs the current state s and aims to output a deterministic action, while the critic inputs only this state-action pair rather than all potential state-action pairs.The architecture of the Actor-Critic network is shown in Figure 3.  Fig. 3 The framework of Actor-Critic network, where the actor is comprised of an Encoder-Decoder architecture and the critic is DQN.The inputs of Actor are history identity pairs and network structure, where history identity pairs were generated by our Actor-Critic network before current epoch (See Eq.( 4)).Remarkably, h 0 is a zero vector.Then this action and current state are input onto the Critic to evaluate the quality of this action.

Architecture of Actor Network
The goal of our Actor network is to generate an action (one matching identity pair) according to the current state.We propose an Encoder-Decoder architecture to achieve this goal.

Encoder for Current State
Encoder aims to generate the representation of current state, which contains two types of information: network structure and previous matched identity pairs.In order to integrate these information, we apply two encoding mechanisms to respectively encode the network structure and the matched pairs, as shown in Figure 3.
In our model, each identity v k (k ∈ (1, N ))is represented as a low dimensional and dense vector, which is denoted as u k and u k ∈ R d (d is the dimension of the identity embedding).These identity embeddings are pre-trained by Node2vec [1].To represent the social network structure, we weighted sum all identity embedding to generate the network embedding e, which is inspired by [13].
where α k denotes the weight of the identity v k .Here, we define α k as (similar to [13]), where ζ is a constant and d(v k ) represents the degree of identity v k .We denote the representation of original network and target network as e O and e T , then we concatenate them to get s net and s net ∈ R 2d .That is: At time t, we need to encode all previously matched identity pairs from time 1 to t − 1, i.e., an action sequence {a 1 ,a 2 , ..., a t−1 }.In each action a i (i ∈ [1, t − 1]), the identity from G O , G T are respectively denoted as v O i and v T i , and their embedding representation are denoted as u O i and u T i .Besides, we take the feedback of a i , which is denoted as g i , into account.g i is a one-hot vector, in which the value of each dimension is equal to the immediate reward at the corresponding time stamp.Through a neural network layer, we get the encoding vector of g i as g i : where g i ∈ R |g| and g i ∈ R |G| , and R |g| is equal to the number of steps in each episode and R |G| depends on the dimension of W G .Then, we get the representation of one history matched pair at time i by concatenating u O i , u T i and g i : where the dimensional of In order to capture the long-term influence of the previously matched pairs, we use a Long Short-Term Memory (LSTM) network to encode the history linkage sequence into a fixed-size vector: Furthermore, to distinguish different contribution of the previous actions, we employ attention mechanism [2], which allows model to adaptively focus on different parts of the input: -0.5em where the dimension of s pair t is equal to s net , and we leverage a location-based attention mechanism [16] to compute γ i from the hidden state h i Then, the embedding of current state can be represented as follows: where s t ∈ R 2d , s pair t and s net represent the embedding of history matching information and network structure respectively.
It is noteworthy that, at time 1, the current state just contains two social network structures because there is not matched identity pair.That is to say, h 0 = 0, s pair t is equal to zero vector and s t is equal to s net .Note that, the decoded a O t and a T t maybe not in the identity embedding space.Thus we need to map them into the real embedding space via the transformation [38].As mentioned above, the action space is very large.In order to correctly map a t into the validate identity, we select the most similar u t ∈ U as the valid identity-embedding.In this work, we compute the cosine similarity to get the valid identity in given networks: where {v O t , v T t } represents the valid identity pair a t .We pre-compute the value of U O U O and U T U T to decrease the computational cost.Since the right alignment identities could be chosen in the following steps, we ignore those identities if they are correctly matched in the previous steps.

Reward
The agent generates the new identity pair and receives the immediate reward r tm from networks, which is also the feedback of this identity pair, where Groundtruth is a set of known aligned identity pairs.Since current action result has a long-term impact on subsequent decisions, we introduce a discount factor λ to metric the weight of reward: where λ t ∈ {0, 1} represents how much influence the action a t will generate on the following steps.Simply, λ t is defined as 1 t , where t represents the current time stamp.

Architecture of Critic Network
Critic Network aims to judge whether the action a t generated by Actor suits the current state s t .Generally, the Critic is designed to learn an action-value function Q(s, a), while the actor updates its' parameters in a direction of improving performance to generate next action according to Q(s, a) in the following steps.However, in real UIL, the state and action space is enormous and many state-action pairs may not appear in the real traces, which makes it hard to update their Q-values.Thus, we choose Deep Neural Network as an approximator to estimate the action-value function.In this work, we refer to a neural network as Deep Q-value function (DQN) [35].
Firstly, we need to feed user's current state s and action a into the DQN.To generate user's current state s, the agent follows the same strategy from Eq.(1) to Eq.( 6).As for action a, we utilize the same strategy in decoder to compute a low-dimensional dense action vector a.Then, this action is evaluate by the DQN, which returns the Q-value of this state-action pair Q(s, a).

Training and Test
Generally, we utilize DDPG [14] algorithm to train the proposed Actor-Critic framework, which has four neural network: critic, actor, critic-target and actortarget, where target networks are the copy of critic and actor respectively.The critic is trained by minimizing the mean squared error loss with the corresponding target given by: where ρ is the discount factor in RL, R is the accumulative reward and s is the previous state, f θ π (s ) = a .Besides, β c and θ π represent all parameters in Critic and Actor respectively, and f θπ represents the policy.The Critic is trained from samples stored in a replay buffer [18].Similarity, actions also stored in the replay buffer generated by the strategy in Actor decoder section.This allows the learning algorithm to dynamic leverage the information of which action was actually executed to train the critic.
The first term R + ρQ β c (s , f θ π (s )) in Eq.( 9) is the output of target, namely y, for current iteration.And parameters from the previous iteration θ π are fixed when optimizing the loss function L(β c ). Computing the full expectations' gradient are not efficient.Thus, we optimize the loss function by Stochastic Gradient Descent (SGD).The derivatives of loss function L(β c ) with respective to parameters β c are presented as follows: The Actor is updated by using the policy gradient: The training algorithm for the proposed framework RLink is presented in Algorithm 1.In each iteration, there are two stages, i.e., 1) generating an action (lines 8-11), and 2) parameter updating (lines [13][14][15][16][17].For generating an action, given the current state s t , the agent firstly encode current state as a vector (line 8) and then generate a pair of nodes according to this vector (line 9); then the agent observes the reward r t and update state to s t+1 (line 10); finally the agent stores transitions (s t , a t , r t , s ( t + 1)) into replay buffer D. For the parameter updating stage: the agent samples mini-batch of transitions (s, a, r, s ) from D (line 13), and then updates parameters of Actor and Critic following a standard DDPG procedure (lines 14-16).Finally, the parameters of target network β c and θ π is updated via the soft update way (line 17).
To evaluate the performance of our model, the test procedure is designed as an online test method, which is similar to the action generation stage in the training procedure.After the training procedure, proposed framework RLink learns parameters θ π and β c .In each iteration, the agent generates a pair of identity a t following the trained policy f θπ .And then the agent receives the reward r t from networks and updates the state to s t+1 .

Experiment
In this section, we compare our RLink with the state-of-the-art methods on types of cross-network datasets.

Datasets
In order to verify our method in different types of networks, we conduct experiments on the following network collections: Foursquare-Twitter, Last.fm-Myspace and Aminer-Linkedin, which are commonly used in the UIL task.The first dataset is provided by [36] and others are are collected by [37].These datasets are introduced as follows and the statistic information is shown in Table 1 which only contain social links as user identities' feature.
-Foursquare-Twitter is a pair of social networks, where users share their current location and other information with others.-Last.fm-Myspace is a pair of online social networks, where users could search music and share their interested music with others.-Aminer-Linkedin is a pair of citation networks where contains users' academic achievements and users could search interested community.

Comparative Methods
In this section, to evaluate the performance of RLink for user identity linkage, we choose the following state-of-the-art methods as competitions, including: -IONE [12] IONE predicts anchor links by learning the followership embedding and followeeship embedding of a user simultaneously.
-DeepLink [39] DeepLink employs unbiased random walk to generate embeddings, and then use MLP to map users.-MAG [32] MAG uses manifold alignment on graph to map users across networks.
-PAAE [43] PAAE employs an adversarial regularization to capture the robust embedding vectors and maps anchor users with an alignment autoencoders.
-SiGMa [10] SiGMa was designed to align two given network by propagating the confidence score in the matching network.We use the name matching method to generate the seed set and utilize the output scores of SVM as the pairwise similarity.Note that SiGMa is an unsupervised method.-SDM: This method is a simpler deep model, which is similar to typical sequence prediction methods [3,11] and utilizes LSTM to process sequence matching.The matching sequence is generated via ranking the similarity of embedding.At each step i, LSTM predicts an user identity in Target SN based on current user identity embedding and previous hidden information (h i−1 ).Finally, the L2 loss would guide the training and testing process of this model.

Evaluation Metrics
To perform the user identity linkage, we utilize two standard metrics [27] to evaluate the performance, including Precision@k (P@k), recall and MAP.Note that higher the value of each these measures, the better the performance.
Precision@k is the metric for evaluating the linking accuracy, defined as: where 1 i {success@k} measures whether the positive matching identity exists in top − k(k <= n) list, and n is the number of testing anchor nodes.
The recall is the fraction of the number of real corresponding user pairs that have been found over the total amount of real matched user pairs (Groundtruth B).
MAP is used for evaluating the ranking performance of the algorithms, defined as: where ra is the rank of the positive matching identity and n is the number of testing anchor nodes.

Hyper-parameter setting
For each pair of networks, we first resort the ground truth data set by identity number and then use the first r as the training data and the later 1 − r as the testing data, where r means the training ratio.The dimension of the identity embedding is set to 128, whether the input of the encoder or the output of the decoder, i.e., |U | = |G| = 128.And the weighting parameter ζ is fixed to 10 −3 .For parameters in Actor, the number of LSTM cell units is set to 256 and batch size is 64.In Critic, the number of hidden layer is 4. The batch size in replay buffer is 64 and we set 200 sessions in each episode.Besides, in training procedure, we set learning rate η = 0.001, discount factor ρ = 0.9, and the rate of target networks soft update τ = 0.001.

Comparisons and Analysis
We compare our proposed model RLink with the following recent user identity linkage methods.Note that SiGMa, SDM and RLink are prediction methods which determine whether two user identities from original and target are matching or not.Therefore, the evaluation of those methods becomes P@k = P@1 = MAP.From Table 2, we can see that: -RLink is significantly better than previous user identity linkage methods.This result demonstrates that our method which considers UIL as a sequence decision problem and makes decisions from a global perspective is useful.Besides, reinforcement learning based methods, which learns to directly optimize the overall evaluation metrics, works better than other deep learning methods, such as DeepLink and PAAE.-SDM and SiGMa achieve higher precision than other baselines, which demonstrates that considering the influence of previous matched information is beneficial for UIL task.Although SigMa achieves the highest precision on Last.fm-Myspace, it suffers from a lowrecall due to its fixed greedy matching strategy.By contrast, SDM and RLink promotes about 30% and 40% recall over SigMa on respectively, which proves previous matched result has a long-time impact on the following matching process.-IONE, PAAE and DeepLink perform better than MAH and MAG, which means that considering more structural information, such as followership/ followee-ship and global network structure, could achieve higher precision.Besides, the precision of PAAE and DeepLink are higher than IONE, which demonstrates that considering global network structure and utilizing deep learning model could improve the performance of UIL.-Besides, we note that all methods perform better on Aminer-Linkedin than they do on Last.fm-Myspace, maybe because the scale of the training data in Aminer-Linkedin is larger.

Parameter Sensitivity
In this section, we analyze the sensitivity of three parameters which are different K in P@K, training ratio r and embedding dimension d.The performance of those parameters on all three testing datasets is similar.We present the performance on Twitter-Foursquare due to the limited spaces.Precision on different K.For different K in P@K, we reports the precision of different methods on variable K between 1 and 30.RLink outperforms all the comparison methods consistently and significantly given different @K settings.IONE performs better than MAG and MAH, showing that constructing the incidence matrices of the hypergraph could fail to differentiate the follower-ship and followee-ship.Both RLink and DeepLink perform better than MAG, MAH and IONE, showing that deep learning methods could Precision on different training ratio r.For different training ratios, we reports the Precision@30 of different methods on variable training ratio between 10% and 90%.RLink outperforms all the comparison methods when the ratio rose to 60%.The ratio of anchor nodes used for training greatly affects the performance of RLink.Especially for ratio settings as 60% to 70%, the performance enhancement is significant.While the result shows that with the increase of training data, the precision of UIL firstly increases significantly and then does not increase drastically as the ratio rose to 70%.And the comparative methods achieve good performance when the training ratio is setting around 90%, which demonstrates the robust adaptation of our method RLink.Besides, from Figure 4b, we found that DeepLink performs better than RLink when the training ratio is less then 40%, indicating that RLink might need more train dataset than DeepLink.
Precision on different embedding dimension d.According to Figure 4c, both MAG and MAH achieve good performance when the dimensionality setting is around 800, while RLink, IONE and DeepLink achieve good performance when the dimensionality is around 100.The complexity of the learning algorithm is highly depending on the dimensionality of the subspace.And low dimensional representation also leads to an efficient relevance computation [12].Therefore, we conclude that RLink, IONE and DeepLink is significantly more effective and efficient than MAH and MAG.Besides, RLink achieves the best results when the dimensionality setting is bigger than 80.

Evaluation of Actor-Critic Framework
In this paper, we use DDPG algorithm to train the RLink model.To evaluate the effectiveness of the Actor-Critic framework, we compare the performance of DDPG with DQN.From Figure 5, we can see that DQN performs similar to DDPG, but the training speed of DQN is much slower.As shown in Figure5 (a), DQN needs 1200 episodes to converge, which is almost four times of the episodes DDPG needs.In Linked-Aminer, as shown in Figure5(b), both DDPG and DQN need more training episodes, i.e., 750 and 2000 episodes respectively, when the size of action space rose to 4153×4153.Besides, DQN performs worse than it does in Last.fm-Myspace, which demonstrates that DQN is not suitable for the large action space.In summary, DDPG achieves better performance and faster training speed than DQN, which indicates that Actor-Framework is suitable for UIL with enormous action space.

Effect of Long-term Reward
To evaluate the effectiveness of long-term reward, we compare it with an immediate reward via Q-value performance.Q-value performance is a judgment of whether a suits s.From Figure 6, we can find that both long-term and immediate reward would converge to a similar value, but the Q-value of immediate reward is increased first and then decreased.The higher Q-value in the training procedure indicates that the immediate reward may get trapped in a local optimum.By contrast, the long-term reward makes our model make decisions from a global perspective and the Q-value smoothly converge.

Effectiveness of RLink Components
This experiment is designed to validate the effectiveness of main components in the Actor network, including the input features, attention layer, LSTM layer, action transformer and decoder.We systematically eliminate the corresponding component and define the following variants of RLink.
-RLink LIN E (RLink with LINE as pre-trained embedding method): In this variant, we replace the pre-trained embedding method Node2vec by LINE [33] to evaluate the diversity of different pre-trained identity embeddings.The result is shown in Table 3.We can find that RLoml LIN E adapt LINE to pre-train user identity embedding vectors, which achieves similar performance to RLink.This phenomenon indicates that the pre-trained identity embeddings could not have great influence on the performance of UIL.RLink-FEED performs the worst among all variants, which demonstrates that the feedback information significantly promotes the performance of our model.RLink-LSTM performs worse than RLink, which suggests that capturing the long-term memory of the history matched information by LSTM is very beneficial for the subsequent matching.RLink-ATT proves that incorporating attention mechanism can better capture the influence of each previous decision than only LSTM.RLink-TRANS proves that action transformer could map a val and a cur exactly.RLink M LP proves that simply neural network in the decoder phase could perform better than more complicated model (MLP).39,19].However, those two-step methods needs two subject which is difficult to optimize.[12] proposes an unified framework to address this challenge, where the embeddings of multiple networks are learned simultaneously subject to hard and soft constraints on common users of the network.[41] introduces an active learning method to over the sparsity of labeled data, which utilizes the numerous unlabeled anchor links in model building.Finally, inspired by the recent successes of deep learning in different tasks, especially in automatic feature extraction and representation, [39] propose a deep neural network based algorithm for UIL.

Reinforcement Learning
Reinforcement Learning (RL) is one of the most important machine learning methods, which gets optimal policy through trail-and-error and interaction with dynamic environment.Generally, RL contains two categories: modelbased and model-free.And the most frequently used method is model-free reinforcement learning methods which can be divided into three categories: Policybased RL, Q-learning and Actor-Critic.The policy-based RL [4,7] learns the policy directly which compute probability distribution of every action.The Q-learning [17,31] learns the value function which evaluate the value of each potential action-state pair.And Actor-Critic [38,30] learns the policy and value function simultaneously which aims to evaluate the quality of a deterministic action at each step.
Recently, due to various advantages of RL, it has been successfully applied in many fields, such as Game [17,28], Computer Vision [42] and Natural Language Processing [4].However, due to the complexity of online social network analysis tasks, works based on RL is less than in other fields.[29] is an early work based on reinforcement learning in social network which argues modeling network structure as dynamic increases realism without rendering the problem of analysis intractable.Existing methods utilize DQN or Q-learning framework, such as [8] applies DQN to address Graph Pattern Matching problem and [23] applies Q-Learning to search expert in social network.Inspired by the above works, we consider UIL as a markov decision problem and apply the reinforcement learning framework.However, computing Q-value of each identity pair is time-consuming.So, we adapt Actor-Critic framework to address UIL in this paper.

Conclusion
In this paper, we consider user identity linkage as a sequence decision problem and present a reinforcement learning based model.Our model directly generate a deterministic action based on previous matched information via Actor-Critic framework.By utilizing the information of previously matched identities and the social network structures, we can optimize the linkage strategy from the global perspective.In experiments, we evaluate our method on

Fig. 1
Fig. 1 An example of User Identity Linkage.The blue link represents friend relation in social network, and orange line represents matched identity pair.User identity sley (in Foursquare) has two candidate identities in Twitter: sley and sley 1.If considering previously matched information, sley in Foursquare is similar to sley 1 in Twitter because they share more similarity friends.

Fig. 5
Fig. 5 Comparison between DDPG and DQN: The x-axis shows the training episodes.The y-axis shows the total reward of each episode.Red line represents DDPG and green Line is DQN.

Fig. 6 Q 1 t / − 1 t-
Fig. 6 Q-value performance with different reward on Last.fm-Myspace and Linkedin-Arminer.The x-axis shows the training episodes.The y-axis shows the normalize Q-value of each session.The value of immediate reward is equal to 1/ − 1, while long-term reward is 1 t / −1 t A Procedure of Reinforcement Learning based User Identity Linkage.The blue link represents friend relation in social network, and orange line at S i represents matched identity pair.At time i, agent generates a pair of matching identities as action according to current state.After environment performs this action, the state would be changed at time i + 1 and next action can be generated based on S i+1 .

Table 1
Statistics of experimental datasets

Table 2
Performance comparison on user identity linkage