Enhancing social media post popularity prediction with visual content

Abstract

Our study presents a framework for predicting image-based social media content popularity that focuses on addressing complex image information and a hierarchical data structure. We utilize the Google Cloud Vision API to effectively extract key image and color information from users’ postings, achieving 6.8% higher accuracy compared to using non-image covariates alone. For prediction, we explore a wide range of prediction models, including Linear Mixed Model, Support Vector Regression, Multi-layer Perceptron, Random Forest, and XGBoost, with linear regression as the benchmark. Our comparative study demonstrates that models that are capable of capturing the underlying nonlinear interactions between covariates outperform other methods.

Appendices

Appendix

Sampling process

In this paper, we sought to leverage text and image data of social media data for the purpose of predicting the number of “Likes”. However, to ensure the quality and relevance of the data used in our study, we implemented a careful selection process.

Our sample consists of influencers with a public account, who among their latest 100 posts, have uploaded a minimum of five sponsored or advertisement posts. This is determined by hashtags, such as #AD or #Sponsored. We further restrict our sample to those who have uploaded at least 100 posts at the time of crawling. Instead of relying on random users’ public data, which often includes individuals who may not frequently post images or share poor quality and meaningless content, we adopted a targeted approach by focusing on active influencers. By selecting users who demonstrate a consistent pattern of posting visually appealing and engaging photos, we aimed to capture a representative sample of users whose content and audience interactions are more likely to yield meaningful insights for predicting the number of “Likes”. This approach allowed us to enhance the reliability and validity of our analysis, ensuring that the data used accurately reflects the dynamics and factors influencing user engagement on social media. This sample selection process allowed us to focus on users who provide regular uploads, post meaningful content, and tend to post diverse topics to actively engage the audience on the platform.

Additionally, we wanted that the number of observations for each user is large for several reasons. First, in the popularity prediction task, it is critical to consider individual user effect because many user-specific unobservable factors, such as user engagement habits, content style, creativity and degree of popularity of the user, may substantially influence post popularity. Ensuring enough observations for each user enables us to disentangle the user-specific fixed effect. Next, examining a substantial number of posts per user allows us to capture a more comprehensive picture of users’ posting behavior and content patterns. Additionally, analyzing a larger sample size helps mitigate the influence of outliers or temporary fluctuations, providing a more accurate representation of their typical posting style and engagement levels.

Specifically, we conducted the following procedure for sample construction:

• 1. To select an initial user, we search for the hashtag ‘$\#$Sponsor’ and choose the user who appears at the top of the search feed. Note that only the public account users will appear in the search feed. This user is assigned as “parent user”.

• 2. We extract the users who are present in the “parent user”’s following list. These users are assigned as “children users”.

• 3. Among the “children users”, we select users who meet the criteria mentioned above: (1) having more than 100 posts and (2) having more than five sponsored/advertisement posts among their most recent 100 posts.

• 4. These selected users are assigned as “parent user”.

• 5. We repeat 2, 3, and 4.

We collected data of 40 users, and to give the same information weight to the selected users, we crawled the 100 most recent posts for each user. Although we collected 100 post information for all users, the total number of uploaded images may vary for each user, as each post can contain anywhere from one to ten images. The data crawling period is between 6 February 2022 and 24 March 2022, with the resulting analysis sample of 40 users, 3807 posts, and 13,774 images. From 4000 collected posts, those consisting only video contents without any image are excluded.

There are a few aspects of the sampling process that need to be mentioned. First, in our data collection process, usernames are not acquired. Instead, individuals are assigned randomized identification numbers for the purpose of anonymity and confidentiality. User individuals effects are addressed by using dummy variables for each user. Second, restricting the user selection pool to the following account list of the previously chosen user offers a realistic and practical approach for data collection. This is because users tend to follow accounts with similar interests or content, making it more likely to identify individuals who actively engage with sponsored posts.

Seeded-LDA

1.1 The settings of Seeded-LDA

The LDA’s hyperparameter is the parameter of the prior distribution of words and documents. In general, $\alpha$ is a parameter of the document distribution, and $\beta$ is a parameter of the word distribution. The tuning grid was set to 0.05, 0.1, 0.5, 1, 5, and 10, and the hyperparameter was selected based on the logarithmic ratio among the measurements of coherence. For caption topics, ($\alpha$,$\beta$) was set to (10,10), and for Image label topics, ($\alpha$,$\beta$), the value of (10,0.1) was selected. The seed words for each topic are listed in Table 5.

Table 5 The list of seed words used for each topic in performing Seeded-LDA for caption and image labels is based on frequency

1.2 Sensitivity of the number of topics

Seeded-LDA can be sensitive to the selection of several pre-determined topics and seed words. To investigate the impact of these factors, we conduct experiments to evaluate topic diversity of our predefined 5 topics, varying the number of topics from 2 to 5. For each value of k, we conduct $5 \atopwithdelims ()k$ experiments.

In each experiment, k seed sets are selected from a pool of predefined 5 seed sets used in Table 5. We evaluate topic diversity in each experiment using Jaccard similarity. For two word sets $T_i$ and $T_j$, Jaccard similarity is defined as:

$$\begin{aligned} JS(T_i,T_j) = \frac{|T_i \cap T_j|}{|T_i \cup T_j|} \end{aligned}$$

For $1 \le i < j \le k$, where k is the number of topics, topic diversity is calculated by averaging pairwise Jaccard similarities between topics:

$$\begin{aligned} \text {Topic Diversity} = \frac{2}{k(k-1)} \sum _{i<j} JS(T_i,T_j) \end{aligned}$$

Topic diversity measures the extent to which the topics captured by captions and image labels represent diverse aspects of the data and quantifies the distinctiveness between topics.

The value of topic diversity for k’s 2–5 are illustrated in Fig. 7. For captions, employing larger topic numbers results in higher overlap between the topics (left panel). This could be attributed to the common terms regularly used by users, causing them to frequently appear in various topics. As an example, the term “cosmetic” often appear in both event and sponsorship posts. Image labels, by contrast, consistently exhibit a lower overlap, indicating that they encompass a broader range of information embedded within the data (right panel).

Fig. 7

Topic diversity of our seed sets. The numbers in parentheses denote the count of overlapping points in the value of topic diversity. The red line connects the median values for all possible combinations of specific topic numbers

1.3 Sensitivity of seed selection

To explore the sensitivity of our results to the chosen seed word sets, we construct an alternative seed word sets using LDA for comparison, which we refer to as ‘contrastive set’. We first performed LDA with five topics. Then, for each topic, we identify the words with the highest information gain. The information gain of a word w to a topic T is calculated as:

$$\begin{aligned} IG(T,w) = H(T) - H(T|w), \end{aligned}$$

where H(T) represents the entropy of the topic, and H(T|w) represents the conditional entropy of the topic given the presence of word w. We then select the words with the highest information gain for each topic as the contrastive seed words, removing the words that overlapped across topics. To maintain consistency with the original seed set size, we chose 20 and 6 final seed words per topic for captions and image labels, respectively. The selected constrastive seed word sets are exhibited in Table 6.

Table 6 The list of seed words selected by Information gain

To evaluate the effectiveness of seed word sets, we assess the coherence of topics generated using these sets. We employ the average Normalized Pointwise Mutual Information (NPMI) as the evaluation measure. The average NPMI of a topic is evaluated across all representative word pairs within the given topic:

$$\begin{aligned} \text {NPMI}(T) = \frac{2}{N(N-1)} \sum _{i=1}^{N-1} \sum _{j=i+1}^{N} \frac{\log _2 \frac{P(r_i(T),r_j(T))}{P(r_i(T),r_j(T) )P(r_j (T))}}{-\log _2 P(r_i (T),r_j (T))}. \end{aligned}$$

where N, $r_i (T)$ for $i=1, \cdots , N$, and P denote the number of representative words, the $i-$th representative word of topic T, and the frequency of the input arguments appearing across the documents. The representative words $r_i(T)$ of a topic T are selected based on their high relevance as proposed by Sievert and Shirley (2014). The relevance $r(\cdot | \ \cdot )$ is defined as follows:

$$\begin{aligned} r(w|T) = \lambda \cdot P(w|T) + (1-\lambda ) \cdot \frac{P(w|T)}{P(w)}. \end{aligned}$$

Fig. 8 displays the average NPMI score of each topic achieved by the proposed seed word sets and the contrastive seed word sets. This analysis reveals a similar trend for both our proposed seed word sets and the contrastive sets. This indicates that the topic coherence, as measured by NPMI score, exhibits relatively low sensitivity to the specific choice of seed words in this dataset.

Fig. 8

Topic coherence of two seed sets

Method settings

1.1 Linear mixed model

In our analysis, the covariates that yield singular results when modeled in the simple Linear Mixed Model (Linear Mixed Model with only one covariate), are set to have fixed effects only. These covariates include ‘Period’, ‘Weekdays’, ‘Hour’, ‘Season’, ‘Topic (event)’, ‘G (green)’, and ‘BG (blue green)’. In addition to these seven covariates, the ‘Time difference’ is also modeled to have only a fixed effect owing to computational issues. The remaining variables are modeled to have both random and fixed effects.

1.2 Support vector regression

In our analysis, kernel Support Vector Regression is used to account for structures of covariates in a more flexible way. Specifically, the original p-dimensional covariates are mapped to infinite dimensional space by the function $\phi : \mathbb {R}^{p} \rightarrow \mathbb {R}^{\infty }$ where

$$\begin{aligned} \langle \phi (\textbf{x}_1), \phi (\textbf{x}_2) \rangle = \exp \left( -\gamma \vert \vert \textbf{x}_1 - \textbf{x}_2 \vert \vert _2^2\right) , \end{aligned}$$

(1)

and $\phi (\textbf{x})$s are used as infinite- dimensional covariates replacing the p-dimensional covariates $\textbf{x}$. The prediction of y given the covariate vector $\textbf{x}$ becomes as follows:

$$\begin{aligned} \hat{y}&= \langle \hat{\beta }_1, \phi (\textbf{x}) \rangle + \hat{\beta }_0, \nonumber \\ (\hat{\beta }_0, \hat{\beta }_{1})&= \underset{(\beta _0, \beta _{1}) \in \mathbb {R} \times \mathbb {R}^{\infty } }{\arg \min } \frac{1}{2} ||\beta _{1}||_2^2 + C \sum _{i} (\xi _{i} + \xi _{i}^{*}) \nonumber \\ \text {subject to} \;&\left( \langle \beta _1, \phi (\textbf{x}_{i}) \rangle + \beta _0\right) - y_{i} \le \epsilon + \xi _{i}, \nonumber \\&y_{i}-\left( \langle \beta _1, \phi ( \textbf{x}_{i} ) \rangle + \beta _0\right) \le \epsilon + \xi _{i}^{*}, \nonumber \\&\text {and} \quad \xi _{i}, \ \xi _{i}^{*} \ge 0. \end{aligned}$$

(2)

1.3 Parameters settings

SVR involves three tuning parameters: $\epsilon$, C in (2), and $\gamma$ in (1). $\epsilon$ is the allowed range of residuals around the regression line, C is the regularization parameter of the residuals that exceed the allowed range $\epsilon$, and $\gamma _{\text {svr}}$ controls the smoothness of the radial kernel.

RF and XGB share several common tuning parameters related to their tree structures, namely, the total number of trees (B), maximum depth of a tree (d), fraction of covariates to be used for each split of a tree (v), fraction of subsamples to be used in building each tree (s), and minimum number of observations that each terminal node should contain (m).

XGB has additional tuning parameters $\rho$ and $\gamma _{\text {XGB}}$ that control overfitting: $\rho$ is the learning rate of gradient boosting, and $\gamma _{\text {XGB}}$ is the minimum loss reduction required for a tree split. Here, $\gamma _{\text {XGB}}$ plays the same role as $\gamma$ described in 4.5 by penalizing a large tree. Another regularization parameter, $\lambda$ in 4.5 is set to 1 by default.

MLP involves two parameters for the hidden layer: hidden layer size h and activation function. We employ Rectified Linear Unit (ReLU) as the activation function for $h=5$. For optimization, MLP selects the optimizer and the learning rate $\rho$. We opt for the Adam optimizer, initializing the learning rate at $\rho =0.001$ and adaptively adjusting it during training. We use a batch size of full data to ensure a fair comparison with other methods in terms of cross validation and train the model for 500 epochs.

The R packages ‘e1071’, ‘ranger’, and ‘XGBoost’ are used for SVR, RF, and XGB, respectively. The python package ‘keras’ are used for MLP. The tuning parameter settings are presented in Table 7.

Table 7 Tuning parameter settings

Fig. 9

Covariate Importance Plot. Mean SHAP values are displayed for each covariate used in the XGB results, except for user dummies

Fig. 10

Covariate Summary Plot. For each covariate, the corresponding SHAP values are visualized, except for user dummies and ‘Time Difference’ for visibility

Fig. 11

Dependence Plot for the top 16 covariates

TreeSHAP results

Figure 9 displays the mean absolute SHAP values of all covariates in the XGB result. Figure 10 shows the observation-wise SHAP values, in which the covariate value of each observation is represented by a color gradient. The closer the color is to dark purple, the greater the value of each covariate.

Figure 11 shows the SHAP dependence plot for the top 16 covariates in order of SHAP values. This plot shows a scatterplot of the SHAP values against the covariate values, along with the trend line for all observations.

The top left corner presents the plot for ‘Time Difference’. Since we considered the latest 100 posts for a given user in the data collection process, most observations are posts uploaded within a year. The trend line of the plot indicates that the older the post, the greater the decline in the importance of the covariate for prediction of “Likes” until around the first 100 days. This suggests that people may not react to old posts because they are mainly interested in new posts on social media. Therefore, considering such characteristics of social media, where new posts are constantly uploaded, it can be said that it is important to use ‘Time Difference’ information.

Another time lapse variable, ‘Period’, was also important, displayed in the rightmost corner. Most users in our dataset upload new posts within 250 h (approximately 10 days) of uploading the previous post. For ‘Period’ plot, when the upload cycle of posts is short, the influence on “Likes” is low, but as the posting cycle gradually increases, the importance tends to increase. That is, when uploading a post, it may be desirable for the user to upload the post with a certain time interval rather than a cycle that is too short for higher popularity.

Another variable, ‘N. of Image’ refers to the number of images uploaded in a single post. There are various distributions of ‘N. of Image’, starting from 0 (when only reels are posted). An inverse U-shaped relationship is observed, which suggests that both too few images and too many images in a single post are not desirable, and that there may exist an optimal number of images per post.

Among Image label topics, the covariate importance was positively higher when there were ‘Body’ and ‘Fashion’ topics in the post than when there were no such topics, but it was negatively higher when ‘Beauty’ topic was included in the post.

Comparison of covariate construction approaches

In this section, we present a comparative analysis on covariate construction methods using ‘User’ and ‘Time difference’ as baseline covariates. Based on the previous section’s findings, XGBoost is chosen as the prediction model due to its effectiveness in describing image-based social media data. We evaluate the performance of our proposed covariate construction approaches alongside other methods in image covariate (image and representative color), and non-image covariate (caption) construction, respectively. More precisely, we focus on contrasting with neural network-based methods, which is known to display strong performance but often challenging to interpret. Previous studies have utilized sequential steps of multimodal fusion processes, employing neural networks (NNs), to obtain unique overall vector representation for captions or images (e.g., Wang et al. (2023)). The resulting vector representation encapsulates fused meanings from various multimodal sources, including text from post captions, image captions, and additional features. However, this compromises the interpretability of the resulting vector representation. The comparison results are exhibited in Table 8.

1.1 Image covariate

Table 8 Test errors for different variable construction methods

For image covariate, we employ two comparison methods to the proposed Google Vision API-based approach, each based on deep convolutional neural networks (DCNNs) and Bootstrapping Language-Image Pre-training (BLIP), referred to as ‘Image class’, and ‘Image caption topic’, respectively, with our proposed approach denoted by ‘Image label topic’. Convolutional neural network-based models demonstrate outstanding performance in image processing. Thus, we employ comparison methods leveraging DCNNs and utilized their output image categories, following approaches outlined in works such as Huang et al. (2017), Hessel et al. (2017), Chen et al. (2016), and Overgoor et al. (2017). Specifically, we employ ResNet-50 architecture (He et al., 2016) on an image and extract the resulting category with the highest probability. This process generates a binary vector of 700 dimensions, in which each element is an indicator of its associated image category. In the resulting vector, the element corresponding to an image category is encoded as 1, if the given image belongs to that category. We then sum the binary vectors of images per post to form the post-level image covariate. Given that our approach involves extracting word labels from a given image, we also include performance comparison with a method that initially generates a caption from an image, followed by applying LDA on the caption, as proposed in Zhang et al. (2018). For generating image caption, we utilized BLIP (Li et al., 2022) which is a convolutional neural network-based (CNN) approach.

For color covariate, we present a comparison of the prediction performance between Hue-Saturation-Value (HSV) color features, as used in Trzciński and Rokita (2017), and the Munsell space employed in this study, referred to as ‘HSV space’ and ‘Munsell space’, respectively. To construct representative colors based on HSV color space, which is a color space commonly used in image analysis tasks, we first convert the RGB values of the image to the HSV. Subsequently, the converted HSV values are assigned to one of eight discrete classes: black/white, blue, cyan, green, yellow, orange, red, and magenta. Each color category is then summarized at the post-level, in the same manner as detailed in Sect. 3.2.2.

Table 8 provides the test errors of various feature extraction methods. Among the compared image content processing methods, the proposed ‘Image label topic’ exhibits the lowest error for both RMSE and MAE. Furthermore, ‘Image class’ exhibits lower error compared to ‘Image caption topic’. The results tend to improve as the image content is more directly utilized. For instance, ‘Image label topic’ utilizes more direct information than ‘Image class’. While ‘Image class’ assigns a single category to an image, which can lead to an overly simplified representation, ‘Image label topic’ methods utilize multiple labels extracted from a given image, resulting in a more direct and informative representation. In addition, unlike ‘Image class’, ‘Image caption topic’ requires an additional step of converting the image into text. This indirect approach may be less suitable for social media platforms, where images are often used to convey a general impression rather than provide detailed information. In such contexts, the textual description of an image generated by ‘Image caption topic’ might introduce irrelevant details that obscure the core message of the image, acting as noise in covariate construction. These findings suggest that image posts on social media platforms carry information that can be represented solely by the image itself. Between two image color processing methods, ‘HSV space’ and ‘Munsell space’ show close results. The inherent similarity between HSV and Munsell spaces may contribute to the observed indifference in the outcome with respect to the chosen space.

1.2 Non-image covariate

For caption covariate, we compare the caption processing performance between the proposed Seeded-LDA approach, pre-trained neural network based model (BERT), and sentiment analysis algorithm (VADER), which are referred to as ‘Caption topic’, ‘Caption deep feature’, and ‘Sentiment score’ in Table 8. In our comparison, BERT is utilized to explore an attention-based method. Specifically, the caption covariates are extracted leveraging BERT, as employed in Ding et al. (2019). We additionally compare caption features extracted through sentiment analysis using the VADER algorithm (Hutto & Gilbert, 2014), as employed prior research in popularity prediction (e.g., Keneshloo et al. (2016), Saeed et al. (2022)). The VADER outputs sentiment scores for 4 emotional categories–positive, negative, neutral, and compound–with the scores ranging between 0 and 1 where higher values represents stronger emotional intensity.

Among the caption processing methods, ‘Caption topic’ achieves best results than both ‘Caption deep feature’ and ‘Sentiment score’. While both caption topic and caption deep feature represent textual content, caption deep feature exhibits a considerably higher dimensionality (768) compared to caption topic (5). This high dimensionality may lead to performance degradation. Furthermore, ‘Sentiment score’ showed limited results, which might stem from the inherent insufficiency of sentiment analysis. Captions often convey richer information than sentiment alone.