Twitter permeability to financial events: an experiment towards a model for sensing irregularities

Progressive usages of technology in the stock markets have led to a continuous growth in their business. Businesses and individual investors alike can take decisions and invest within minutes if not in seconds [24]. Helping both businesses and individual investors harvest information from diverse sources such as the company itself, through their website or Regulatory News Service (RNS), news agencies, brokers, stock market and individual investors, is a difficult and time consuming task. At the beginning of the emergence of social media, stock discussion forums pioneered an alternative source of information for investors, specially retailers, supplementing the traditional news media. A plethora of academic works study the capability of these highly-specialized social channels to predict returns and to detect abnormal behaviours in the stock market [3, 9, 15, 47]. From the investor’s perspective, success highly depends on the quality and the rapidness of the information to support the decision-making. As online social media invaded the habits of people, also companies, brokers and other key roles in the financial market began to share more and more factual information and informed opinion in social media. The task of a financial analysts becomes progressively more challenging if we consider that not only have they to take decisions in seconds, but also deal with an increasing number of data sources comprising of a continuous range of different features and quality. Although social media may be a vehicle to fight against asymmetric information in the financial market [5], the volume, velocity and variety of its data make investing a heroic task, especially for individual investors.

Given today co-existence of social media and traditional news media (newspaper or online news media), the authors in [25] compare the performance of social media and news media in predicting returns in the Australian market. It is clear that social media outperforms traditional media on the capability for readers to interact or further spread information. Although the study shows a significant effect of sentiments on one online discussion forum focused on the Australian financial market, it found that the effect of new media was not significant. The most interesting finding was that sentiments on online investor forums do not seem to mirror the ones from news media.There is in fact, an ongoing debate about to what extent social media analytics would be the new predictor for the future returns of stock-exchange traded financial assets. We may even wonder if social media can lead the direction of the financial market at some point in the future as some authors suggested [59].

Although organizations have always used various corporate disclosure channels to communicate directly and indirectly with investors, they are now relying on social media to provide up-to-date information to investors. As the heart of publicly accessible Social Media, Twitter has become a vital source for open source social intelligence about news, natural disasters, terrorist attacks, public health, politics, etc. Also, Twitter is one of the most currently used platforms to share financial information from companies, brokers, news agencies or individual investors. Some recent works in the literature study this changes in reporting financial news [32, 56, 57]. As Twitter usage in this context is definitively increasing; it is important to stress that, according to [50], stock microblogs exhibit three distinct characteristics about stock message boards: (1) Twitter’s public timeline may capture the natural market conversation more accurately and reflect up to date developments; (2) Twitter reflects a more ticker-like live conversation, which allows micro-bloggers to be exposed to the most recent information of all stocks and does not require users to actively enter the forum for a particular stock; and (3) micro-bloggers have a strong incentive to publish valuable information to maintain reputation (increase mentions, the rate of retweets, and their followership), meanwhile financial bloggers can be indifferent to their reputation in the forum. Providing sensing, harvesting and analysing methods and tools of such information could be very useful for many stakeholders such as businesses and individuals making decisions to invest, stock market analysts and law enforcement agencies.

This paper analyses the permeability of Twittersphere, the total universe of Twitter users and their habits, to a relevant event for the financial market. Our hypothesis is that Twitter (although not a specific financial forum) is permeable to financial events and this permeability can be analysed by monitoring some specific features related to companies. Although several works in the literature have studied the correlation between Twitter and financial movements (to exploit its predictive power), our objective does not address regularities of the market in a longitudinal study that should take into consideration a long time series. On the contrary, we look for assessing the level of permeability of Twitter to financial events. For that, we study a remarkable event in January 2017: the announcement about the merge of Tesco PLC (hereinafter Tesco) and Booker Group PLC (hereinafter Booker) to create the UK’s Leading Food Business on the 27th January 2017. Both companies, Tesco and Booker are listed in the main market of LSE (London Stock Exchange). The findings in the variation of Twitter volume and content show that the use of general purpose social media data is permeable to financial-specific events. The work presented is the first step in considering Twitter as a relevant feeder for taking decisions regarding the financial market and detecting irregularities in that market. An initial analysis of this experiment in [21] reported important insights when considering Twitter volume. This paper advances in the analysis of the Twitter permeability at that event by considering also the content of posts.

This paper is structured as follows. Section 2 introduces the related work the usage of digital information sources to assist decision-taking in the financial market, and Twitter in particular (Section 2.1). From this review of related works, Section 2.2 also highlights the contribution in this paper. In Section 3, we describe Twitter efforts to accommodate financial information in a general-purpose microblogging platform and we propose a model of the position of Twitter in the financial universe as well as the path flows of information. Also, we introduce the experiment and two specific research questions around our hypothesis. The experiment methodology (Section 4) takes advantage of the extraction characteristics of Twitter APIs to construct a specific dataset for the merger, which is described in Section 4.1 along with features related to the rapidness and synchronisation of Twitter with the LSE regarding the announcement of the financial event and the stock share prices (Section 4.2). Then, a series of investigations are conducted over the extracted dataset to address the research questions and the impact is analysed according to its disturbance in terms of Twitter volume (Section 5); in terms of hashtag dynamics and topic modelling (Section 6 and Section 7); in terms of sentiment towards the actions (Section 8); and in terms of geographical distribution of the posting (Section 9). Finally, Section 10 discusses our findings and their limitations, meanwhile Section 11 summarises the main conclusions and introduces our ongoing work in the study of permeability of Twitter to financial events.

The modernisation and digitalisation of the financial market has been accompanied with the enormous increasing of information available for traders and investors, which can make extremely difficult for one person or a small group to conduct an analysis of a financial asset. In the application of computational intelligence to the financial market, mining textual content (financial news, financial reports , and even information in micro-blogs) is considered a relevant source of information for predicting future market behaviour [10]. To do so, researches have proposed the extraction of relevant features from the textual content. From the very beginning, sentiment was considered one of these relevant features to improve the accuracy of financial time series forecasting [54]. More recently, [27] proposes considering professional opinions in social media and financial news as supplementary sentiment sources, along with firm characteristics, so that, a methodology for joint effects of sentiment is proposed.

Cavalcante et al. [10] introduced a comprehensive review on the usage of computational intelligence in the financial market. The authors also suggest a systematic proposal for the building of intelligent trading systems. To sum up, most of the research in this field address the predictive power of social media problem and the differences appear in information sources, their extracted features, and also the predictive methods and the values to be predicted. [7] have shown that trading volumes of stocks traded in NASDAQ-100 are correlated with their query volumes (i.e., the number of users requests submitted to search engines on the Internet). Gunduz and Cataltepe [22] proposed a forecasting method which combines the analysis of news articles from Turkish finance websites, the extraction of feature vectors and stock prices to predict future market movements. The experiment used a naive Bayes algorithm to construct a prediction model which integrate the feature vectors and stock prices. Also, [35] used text mining of financial news-headlines to predict movements in the FOREX market; the extracted features include both semantics and sentiment in the content. Deep learning has been also applied to model both short-term and long-term influences of events on stock price movements in [17]. Finally, The combination of public news with the browsing activity of the users of Yahoo! Finance to forecast intra-day and daily price changes of a set of 100 highly capitalised US stocks was explored in [43]. The work showed that, when taken alone, sentiment analysis or browsing activity have very small or no predictive power and uncovered a “wisdom-of-the-crowd” effect that allows to exploit users’ activity to identify and weigh properly the relevant and surprising news.

But, Twitter is neither the only nor the biggest source of information about financial markets and, in this respect, Figure 1 shows a reference model for the Twitter position regarding this universe of information and its relationships within the stock market. At any moment (time) and from anywhere (location), a variety of contributors in the financial sector (verified and unverified Twitter accounts) may post (tweet or retweet) pieces of information referring to a company by using either (1) the unique symbol $cashtag, (2) #hashtags related with the company for a specific purpose or related to a specific event and /or (3) simply the company name in its different forms (official, abbreviation, colloquial, etc.). At the same time that main agents in the financial market post into Twitter, the Twitter content maintains references to the financial universe outside via mentions, URLs, etc. Given apart the flows which characterise the permeability of the layer in between Twitter and the financial market, the information naturally spreads throughout the Twittersphere by the common re-tweeting and following mechanism in the platform. Even if the piece of news (post) comes from a human spread rumour or some cyber-attack which ends up being false information, [26] sustains about “the power of networked social and financial systems to connect autonomously and to produce a present without human oversight or governance”. In our opinion, further actions and research are needed to achieve a safer financial ecosystem for traders and investors.

In this section, we describe the experimental methodology which will be used to answer our main hypotheses: Twitter (although not a specific financial forum) is permeable to financial events and this permeability can be analysed by monitoring some specific features related to companies. To address this high-level research question, we selected a random but relevant even, according to its impact on the financials spheres but also on the general public, at the beginning of year 2017. The most important feature for the selection of the event is the latter, that is, an event which being intrinsically financial, is of common interest for the general audience so that permeability can be measured beyond the spheres of financial experts in Twitter.

The event under study, is the merge of Tesco and Booker on the 27^th January 2017, announced by the RNS (Regulatory News Service) of LSE (London Stock Exchange) at 7:00 a.m. GMT (Greenwich Mean Time). This action is modelled according to the reference model in Fig. 1, that is, we model Tesco on Twitter with the conceptual triplet (cashtag, hashtags, keyword), that is ($TSCO, #[Tesco], “Tesco”). This model represents the financial perspective of Tesco on Twitter ($TSCO), specific comments about some Tesco issue on Twitter (#[Tesco]: #hashtag in some tweet with the keyword “Tesco”), and general references to Tesco on Twitter (“Tesco”). As mentioned, our objective is a monitoring platform that continuously harvests signs scattered all over social media platforms to identify irregularities, such a platform will be supported by a data fusion model which extract relevant features and weight the contributions from different online information sources. Although in such an approach not only Tesco but Booker would be continuously monitored, the experiment faces the isolated behaviour of Tesco (one of the parts in the action) due to significance reasons, as the part with more presence in social media and due to design reasons, as the company being the focus of real-time irregularity monitoring.

Open image in new window

Regarding permeability to financial events, we may hypothesise that the permeability and the impact is not alike in the three perspectives which constitute the triplet. The cashtag is invariably linked to financial news of a company but hashtags have a completely different dynamic. The hashtags related to a specific company will emerge and disappear dynamically according to the company decisions, marketing campaigns, consumer behaviour, etc. Tesco being a well-known company in the UK, the impact of financial events in hashtags might be limited and just visible in case of general-public financial events. Presumably, financial events should have a bigger impact on cashtag tweets (according to volume and context) than on hashtags (just altering their dynamics) or on topics of Tweets content. Nevertheless, this presumably different behaviour should be inspected.

Given our hypothesis: “Twitter (although not a specific financial forum) is permeable to financial events and this permeability can be analysed by monitoring (1) the name of companies as a keyword (“Tesco” in this case), (2) the Cashtag of the company ($TSCO) and (3) the hashtags related to that company.”

On the basis of the proposed RQs, and even though we are reporting a single case study, we would like to highlight that our triplet approach can be applied to the general monitoring of financial event on Twitter disregarding the market and the sector if we consider announcement services and stock prices in other regulated financial markets.

There are three different ways to obtain Twitter data: Search API, Streaming API and Firehose. The Twitter Search API provides the endpoints to recover tweets that were published in the previous two weeks, with the possibility of filtering according to several criteria. On the other hand, Twitter Streaming API returns 1% of the tweets that match some search parameters in real time. Finally, Twitter Firehose provide access to the 100% of the tweets, but it is not a free-access API. Twitter APIs are constructed around four main “objects”: Tweets, Users, Entities (hashtags, URLs, mentions and media in a tweet) and Places. From theses object and taking time as the Twitter’s backbone, we consider a Twitter model which contemplates 3 orthogonal perspectives: The content, the social structure and their spatio-temporal context (see Fig. 2).

Open image in new window

There are some differences between the Searching API and the Streaming API illustrated in Fig. 2. One difference being the time direction - the most relevant one to our experiment. The Search API goes back in time, whilst the streaming API goes forward. Moreover, there are other differences related to mainly the format and the rate limit rules. As it is shown in Fig. 2, the search and Streaming API does not return data in exactly the same format but the differences in format are irrelevant for data analysis where preprocessing can define a uniform intermediary format. Regarding their extracting capacity, forums contain plenty of discussion about this issue which has not ever made enough clear from Twitter officially.

4.1 The dataset

The extraction strategy firstly used the Twitter Search API to recover the information backwards before the announcement on 27th of January 7:00 a.m. GMT and the streaming API was used to recover information forwards until 27th February (one month later). This data streaming collection, just after the announcement, was used to visualise the impact of the announcement and the time the behaviour of tweets concerning Tesco resumes a regular pattern again. The results of the combination of the search and streaming results is shown in Fig. 3.

Open image in new window

Once the behaviour becomes regular, we used the Search API again to obtain a regular dataset to compare with the Search API results recovered just after the merge action was announced. Clearly, the Twitter Search API is not appropriate for continuous analytical monitoring and as a data source to allow real-time decision making. It is not intended and does not fully support the repeated constant searches that would be required to deliver 100% coverage. However, the experiment in this paper is limited to one individual company, 2 keywords and timelines in the scale of weeks. Therefore, the Search API provides a better coverage than the Streaming API (1% according to the Twitter official information), if the superior filtering characteristics of the Search API are used. Nevertheless, as the Search API has a limit on the number of tweets recovered, to get the whole data during the period under study (see Table 1), we repeatedly asked Twitter for the most recent results backwards by windowing the searches according to the publication date and merging results according to the post Id. In this way, we guarantee a fair comparison according to the volume of data since, in any manner, we should compare the Search API with Streaming API results. According to that, and to give response to the research questions, we use the Search API queries to cover the time periods defined in Table 1. Mainly, the 4 periods in this table capture the Twitter volume retrieved by the Search API during the very same days of the week, Wednesday to Sunday, around the announcement on Friday 27^th of the merge (pre-announcement and post-announcement periods) and when behaviour becomes regular at the corresponding week days. Two types of Search API queries were considered, (1) query with the term ‘tesco’ for the Tweet:entities and the Tweet:text entities, to capture post related with Tesco and post also containing a hashtag related with Tesco (a total of 70,793 tweets) and (2) query with the term ‘$TSCO’ for Tweet:entities (a total of 151 tweets). It is fair to mention that ‘tesco’ volume is several orders of magnitude higher than ‘$TSCO’ given the high visibility of the company Tesco in social media as a vehicle for marketing and consumer engagement.

Table 1

Time Periods (GEM Time) in the experiment (extracted with Twitter Search API)

Name/Period	‘tesco’		$TSCO
	incl. #[tesco]
	Total	Per/hour	Total	Per/hour
Pre-announcement
25th Jan 00:00- 27th Jan 07:00	11,817	214.85	12	0.218
Post-announcement
27th Jan 07:00 - 29th Jan 23:59	25,547	393.03	91	1.4
Regular 2-weeks-after
8th Feb 00:00-10th Feb 07:00	13,417	243.94	26	0.473
Regular 2-weeks-after
10th Feb 07:00 - 12th Feb 23:59	20,012	307.88	22	0.338

4.2 Rapidness & synchronization in the dataset

Although our analysis focuses on the permeability of Twitter to financial events, our objective, and part of our future work, is the use of Twitter as a sensor of irregularities in the stock market so rapidness and synchronisation of Twitter as a channel to the stock market: rapidness in its response to the RNSs of LSE (London Stock Exchange) and synchronisation with the share prices also in LSE are relevant. Regarding the rapidness, the experiment definitively shows the good characteristics of Twitter. The first tweet referring to the RNS was at 7:03 a.m. on 27th, just 3 minutes before the RNS announcement about the Tesco and Booker merge (Fig. 4). Beyond the very first tweet, it is remarkable the rapidness of the peak response to the announcement in both datasets, so that the 27th Twitter time series (‘tesco’ and $TSCO) can be considered abnormal time series when a regular Friday is taken as a reference. We highlight that the peak starts from 7:00 to 8:00 both in the #TSCO and $TSCO dataset (see Fig. 5).

Open image in new window

Open image in new window

Regarding the synchronisation with the share prices at LSE (Fig. 6 and Table 2), it is fair to mention that although the share prices were abnormally low the day before the announcement, we haven’t found any reference to the financial vocabulary considered for the action during this period. Moreover, we cannot extract a sound finding given to the concatenation of two events with financial impact in Tesco PLC: the legal actions against Tesco PLC overstatement (see Section 7) and the merge of Tesco and Booker. In terms of rapidness and synchronisation, we appreciate that the Twitter permeable layer exhibits rapidness in response to the financial market but we barely appreciate a clear synchronisation with the stock share prices.

Open image in new window

Table 2

Values of share prices and Twitter volume

Period	Price closing time	‘tesco’	$TSCO	Total
24th 16:30 to 25th 16:30	188.45	7,697	6	7,703
25th 16:30 to 26th 16:30	188.5	8,356	4	8,360
26th 16:30 to 27th 16:30	206.55	15,295	38	15,333
28th 16:30 to 29th 16:30	—	10,077	8	10,085
29th 16:30 to 30th 16:30	—	4,539	3	4,542

In this section, we detail the impact of the event by analysing the variation in the number of tweets (volume) so that a quantitative measure of Twitter permeability to a financial event can be observed. During this part of the analysis some irregularities were discovered which related to an inconsistency in the named scheme of tickers in Twitter. In particular, to our knowledge, Twitter has not promoted the specific distinction among financial markets so that the uniqueness of ticker symbols inside a market disappear in the Twittersphere. That is the case of $TSCO cashtag which corresponds to Tesco PLC in the LSE and to Tractor Supply Company in the NASDAQ, the second stock exchange in USA. So, the returned results to a $TSCO query included tweets related to Tesco PCL and also to Tractor Supply Company. If the cashtags is the entity to aggregate information around a specific company and, consequently, to allow the spreading of such information, some kind of market prefix should be used, especially in the times when companies are increasingly global.

Figure 7 shows the temporal series in tweets per hour (TPH) scale. Although it is quite obvious that the number of TPH in the ‘tesco’ dataset is up several orders of magnitude higher than those of the $TSCO dataset, the peak behaviour is more acute in the $TSCO one. As it is shown in Table 3, considers the hourly volume of the ‘tesco’ dataset on the 27th January. There are no outliers during the day, with the peak value of 2,057 tweets occurring in the sample from 8:00 to 9:00. Nevertheless, there are 3 outliers in the $TSCO dataset: samples 8:00-9:00, 9:00-10:00, corresponding to the time just after the announcement and 12:00-13:00, being consistent with previous studies about social timing in [34], which showed peak activity during lunch time in different cities around the world.

Open image in new window

Table 3

Peak behaviour on the 27th January for ‘tesco’ and $TSCO

Open image in new window

Apart from the peak comparison, we inspect the disturbance on other dataset features before and after the event, also comparing these dates with the regular behaviour 2 weeks later. In Table 4, we compare the behaviour of the main features of ‘tesco’ data (in green) and $TSCO’ data (in blue).

Table 4

Variability of features in ‘tesco’ and $TSCO data before and after the announcement in terms of regular behaviour

Open image in new window

Firstly, we can observe the increase of Tweets per hour during the post-announcement, compared to the regular period, is more acute in the $TSCO data (1.40 vs 0.34) than in the ‘tesco’ data (393,03 vs 307.86).Secondly, the percentage of tweets which contain a URL are significantly higher in the $TSCO data (from 75% to 82.61% in the different periods) with respect to the ‘tesco’ one (from 29.51% to 48.41%), which is a result of the professional and financial orientation of the $TSCO data as a channel to spread facts and news rather than opinions and sentiments. Finally, the retweeting activity is higher in the announcement periods (pre- and post- with 43.85% and 31.59%) compared to the regular periods (38.69% and 28.36%) for the ‘tesco’ data. The increase of retweeting is, by nature, linked to the need or desire of spreading a piece of content but, the reason behind may be different as, in fact, it is in our case study: retweeting the ‘tesco’ keyword is mainly related with a Tesco campaign involving retweeting (see section VI for the details) meanwhile retweeting in $TSCO data is mainly linked to spreading the information about the merge (post-announcement) and about other financial news related with Tesco PLC. The information about this retweeting activity is expanded upon in the following sections of this paper. Finally, we observe the invariability on the number of verified users either along all the periods and along the ‘tesco’ and $TSCO data.

Both the peak behaviour (Table 3) and the feature comparison (Table 4) provide an answer to RQ1, that is, the event impact on $cashtag-content is greater than on #hashtag-content or on general content.

Given the volume of Tweets, the spreading of the event throughput the Twittersphere should presumably have two effects: the change on the hashtag dynamics and the change on the main topics in tweets. More than this effect, what we would like to measure is to what extent this disturbance can be perceived differently in ‘tesco’ and $TSCO data. Presumably, and contrarily to the volume analysis, the content impact is expected to be more marked or noticeable in the ‘tesco’ data. So that, financial topics, which are the core of cash-tagged tweets, would conquer the whole Tesco content as a result of the good permeability to this financial event in Twitter, should be studied.

Focusing on the analysis on hashtags as a measure of ephemeral or nor so ephemeral interest, Table 5 shows the disturbance of the event in hashtag dynamics in the ‘tesco’ data, which is considered data related with the company without any specific financial bias. In terms of the 5 most frequent hashtags during the periods considered in the experiment, we clearly perceive that, only during the period Post-Announcement, financial hashtags emerge: #teschoshareprice and #booker. We consider #booker a hashtag related with the action given its co-occurrence with the keyword ‘tesco’, the reverse may be not true. Also, we discover a persistent hashtag during all the periods: #essothursdays, related with the equally named promotion campaign which allows Twitter and Facebook users to obtain 100 Tesco voucher provided that they follow the Twitter account @GB_Esso and retweet the competition tweet.

Table 5

Disturbance on Hashtag Dynamics for the ‘tesco’ dataset

Open image in new window

Regarding hashtag disturbance at a more general level, the bottom part of Table 5 shows the percentage of occurrence of the main hashtag of the company (#tesco), hashtags related with main business of the company (#-main business), and hashtags related with merge and financial aspects (#-action and financial). Also, Figure 8 shows hashtags in terms of semantic families. The left chart shows the main topics reported in the ‘tesco’ data and the right chart distinguishes the three main categories within these topics: “Tesco” in grey, “Main Business” in yellow and “Action and Financial news” in green). It is clear that, from the point of view of the hashtags dynamics, the merge reporting achieves a high position in the ‘tesco’ dataset after the announcement. During the pre-announcement period, hashtags (apart for the generic one #tesco in grey) are related to marketing campaigns (yellow in the aggregated chart on the right). It is only during the post-announcement period that financial-related hashtags emerge: #tescoshareprice being 40.80% and #booker being the 11.16% of the top hashtags. Also, it is remarkable that the emergence of the hashtags related to the breaking news (#bbcbreaking and #nieuwstwitter) and that even #tesco disappears in the top-5 hashtags during the post-announcement period (Table 5). All of financial-related and breaking news hashtags completely disappear in the period we consider as regular in terms of Tesco experiment (see also Table 5).

Open image in new window

Lastly, regarding the analysis of hashtags in the $TSCO dataset, it is not strange than during the regular period none of the tweets in the dataset contains a hashtag, so no extra specific information to follow and spread information is provided, apart from the pseudo-hashtag $TSCO. However, during the post-announcement period 15,38% of the tweets in the $TSCO dataset contains at least one hashtag. Nevertheless, the number of tweets during this period is 91 of which 16 include at least one hashtag. Although we prefer not to analyse the hashtag dynamics with this small number, we can appreciate a disturbance with regards to the regular period. This result talks about the malfunction of #hashtags in the $TSCO dataset. If #hashtags are the Twitter tool to aggregate semantically-related content and spread it, this feature is almost absent in the $TSCO dataset. We cannot perceive a hashtag disturbance related to the action, because in the financial island of Twitter the $cashtag is the main Twitter tool to aggregate and spread content. What we can only appreciate with respect to the regular behaviour is the increase on the number of hashtags which accompany the pseudo-cashtag $TSCO.

According to these results we can establish the fulfilment of RQ1, in that Twitter permeability to the merge can be perceived not only on $cashtag-content, which has a clear financial orientation, but also in the #hashtag dynamics.

When analysing the impact of the action on the Twitter content, the complete response to RQ1 (can a financial load be also perceived in the #hashtag dynamics as in the general content?) should be studied also from the point of view of the main topics people post about Tesco. For that, we should extract these topics from the tweets content in ‘tesco’ data and compare them with the ones in the specifically financial $TSCO data. Unfortunately, the number of tweets in the $TSCO dataset is relatively low. According to this, we decided to automatically split the ‘tesco’ and $TSCO data according to an automatic financial annotation by using a simple vocabulary related with general financial terms and specific terms related with the action.

Regarding the representativeness of our vocabulary, the frequency of terms in the ‘tesco’ data was 7.11% and is much lower than the frequency in the $TSCO data which is 47.68%. We remark that these percentages referred to the total corpus (content of all the tweets in the dataset), not the total number of tweets in the datasets. Regarding the number of tweets containing at least one term in the vocabulary, we observe the following results in ‘tesco’ dataset: 2.07% (245 of 11817) before the announcement and 9.52% (2437 of 25547) after the announcement. This difference is obviously not so big in the $TSCO dataset: 25.00% (3 of 12) before the announcement and 27.47% (25 of 91) after the announcement (the increasing can be explained because of the inclusion of terms specifically related with the action). These differences in percentages is what these datasets should presumably exhibit according to the clear financial orientation of $TSCO as a pseudo-hashtag for the financial facts related with Tesco PLC. So that, our simple vocabulary allows us to measure the financial load of a set of tweets.

According to the vocabulary, we split the ‘tesco’ dataset (pre- and post- announcement) into a set of tweets containing at least one term in the vocabulary, hereinafter referred as ‘tesco’ financially-oriented subset; and the set of tweets not containing any term in our vocabulary, hereinafter referred as ‘tesco’ general subset. Then, for uncovering the main topics, we adopted a simple bag-of-words model [55]. Under this approach tweet words in the status content are only considered according to their relative frequency and not according to their order within the document. In the bag-of-word result, we consider the proper cleaning precautions (i) by lowercasing and removing stop words; and (2) by specifically considering Twitter jargon (i.e. HT, QOTP). Finally, terms are contextualised according to the tweet semantics and categorised into topics. As a result, we obtain the contextualised & categorised occurrence of terms in Table 6, and the evolution of the size of the considered categories on the ‘tesco’ corpus in Fig. 9.

Open image in new window

Table 6

Term occurrences in subsets of the ‘tesco’ dataset (General and Financially-oriented)

Open image in new window

More specifically, Table 6 contains the contextualisation of the 20-most frequent terms inside the complete content of the tweet according to an n-gram strategy and manual annotation. We can obviate the term ‘tesco’ in its different forms (tesco, tesco’s in grey) from the analysis, since the dataset has been obtained from different ‘tesco’ search queries to Twitter. Regarding the general annotated subset of the ‘tesco’ dataset, before 27th 7:00 a.m. all most frequent terms are either stop words (including Twitter jargon) and terms related with the main Tesco PLC business. Even though, we can distinguish a set of terms related with a specific Tesco campaign (the one previously mentioned in the appearance of #essothursdays in blue) and also a topic ‘January blues’ which can be considered seasonal in the UK, even in the context of a dataset specific for the company Tesco PLC. Just after the announcement, the scene totally changed and neither the campaign nor the seasonal topic appear but they are replaced by terms related to the Tesco PLC merge with Booker. Therefore, topic discovering exhibits the same kind of disturbance than hashtags, that is, the impact of the merge can be perceived in the ‘tesco’ general subset after the announcement. After this analysis, we can establish the fulfilment of the complete RQ1 : Given that $cashtag-content has a clear financial orientation, this financial burden can also be perceived in the #hashtag dynamics and in the general content. So that, Twitter is highly permeable to the financial event considered in this experiment.

Regarding the financially-oriented subset of ‘tesco’, the most frequent terms are obviously related with Tesco PLC financial perspective before and after the announcement but, once again, the scene changes totally from one day to another. With the financial lens the filtering vocabulary provides, a topic emerges referring to another relevant financial event related with Tesco PLC. On Jan 24th, a Tesco spokesman announced that the firm was facing a claim for damages about profit overstatement. This topic is, therefore, relevant enough to emerge in the financial subset but is disguised by the contents related with the main business in the general subset. What we have is a financial event, in the same company, which does not surpass the imaginary threshold of permeability to the general Twittersphere. This can be clearly perceived in the Fig. 9 where bubbles are coloured according to the main topics (green scale for financial and yellow scale for main business, grey for Tesco in the middle of these two worlds) and sized according to the number of term occurrences. A big green bubble appears in the general post-announcement subset but, in the financially-oriented subset, the green bubble related to the overstatement is replaced by the green bubble related to the merge.

Coming back to the fundamental research question addressed in this paper, we sustain and successfully confirmed that “Twitter (although not a specific financial forum) is permeable to financial events and this permeability can be analysed by monitoring (1) the name of companies as a keyword, (2) the Cashtag of the company and (3) the hashtags related to that company.” The results of the experiment on the announcement about the merge of Tesco PLC and Booker Group PLC on the 27th January 2017 show that the Twittesphere is permeable to the financial market dynamics thanks to the tweets of a variety of different contributors. The merge impacted on all the Tesco-related content in Twitter in terms of volume, having a higher impact on $cashtag-content but is altering the tweets’ topics in comparison to regular behaviour (altering #hashtag dynamics and tweets’ content). Also, with the precautions of a single experiment, we also observed changes in polarity and in the geographical distribution of the contributes through time. Finally, the good freshness characteristics of Twitter as news media is confirmed by the rapidness of response to the RNS announcement.

Therefore, the experiment was successful in confirming that a far-impacting financial event causes disturbance in all the features considered in relation with Twitter permeability: information volume, content and sentiment as well as geographical provenance. Nevertheless, the experiment had a little success in identifying some rumour or sign of the announcement prior to the event. Even considering that the experiment was not deployed over the whole Firehose Twitter data, uncovering rumours before the announcement turns definitively into a hard task, if the spreading of rumours in real life is not mimicked inside Twitter, that means, if the rumour is not in Twitter at all. At this respect, and according to [30], social media data can only be generalised to human behaviour when social media provides a representative description of human activity. Twitter is a social media which, at least, exhibits some demographic bias. Moreover, Twitter may be providing a skewed representation of content. Although well-known rumour detection algorithms [51, 53] can be applied to Twitter, an alternative approach can be the fusion of financial information from different data sources in a way that we can mitigate the inevitable bias in a single source, and, at the same time, combine their weaknesses and strengthens in a proper representation of the real financial activity.

As mentioned, the experiment in this paper addresses a well known financial market event as a first step to adapt the methodology to low impacting events. Presumably, financial events with fully coverage in the media (social media, news, tv) would have a major impact on the whole Twittersphere, meanwhile financial events in sectors not so attractive to the general public as retailing or, not so covered by non financial media, would have lower impact on Twittersphere but remain impacting inside Twitter financial sphere. The permeability and patterns of change in hashtag and cashtag should be adjusted according to theses conditions. The real deployment of a continuous real-time monitoring system for irregularities coming from financial actions will be based on mining social media and establishing patterns for the time series of cashtag and hashtags and topics (bag-of-words) so we can apply and ARIMA (Autoregressive moving average model) to avoid noise and distinguish the stationary behaviour form the tending one. The continuous adjustments of the patterns, the bag of words and outlier detection techniques will be easily scalable to the huge quantity of data by a map-reduce implementation and a cloud deployment.

This paper inspects the permeability of Twitter to financial events in order to provide evidence which allows Twitter to be used as a social sensor for the financial and stock market. To do that, this permeability should be checked and measured. This a single experiment for a single financial that had been fully covered by traditional media as well as social media. Bearing this in mind, we can conclude that the event in the financial market invaded the Twittersphere on the 27th January 2017, just after the RNS announcement at 7:00, and that the behaviour of the triplet ($TSCO, #[tesco], “tesco”) was altered in comparison with the regular behaviour around the company involved in the financial event. At the same time, the experiment shows that other financial events that affected the company (overstatement) during the very same period were only permeable to the more financial oriented tweets and users. As many other social networks, homophily [14] exists in the context of Twitter, that is, the tendency to interact with similar individuals in respect to several dimensions such as age, location, and occupation, etc. The experiment in this paper shows that the high impact of the merge action crosses the fragile boundary between users with specific financial interests (highly connected to each other) to the general audience in Twitter. However, this work has not studied the profile of the users in general audience that are captured by the Tesco merge, or, to put in another way, to which kind of homophilic words the information flows out from the experts in the stock market.

As mentioned before, this Twitter study is framed by a a joint project that pursue the construction of a Data Fusion Model and an Ecosystem to monitor the financial market, which is increasingly demanded given the irruption of FinTech technologies and the subsequent increase of non expert traders, automatic trading systems, cryptocurrencies, HFT, etc. The lack of integration between financial stock market data, social media comments, financial discussion board posts and broker agencies means that the benefits of data fusion are not being realised to their full potential. Our proposed ecosystem [20], inspired by the data fusion model introduced by JDL (Joint Directors of Laboratories [52]), pursue the fusing of disparate data sources relating to financial stocks; data with a diverse set of features from different data sources will supplement each other in order to obtain a Smart Data Layer, which will assist in scenarios of irregularity detection.