(Im)balance in the Representation of News? An Extensive Study on a Decade Long Dataset from India

Abstract

(Im)balance in the representation of news has always been a topic of debate in political circles.

The concept of balance has often been discussed and studied in the context of the social responsibility theory and the prestige press in the USA. Comprehensive analysis of all these measures across a large dataset of the post-truth era comprising different popular news media houses over a sufficiently long temporal scale in a non-US democratic setting is lacking. For this study, we amass a huge dataset of over four million political articles from India for 9+ years and analyze the extent and quality of coverage given to issues and political parties in the context of contemporary influential events for three leading newspapers. We use several state-of-the-art NLP tools to effectively understand political polarization (if any) manifesting in these articles over time. We also observe that only a few locations are extensively covered across all the news outlets. Cloze tests show that the changing landscape of events get reflected in all the news outlets with border and terrorism issues dominating in around 2010 while economic aspects like unemployment, GST, demonetization, etc. became more dominant in the period 2014–2018.

Appendices

A Appendix

1.1 A.1 Summary Based on Time Series Clustering

For each individual imbalance metric and each newspaper, one can obtain a time series of the directed scores spanning over 9+ years. While one can always look into each such time series data to make an inference, our idea was to look for universal characteristics of imbalance across the three news outlets. To this end, we cluster the time series of imbalance scores using the standard dynamic time warping (DTW) approach. We use hierarchical agglomerative clustering to understand the similarity among the newspapers based on their temporal imbalance characteristics. In addition, this clustering technique also summarizes which of the metrics have remained closer to each other over time.

Fig. 3.

Dendogram of different measures of imbalances across different newspapers

Fig. 4.

Dendogram of different newspapers across different measures of imbalances.(Acronyms used and their full forms: cov_h: coverage imbalance in headings; cov: coverage imbalance in content; sup: superlative and comparative imbalance; pos: positive sentiment imbalance; neg: negative sentiment imbalance; subj: subjectivity imbalance)

Fig. 5.

Temporal variation of imbalance in coverage of content and positive sentiments in the news articles for the different media houses.

Observations. We have seven different imbalance metrics namely – headlines coverage, content coverage, point of view, positive sentiment, negative sentiment, subjectivity and superlatives/comparatives. For a given news outlet therefore we shall have seven corresponding time series each spanning over 9+ years. Since there are three major news outlets in our dataset we have 21 time series in all. We cluster these 21 time series using DTW as discussed in the previous section. As evident from the results presented in the form of dendograms in Fig. 3, Times of India and India Today exhibit stronger clustering across almost all the imbalance measures pointing to an interesting universal characteristic. In order to delve deeper into the dynamics, we present in Figs. 5(a) and 5(b) respectively, two representative time series plots of imbalance scores – content coverage imbalance and positive sentiment imbalance.

Content Coverage Imbalance: In Fig. 5(a), we plot the directed imbalance scores of content for each of the media houses over time. The first noticeable trend is that the media houses have distinct relative imbalance very consistent over the timeline. The Hindu has been especially consistent in maintaining a 5–10% shift in coverage toward the Congress party than the other two news organizations. Consequent to relatively higher leaning of The Hindu in coverage of the Congress party, The Hindu always remained Congress leaning (below zero in the curve) unlike its two peers. The shift between TOI and India Today is not that apparent thus providing the empirical justification of the results obtained from the DTW clustering.

Positive Sentiment Imbalance: In Fig. 5(b), we observe that The Hindu has an imbalance score higher than the zero mark throughout till 2017 showing higher density of positive sentiments toward BJP. This behavior is in drastic contrast with the other two news outlets thus providing the justification in support of the DTW results. Of course, the election year (2014, which also marked significant change in vote share and public sentiment) observes high positive sentiments in favor of BJP in general.

As a next step, we cluster the seven different time series corresponding to the respective imbalance metrics for each of the news outlets separately (see Fig. 4). For all the news outlets we again observe a universal pattern whereby the tonality based imbalances are clustered more strongly exhibiting their distinctions with the coverage and point of view imbalances.

Fig. 6.

Temporal variation of popularity of each party for the India Today corpus. The trends are very similar for the other two news outlets.

1.2 A.2 WEAT Scores to Determine Party Popularity

We calculate the differential association over time and plot that over the years. We use this distance as a proxy for popularity as portrayed by that particular news media. We illustrate the algorithm next.

Word Embedding Association Test (WEAT). In order to understand how the popularity of the political parties among the people of India has changed over time we calculate the year specific word embeddings on our corpus using methods used to measure semantic shift in words over time.

Previous research Azarbonyad et al. (2017) suggests that frequently used words have the least shift of their meaning over time. Hence we use top 1000 words in our corpus (excluding the words BJP and Congress consciously) to align the word embeddings trained for different time periods. We train word2vec Mikolov et al. (2013) with SGNS (skip-gram with negative sampling), to create embeddings for each of the year in our dataset. Let $W(t) \in \mathbb {R}^{d \times V}$ be the matrix of word embeddings learnt for year t and for vocabulary V. Following Hamilton et al. (2016), we jointly align the word embeddings while generation, using the top 10000 common words present across time periods $t_1$ and $t_2$ by optimizing:

$$\begin{aligned} R^t=\mathop {\mathrm {arg\,min}}\limits _{Q_{t_1t_2}^TQ_{t_1t_2}=I} ||QW^{t_1} -W^{t_2}|| \end{aligned}$$

For simplicity we assume $Q_{t_1t_2}=I \forall t_1, t_2$. After alignment, we measure the WEAT score of the words BJP and Congress with the opposite set of words $A_1=\,${good, honest, efficient, superior} and $A_2=${bad, dishonest, inefficient, inferior} using the algorithm presented in Brunet et al. (2019).

The differential association of a word c with word sets $A_1$ and $A_2$ is given by

$$\begin{aligned} g(c, A_1, A_2, w) = \tiny {\begin{array}{c}mean \\ a \in A_1\end{array}} \cos {(w_c, w_a)} - \tiny {\begin{array}{c}mean \\ b \in A_2\end{array}} \cos {(w_c, w_a)} \end{aligned}$$

where w is the set of word embeddings, $w_x$ is the word embedding for the word x.

Now, the WEAT score is calculated as

$$\begin{aligned} B_{weat}(w)=\frac{g'(s_1) - g'(s_2) }{\tiny {\begin{array}{c}SD \\ s_3 \in S_1,S_2\end{array}} g(s_3 , A_1, A_2, w)} \end{aligned}$$

where

$$\begin{aligned} g'(s_i) = \tiny {\begin{array}{c}mean \\ s_i \in S_i\end{array}} g(s_1 , A_1, A_2, w) \end{aligned}$$

Here the word sets $S_1$ and $S_2$ are the keywords related to the political parties, as already discussed previously.

From Fig. 6, it is evident that BJP gained popularity in news very fast post 2011, surpassing popularity of Congress in 2014, the year of legislative assembly election when incumbent BJP overthrew the ruling Congress government. We also see the popularity of Congress increasing again since 2016, the year of demonetization, that possibly had a strong impact on the economy of India and specially on the poorest ones of the country.¹³

1.3 A.3 State Level Analysis of Inequality in Coverage

Now we attempt to understand if the situation is similar in state level and if yes, then which states are covered poorly. We collect all the state names from the census report of 2011 (see footnote 12) and search for their occurrence across the corpus. We note the number of articles a specific state is mentioned in and plot the same. We do this experiment for all the three newspapers and for each newspapers, we once plot for only 2010, once only for 2018 to understand the evolving trend.

Observations: Figure 7 holds the evidence that the states of Jammu & Kashmir, states in the north-east and some non-Hindi speaking states like Orissa or Jharkhand are squarely ignored by all the three newspapers. Hindu seems to stand out in coverage of states from the other two newspapers covering mostly south Indian states. Looking at the maps comparatively from 2010 to 2018, it seems that the situation is improving and more states are getting covered by the national newspapers over time though equality among the states may be a long way.

B Further Insights

In order to obtain further insights, we perform a cloze task Taylor (1953), i.e., a task that requires completion of a sentence by correctly predicting the masked/hidden word. For instance, in the following cloze task – “Sun is a huge ball of $\langle mask \rangle$, “fire” is a likely completion for the missing word. Given a cloze test, well-known language models like RoBERTa Liu et al. (2019), produce a sequence of tokens with their corresponding probabilities to fill the given blank in the input sentence. We train RoBERTa (initialized with RoBERTa-base Liu et al. (2019)) for each of our newspapers for each year present in the corpus separately for 20000 iterations following the language model training procedure described in Khalidkar et al. (2021). This results in a total $3*9=27$ different models. We use these models (representative/mouthpiece of each newspaper at different times of the 9 years in our corpus) to answer the following three questions – (a) can one track the changing priorities for India as depicted by each news media house? (b) how are these newspapers reporting popularity of one party over the other, for these 9 years? and (c) how are newspapers presenting perception about the Indian economy?

Table 3. Top tokens increasingly and decreasingly accepted as answer in 2018 for the cloze task (a) & (c).

Can One Track Changing Priorities? To understand the changing priorities of India as a nation over the last decade, we propose the following cloze task query – “The main issue in India is $\langle mask \rangle$”. We attempt to understand how RoBERTa’s answer changes for this specific query from 2010 to 2018. To this purpose, we take a union of top 50 tokens given as output for RoBERTa$_{2010}$ and RoBERTa$_{2018}$. We then rank the top tokens which underwent maximum positive change from 2010 to 2018 as an answer to the cloze test (i.e. the tokens which are more accepted as answer in 2018 than in 2010 for the cloze test). We also rank the top tokens which underwent maximum negative change from 2010 to 2018 as an answer to the cloze test (i.e. the tokens which are less accepted as answer in 2018 than in 2010 for the cloze test). We show maximum of 15 such tokens in order of probability (higher to lower).

Analysis and Observations: From Table 3, we see a similar pattern reverberating across the news media houses. The focus of India in 2018 is more on economic issues like unemployment, jobs, corruption, poverty, GST, food and reservation and less on border issues like Kashmir, Pakistan, Afganistan and security. More basic demands like food, housing, water and agriculture are popping up in 2018. We showed these results to 9 Indians in verse with the events in Indian government. All of them unanimously agreed that these are due to the changing landscape of events affecting India from 2010 to 2018. The period 2008–2010 saw a lot of coordinated bombing and shooting attacks by terrorists on Mumbai, the economic capital of India resulting in mass killings and injuries. These issues mainly related to the India-Pakistan border conflicts emerge in the words popping up in the 2010 newspapers. Between 2014–2018, on the other hand, India saw various economic reforms in the form of introduction of GST, demonetization, stress on online transactions and implementation and linking of AADHAR (an unified database of citizens like social security number in US) with banking for continuation of banking services. All these together led to the increase of priority of economy related words in these news outlets.

How are These Newspapers Reporting Popularity? We attempt to understand how popularity of one party over the other is reported in these newspapers and how they are similar or different from each other. We define voting preference toward a specific political party $\langle p \rangle$, $\forall \langle p \rangle \in${“BJP”,“Congress”} as:

$$\begin{aligned} V_{pop}(\langle p \rangle ) = P_{RoBERTa}(\langle mask \rangle = \langle p \rangle | input =prompt) \end{aligned}$$

(2)

where

$$\begin{aligned} prompt=\text {``This election people will vote for } \langle mask \rangle \text {.''} \end{aligned}$$

(3)

Fig. 7.

Number of articles mentioning each state for different newspapers across different time periods

Fig. 8.

Popularity of BJP over Congress quantified from the results of Cloze test 4, plotted over the years

Further, we normalize these values to probabilities toward any of the two parties, arbitrarily selected to be BJP (plotting both is redundant as $p_{congress}=1-p_{bjp}$) as

$$\begin{aligned} Pr_{pop}(``BJP\mathrm{"})=\frac{V_{pop}(``BJP\mathrm{"})}{V_{pop}(``BJP\mathrm{"})+V_{pop}(``Congress\mathrm{"})} \end{aligned}$$

(4)

How are These Newspapers Reporting Economic Prosperity? We attempt to understand how media houses are reporting economic prosperity of India over time. Using the probe “The economy of India is $\langle$mask$\rangle$”, we report the most probable outputs in Table 3.

Observations: We plot the probabilities in favor of “BJP” over time for each news media house in Fig. 8. We observe that once again all the news media groups show a very similar pattern with the period 2012–2013, a year before the national election, to be the inflection point. The opposition ‘BJP’ could defeat the incumbent ‘Congress’ government with a large margin following gain in popularity in 2010–2011 largely due to corruption charges against the ‘Congress’ which resulted in nationwide protest and a very influential anti-corruption movement in the capital. We see the popularity of ‘BJP’ with respect to ‘Congress’ only rose in the years following the election which seems intuitive as ‘BJP’ won the 2019 election also with huge majority and increased vote share than 2014¹⁴. Also, an interesting observation is that a huge policy failure like demonetization which arguably influenced the fall of GDP due to extreme shrinkage of money in circulation¹⁵ and nationwide increase in economic inequality did not decrease the popularity of ‘BJP’ very significantly though a dip in popularity is observed in 2016 for all the news media houses. For cloze test (c), we see Hindu and India Today both reporting similarly about the economy with higher negative words for economy in 2018 which resonates with the ground truth of GDP growth rate for India but ToI interestingly reports the opposite trend.