Sentiment and position-taking analysis of parliamentary debates: a systematic literature review
Parliamentary and legislative debate transcripts provide access to information concerning the opinions, positions, and policy preferences of elected politicians. They attract attention from researchers from a wide variety of backgrounds, from political and social sciences to computer science. As a result, the problem of computational sentiment and position-taking analysis has been tackled from different perspectives, using varying approaches and methods, and with relatively little collaboration or cross-pollination of ideas. The existing research is scattered across publications from various fields and venues. In this article, we present the results of a systematic literature review of 61 studies, all of which address the automatic analysis of the sentiment and opinions expressed, and the positions taken by speakers in parliamentary (and other legislative) debates. In this review, we discuss the existing research with regard to the aims and objectives of the researchers who work in this area, the automatic analysis tasks which they undertake, and the approaches and methods which they use. We conclude by summarizing their findings, discussing the challenges of applying computational analysis to parliamentary debates, and suggesting possible avenues for further research.
KeywordsSentiment analysis Opinion mining Text as data Parliamentary debates Legislative debates
Debate transcripts from legislatures such as the United Kingdom (UK) and European Union (EU) parliaments and the United States (US) Congress, among others, provide access to a wealth of information concerning the opinions and attitudes of politicians and their parties towards arguably the most important topics facing societies and their citizens, as well as potential insights into the democratic processes that take place in the world’s legislative assemblies.
In recent years, these debates have attracted the attention of researchers from diverse fields and research backgrounds. These include, on one hand, computer scientists working in the field of natural language processing (NLP), who have investigated the application and adaptation to the political sphere of methods developed for sentiment analysis of product reviews and blogs, and who have also tackled other related tasks in this domain, such as topic detection. In addition, political and social scientists, traditionally relying on expert coding for the analysis of such transcripts, have increasingly been exploring the idea of viewing ‘text as data’ , and using computational methods to investigate the positions taken by debate participants.
As a result, a wide range of approaches to the problem of automatic debate analysis have been adopted, with research on this problem varying widely in its aims and methods. Within this body of work, there exist many inconsistencies in the use of terminology, with studies in some cases referring to very similar tasks by different names; while in others, the same term may mean quite different things. For example, while Chen et al.  and Kapočiūtė-Dzikienė and Krupavičius  both attempt to classify debate speakers according to party affiliation, the former refer to this as ‘political ideology detection’, and the latter as ‘party group prediction’. Conversely, a single term like ‘sentiment analysis’ may be used to refer to, among other things, support/opposition detection , a form of opinion-topic modeling , and psychological analysis . The approaches adopted range from statistical analyses to predictive methods, including both supervised classification and unsupervised topic modeling. There are also contrasting approaches to modeling the textual data, the level of granularity of the analyses, and, for both supervised learning methods and the evaluation of other approaches, the acquisition and application of labels used to represent the ground-truth speaker sentiment.
With regard to synthesis of the existing research on this topic, Kaal et al.  assembled researchers from diverse fields to investigate the problem of text analysis in political texts, Glavaš et al.  presented a tutorial addressing similar themes to this review, and both Hopkins and King  and Monroe et al.  discussed the general differences in the aims and objectives of social scientists and computer scientists when working on such problems. However, as far as we are aware, there exists no comprehensive written overview, systematic or otherwise, of research in this area. The aim, therefore, of this review is to bring together work from different research backgrounds, locating and appraising literature concerning computational sentiment and position-taking analysis that has been undertaken to date on the domain of parliamentary and legislative debate transcripts. We assess the research objectives, the types of task undertaken, and the approaches taken to this problem by scholars in different fields, and present suggested directions for future work in this area.
In the NLP literature, the terms opinion mining and sentiment analysis are used more or less interchangeably (for a discussion of this, see Pang and Lee ), and are employed to describe both the specific task of determining a document’s sentiment polarity (that is, positive or negative, or sometimes neutral), and the more general problem area of automatically identifying a range of emotional and attitudinal ‘private states’ (that is, non-observable, subjective states),1 such as ‘opinion, sentiment, evaluation, appraisal, attitude, and emotion’ . In a recent survey, Yadollahi et al.  list nine such different sentiment analysis sub-tasks.
Researchers in the political and social sciences, meanwhile, appear to lack a single term to describe the act of determining from text the positions taken by legislators. In the literature, such tasks are variously referred to as ‘political scaling’, ‘position scaling’, ‘ideal point estimation’, and a range of other task- and data set-specific terms. One phrase that appears throughout such work is ‘text as data’ [e.g., 23, 36, 56]. We, therefore, include this term in our systematic search to capture work from the field that utilizes computational methods for speaker position analysis.
For the purposes of this article, we use ‘sentiment analysis’ as a general umbrella term, encompassing any tasks concerned with the extraction of information relating to speakers’ opinions and expressed positions, and ‘sentiment polarity classification’ for the more specific, binary or ternary classification task.
Research questions and objectives
RQ1: What are the research backgrounds of the authors of papers published in this area, and to what extent is the work multi-disciplinary?
RQ2: From which parliaments and other legislatures have debates been analyzed?
RQ3: What are the objectives of researchers from different backgrounds working on sentiment and position analysis of parliamentary debates?
RQ4: What sentiment analysis task sub-types have been undertaken to conduct analysis of parliamentary debates?
RQ5: What approaches have been taken to sentiment/position analysis of parliamentary debates?
RQ6: What conclusions can be made about the reported performance and outcomes of the sentiment/position analysis systems that have been described?
Review scope and method
For this review, we followed the established systematic review guidelines of the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement . The use of systematic review methodology to conduct this review has enabled us to uncover a substantial body of relevant work, but has meant the exclusion of some potentially interesting studies. We were unable to include a number of known relevant results uncovered by an initial scoping search of the Google Scholar platform, which, due to the lack of transparency of its search algorithm, does not facilitate replication. While somewhat limiting in this sense, the decision to adhere to a systematic methodology provides a replicable and transparent method of synthesizing and summarizing the literature and identifying future research priorities.
The review covers all literature retrieved by systematic search of five digital library databases and repositories [see Fig. 1(1)]. These were selected as they provided high coverage of the results obtained by the prior scoping search. That search also provided the basis of our keyword search terms, which we developed to return results that included, as a minimum, all the relevant publications previously found (Fig. 1(2)).2
We categorize the research background of each study according to the institutional affiliation(s) of its author(s) and the nature of its venue of publication, coding them as either computer science, political/social science, or multi-disciplinary. We consider a study to be multi-disciplinary if it (a) is written by authors from two or more research backgrounds, or (b) the paper is published at a venue associated with a different research background than that of its author(s)’ affiliations(s). While, it is of course possible that the work which we class as being from computer science or the social sciences actually involves some level of inter-disciplinary collaboration that does not fit within our definition (for example, we do not investigate the authors’ academic histories), this is a straightforward yet systematic way of obtaining a general overview of the research community working in this area.
Parliaments and legislatures
Nearly, all the included studies focus on one single legislature for analysis, with only Sakamoto and Takikawa  and Proksch et al.  comparing their approaches (to the analysis of the level of polarization, i.e., ideological division, in parliaments) on transcripts from two or more different chambers. The US Congress is by far the most popular legislature for analysis, attracting the attention of 31 of the studies. This can partly be attributed to the global power and influence of the US and of the English language, but is also explained by the widespread use by NLP researchers of the ConVote corpus  as a benchmark data set for the evaluation of sentiment analysis systems. Indeed, including its original authors, 17 of the publications use this data set, 15 of which are written from a computer science background, with Hopkins and King  (social science) and Iyyer et al.  (multi-disciplinary) the exceptions. In some cases, ConVote is used alongside one or more other non-legislative data sets (such as product reviews) for the evaluation of text classification methods [5, 15, 16, 28, 30, 38, 41, 79, 80, 81]. In fact, only a little over half (37) of the studies are exclusively concerned with the analysis of legislative debates. On the whole, political and social scientists seemingly prefer to construct their own data sets from the congressional record to suit their research aims, while Sakamoto and Takikawa  (multi-disciplinary computational social science) also do so.
Sources of publicly available parliamentary and legislative debate data used in studies
Talk of Norway
Kronika Sejmowa (Sejm Chronicle)
Nearly all the included publications consist of analysis of parliamentary or legislative data in a single language. Exceptions include Sakamoto and Takikawa  who use two corpora in different languages (English and Japanese), and Glavaš et al.  who use a multilingual dataset (German, French, English, Italian, and Spanish), as do Proksch and Slapin  (English, French, and German translations). In the latter case, while the original data are multilingual (23 official languages of the European Parliament), the transcripts have been translated to these three languages. Similarly, for speeches not originally in English, Baturo et al.  use official translations from the other official languages of the UN (Arabic, Chinese, French, Russian, and Spanish). By far, the most prominent language is English, analyzed in 55 studies. This is followed by French and German, used in six studies each, and a long tail of languages that only appear in one study each (Czech, Dutch, Finnish, Italian, Japanese, Lithuanian, Norwegian, Polish, Spanish, and Swedish).
Research objectives, tasks, and approaches
We investigated three aspects of the studies under review: the authors stated research objectives; the task types undertaken to achieve those aims; and the approaches taken to tackling those tasks. For the latter, we report the granularity at which analysis is undertaken, the methods used, and, where applicable, the labels used to represent ground-truth sentiment/position.
Included studies by background and stated research objectives
Political and social sciences
System/method performance evaluation
Although the work of computer scientists generally focuses on system evaluation, they often state secondary application objectives which encompass motivations relating to contributions to civic technology or the development of tools for social scientists. For example, Burfoot  suggests that ‘tools could assist researchers in understanding the nuances of contentious issues on the web by highlighting areas in which different sites or pages agree and disagree’, while Budhwar et al.  hope that their work will ‘give ordinary citizens a powerful tool to better organize and hold accountable legislators without the costs of physical presence and full-time lobbying representation’.
While the production of corpora and data sets are among the secondary contributions of many of the featured papers [e.g., 2, 66, 74], in the cases of Lapponi et al.  (linguistically annotated corpus) and Rauh  (sentiment lexicon), this is their principal objective.
Hopkins and King  claim that a fundamental difference between the objectives of computer scientists and political or social scientists is that, while the former are interested in making predictions about latent characteristics of individuals documents (such as sentiment polarity), the latter are more concerned with characterizing corpora or collections of documents as a whole, for example by the proportion of positive or negative examples it contains. This point is supported by Monroe et al. , who agree that individual document classification is ‘inappropriate to the task’, because, they suggest, under this model, representation of the whole data-generation process is backwards—where classification presumes that class labels are manifestations of the underlying latent phenomena of interest, the reality is the other way around: people first hold opinions or positions, and subsequently express them in speech or writing.
Despite this dichotomy, as we see in the next section, we do find cases of political scientists tackling classification  or computer scientists undertaking the scaling task from political science .
Agreement and alignment detection: analysis of the similarity of the position taken by a speaker and another entity (another speaker, or a person or organisation outside of the debate in question), \(n = 5\) [3, 18, 33, 48, 67].
Ideology and party affiliation detection: in the literature, a speaker’s party affiliation is often used as a proxy for their ideological position. This may be performed as either topic modeling or classification, \(n = 14\) [6, 9, 13, 16, 17, 28, 29, 32, 35, 37, 38, 50, 51, 73].
Sentiment/opinion polarity classification: binary or ternary analysis. As votes are frequently used as opinion polarity labels, this includes the task of vote prediction, \(n = 28\) [1, 4, 5, 6, 7, 12, 13, 14, 15, 18, 26, 30, 33, 41, 52, 55, 56, 60, 62, 64, 66, 67, 68, 72, 74, 78, 79, 80, 81].
As all the tasks undertaken concern the analysis of the positions taken by debate participants, there is considerable overlap between them. Furthermore, there is sometimes some discrepancy between the name given to a task and the actual task performed. For example, Onyimadu et al.  refer to the task which they perform as both ‘sentiment analysis’ and ‘stance detection’, although it could be said that they actually carry out only sentiment polarity classification as they do not specify a pre-chosen target, a requirement of stance detection (as defined by Mohammad et al. ). Meanwhile, Thomas et al.  refer to this task as ‘predicting support’, and Allison , working on the same problem and the same data set, calls it variously sentiment ‘detection’ and ‘classification’. Although Rheault et al.  consider their work to be a form of emotion detection, they actually perform a form of sentiment polarity analysis at the whole legislature level, while Akhmedova et al.  simply refer to the task as an ‘opinion mining problem’. Other terms used to refer to this include ‘attitude detection’ , ‘vote prediction’ , ‘emotional polarity’ measurement , ‘predicting the polarity of a piece of text’ , ‘sentiment classification’ [78, 79], and simply ‘sentiment analysis’ [57, 60, 64, 66, 81].
In some cases, more than one task is investigated. For example, by switching party labels for vote labels, Burfoot  uses the same method to perform both sentiment polarity and party affiliation (or ideology) detection. Sentiment polarity analysis is often used as part of an NLP pipeline as a sub-task of a different opinion mining task, such as agreement detection . Similarly, Kauffman et al.  use sentiment analysis as a sub-task and the output scores as features for alignment detection, while Duthie and Budzynska  do similar for ethos detection, and Budhwar et al.  aim to predict vote outcome using the results of sentiment polarity analysis as features for the task. Conversely, Burfoot  applies the results of classification by party affiliation to predict speaker sentiment.
Balahur et al.  combine polarity with party classification, a task that we consider to be a form of ideology detection, but which they name ‘source classification’. Indeed, this is another task that suffers from a lack of clarity over terminology, with some studies considering party affiliation to be a proxy for ideology [17, 29, 32, 73], while others do not make this connection, extracting information about speakers’ ideologies from their sentiment towards different topics [9, 16, 51], or training a model on examples that have been explicitly labeled by ideology, and not party membership . Yet others perform party classification, making no mention of the relationship between party membership and ideology [6, 13, 35, 37]. Alternatively, Abercrombie and Batista-Navarro  explicitly assume that membership of the same party does not guarantee homogeneity of ideologies, investigating intra-party differences of opinion and positions. Also concerned with ideology, position scaling, which we code here as a separate task, can be performed on different dimensions, one of the most common being the left- to right-wing (ideological) scale.
The literature contains several efforts to simultaneously extract topics and speakers’ attitudes towards them (opinion-topic analysis). A common approach is to combine topic modeling with forms of stance detection. Nguyen et al.  use a supervised form of hierarchical Latent Dirichlet Allocation  to extract topics and polarity variables. Van der Zwaan et al.  generate separate topic models for different grammatical categories of words in efforts to obtain this information. And Nguyen et al.  perform supervised topic modeling to capture ideological perspectives on issues to produce coarse-grained speaker ideology analysis. Topic modeling has also been undertaken by Sakamoto and Takikawa , who use it to analyze polarization, a task also tackled by Jensen et al. . Meanwhile, Vilares and He  also perform opinion-topic modeling to extract speakers’ perspectives—‘the arguments behind the person’s position’—on different topics.
A number of other tasks which fit under the broader definition of sentiment analysis have also been tackled. Polarization analysis is undertaken in both Jensen et al.  and Sakamoto and Takikawa , who investigated changes in the extent to which language in Congress is polarized over time. Meanwhile, emotion detection is the subject of three studies. Dzieciątko  classifies speakers as expressing happiness, anger, sadness, fear, or disgust, while Rheult  attempts to identify the level of anxiety exhibited by speakers, and Honkela et al.  analyze a corpus of congressional speeches under the PERMA (Positive emotion, Engagement, Relationships, Meaning, and Achievement) model . Agreement detection, an end in itself for Ahmadalinezhad and Makrehchi  and Kauffman et al. , is used by Burfoot , Burfoot et al.  and Burford et al.  to predict speaker sentiment, while Salah et al.  use agreement information to construct debate graphs. Finally, Naderi and Hirst  automatically compare speeches with another type of labeled text (statements from online debates) to identify positive and negative framing of arguments.
Although there exist exceptions (see above), a notable difference between the focus of tasks undertaken by NLP researchers and social scientists is that the former tend to perform analysis with regard to the target of expressed sentiment (a specific proposal [e.g., 5], piece of legislation [e.g., 74], topic [e.g., 75], or other entity), while the latter generally analyze speakers’ aggregated speeches, ignoring the targets of individual contributions, and instead attempting to project actors onto a scale (such as left–right) [27, 34, 36, 40, 57, 69]. Grimmer and Stewart  note that this can be problematic as manual ‘validation is needed to confirm that the intended space has been identified’, and suggest automatic detection of relevant ‘ideological statements’ (or opinion targets) as an important challenge.
We consider the granularity (level of analysis), methods, features, and ground-truth labels used (in the cases of both supervised learning methods and evaluation of other methods) for each study.
While in other domains such as product reviews, sentiment analysis is typically carried out at the document, sentence, and aspect levels, here, we have found a number of approaches to segmenting the transcripts for analysis, including breaking them down to the sub-sentence level and aggregating sentiment over entire corpora. We also find differences in the terminology used to refer to these levels.
The vast majority of studies conduct analysis at the speech level (\(n = 39\)). However, ‘speech’ appears to mean different things in different publications, and in some, it is not immediately clear just what the unit of analysis actually is.3 A speech may be considered to be the concatenated utterances of each individual speaker in each debate (\(n = 16\)). Alternatively, analysis may be conducted at the utterance or ‘speech segment’ level (that is, an unbroken passage of speech by the same speaker) (\(n = 24\)), although Akhmedova et al.  refer to these as ‘interventions’, and Bansal et al.  as ‘individual conversational turns’. While several researchers who use Thomas et al.’s  ConVote corpus claim to analyze ‘speeches’, the data set (usually used unaltered) is, in fact, labeled at the speech segment level. Similar use of terminology can be found in other work, such as Vilares and He .
Other approaches are to analyze speeches at the coarser political party (or bloc/coalition) level [20, 21, 56, 57, 65, 75], or the finer sentence [18, 48, 52, 60] or phrase  levels. Although Rudkovsky et al.  detect sentiment in sentences, they aggregate these scores to provide speech-level results. Iyyer et al.  break speeches down to both these levels, while Naderi and Hirst  do so for sentences and paragraphs.
At the highest possible level of granularity, four studies consider sentiment over entire corpora. Dzieciątko  and Rheault et al.  aggregate sentiment scores for all speeches, presenting analysis of the Polish and UK parliaments respectively. Meanwhile, Honkela et al.  compare the overall sentiment of the European Parliament transcripts with other corpora at the whole data set level, and, in addition to party-level analysis, Sakamoto and Takikawa  compare the polarity of Japanese and US datasets.
Finally, Ahmadalinezhad and Makrehchi  consider each document to be a ‘conversation between two individuals’—that is, the two parties’ combined utterances—to classify these as being either in agreement or disagreement.
Overall, computer scientists tend to work at finer-grained levels (speech, speech segment, paragraph, sentence, or phrase), while in political science, the preferred units of analysis are the actor (or individual politician, whose contributions are pooled together), which is the target of most work on position scaling, a task very much associated with that field. This confirms to some extent the assertation of Hopkins and King  that, while computer scientists are ‘interested in finding the needle in the haystack, ... social scientists are more commonly interested in characterizing the haystack’. Exceptions, from the political and social sciences, are Iliev et al. , and Hopkins and King —who actually propose a method of optimizing speech-level classification for social science goals, and from computer science, Glavaš et al. , who also tackle the position scaling problem.
Statistical machine learning: used to learn a predictive function based on the data, \(n = 45\) [1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 13, 14, 15, 16, 17, 18, 21, 25, 26, 27, 28, 30, 32, 33, 35, 37, 38, 41, 48, 50, 51, 55, 61, 64, 65, 67, 72, 73, 74, 75, 76, 78, 79, 80, 81].
In total, roughly three quarters of included studies (\(n = 45\)) make some use of machine learning, and within this area, there are a multitude of different approaches (see Fig. 6). These can be broadly categorized as supervised learning (\(n = 40\)), semi-supervised (\(n = 1\)), or unsupervised (\(n = 11\)) methods. Supervised methods are the preferred approach for text classification, and a wide variety of algorithms are used, including logistic regression (\(n = 10\)), naive Bayes (\(n = 6\)), decision trees (\(n = 3\)), nearest neigbor (\(n = 3\)), boosting (\(n = 1\)), a fuzzy rule-based classifier (\(n = 1\)), maximum entropy (\(n = 1\)), nearest class classification (\(n = 1\)). A further 11 studies make use of neural networks, which range in complexity from ‘vanilla’ feed-forward networks such as the multi-layer perceptron to convolutional and recurrent neural networks, including the use of long short-term memory units (LSTMs) (see Fig. 5). Of these, six are concerned with sentiment polarity analysis and four with ideology detection, which, as previously discussed, are highly similar classification tasks, performed with different class labels. The exception is Duthie and Budzynska , who use recurrent neural networks modules in their ethos mining task.
Rather than testing different classification algorithms, some work focuses on the use of different regularization methods (for logistic regression) [79, 80, 81], while other approaches to improving performance of classifiers include Boosting , and Collective Classification .
Much of the work on unsupervised learning focuses on use of topic modeling methods, the majority of which are variations of LDA. The cross-perspective topic model of van der Zwaan et al. , for example, generates two topic models from the data: one over nouns to derive topics, and the other over adjectives, verbs, and adverbs, which is intended to produce opinion information.
Additionally, seven studies model debates as networks of connected speakers and employ graph-based methods. Bansal et al. , Burfoot , Burfoot et al. [14, 15], and Thomas et al.  all approach sentiment polarity classification as a minimum cuts graph partition problem. Chen et al.  construct an ‘opinion-aware knowledge graph’, propagating known ideology information through the graph to infer opinions held by actor nodes towards entity nodes. For position scaling, Glavaš et al.  use similarity measurements as edges between documents, and propagate the scores of known ‘pivot texts’ to the other nodes.
Language models and feature selection
Although analysis of debate transcripts necessarily utilizes textual features derived from the speeches, there are a variety of approaches to how this is modeled, and which types of features are selected. In terms of language modeling, the majority of studies represent the text as bags-of-words.4 However, some add contextual information with use of word embeddings. While their use is generally restricted to studies from computer science, Rudkovsky et al.  (multi-disciplinary) and Glavaš et al.  (computer science) explore the use of word embeddings for aims normally associated with the social sciences. Relatively little linguistic or structural analysis is undertaken. Exceptions are Ji and Smith , who find that the application of Rhetorical Structure Theory does not produce satisfying results in this domain, because its ‘discourse structure diverges from that of news’, and Balahur et al. , who extract parse trees to determine the target of speech sentiment.
Textual: including word (all studies) or character  n-grams, custom dictionary keyword features , words from particular grammatical categories [28, 32, 35, 47, 48, 52, 72, 75], presence of questions , word embeddings [9, 21, 28, 30, 38, 48, 61, 62], sentence  embeddings, and parse trees [6, 28, 30].
Debate discourse features: including citations [13, 14, 37], interruptions and speech length , n-grams from other neighboring sentences , and utterance statistics (number and duration of speaker’s utterances) .
Speaker metadata features: including bill authorship , debate and speaker IDs [1, 66], debate type , donations [11, 33], geographic provenance , party affiliation [1, 2, 13, 33, 64, 65, 66], and gender .
Polarity scores: the output of opinion polarity analysis used as a feature for prediction of another phenomena (such as sentiment polarity , voting intention , argument frame , or alignment with another entity ).
Relational and knowledge graph features: features based on the relationships between different speeches or speakers, speakers and other entities (such as the targets of expressed opinion), or the measured similarity between speeches [13, 14, 15, 16, 74].
Depending on the nature of the task being tackled, for supervised classification methods, and in some cases, validation of unsupervised methods, several different data sources are used to represent the ground-truth sentiment or position. In most cases, researchers opt to make use of some pre-existing data, with the most common being the speakers’ roll-call or division5 votes (n = 21). Here, the most common approach is to consider each speaker’s vote on a given debate to represent the ground-truth position taken in their speeches, which may be analyzed at the whole (concatenated) speech level (as in Salah ) or broken down into smaller units (as in Thomas et al. ), whereby each vote label is attached to multiple examples. A general difference in approach is that, while computer science studies use these as ground truth, political scientists tends to view speech and vote as unconnected, and even explicitly compare the two on this basis, as in Schwarz et al. . Lowe and Benoit  use human annotations for validation of the output of their scaling method. Whether or not votes are actually reliable as ground-truth is a matter of contention. Although some computer science studies assume this to be the case [e.g., 66, 68, 74], Schwarz et al.  compare speeches to votes for scaling, producing quite different results, and Abercrombie and Batista-Navarro  who compare votes with human produced labels, conclude that it is the latter which more closely reflect sentiments expressed by speakers.
An alternative approach is to use manually annotated labels. While some researchers make use of already existing expert annotations, such as the Chapel Hill expert surveys6 [21, 57], others produce labeled data sets specifically for their purposes. Onyimadu et al.  and Rauh  both had in-house annotators label speech sentences as being positive, negative, or neutral, while Rheault  used crowd-sourced coders to label sentences as anxious or non-anxious. Rudkovsky et al.  also use crowd-sourced labels, but for evaluation rather that training purposes, in their case to assess negativity detection. To create labels for the validation of their scaling of speakers’ positions towards a given topic, Frid-Nielsen  had experts follow the coding scheme of the Comparative Manifestos Project7 to produce policy position labels, although the reliability of these is also controversial .
Other data used as ground-truth labels are the speakers’ DW-nominate scores (scores derived from congressional legislators’ voting records8) [17, 51], their constituency vote shares , ‘issue’ labels from the Library of Congress’s Congressional Research Service9 , word perplexity , sentiment analysis scores obtained from prior experiments on the same data , and party affiliations (\(n = 13\)). While the latter are widely used as a proxy for speaker ideology [9, 29, 38], Hirst et al.  suggest that party membership is actually a confounding factor for this task.
Performance and outcomes
With the research reviewed here having such varied objectives and undertaking many different analysis tasks, it is not possible to directly compare the reported performances of the methods proposed. Nevertheless, in this section, we attempt to summarize some conclusions of the included studies that are potentially relevant to future work in this area.
For classification, machine learning methods, and particularly neural networks, seem to outperform other approaches. Here, just as in other domains, such as product reviews, dictionary-based sentiment analysis methods appear to have been superceded by machine learning approaches. In a direct comparison, Salah  found that machine learning classification methods outperform those utilizing both generic and parliament-specific lexica, while Balahur et al.  improved lexicon-based performance with the addition of a support vector machine classifier. Given this, and also considering the conclusion of Allison  that ‘classifier choice plays at least as important a role as feature choice’, which learning algorithms should be selected for classification in this domain? In the work reviewed here, support vector machines, used in 29 of the studies, are the most popular option—both as a common baseline, and as a default algorithm choice. Although, in NLP in general, the last decade has seen an explosion in interest in deep learning methods, here, we see relatively little use of neural network-based machine learning. Those studies that do directly compare the performance of such methods with other classifiers suggest a tendency towards better performance using neural networks [1, 12, 28, 38].
For position scaling, political and social scientists do not tend to place the same emphasis on performance metrics such as accuracy, preferring to make comparisons between output manual analyses to investigate theory-based hypotheses. Indeed, discussion of technical performance in these papers often focuses on whether or not computational text analysis is valid at all when compared with expert examination. In this respect, [17, 20, 36] conclude that, with some caveats, it is a legitimate approach (Lowe and Benoit  note that their method appears to position some speakers on a different dimension to that of their expert analysis). In the one computer science paper to tackle this problem, Glavaš et al.  report equally promising results on mono- and multilingual data, as well as superior performance using word embeddings over a bag-of-words model.
In the reviewed publications, a large range of feature types are extracted from the transcripts. Most studies rely primarily on the bag-of-words model, and for textual features, the benefits of adding higher order n-grams (bi-, tri-grams, etc.) appear inconclusive. While Plantié et al.  report improved performance with the addition of bigrams to their feature set, Abercrombie and Batista-Navarro  do not see significant improvement with the use of bi- and tri-grams. With the most common method of n-gram feature selection being TF-IDF weighting, Martineau et al. , noting that IDF favors rare features, find that, for the relatively homogenous domain of a particular parliament’s transcripts, their alternative Delta TF-IDF representation leads to better classification performance.
As we have seen, the appropriateness of using metadata features depends on the objectives of the research. However, if optimal classification performance is the goal and information regarding the speakers’ party affiliations is available, this has been found to be highly predictive of expressed sentiment [1, 66]. Inter-document relationship information regarding agreement between speakers also assists in sentiment polarity classification, and has been applied successfully by Bansal et al. , Burfoot , and Thomas et al. , as has network information [14, 15]. The latter show that it is possible to model these relationships for any data set using n-gram overlap. In another approach to modeling debate structure, Balahur et al.  use dependence parsing to find targets, which seems to improve classification and helps to balance results obtained in the positive and negative classes. While Iyyer et al.  also report success in using parse trees as features for classification with a recurring neural network, Ji and Smith  do not find improvement in the parliamentary domain (although they do in news articles).
When it comes to representing ground-truth, votes are not necessarily indicative of the opinions expressed in speeches, but for speech-level polarity analysis, they can be a convenient option. The results of computational analysis by Schwarz et al.  supports manual analysis in political science  to indicate that representatives position themselves differently in their speeches than in their voting behavior. However, the relatively small difference between votes and manual annotations (less than four per cent of their corpus) found by Abercrombie et al.  suggests that relatively small gains are to be had by investing in human labeling where other forms of class label are available.
A number of observations arise about the use of language in this domain. For the UK Parliament, Onyimadu et al.  find that ‘compound opinions’, sarcasm, and comparative structures are all confounding elements for classifiers. In German, Rauh  notes that ‘positive language is easier to detect than negative language’, while Salah et al. [67, 68] make a similar observation for the UK Hansard transcripts. The latter study explains this phenomena as an artifact of the ‘polite parliamentary jargon’ used in Parliament. This point is also backed up by Abercrombie and Batista-Navarro , who observe that the most indicative features, even of negative polarity, are words not typically thought of as conveying negativity. Where negative adjectives and verbs are present, Sokolova and Lapalme  find that these are highly discriminative features.
Conclusion and discussion
Scope for further inter-disciplinary collaboration
Considering the nature of the problem at hand—computational methods for the analysis of political text—it is somewhat surprising how little crossover can be found in this domain between ideas from computer science and political science, and how seldom the methods used by researchers from these different fields are adopted by researchers from the other disciplines. As an explanation for this, Hopkins and King , Monroe et al. , and Lowe and Benoit  provide insights into the differing aims of the two fields. However, despite these differences, researchers in computer science may well be able to benefit from the theoretical expertise of political and social scientists, such as the rigourous labeling schema and expertly coded corpora already existing in the field. Similarly, more political and social scientists could consider going beyond the simple bag-of-words n-gram language models which they currently rely on to investigate the use of more advanced NLP methods of representing text and handling feature sparsity in natural language, such as word embeddings.
Standardization of terminology
A problematic issue that arises from surveying the work included in this review is the wildly inconsistent use of terminology, even within each of the research fields represented here. There is a clear need for greater agreement on which terms to use to refer to the affective targets of interest and the names of the tasks designed to analyze them, as well as the varying levels of granularity at which analysis is performed. These inconsistencies often mean that it is difficult—or even, without further investigation, impossible—for the reader to understand just what is done in a given study.
More fine-grained level of analysis
Studies included in this review approach analysis of legislative transcripts at a wide variety of granularities, from the phrase level to comparisons aggregating sentiment over entire corpora. However, for the sake of convenience, and to make use of existing labels such as votes, the majority conduct analysis at the speech level, or even if they do so at a more fine-grained sentence or phrase level, they tend not to consider the discourse structure of the debates. As Burfoot  points out that parliamentary and legislative debates are complex, with many topics discussed and sentiment directed towards varying targets in ways that a document level classifier can struggle to identify. There is therefore room to develop more complex analyses, capable of recognizing the relationships between entities and targets in fine-grained sections of the transcripts, perhaps using argument mining methods that harnesses theories from fields such as communication theory [e.g., 48] or even philosophy [e.g., 18] to explore the relationships between actors, opinion, targets, and other entities in debates.
There have also been few attempts to link expressed opinion with topic information. While there are some efforts to do so from debate motions , at the political party level , and as a form of perspective analysis , as well as by scaling on pre-defined topic dimensions , the majority of studies simply conduct analysis of sentiment towards a target, such as a Bill or motion, the topic of which is unknown. To provide truly useful information, it may make sense to focus efforts on the extraction of topic-centric opinions and to conduct analysis at the level at which different topics are found in the data.
Use of ground-truth labels
While the majority of studies that focus on supervised classification rely on votes as ground-truth labels, it is debatable whether these actually represent the target phenomena—the opinion or position taken by the speaker. Manual analysis in political science  certainly suggests that, in many legislatures, representatives express different positions in speech than in their votes, a point supported by Schwarz et al. , who compare the scaling of speeches and votes. However, as Abercrombie and Batista-Navarro  point out, gains made from seeking more reliable ground-truth may be small and not worth the associated costs in a practical setting. An alternative approach, for which there is still plenty of scope for further research, is to develop semi-supervised or unsupervised approaches, which require few or no labels.
Other possibilities for the direction of future work
There also exist other possible directions for future work. With an increasing quantity of transcript data becoming available, in the case of some parliaments stretching back over hundreds of years, one such possibility is the analysis of language change over time. While Diermeier et al.  suggest using changes in classification performance to infer changes in agenda control, and Kapočiūtė-Dzikienėa and Krupavičius  found that performance worsened when transcripts from different sessions were used for training and testing, language drift in parliamentary debates remains relatively unexplored. Although Ahmadalinezhad and Makrehchi  note a performance drop when training and testing on different debate data (Canadian Hansard and US election debates), domain adaptation and inter-legislature transfer learning also remains under-explored. Additionally, given the successes achieved with neural networks and deep learning in other domains, as well as the results reported by studies that use such methods, there would appear to be considerable scope for further investigation of their application to legislative debates. Finally, while some of the included studies mention potential applications of their work for civic technology [e.g., 12, 13], with the exception of Bonica’s  CrowdPac Voter Guide, as far as we know, the methods used are not currently being applied to any real-world systems. There is, therefore, room to explore the application of the approaches used to the area of civic technology, and provide tools that could genuinely assist people in processing information about their elected representatives.
The computational analysis of sentiment and position-taking in parliamentary debate transcripts is an area of growing interest. While the researchers working on this problem have varied backgrounds and objectives, in this review, we have identified some of the common challenges which they face. With the majority of work emanating from computer science focusing on unknown targets (Bills or debate motions, the topic of which is not assessed), and political scaling being conducted on very coarse-grained scales (left-right, pro/anti-EU), there has thus far been little effort to direct efforts towards examining the targets of the opinions expressed. For the aims of both political scholarship and civic technology, what is required in many cases is identification of these targets, namely the policies and policy preferences that are discussed in the legislative chambers. It is our belief, therefore, that future work should be directed towards such target-specific analyses.
See Quirk et al. .
To reduce the number of results returned, we also added exclusion terms such as ‘big data’ and ‘twitter’, which we found did not prevent the retrieval of any known relevant publications.
In several cases, it has been necessary to contact the authors for clarification or to manually examine the data sets used to obtain this information.
That is, unstructured, unordered arrays of n-gram (term) counts.
Terms used in the US Congress and UK Parliament, respectively.
Compliance with ethical standards
Conflict of interest
The authors declare that they have no conflict of interest.
- 1.Abercrombie, G., & Batista-Navarro, R. (2018). ‘Aye’ or ‘no’? Speech-level sentiment analysis of Hansard UK parliamentary debate transcripts. In: Proceedings of the eleventh international conference on language resources and evaluation (LREC-2018). European Languages Resources Association (ELRA), Miyazaki, Japan. https://www.aclweb.org/anthology/L18-1659.
- 2.Abercrombie, G., & Batista-Navarro, R.T. (2018). Identifying opinion-topics and polarity of parliamentary debate motions. In: Proceedings of the 9th workshop on computational approaches to subjectivity, sentiment and social media analysis. Association for Computational Linguistics, Brussels, Belgium (pp. 280–285). https://doi.org/10.18653/v1/W18-6241. https://www.aclweb.org/anthology/W18-6241.
- 4.Akhmedova, S., Semenkin, E., & Stanovov, V. (2018). Co-operation of biology related algorithms for solving opinion mining problems by using different term weighting schemes. In: K. Madani, D. Peaucelle, O. Gusikhin (Eds.) Informatics in control, automation and robotics: 13th international conference, ICINCO 2016 Lisbon, Portugal, 29-31 July, 2016 (pp. 73–90). Cham: Springer. https://doi.org/10.1007/978-3-319-55011-4_4.Google Scholar
- 7.Bansal, M., Cardie, C., & Lee, L. (2008). The power of negative thinking: Exploiting label disagreement in the min-cut classification framework. In: Coling 2008: Companion volume: Posters (pp. 15–18). Coling 2008 Organizing Committee, Manchester, UK. https://www.aclweb.org/anthology/C08-2004.
- 9.Bhatia, S., P, D. (2018). Topic-specific sentiment analysis can help identify political ideology. In: Proceedings of the 9th workshop on computational approaches to subjectivity, sentiment and social media analysis (pp. 79–84). Association for Computational Linguistics, Brussels, Belgium. https://doi.org/10.18653/v1/W18-6212. https://www.aclweb.org/anthology/W18-6212.
- 10.Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3(Jan), 993–1022.Google Scholar
- 11.Bonica, A. (2016). A data-driven voter guide for US elections: Adapting quantitative measures of the preferences and priorities of political elites to help voters learn about candidates. Journal of the Social Sciences, 2(7), 11–32. https://doi.org/10.7758/RSF.2016.2.7.02. https://www.rsfjournal.org/content/2/7/11.CrossRefGoogle Scholar
- 12.Budhwar, A., Kuboi, T., Dekhtyar, A., & Khosmood, F. (2018). Predicting the vote using legislative speech. In: Proceedings of the 19th annual international conference on digital government research: governance in the data age, dg.o ’18 (pp. 35:1–35:10). ACM, New York, NY, USA. https://doi.org/10.1145/3209281.3209374.
- 13.Burfoot, C. (2008). Using multiple sources of agreement information for sentiment classification of political transcripts. In: Proceedings of the Australasian language technology association workshop 2008 (pp. 11–18). Hobart, Australia. https://www.aclweb.org/anthology/U08-1003.
- 14.Burfoot, C., Bird, S., & Baldwin, T. (2011). Collective classification of congressional floor-debate transcripts. In: Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies (pp. 1506–1515). Association for Computational Linguistics, Portland, Oregon, USA. https://www.aclweb.org/anthology/P11-1151.
- 15.Burford, C., Bird, S., & Baldwin, T. (2015). Collective document classification with implicit inter-document semantic relationships. In: Proceedings of the fourth joint conference on lexical and computational semantics (pp. 106–116). Association for Computational Linguistics, Denver, Colorado. https://doi.org/10.18653/v1/S15-1012. https://www.aclweb.org/anthology/S15-1012.
- 16.Chen, W., Zhang, X., Wang, T., Yang, B., & Li, Y. (2017). Opinion-aware knowledge graph for political ideology detection. In: Proceedings of the 26th international joint conference on artificial intelligence, pp. 3647–3653.Google Scholar
- 18.Duthie, R., & Budzynska, K. (2018). A deep modular rnn approach for ethos mining. In: Proceedings of the twenty-seventh international joint conference on artificial intelligence, (IJCAI-18), pp. 4041–4047.Google Scholar
- 21.Glavaš, G., Nanni, F., & Ponzetto, S.P. (2017). Unsupervised cross-lingual scaling of political texts. In: Proceedings of the 15th conference of the European chapter of the association for computational linguistics: Volume 2, short papers (pp. 688–693). Association for Computational Linguistics, Valencia, Spain. https://www.aclweb.org/anthology/E17-2109.
- 22.Glavaš, G., Nanni, F., & Ponzetto, S.P. (2019). Computational analysis of political texts: Bridging research efforts across communities. In: Proceedings of the 57th annual meeting of the association for computational linguistics: Tutorial abstracts (pp. 18–23). Association for Computational Linguistics, Florence, Italy. https://doi.org/10.18653/v1/P19-4004. https://www.aclweb.org/anthology/P19-4004.
- 24.Hirst, G., Riabinin, Y., & Graham, J. (2010). Party status as a confound in the automatic classification of political speech by ideology. In: Proceedings of 10th international conference on statistical analysis of textual data/10es Journées internationales d’Analyse statistique des Données Textuelles (JADT 2010), Rome, pp. 731–742.Google Scholar
- 25.Honkela, T., Korhonen, J., Lagus, K., & Saarinen, E. (2014). Five-dimensional sentiment analysis of corpora, documents and words. In T. Villmann, F. M. Schleif, M. Kaden, & M. Lange (Eds.), Advances in self-organizing maps and learning vector quantization (pp. 209–218). Cham: Springer.CrossRefGoogle Scholar
- 28.Iyyer, M., Enns, P., Boyd-Graber, J., & Resnik, P. (2014). Political ideology detection using recursive neural networks. In: Proceedings of the 52nd annual meeting of the association for computational linguistics (Volume 1: Long Papers) (pp. 1113–1122). Association for Computational Linguistics, Baltimore, Maryland. https://doi.org/10.3115/v1/P14-1105. https://www.aclweb.org/anthology/P14-1105
- 29.Jensen, J., Naidu, S., Kaplan, E., Wilse-Samson, L., Gergen, D., Zuckerman, M., & Spirling, A. (2012). Political polarization and the dynamics of political language: Evidence from 130 years of partisan speech [with comments and discussion]. Brookings Papers on Economic Activity, pp. 1–81.Google Scholar
- 30.Ji, Y., & Smith, N.A. (2017) Neural discourse structure for text categorization. In: Proceedings of the 55th annual meeting of the association for computational linguistics (Volume 1: Long Papers) (pp. 996–1005). Association for Computational Linguistics, Vancouver, Canada. https://doi.org/10.18653/v1/P17-1092. https://www.aclweb.org/anthology/P17-1092.
- 33.Kauffman, D., Khosmood, F., Kuboi, T., & Dekhtyar, A. (2018). Learning alignments from legislative discourse. In: Proceedings of the 19th annual international conference on digital government research: Governance in the data age, dg.o ’18 (pp. 119:1–119:2). ACM, New York, NY, USA. https://doi.org/10.1145/3209281.3209413.
- 37.Lefait, G., & Kechadi, T. (2010). Analysis of deputy and party similarities through hierarchical clustering. In: 2010 fourth international conference on digital society (pp. 264–268). https://doi.org/10.1109/ICDS.2010.49.
- 39.Liu, B. (2012). Sentiment analysis and opinion mining, synthesis lectures on human language technologies (Vol. 5). San Rafael: Morgan & Claypool Publishers.Google Scholar
- 41.Martineau, J., Finin, T., Joshi, A., & Patel, S. (2009). Improving binary classification on text problems using differential word features. In: Proceedings of the 18th ACM conference on information and knowledge management, CIKM ’09 (pp. 2019–2024). ACM, New York, NY, USA. https://doi.org/10.1145/1645953.1646291.
- 42.Menini, S., Nanni, F., Ponzetto, S.P., & Tonelli, S. (2017). Topic-based agreement and disagreement in US electoral manifestos. In: Proceedings of the 2017 conference on empirical methods in natural language processing (pp. 2938–2944). Association for Computational Linguistics, Copenhagen, Denmark. https://doi.org/10.18653/v1/D17-1318. https://www.aclweb.org/anthology/D17-1318.
- 43.Menini, S., & Tonelli, S. (2016). Agreement and disagreement: Comparison of points of view in the political domain. In: Proceedings of COLING 2016, the 26th international conference on computational linguistics: Technical papers (pp. 2461–2470). The COLING 2016 Organizing Committee, Osaka, Japan. https://www.aclweb.org/anthology/C16-1232.
- 44.Mikhaylov, S., Laver, M., & Benoit, K. (2008). Coder reliability and misclassification in comparative manifesto project codings. In: 66th MPSA annual national conference.Google Scholar
- 46.Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). The PRISMA group: Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151(4), 264–269. https://doi.org/10.7326/0003-4819-151-4-200908180-00135.CrossRefGoogle Scholar
- 49.Nanni, F., Zirn, C., Glavaš, G., Eichorst, J., & Ponzetto, S.P. (2016) Topfish: topic-based analysis of political position in us electoral campaigns. In: PolText 2016: The international conference on the advances in computational analysis of political text: proceedings of the conference.Google Scholar
- 50.Nguyen, V.A., Boyd-Graber, J., Resnik, P., & Miler, K. (2015). Tea party in the house: A hierarchical ideal point topic model and its application to republican legislators in the 112th congress. In: Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (Volume 1: Long papers) (pp. 1438–1448). Association for Computational Linguistics, Beijing, China. https://doi.org/10.3115/v1/P15-1139. https://www.aclweb.org/anthology/P15-1139.
- 51.Nguyen, V. A., Ying, J. L., & Resnik, P. (2013). Lexical and hierarchical topic regression. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, & K. Q. Weinberger (Eds.), Advances in neural information processing systems 26 (pp. 1106–1114). Curran Associates Inc. http://papers.nips.cc/paper/5163-lexical-and-hierarchical-topic-regression.pdf.
- 59.Quirk, R., Greenbaum, S., Leech, G., & Svartvik, J. (1985). A comprehensive grammar of the english language. London: Longman.Google Scholar
- 63.Richards, L. (2005). Handling qualitative data: A practical guide. London: Sage Publications.Google Scholar
- 65.Sakamoto, T., & Takikawa, H. (2017). Cross-national measurement of polarization in political discourse: Analyzing floor debate in the US the japanese legislatures. In 2017 IEEE international conference on big data (Big Data) (pp. 3104–3110). https://doi.org/10.1109/BigData.2017.8258285.
- 66.Salah, Z. (2014). Machine learning and sentiment analysis approaches for the analysis of parliamentary debates. Ph.D. thesis, University of Liverpool.Google Scholar
- 67.Salah, Z., Coenen, F., & Grossi, D. (2013). Extracting debate graphs from parliamentary transcripts: A study directed at uk house of commons debates. In Proceedings of the fourteenth international conference on artificial intelligence and law, ICAIL ’13 (pp. 121–130). ACM, New York, NY, USA. https://doi.org/10.1145/2514601.2514615.
- 70.Seligman, M. E. P. (2012). Flourish: A visionary new understanding of happiness and well-being. New York: Simon and Schuster.Google Scholar
- 71.Sim, Y., Acree, B.D.L., Gross, J.H., & Smith, N.A. (2013). Measuring ideological proportions in political speeches. In Proceedings of the 2013 conference on empirical methods in natural language processing (pp. 91–101). Association for Computational Linguistics, Seattle, Washington, USA. https://www.aclweb.org/anthology/D13-1010.
- 74.Thomas, M., Pang, B., & Lee, L. (2006). Get out the vote: Determining support or opposition from congressional floor-debate transcripts. In Proceedings of the 2006 conference on empirical methods in natural language processing (pp. 327–335). Association for Computational Linguistics, Sydney, Australia. https://www.aclweb.org/anthology/W06-1639.
- 75.van der Zwaan, J.M., Marx, M., & Kamps, J. (2016). Validating cross-perspective topic modeling for extracting political parties’ positions from parliamentary proceedings. In Proceedings of the twenty-second European conference on artificial intelligence, ECAI’16 (pp. 28–36). IOS Press, Amsterdam, The Netherlands, The Netherlands. https://doi.org/10.3233/978-1-61499-672-9-28.
- 76.Vilares, D., & He, Y. (2017). Detecting perspectives in political debates. In Proceedings of the 2017 conference on empirical methods in natural language processing (pp. 1573–1582). Association for Computational Linguistics, Copenhagen, Denmark. https://doi.org/10.18653/v1/D17-1165. https://www.aclweb.org/anthology/D17-1165.
- 78.Yessenalina, A., Yue, Y., & Cardie, C. (2010). Multi-level structured models for document-level sentiment classification. In Proceedings of the 2010 conference on empirical methods in natural language processing (pp. 1046–1056). Association for Computational Linguistics, Cambridge, MA. https://www.aclweb.org/anthology/D10-1102.
- 79.Yogatama, D., Kong, L., & Smith, N.A. (2015). Bayesian optimization of text representations. In Proceedings of the 2015 conference on empirical methods in natural language processing (pp. 2100–2105). Association for Computational Linguistics, Lisbon, Portugal. https://doi.org/10.18653/v1/D15-1251. https://www.aclweb.org/anthology/D15-1251.
- 80.Yogatama, D., & Smith, N. (2014). Making the most of bag of words: Sentence regularization with alternating direction method of multipliers. In International conference on machine learning, pp. 656–664.Google Scholar
- 81.Yogatama, D., & Smith, N.A. (2014). Linguistic structured sparsity in text categorization. In Proceedings of the 52nd annual meeting of the association for computational linguistics (Volume 1: Long Papers) (pp. 786–796). Association for Computational Linguistics, Baltimore, Maryland. https://doi.org/10.3115/v1/P14-1074. https://www.aclweb.org/anthology/P14-1074.
Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.