Controlling Hallucinations at Word Level in Data-to-Text Generation

Data-to-Text Generation (DTG) is a subfield of Natural Language Generation aiming at transcribing structured data in natural language descriptions. The field has been recently boosted by the use of neural-based generators which exhibit on one side great syntactic skills without the need of hand-crafted pipelines; on the other side, the quality of the generated text reflects the quality of the training data, which in realistic settings only offer imperfectly aligned structure-text pairs. Consequently, state-of-art neural models include misleading statements - usually called hallucinations - in their outputs. The control of this phenomenon is today a major challenge for DTG, and is the problem addressed in the paper. Previous work deal with this issue at the instance level: using an alignment score for each table-reference pair. In contrast, we propose a finer-grained approach, arguing that hallucinations should rather be treated at the word level. Specifically, we propose a Multi-Branch Decoder which is able to leverage word-level labels to learn the relevant parts of each training instance. These labels are obtained following a simple and efficient scoring procedure based on co-occurrence analysis and dependency parsing. Extensive evaluations, via automated metrics and human judgment on the standard WikiBio benchmark, show the accuracy of our alignment labels and the effectiveness of the proposed Multi-Branch Decoder. Our model is able to reduce and control hallucinations, while keeping fluency and coherence in generated texts. Further experiments on a degraded version of ToTTo show that our model could be successfully used on very noisy settings.

level labels to learn the relevant parts of each training instance. These labels are obtained following a simple and efficient scoring procedure based on co-occurrence analysis and dependency parsing. Extensive evaluations, via automated metrics and human judgment on the standard WikiBio benchmark, show the accuracy of our alignment labels and the effectiveness of the proposed Multi-Branch Decoder.
Our model is able to reduce and control hallucinations, while keeping fluency and coherence in generated texts. Further experiments on a degraded version of ToTTo show that our model could be successfully used on very noisy settings.
Keywords Data-to-Text Generation · Hallucinations · Controlled Text Generation

Introduction
Data-to-Text Generation (DTG) is the subfield of Computational Linguistics and Natural Language Generation (NLG) that is concerned with transcribing structured data into natural language descriptions, or, said otherwise, transcribing machine understandable information into a human understandable description [15]. DTG objectives includes coverage, i.e. all the required information should be present in the text, and adequacy, i.e. the text should not contain information that is not covered by the input data. DTG is a domain distinct from other NLG task (e.g. machine translation [70], text summarization [28]) with its own challenges [70], starting with the nature of inputs [55,40]. Such inputs include and are not limited to: databases of records, spreadsheets, knowledge bases, sensor readings. As an example, Fig. 1 shows an instance of the WikiBio dataset, i.e. a data table containing information about Kian Emadi, paired with its corresponding natural language description found on Wikipedia.
Early approaches to DTG relied on static rules hand-crafted by experts, in which content selection (what to say) and surface realization (how to say it) are typically two separate tasks [55,10]. In recent years, neural models have blurred this distinction: various approaches showed that both content selection and surface realization can be learned in an end-to-end, data-driven fashion [38,34,56,49]. Based on the now-standard encoder-decoder architecture, with attention and copy mechanisms [1,59], neural methods for DTG are able to produce fluent text conditioned on structured data in a number of domains [29,70,50], without relying on heavy manual work from field experts.
Such advances have gone hand in hand with the introduction of larger and more complex benchmarks. In particular, surface-realization abilities have been well studied on hand-crafted datasets such as E2E [43] and WebNLG [14], while content-selection has been addressed by automatically constructed dataset such as WikiBio [29] or RotoWire [70]. These large corpora are often constructed from internet sources, which, while easy to access and aggregate, do not consist of perfectly aligned source-target pairs [46,7]. Consequently, model outputs are often subject to over-generation: misaligned fragments from training instances, namely divergences, can induce similarly misaligned outputs during inference, the so-called hallucinations.
In this paper, we specifically address the issue of hallucinations, which is currently regarded as a major issue in DTG [40]. Indeed  that real-life end-users of DTG systems care more about reliability than about readability [54], as unfaithful texts can potentially mislead decision makers, with dire consequences. Hallucinations-reduction methods such as the one presented here have applications in a broad range of tasks requiring high reliability, like news reports [31], in which hallucinations may give rise to fake news, or summaries of patient information in clinical contexts [48,2]. When corpora include a mild amount of noise, as in handcrafted ones (e.g. E2E, WebNLG), dataset regularization techniques [41,9] or hand crafted rules [21] can help to reduce hallucinations. Unfortunately, these techniques are not suited to more realistic and noisier datasets, as for instance WikiBio [29] or RotoWire [70]. On these benchmarks, several techniques have been proposed, such as reconstruction loss terms [70,68,33] or Reinforcement Learning (RL) based methods [47,35,52]. These approaches suffer however from different issues: (1) the reconstruction loss relies on the hypothesis of one-to-one alignment between source and target which does not fit with content selection in DTG; (2) RL-trained models are based on instance-level rewards (e.g. BLEU [44], PARENT [7]) which can lead to a loss of signal because divergences occur at the word level. In practice, parts of the target sentence express source attributes (in Fig. 1 name and occupation fields are correctly realized), while others diverge (the birthday and nationality of Kian Emadi are not supported by the source table).
Interestingly, one can view DTG models as Controlled Text Generation (CTG) ones focused on controlling content, as most CTG techniques condition the generation on several key-value pairs of control factors (e.g. tone, tense, length) [8,19,11]. Recently, Filippova [12] explicitly introduced CTG to DTG by leveraging an hallucination score simply attached as an additional attribute which reflects the amount of noise in the instance. As an example, the table from Fig 1 can be augmented with an additional line (hallucination_score, 80%) 1 . However, this approach requires a strict alignment at the instance-level, namely between control factors and target text. A first attempt towards word-level approaches is proposed by Perez-Beltrachini and Lapata [47] (also PB&L in the following). They design word-level alignment labels, denoting the correspondence between the text and the input table, to bootstrap DTG systems. However, they incorporate these labels into a sentence-level RL-reward, which ultimately leads to a loss of this finer-grained signal.
In this paper, we go further in this direction with a DTG model by fully leveraging word-level alignment labels with a CTG perspective. We propose an original approach in which the word-level is integrated at all phases: we propose a word-level labeling procedure (Section 3), based on cooccurrences and sentence structure through dependency parsing. This mitigates the failure of strict word-matching procedure, while still producing relevant labels in complex settings. we introduce a weighted multi-branch neural decoder(Section 4), guided by the proposed alignment labels, acting as word-level control factors. During training, the model is able to distinguish between aligned and unaligned words and learns to generate accurate descriptions without being misled by un-factual reference information. Furthermore, our multi-branch weighting approach enables control at inference time.
We carry out extensive experiments on WikiBio, to evaluate both our labeling procedure and our decoder (Section 6). We also test our framework on ToTTo [45], in which models are trained with noisy reference texts, and evaluated on references reviewed and cleaned by human annotators to ensure accuracy. Evaluations are based on a range of automated metrics as well as human judgments, and show increased performances regarding hallucinations reduction, while preserving fluency. Importantly, our approach makes training neural models on noisy datasets possible, without the need to handcraft instances. This work shows the benefit of word-level techniques, which leverage the entire training set, instead of removing problematic training samples, which may form the great majority of the available data.

Related work
Handling hallucinations in noisy datasets. The use of Deep Learning based methods to solve DTG tasks has led to sudden improvements in state of the art performances [29,70,36,49]. As a key aspect in determining a model's performance is the quality of training data, several large corpora have been introduced to train and evaluate models' abilities on diverse tasks. E2E [43] evaluates surface realization, i.e. the strict transcription of input attributes into natural language; RotoWire [70] pairs statistics of basketball games with their journalistic descriptions, while WikiBio [29] maps a Wikipedia info-box with the first paragraph of its associated article. Contrary to E2E, the latter datasets are not limited to surface realization. They were not constructed by human annotators, but rather created from Internet sources, and consist of loosely aligned table-reference pairs: in Wik-iBio, almost two thirds of the training instances contain divergences [7], and no instance has a 1-to-1 source-target alignment [46].
On datasets with a moderate amount of noise, such as E2E, data pre-processing has proven effective for reducing hallucinations. Indeed, rule-based [9] or neuralbased methods [41] have been proposed, specifically with table regularization techniques, where attributes are added or removed to re-align table and target descrip-tion. Several successful attempts have also been made in automatically learning  alignments between the source tables and reference texts, benefiting from the regularity of the examples [21,61,17]. For instance, Juraska et al. [21] leverage templating and hand-crafted rules to re-rank the top outputs of a model decoding via beam search; Gehrmann et al. [17] also leverage the possible templating formats of E2E's reference texts, and train an ensemble of decoders where each decoder is associated to one template; and Kasner and Dusek [23] produce template-based lexicalizations and improve them via a sentence fusion model. The previous techniques are not applicable in more complex, general settings. The work of Dusek et al. [9] hints at this direction, as authors found that neural models trained on E2E were principally prone to omissions rather than hallucinations. In this direction, Shen et al. [61] were able to obtain good results at increasing the coverage of neural outputs, by constraining the decoder to focus its attention exclusively on each table cell sequentially until the whole table was realized. On more complex datasets (e.g. WikiBio), a wide range of methods has been explored to deal with factualness such as loss design, either with a reconstruction term [70,68] or with RL-based methods [47,35,52]. Similarly to the coverage constraints, a reconstruction loss has proven only marginally efficient in these settings, as it contradicts the content selection task [68], and needs to be well calibrated using expert insight in order to bring improvements. Regarding [7] using self-critical RL. Note that data regularization techniques have also been proposed [65,68], but these methods require heavy manual work and expert insights, and are not readily transposable from one domain to another.
From CTG to controlling hallucinations. Controlled Text Generation (CTG) is concerned with constraining a language model's output during inference on a number of desired attributes, or control factors, such as the identity of the speaker in a dialog setting [32], the politeness of the generated text or the text length in machine-translation [60,24], or the tense in generated movie reviews [19]. Earlier attempts at neural CTG can even be seen as direct instances of DTG as it is currently defined: models are trained to generate text conditioned on attributes of interest, where attributes are key-value pairs. For instance, in the movie review domain, Ficler and Goldberg [11] proposed an expertly crafted dataset, where sentences are strictly aligned with control factors, being either content or linguistic style aspects (e.g. tone, length).
In the context of dealing with hallucinations in DTG, Filippova [12] recently proposed a similar framework, by augmenting source tables with an additional attribute that reflects the degree of hallucinated content in the associated target description. During inference, this attribute acts as an hallucination handle used to produce more or less factual text. As mentioned in Section 1, we argue that a unique value can not accurately represent the correspondence between a table and its description, due to the phrase-based nature of divergences.
Based on the literature review, the lack of model control can be evidenced when loss modification methods are used [68,34,52], although these approaches can be efficient and transposed from one domain to another. On the other hand, while Fig. 2 The reference sentence of the example shown in Fig. 1. Every token is associated to its Part-of-Speech tag and hallucination score st. Words in red denote st < τ . The dependency parsing is represented by labeled arrows that flow from parents to children. Important words are kian, emadi, 29, july, 1992, british, track, and cyclist.
CTG deals with control and enables choosing the defining features of generated texts [12], standard approaches rely on instance-level control factors that do not fit with hallucinations, which rather appear due to divergences at the word level. Our approach aims at gathering the merits of both trends of models and is guided by previous statements highlighting that word-level is primary in hallucination control. More particularly, our model differs from previous ones in several aspects: (1) Contrasting with data-driven approaches (i.e. dataset regularization) which are costly in expert time, and loss-driven approaches (i.e. reconstruction or RL losses) which often do not take into account key subtasks of DTG (contentselection, world-level correspondences), we propose a multi-branch modeling procedure which allows the controllability of the hallucination factor in DTG. This multi-branch model can be integrated seamlessly in current approaches, allowing to keep peculiarities of existing DTG models, while deferring hallucination management to a parallel decoding branch. (2) Unlike previous CTG approaches [32, 60,11,12] which propose instance-level control factors, the control of the hallucination factor is performed at the wordlevel to enable finer-grained signal to be sent to the model.
Our model is composed of two main components: (1) a word-level alignment labeling mechanism, which makes the correspondence between the input table and the text explicit, and (2) a multi-branch decoder guided by these alignment labels. The branches separately integrate co-dependent control factors (namely content, hallucination and fluency). We describe these components in Sections 3 and 4, respectively.

Word-level Alignment Labels
We consider a DTG task, in which the corpus C is composed of a set of entitydescription pairs, (e, y). A single-entity table e is a variable-sized set of T e key-value pairs x := (k, v). A description y := y 1:T y is a sequence of T y tokens representing the natural language description of the entity; we refer to the tokens spanning from indices t to t of a description y as y t:t . A description is made of statements, defined as text spans expressing one single idea (Appendix A presents in detail the statement partitioning procedure). We refer to the first index of a statement as t i , so that y t i :t i+1 −1 is the i th statement itself. Fig. 1 shows a WikiBio entity made by 8 key-value pairs together with its associated description.
First, we aim at labeling each word from a description, depending on the presence of a correspondence with its associated table. We call such labels alignment labels. We drive the word-level labeling procedure on two intuitive constraints: (1) important words (names, adjectives and numbers) should be labeled depending on their alignment with the table, and (2) words from the same statement should have the same label.
With this in mind, the alignment label for the t th token y t is a binary label: l t := 1 {s t >τ } where s t refers to the alignment score between y t and the table, and τ is set experimentally (see Sec. 5.3). The alignment score s t acts as a normalized measure of correspondence between a token y t and the table e: where the function align estimates the alignment between token y t and a key-value pair x from the input table e, and norm is a normalization function based on the dependency structure of the description y. Fig. 2 illustrates our approach: under each word we show its word alignment score, and words are colored in red if this score is lower than τ , denoting an alignment label equal to 0. Below, we describe these functions (Appendix A contains reproducibility details).
Co-occurrence-based alignment function (align(·, x)). This function assigns to important words a score in the interval [0, 1] proportional to their co-occurrence count (a proxy for alignment) with the key-value pair from the input table. If the word y t appears in the key-value pair x := (k, v), align(y t , x) outputs 1; otherwise, the output is obtained scaling the number of occurrences co y t ,x between y t and x through the dataset: where M is the maximum number of word co-occurrences in the dataset vocabulary and the row x, m is a threshold value, and a := 1 (M −m) 2 . Score normalization (norm(·, y)). According to the already stated assumption (2) -words inside the same statement should have the same score -, we first split the sentence y into statements y t i :t i+1 −1 , via dependency parsing and its rule-based conversion to constituency trees [18,73,20,3]. Given a word y t associated to the score s t and belonging to statement y t i :t i+1 −1 , its normalized score corresponds to the average score of all important words in this statement: This in-statement average depends on both the specific word and its context, leading to coherent hallucination scores which can be thresholded without affecting the syntactical sentence structure, as shown in Fig. 2.

Multi-Branch Architecture
The proposed Multi-Branch Decoder (MBD) architecture aims at separating targeted co-dependent factors during generation. We build upon the standard DTG architecture, an encoder-decoder with attention and copy mechanism, which we modify by duplicating the decoder module into three distinct parallel modules. Each control factor (i.e. content, hallucination or fluency) is modeled via a single decoding module, also called branch, whose output representation can be weighted according to its desired importance. At training time, weights change depending on the word currently being decoded, inducing the desired specialization of each branch. During inference, weights are manually set, according to the desired tradeoff between information reliability, sentence diversity and global fluency. Text generation is thus controllable, and consistent with the control factors. Figure 3 illustrates a training step over the sentence "Giuseppe Mariani was an Italian art director ", in which Italian is a divergent statement (i.e. is not supported by the source table). While decoding factual words, the weight associated to the content (resp. hallucination) branch is set to 0.5 (resp. 0) while during the decoding of Italian, the weight associated to the content (resp. hallucination) branch is set to 0 (resp. 0.5). Note that the weight associated to the fluency branch is always set to 0.5, as fluency does not depend on factualness.
The decoding modules' actual architecture may vary, as we framed the MBD model from a high level perspective. Therefore, all types of decoder can be used, such as Recurrent Neural Networks (RNNs) [57], Transformers [67], and Convolutional Neural Networks [16]. The framework can be generalized to different merging strategies as well, such as late fusion, in which the final distributions are merged, instead of the presented early fusion, which works at the decoder states level.
In this paper, experiments are carried out on RNN-based decoders, weighting their hidden states. Section 4.1 presents the standard DTG encoder-decoder architecture; Section 4.2 shows how it can be extended to MBD, together with its peculiarities and the underlying objectives and assumptions.

Standard DTG architecture
Neural DTG approaches typically use an encoder-decoder architecture [70] in which (1) the encoder relies on a RNN to encode each element of the source table into a fixed-size latent representation h j (elements of the input table are first embedded into T e N -dimensional vectors, and then fed sequentially to the RNN [70]), and (2) the decoder generates a textual description y using a RNN augmented with attention and copy mechanisms [59]. Words are generated in an auto-regressive way. The decoder's RNN updates its hidden state d t as: where y t−1 is the previous word and c t is the context vector obtained through the attention mechanism. Finally, a word is drawn from the distribution computed via a copy mechanism [59].

Controlling Hallucinations via a Multi-Branch Model
Our objective is to enrich the decoder in order to be able to tune the content/hallucination ratio during generation, aiming at enabling generation of hallucination-free text when needed. Our key assumption is that the decoder's generation is conditioned by three co-dependent factors: -Content factor constrains the generation to realize only the information included in the input; -Hallucinating factor favors lexically richer and more diverse text, but may lead to hallucinations not grounded by the input; -Fluency factor 2 conditions the generated sentences toward global syntactic correctness, regardless of the relevance. Based on this assumption, we propose a multi-branch encoder-decoder network, whose branches are constrained on the above factors at word-level, as illustrated in Fig. 3. Our network has a single encoder and F = 3 distinct decoding RNNs, noted RNN f respectively, one for each factor. During each decoding step, the previously decoded word y t−1 is fed to all RNNs, and a final decoder state d t is computed using a weighted sum of all the corresponding hidden states, where d f t and ω f t are respectively the hidden state and the weight of the f th RNN at time t.
Weights are used to constrain the decoder branches to the desired control factors (ω 0 t , ω 1 t , ω 2 t for the content, hallucination and fluency factors respectively) and sum to one.
During training, their values are dynamically set depending on the alignment label l t ∈ {0, 1} of the target token y t (see Sec. 5.3). While a number of mappings can be used to set the weights given the alignment label, early experiments have shown that better results were achieved when using a binary switch for each factor, i.e. activating/deactivating each branch, as shown in Fig. 3 (note that fluency should not depend on content and therefore its associated branch is always active).
During inference, the weights of the decoder's branches are set manually by a user, according to the desired trade-off between information reliability, sentence diversity and global fluency. Text generation is then controllable and consistent with the control factors.

Datasets
We evaluated the model on two representative large size datasets. Both have been collected automatically and present a significant amount of table-text divergences for training. Both datasets involve content selection and surface realization, and represent a relatively realistic setting.
WikiBio [29] contains 728, 321 tables, automatically paired with the first sentence of the corresponding Wikipedia English article. Reference text's average length is 26 words, and tables have on average 12 key-value pairs. We use the original data partition: 80% for the train set, and 10% for validation and test sets. This dataset has been automatically built from the Internet; concerning divergences, 62% of the references mention extra information not grounded by the table [7].  (1) highlighting the input cells realized by the output, and (2) removing divergences and making the sentence self-contained (e.g. replacing pronouns with their invoked noun or noun phrase). Fig. 6 exemplifies the difference between noisy and clean ToTTo sentences. In our experiments, we limit the input to the highlighted cells and use the original, noisy sentence as output. Noisy texts' average length is 17.4 words, and 3.55 table cells are highlighted, on average.

Baselines
We assess the accuracy and relevance of our alignment labels against the ones proposed by Perez-Beltrachini and Lapata [47], which is, to the best of our knowledge, the only work proposing such a fine-grained alignment labeling.
To evaluate our Multi-Branch Decoder (MBD), we consider five baselines: stnd [59], a LSTM-based encoder-decoder model with attention and copy mechanisms. This is the standard sequence-to-sequence recurrent architecture.
stnd filtered, the previous model trained on a filtered version of the training set: tokens deemed hallucinated according to their hallucination scores, are removed from target sentences.
hsmm [71], an encoder-decoder model with a multi-branch decoder. The branches are not constrained by explicit control factors. This is used as a baseline to show that the multi-branch architecture by itself does not guarantee the absence of hallucinations.
hier [34], a hierarchical sequence-to-sequence model, with a coarse-to-fine attention mechanism to better fit the attribute-value structure of the tables. This model is trained with three auxiliary tasks to capture more accurate semantic representations of the tables. hal WO [12], a stnd -like model trained by augmenting each source table with an additional attribute (hallucination ratio, value).
We ran our own implementations of stnd, stnd filtered and hal WO . Authors of hier and hsmm models kindly provided us their WikiBio's test set outputs. The metrics described in Sec. 5.4 were directly applied on them.

Implementation Details
During training of our multi-branch decoder the fluency branch is always active (ω 2 t = 0.5) while the content and hallucination branches are alternatively acti-vated, depending on the alignment label l t : ω 0 t = 0.5 (content factor) and ω 1 t = 0 (hallucination factor) when l t = 1, and conversely. The threshold τ used to obtain l t is set to 0.4 using human tuning to optimize for highest accuracy 3 . All hyperparameters were tuned in order to optimize the validation PARENT F-measure [7]. In particular, we use the [0.4 0.1 0.5] weight combination during inference. See Sec. 6.2 for a discussion about weight combinations and Appendix B for other implementation details. 4

Metrics
To evaluate our model, we carried out (1) an automatic analysis and (2) a human evaluation for a qualitative analysis of generated sentences.
For the automatic analysis, we use five metrics: -BLEU [44] is a length-penalized precision score over n-grams, n ∈ 1, 4 , optionally improved with a smoothing technique [4]. Despite being the standard choice, recent findings show that it correlates poorly with human evaluation, especially on the sentence level [42,53], and that it is a proxy for sentence grammar and fluency aspects rather than semantics [7].
-PARENT [7] computes smoothed n-gram precision and recall over both the reference and the input table. It is explicitly designed for DTG tasks, and its Fmeasure shows "the highest correlation with humans across a range of settings with divergent references in WikiBio." [7] -The hallucination rate computes the percentage of tokens labeled as hallucinations (Sec. 3).
-The average generated sentence length in number of words.
-The classic readability Flesch index [13], which is based on words per sentence and syllables per word, and is still used as a standard benchmark [27,62,63,64].
Finally, we perform qualitative evaluations of the results obtained on WikiBIO and ToTTo, following the best practices outlined by van der Lee et al. [30]. Our human annotators are from several countries across Europe, between 20 and 55 years old and proficient in English. They have been assigned two different tasks: (i) hallucination labeling, i.e. the selection of sentence pieces which include incorrect information, and (ii) sentence analysis, i.e. evaluating different realizations of the same table according to their fluency, factualness and coverage. Scores are presented as a 3-level Likert scale for Fluency (Fluent, Mostly fluent, or Not fluent) and Factualness (likewise), while coverage is the number of cells from the table that have been realized in the description. To avoid all bias, annotators are shown a randomly selected table at a time, together with its corresponding descriptions, both from the dataset and the models that are being evaluated. Sentences are presented each time in a different order. Following Tian et al. [66], we first tasked three expert annotators to annotate a pilot batch of 50 sentences. Once confirmed that Inter-Annotator Agreement was approx. 75% (a similar finding to Tian et al. [66]), we asked 16 annotators to annotate a bigger sample of 300 instances (where each instance consists of one

Results
We perform an extensive evaluation of our scoring procedure and multi-branch architecture on the WikiBio dataset: we evaluate -the quality of the proposed alignment labels, both intrinsically using human judgment and extrinsically by means of the DTG downstream task and -the performance of our model with respect to the baselines. Additionally, we assess the applicability of our framework on the more noisy ToTTo benchmark, which represents a harder challenge for today's DTG models.
6.1 Validation of Alignment Labels.
To assess the effectiveness of our alignment labels (Sec. 3), we first compare the alignment labels against human judgment, and then explore their impact on a DTG task. As a baseline for comparison we report performances of PB&L.
Intrinsic performance. Tab. 1 (top) compares the labeling performance of our method and PB&L against human judgment. Our scoring procedure significantly improves over PB&L: the latter only achieves 46.9% accuracy and 29.7% F-measure, against 87.5% and 68.7% respectively for our proposed procedure. Perez-Beltrachini and Lapata [47] report a F-measure of 36%, a discrepancy that can be explained by the difference between the evaluation procedures: PB&L evaluate on 132 sentences, several of which can be tied to the same table, whereas we explicitly chose to evaluate on 300 sentences all from different tables in order to minimize correlation. We remark that beyond F-measure, the precision of PB&L's scoring procedure is at 21.3% compared to 80.6% for ours, and recall stands at 49.2% against 59.8%. We argue that selecting a negative instance at random for training their classifier leads the network to incoherently label words, without apparent justification. See Figure 4 for two examples of this phenomenon; and Appendix D for other comparisons. In contrast, our method is able to detect hallucinated statements inside a sentence, without incorrectly labeling the whole sentence as hallucinated.
Impact on a DTG downstream task. Additionally, we assess the difference of both scoring procedures using their impact on the WikiBio DTG task. Specifically, Tab. 1 (bottom) shows the results of training MBD using either PB&L's or our labels. We observe significant improvements, especially in BLEU and PARENTrecall (40.5% vs 32.2% and 45% vs 39.3%), showing that our labeling procedure is more helpful at retaining information from training instances (the system better  picks up what humans picked-up, ultimately resulting in better BLEU and recall).

Automatic System Evaluation
Comparison with SOTA systems. Tab. 2 shows the performances of our model and all baselines according to the metrics of Sec. 5.4. Two qualitative examples are presented in Figure 5 and more are available in Appendix D.
First of all, reducing hallucinations is reached with success, as highlighted by the hallucination rate (1.43% vs. 4.20% for a standard encoder-decoder and 10.10% for the best SOTA model on BLEU). The only model which gets a lower hallucination rate (0.74%, corroborated by its PARENT-precision of 80.9%), stnd filtered, achieves such a result at a high cost. As can be seen in Figure 5 where its out-  Table 2 Comparison results on WikiBio. ↑ (resp. ↓ ) means higher (resp. lower) is better. "Gold" refers to the gold standard, i.e. the reference texts included in the dataset.
put is factual but cut short, its sentences are the shortest and the most naive in terms of the Flesch readability index, which is also reflected by a lower BLEU score. The high PARENT precision -mostly due to the shortness of the outputs -is counterbalanced by a low recall: the F-measure indicates the overall lack of competitiveness of this trade-off. This shows that the naive approach of simply filtering training instances is not the appropriate solution for hallucination reduction. This echoes [12] who trained a vanilla network on the cleanest 20% of the data and found that predictions are more precise than those of a model trained on 100% but that PARENT-recall and BLEU scores are low. At the other extreme, the best model in terms of BLEU, hier, falls short regarding precision, suggesting that often the generated text is not matched in the input table; this issue is also reflected by the highest hallucination rate of all models (10.10%). A reason could be the introduction of their auxiliary training tasks which often drive the decoder to excess in mimicking human behavior. While BLEU score improves, overall factualness of outputs decreases, showing that the model picks up domain lingo (how to formulate ideas) but not domain insight (which ideas to formulate) (see Figure 5). This is in line with [53,12] who argue that BLEU is an inappropriate metric for generation tasks other than machine translation.
The analysis of hsmm, and especially of its relatively weak performance both in terms of BLEU and PARENT, highlights the insufficiency of the multi-branch architecture by itself. This reinforces the need of the additional hallucinations supervision provided by our labeling procedure.
Finally, in the comparisons with hal WO , we can see that while it achieves one of the highest performances in term of precision (79.5%), this comes at the cost of the lowest recall (40.5%) of all models and thus poor F-measure. This confirms our hypothesis that, while effective at producing mostly factual content, modeling hallucination only as a fixed value for a whole instance is detrimental to the content generation procedure. Finer-grain annotations are required, as shown by our model recall (46.4%), coupled with a robust precision (79.0%).
Weight impact on decoding. As we deal with a CTG system, we can guide our network at inference to generate sentences following desired attributes. The impact of different weight combinations is explored in Tab. 3. In particular, we can see that changing weights in favor of the hallucination factor (top five lines) leads to decreases in both precision and recall (from 80.37% to 57.88% and 44.96% 4.82% respectively). We also observe that strongly relying on the hallucinating branch dramatically impacts performances ([0.0 0.5 0.5] obtains near 0 BLEU and

Human evaluation
To measure subtleties which are not captured by automatic metrics, we report in Tab. 4 human ratings of our model, two baselines and the gold. These baselines have been selected because they showcase interesting behaviors on automatic metrics: hier obtains the best BLEU score but a poor precision, and stnd filtered gets the best precision but poor BLEU, length and Flesch index. First, coherently with [7], we found that around two thirds of gold references contain divergences from their associated tables. Such data also confirm our analysis on the stnd filtered baseline: it's training on truncated sentences lead to an unquestionable ability to avoid hallucinations, while dramatically impacting both its fluency and coverage, leading to less desired outputs overall, despite the high PARENT-precision score.
The comparison between hier and MBD shows that both approaches lead to similar coverage, with MBD obtaining significantly better performances in terms of factualness. We also highlight that MBD is evaluated as being the most fluent one, even better than the reference (which can be explained by the imperfect pre-processing done by Lebret et al. [29]).

ToTTo: a considerably noisy setting
The ToTTo dataset is used in the following experiments to explore models' robustness to the impact of extreme noise during training. As stated in Section 5.1, we use as inputs only the highlighted cells, as content selection is arbitrary (i.e. the cells were chosen depending on the target sentence, and not vice versa  Table 5 Comparison results on ToTTo. ↑ (resp. ↓ ) means higher (resp. lower) is better. In human evaluation for Fluency, reported are for "Fluent" and "Mostly Fluent", with only "Fluent" in parentheses. Same for Factualness. (b) Fig. 6 Qualitative examples of MBD and hal WO on ToTTo. hal WO 's poor generation quality is not detected by discrete metrics. In contrast, MBD generates fluent and naively factual sentences. Note that stnd and stnd filtered have the same behavior as on WikiBio: the former produces fluent but nonsensical text; the latter generates very un-fluent, but factual, text.
other hand, we use as targets the noisy references, which may contain both divergences and lexical issues. This setting is particularly challenging and is more effective in recreating a representational, hallucination-prone real-life context than WikiBio. Other datasets [43, 14,69] available in literature are too similar to Wik-iBio concerning their goals and challenges, and are therefore less interesting in this context. Table 5 reports the performances of stnd, stnd filtered, hal WO and MBD with regards to automatic metrics and human evaluation. Compared to their respective performances on WikiBio, all models show significantly decreased scores. They struggle at generating syntactically correct sentences but, at the same time, they have still learned to leverage their copy mechanism and to stick to the input. This behavior is illustrated in both examples of Fig. 6. In particular, hal WO 's high PARENT-precision score (77.64%) seems to be due to its tendency to blindly copy input data without framing them in a sentence structure, as its low BLEU and PARENT-recall scores suggests (17.06% and 22.65%). These lower scores are good indicators that the ToTTo task, as framed in this paper, is difficult. Following the same evaluation protocol than for WikiBio, we report human ratings of different models, also included in Table 5.
MBD's factualness is judged favorably, with 55.9% hallucination-free texts, and up to 85.3% texts with a single error at most. In contrast, hal WO stands at 32.4% and 61.8% for error-free texts and single-error texts respectively. Interestingly, stnd filtered obtains the second best performance (70.6% texts with a single error).
Fluency scores are also meaningful: hal WO and MBD respectively obtain 61.7% and 91.2%. Word-based filtering is not suitable for noisy datasets, as shown by stnd filtered 's worse fluency score, 29.4%.
As for coverage performances, our model MBD obtains the maximum coverage score 3.613, surpassing all baselines by at least 0.789 slots (the second best coverage score is obtained by stnd at 2.824), and getting very close to the Gold value (which stands at 3.618). These performances, and qualitative examples of Figure 6, suggest that stnd filtered and hal WO try to reduce hallucinations at the cost of missing some input slot, while MBD effectively balances both goals.
The analysis of Factualness, Fluency and Coverage can be enhanced using qualitative error analysis on randomly sampled generated texts (we report two such examples in Figure 6). In particular, we want to highlight the following considerations: -As most training examples are very noisy, sentence-level models fail at learning from them. stnd filtered has been trained on factual statements only, at the cost of using mostly incomplete sentences during training. On both examples of Figure 6, it generated truncated sentences, missing their subjects. Its relatively high Factualness and low Fluency scores indicate that it did not learn to produce diverging outputs, nor complete sentences. Differently, hal WO generates incorrectly ordered sequences of words extracted from the table (Fig. 6a), or repetitions (Fig. 6b). The low number of training instances containing the input pair (hallucination ratio, 0) does not allow to learn what a non-hallucinated sentence actually consists in. -In contrast, our proposed finer-grained approach proves helpful in this setting, as shown by the human evaluation: sentences generated by MBD are more fluent and more factual. The multi-branch design enables the model to leverage the most of each training instance, leading to better performances overall. -Finally, we acknowledge that despite over-performing other models, MBD obtains only 55.9% of factual sentences. For instance, in Figure 6b, our model does not understand that a range consists of two numbers. The difficulty of current models to learn on very noisy and diverse datasets shows that there is still room for improvement in hallucination reduction in DTG.

Conclusion
We proposed a Multi-Branch decoder, able to leverage word-level alignment labels in order to produce factual and coherent outputs. Our proposed labeling procedure is more accurate than previous work, and outputs from our model are estimated, by automatic metrics and human judgment alike, more fluent, factual, and relevant.
We obtain state-of-the-art performances on WikiBio for PARENT F-measure, and show that our approach is promising in the context of a noisier setting. We designed our alignment procedure to be general and easily reproducible on any DTG dataset. One strength of our approach is that co-occurrences and dependency parsing can be used intuitively to extract more information from the tables than a naive word matching procedure. However, in the context of tables mainly including numbers (e.g., RotoWire), the effectiveness of the co-occurrence analysis is not guaranteed. A future work will be to improve upon the co-occurrence analysis to generalize to tables which contain less semantic inputs. For instance, the labeling procedure of Perez-Beltrachini and Lapata [47] might be revised so that adverse instances are not selected randomly, which we hypothesize would result in more relevant labels.
Finally, experiments on ToTTo outline the narrow exposure to language of current models when used on very noisy datasets. Our model has shown interesting properties through the human evaluation but is still perfectible. Recently introduced large pretrained language models, which have seen significantly more varied texts, may attenuate this problem. In this direction, adapting the work of [5,22] to our model could bring improvements to the results presented in this paper.

Declarations
• Fundings: We would like to thank the H2020 project AI4EU (825619) and the ANR JCJC SESAMS (ANR-18-CE23-0001) for supporting this work. This research has been partially carried on in the context of the Visiting Professor Program of the Italian Istituto Nazionale di Alta Matematica (INdAM).

A Alignment labels reproducibility
We consider as important words, i.e. nouns, adjectives or numbers, those which are Part-of-Speech tagged as NUM, ADJ, NOUN and PROPN.
In order to apply the score normalization function norm(·, y), we separate sentences y into statements y t i :t i+1 −1 . To do so, we identify the set of introductory dependency relation labels 7 , following previous work on rule-based systems for the conversion of dependency relations trees to constituency trees [18,73,20,3].
Our segmentation algorithm considers every leaf token in the dependency tree, and seeks its nearest ancestor which is the root of a statement.
Two heuristics enforce the score normalization: (i) conjunctions and commas next to hallucinated tokens acquires these lasts' hallucination scores, and (ii) paired parentheses and quotes acquire the minimum inner tokens' hallucination score.
Part-of-Speech tagging has been done using the HuggingFace's Transformers library [72] to fine-tune a BERT model [6] on the UD English ParTUT dataset [58]; Stanza [51] has been exploited for dependency parsing.

B Implementation details
Our system is implemented in Python 3.8 8 and PyTorch 1.4.0 9 . In particular, our multi-branch architecture is developed, trained and tested as an OpenNMT [26] model. Sentence lengths and Flesch index [13] are computed using the standard style Unix command.
Differently to Perez-Beltrachini and Lapata [47], we did not adapt the original WikiBio dataset 10 in any manner: as we work on the model side, we fairly preserve the dataset's noisiness.
Word-level alignment labels are computed setting m = 5, following Mikolov et al. [39]. As stated in Sec. 5.3, the threshold τ 's value is optimized for highest accuracy via human tuning: Table B1 shows accuracy scores of our proposed automated labels for different values of τ .
We share the vocabulary between input and output, limiting its size to 20000 tokens. Hyperparameters were tuned using performances on the development set: Tab. B2 reports the performances of our best performing MBD on the development set. Our encoder consist of a 600-dimensional embedding layer followed by a 2-layered bidirectional LSTM network with hidden states sized 600. We use the general attention mechanism with input feeding [37] and the same copy mechanism as See et al. [59]. Each branch of the multi-branch decoder is a 2-layered LSTM network with hidden states sized 600 as well.
Training is performed using the Adam algorithm [25] with learning rate η = 10 −3 , β 1 = 0.9 and β 2 = 0.999. The learning rate is decayed with a factor of 0.5 every 10000 steps, starting from the 5000th one. We used minibatches of size 64 and regularized via clipping the gradient norm to 5 and using a dropout rate of 0.3. We used beam search during inference, with a beam size of 10.
All experiments were performed on a single NVIDIA Titan XP GPU. Number of parameters and training times are shown in Table B3. Same model's differences between WikiBio and ToTTo are justified by the different datasets' number of instances and input vocabulary sizes.

C Annotation interface
The human annotation procedure is done via a web application specifically developed for this research. Fig. C1a shows how the hallucination tagging user interface looked like in practice, while in Fig. C1b  : susan maria blu -lrb-born july 12 , 1948 -rrb-, sometimes credited as sue blu , is an american voice actress , voice director and casting director in american and canadian cinema and television . PB&L: susan maria blu -lrb-born july 12 , 1948 -rrb-, sometimes credited as sue blu , is an american voice actress , voice director and casting director in american and canadian cinema and television . Ours: susan maria blu -lrb-born july 12 , 1948 -rrb-, sometimes credited as sue blu , is an american voice actress , voice director and casting director in american and canadian cinema and television .   : ryan david moore -lrb-born december 5 , 1982 -rrb-is an american professional golfer , currently playing on the pga tour . PB&L: ryan david moore -lrb-born december 5 , 1982 -rrb-is an american professional golfer , currently playing on the pga tour . Ours: ryan david moore -lrb-born december 5 , 1982 -rrb-is an american professional golfer , currently playing on the pga tour .

Gold
pat burke is an irish gaelic footballer who plays for dublin and kilmacud crokes . stnd pat burke is a gaelic footballer from dublin , ireland . stnd filtered pat burke is a gaelic footballer for dublin . hsmm pat burke ( born in dublin ) is a former irish gaelic footballer who played as a gaelic footballer . hier pat burke is a former gaelic footballer for dublin . MBD[.4, .1, .5] pat burke is a gaelic footballer from county dublin .   This list ranks the top 10 islands of the Maldives by area . Some islands in the Maldives , although geographically one island , are divided into two administrative islands ( for example , Gan and Maandhoo in Laamu Atoll ) .

Gold
Hulhumeedhoo is the 5th largest island in Maldives . stnd It has a area of Hulhumeedhoo km² ( Islands sq mi ) . stnd filtered is the fourth of the Maldives in Maldives .

hal WO
Hulhumeedhoo is the largest islands of the Maldives by area size . MBD[.4, .1, .5] Hulhumeedhoo is the fifth largest island by area size .

Table D15
A ToTTo input table, coupled with the corresponding sentence and the modelsgenerated outputs.