Lexical Complexity Controlled Sentence Generation for Language Learning

Abstract

Language teachers spend a lot of time developing good examples for language learners. For this reason, we define a new task for language learning, lexical complexity controlled sentence generation, which requires precise control over the lexical complexity in the keywords to examples generation and better fluency and semantic consistency. The challenge of this task is to generate fluent sentences only using words of given complexity levels. We propose a simple but effective approach for this task based on complexity embedding while controlling sentence length and syntactic complexity at the decoding stage. Compared with potential solutions, our approach fuses the representations of the word complexity levels into the model to get better control of lexical complexity. And we demonstrate the feasibility of the approach for both training models from scratch and fine-tuning the pre-trained models. To facilitate the research, we develop two datasets in English and Chinese respectively, on which extensive experiments are conducted. Experimental results show that our approach provides more precise control over lexical complexity, as well as better fluency and diversity.

Appendices

A Complexity Embedding Id

The English words have six levels. And the Chinese words have seven levels (Diff 1–7). We give the design of the complexity embedding id for this two language in the Table 7. Note that, if a word is out of the complexity level vocabulary, its complexity is “$\langle out \rangle$” which is mapping into id 7 in English corpus and 8 in Chinese corpus. In addition, the special tokens such as “$\langle s \rangle$” “$\langle pad \rangle$” “$\langle \backslash s \rangle$” “$\langle unk \rangle$” are the common meaning in data preprocessing for model training.

B Details of Datasets Construction

1.1 B.1 English Dataset

We adopt the English word complexity levels in the Common European Framework of Reference for Languages (CEFR)⁵ which is divided into six complexity levels (A1, A2, B1, B2, C1, and C2). First, we need to restrict the words in the corpus to ensure most of the words are in the complexity level vocabulary. Then, we need to extract keywords from the sentences. In this process, we command the number of keywords is related to the length of the sentence, and the number of keywords is between 1 to 5. Finally, we obtain the complexity information of each sentence through the complexity level vocabulary. The English raw corpus is collected from the monolingual English News dataset in ACL2019 WMT. We select those sentences which have 90% words in the complexity level vocabulary of CEFR. After the processes mentioned above, we get 199k samples in the English corpus, and we split the train, validation and test dataset as shown in the Table 8.

Table 7. Complexity Embedding Id.

1.2 B.2 Chinese Dataset

The word complexity levels in Chinese Proficiency Grading Standards for International Chinese Language Education (CPGS)⁶ is divided into six complexity levels (1 to 7). The Chinese raw corpus is collected from 500 textbooks for Chinese learners. These textbooks contain two types of text: essay and dialogue. We split these texts into sentences and throw away those short sentences. If the raw text is a dialogue, after splitting, we need to remove the speaker’s name to guarantee it is a proper sentence. Then, we command the number of keywords is related to the length of the sentence, and the number of keywords is between 1 to 5. After the processes mentioned above, we get 156k samples in the Chinese corpus, as shown in the Table 8.

1.3 B.3 Analysis of the Datasets

Coverage of Words with Levels. We first analyze the two datasets from the coverage rate of complexity level vocabulary. Due to the requirement of complexity level, the target text is proper to cover most of the vocabulary of complexity level. Both of the two datasets have covered over 93% of the vocabulary of complexity levels.

Fig. 3.

Distributions of the number of keywords and complexity levels.

Table 8. Statistics of the two datasets.

Distributions of the Number of Keywords and Complexity Levels. One or multiple complexity levels and keywords are given as the input to generate sentences. We give the distribution of the number of keywords and the complexity levels in Fig. 3. From the statistics of (a) and (c) in Fig. 3, the number of keywords in all samples has covered the range of 1 to 5 both in the English and Chinese datasets, but the distributions are quite different. On account of the average sentence length of English news data is longer than the Chinese corpus, the number of keywords in English is larger. From the statistics in (b) and (d) of Fig. 3, the number of complexity levels distribution of the Chinese dataset is close to a standard normal distribution, and the English dataset concentrates on a wider range of complexity levels. This indicates that in the English dataset it tends to use more words of different complexity levels in the same sentence.

C Algorithm of Reranking

The algorithm is the detail of reranking method. We select the sentence that best meets the lexical complexity requirements from the N-best candidates, and $N=10$. On the test set, We take the sum of ACC score and F1 score. The, we choose the candidate that has the largest score.

D Case Study

We choose some cases of the fine-tuning pattern from two datasets. The English cases are in the Table 9, and the Chinese cases are in the Table 10. In both tables, the required keywords as well as appearing in the sentences are shown in blue font, and certain given grades as well as words actually appearing in the sentences for the corresponding grade are shown in red font.

E Related Methods

1.1 E.1 Controlled Decoding

The gradients of an external discriminator is directly used to the generation of a pre-trained language model toward the target topic [8]. The output probabilities of a language model is modified by using the output of a discriminator that determines whether the future text will contain the desired attribute. Different from the controlled decoding methods, our method considers the constraint of lexical complexity during both training and prediction.

1.2 E.2 Prompting

The prompting method has emerged as a new way to perform natural language processing by conditioning on extra information. Brown et al. propose to use a task description and a few examples to adapt the GPT-3 model to downstream tasks, which is referred to as in-context learning [4]. Their prompts are manually designed. Gao et al. present LM-BFF for automatic prompts generation [11]. Liang et al. propose prefix-tuning, which uses continuous vectors as prompts [24]. Compared to the prompting method, our method fuses more fine-grained information on lexical complexity in model training.

1.3 E.3 Reranking

The reranking approach has been proved to have excellent performance in machine translation [31] and text generation [37]. The reranking method rescores the n-best candidates through a model or a function and selects the highest scoring candidate as the final prediction [16]. Unlike the reranking method, our method do not need to process the outputs after decoding.

Table 9. Generated examples from the English dataset.

F Limitation

Our proposed task has wide applications in the field of language teaching, and the proposed method has precise control over lexical difficulty. However, the task requires that the lexical complexity is known first. The vocabulary difficulty table is the experience summed up by the predecessors, and it is difficult to apply to all vocabulary. Therefore, we are actively exploring how to make the model automatically understand all vocabulary difficulties so that it can cover a wider vocabulary at generation.

Table 10. Generated examples from the Chinese dataset.