An Evaluation of NLP Methods to Extract Mathematical Token Descriptors

Abstract

Mathematical formulae are a foundational component of information in all scientific and mathematical papers. Parsing meaning from these expressions by extracting textual descriptors of their variable tokens is a unique challenge that requires semantic and grammatical knowledge. In this work, we present a new manually-labeled dataset (called the MTDE dataset) of mathematical objects, the contexts in which they are defined, and their textual definitions. With this dataset, we evaluate the accuracy of several modern neural network models on two definition extraction tasks. While this is not a solved task, modern language models such as BERT perform well ($\sim$90%). Both the dataset and neural network models (implemented in PyTorch jupyter notebooks) are available online to help aid future researchers in this space.

Supported by University of Illinois at Urbana-Champaign - College of Engineering.

Appendices

Appendix

1 Models

Below is a more detailed overview of the models used in this manuscript. Long Short-Term Memory (LSTM) RNNs were used for all of the models (with the exception of BERT).

1.1 1.1 Vanilla Sequence-to-Sequence Model

Fig. 3.

The Vanilla Seq2Seq architecture

To find the best output sequence, generative models use the probability chain rule, which states that the probability of a sequence is the conditional probability of its tokens. We represent this in the following relation, where $T$ is the training data and $\theta$ is the parameters of the RNN:

$$\begin{aligned} p_{\theta }(y|T; \theta ) = \prod ^{i}{p_{\theta }(y_{i}| y_{i-1}, y_{i-2}, \ldots , y_{1}, T;\theta )} \end{aligned}$$

Seq2Seq-based models only take one sequence as its input. Therefore, a separation token (seen as $<SEP>$ in Fig. 3) was used to concatenate the variable name and it’s context. This was done for the Vanilla Seq2Seq model, Transformer Seq2Seq model, Pointer network, and BERT model (Fig. 4).

1.2 1.2 Transformer Seq2Seq Model

Fig. 4.

The Transformer Seq2Seq architecture

To create a context vector, an annotation for each input word is generated based on the word itself and its surrounding words. After this, the context vector for the $i$-th output word is calculated as the weighted sum of the annotations $h$:

$$\begin{aligned} c_{i} = \sum ^{k}_{j=1}{\alpha _{ij}h_{j}} \end{aligned}$$

where i is the indicates the length of the output sequence and $k$ is the length of the input sequence. The attention weights are calculated as such:

$$\begin{aligned} \alpha _{ij} = \frac{\exp {(a(s_{i - 1}, h_{j}))}}{\sum ^{T_{x}}_{k=1}{\exp {(a(s_{i - 1}, h_{j}))}}} \end{aligned}$$

where $a(s_{i - 1}, h_{j})$ is the alignment function which ranks how likely an input word is related to an output word. The alignment function takes the decoder hidden state of the $i$-th output word and the annotation of the $j$-th input word and learns the weights for each input annotation-output word pair via a feed-forward layer.

1.3 1.3 Pointer Network

To find the most likely answer out of the input tokens, the Pointer Network computes attention weights of variable length as such:

$$\begin{aligned} \alpha ^{i}_{j} = \mathop {\textrm{softmax}}\limits {(u^{i}_{j})} \end{aligned}$$

$$\begin{aligned} u^{i}_{j} = v^\top \tanh {(W_{1}e_{j} + W_{2}d_{i})} \end{aligned}$$

where $e_{j}$ is the encoder state at the $j$-th input word, $d_{i}$ is the decoder state at the $i$-th input word, and $v$, $W_{1}$, $W_{2}$ are learnable parameters (Fig. 5).

Fig. 5.

The Pointer Network architecture

1.4 1.4 Match-LSTM Model

Fig. 6.

The Match LSTM architecture

A visual representation of the model is given in Fig. 6. The attention weights in the Match-LSTM layer is calculated as such:

$$\begin{aligned} \alpha _{i} = w^{\top }u_{i} + b \end{aligned}$$

$$\begin{aligned} u_{i} = \tanh {(W^{v}h^{v} + (W^{c}h_{i}^{c} + W^{r}h_{i-1}^{r} + b^{c}))} \end{aligned}$$

Here, $W^{v}, W^{c}, W^{r}, b^{c}, w, b$ are learned weights, $h_{i}^{c}$ and $h^{v}$ are the hidden states of the $i-th$ word of the context text and the variable name respectively, and $h_{i-1}^{r}$ is the previous hidden state of the Match-LSTM. The attention weights are then combine with $h_{i}^{c}$ and $h^{v}$ and processed in the Match-LSTM. This process is done in the forward and backward direction and all resulting hidden states are coalesced into a final hidden state vector. The final hidden state vector is then processed by a pointer layer, which calculates the most likely position of the definition of the variable in the context text. The following formulas show how the probability $\beta$ is calculated:

$$\begin{aligned} \beta _{j} = \mathop {\textrm{softmax}}\limits {(v^{\top }s_{j} + b \otimes e_{C + 1})} \end{aligned}$$

$$\begin{aligned} s_{j} = \tanh {(V\bar{H^{r}} + (W^{a}h_{k - 1}^{a} + b^{a}) \otimes e_{C + 1})} \end{aligned}$$

where $V, W^{a}, b^{a}, v$ and $b$ are learned weights, $\bar{H^{r}}$ is the vector of concatenated Match-LSTM hidden states, $h_{k - 1}^{a}$ is the previous hidden state of the pointer LSTM and $e_{C + 1}$ is a vector of ones with size C, where C is the length of the context text. $\beta _{j}$ is combined with $h_{j}^{c}$ and processed by an LSTM.

1.5 1.5 Bert-Based Model

In these methods, a pre-trained Hugging Face BertForQuestionAnswering model was fine-tuned to perform the definition-extraction task. The tokenizer used to process the text was a custom Hugging Face tokenizer created by [18]. This tokenizer was trained on a mathematical language corpus comprised of a wide range of education materials used in primary-school, high-school, and college-level courses (Fig. 7).

Fig. 7.

The BERT architecture