Evaluating Transformers’ Sensitivity to Syntactic Embedding Depth

Abstract

Various probing studies have investigated the ability of neural language models trained only on word prediction over large corpora to process hierarchical structures, a hallmark of human linguistic abilities. For instance, it has been shown that Long Short-Term Memory (LSTM), a type of Recurrent Neural Networks (RNNs), can capture long-distance subject-verb agreement patterns and attraction effects found in human sentence processing. However, although it’s found in human experiments that attractors that are syntactically closer to but linearly farther from the verb elicit greater attraction effects than those that are linearly closer but syntactically farther, LSTM shows the opposite pattern, suggesting that it lacks knowledge regarding syntactic distance and is more sensitive to local information. The current article investigates whether state-of-the-art Generative Pre-trained Transformers (GPTs) can capture the prominence of syntactic distance. We experimented with various versions of GPT-2 and GPT-3 and found that all of them succeeded at the task. We conclude that GPT may be able to better model human linguistic cognition than LSTM, corroborating previous research, and that further investigating what mechanisms enable GPT to do so may inform research on human syntactic processing.