Exploratory mapping of theoretical landscapes through word use in abstracts

Abstract

We present a case study of how scientometric tools can reveal the structure of scientific theory in a discipline. Specifically, we analyze the patterns of word use in the discipline of cognitive science using latent semantic analysis, a well-known semantic model, in the abstracts of over a thousand academic papers relevant to these theories. Our results show that it is possible to link these theories with specific statistical distributions of words in the abstracts of papers that espouse these theories. We show that theories have different patterns of word use, and that the similarity relationships with each other are intuitive and informative. Moreover, we show that it is possible to predict fairly accurately the theory of a paper by constructing a model of the theories based on their distribution of word use. These results may open new avenues for the application of scientometric tools on theoretical divides.

Appendices

Appendix 1: Search filters

For the search procedure, quotes were used on the keywords that generated the most confusion on the results due to how common the words used are. Only references classified as articles, proceedings paper, reviews, book chapters and editorial material were downloaded.

The following subdiscipline categories provided by WoS were used (in alphabetical order):

Behavioral Sciences; Computer Science—Artificial Intelligence; Computer Science—Cybernetics; Computer Science—Information Systems; Computer Science—Interdisciplinary Applications Computer Science—Theory Methods; History Philosophy of Science; Language—Linguistics; Linguistics; Neurosciences; Philosophy; Psychology; Psychology—Applied; Psychology—Biological; Psychology—Developmental; Psychology—Educational; Psychology—Experimental; Psychology—Mathematical; Psychology—Multidisciplinary; Psychology—Social; Robotics; Social Sciences—Interdisciplinary

Appendix 2: Entropy calculation

The formula with which each cell was weighted in our analysis, from (Martin and Berry 2007, p. 38). Each cell was weighted locally with the logarithm of frequency plus 1, $\log \left( f_{ij} + 1 \right)$. Then, that value was multiplied by the entropy of each one of the terms:

$$\begin{aligned} 1 + \sum \limits _{j} \frac{P_{ij} \times \log _{2}P_{ij}}{\log _{2}n} \end{aligned}$$

where $P_{ij}$ is the number of times the term i appears in document j, divided by the number of times the term appears in all of the documents, and n is the total number of documents in the dataset. This formula assumes that terms that appear in fewer documents are more informative than terms that appear in more documents. Thus, the values of the former are relatively increased, while the values of the latter are relatively diminished.

Appendix 3: Other dendrograms

In Figs. 16 and 17, we present the dendrograms that obtain by using the values lower ($D = 3$) and ($D = 5$) than the range of the stable dendrogram presented in Fig. 2. Figure 18 shows the same D as the one used to produce Fig. 2 ($D = 10$), but including a randomization of the theory labels attached to each paper (Figs. 19, 20, 21, 22, 23).

Fig. 16

$D = 3$

Fig. 17

$D = 5$

Fig. 18

$D = 10$. Randomized theories. Note the low distance values between the tree branches in comparison to Fig. 2, on the y axis

Appendix 4: Selection of number of dimensions to evaluate

Fig. 19

Mean performance of the prediction by changing the number of dimensions allowed to be evaluated when selecting the best predictors for each theory. Peak performance is achieved by limiting it to 80 dimensions (red line). However, performance is robust, so this parameter can be changed without much decrease in performance. Results aggregate over 1000 iterations. (Color figure online)

Appendix 5: Prediction performance across values of D

Fig. 20

Mean performance of the GLM of each theory. D is the number of dimensions used for the model. Results aggregate over 10, 000 iterations

Fig. 21

Mean performance of the 8 models across values of D. Aggregated over 10, 000 iterations

Fig. 22

Mean performance of the GLM of each theory using new data set. D is the number of dimensions used for the model. Results aggregate 10, 000 iterations

Fig. 23

Mean performance of the 8 models across values of D using new data set. Results aggregate 10, 000 iterations

Appendix 6: Prediction performance in randomization condition

Figure 24 shows the prediction confusion matrix for $D = 20$ the eight different theories with randomization of labels. The maximum value shown is 16.9% of confusion (embodied - ecological) and the minimum value is 8% (distributed - enactive). Figure 25 shows the mean performance for the prediction (y-axis) for each theory (x-axis) when theory labels are randomized.

Fig. 24

Confusion matrix with randomized theories, $D = 20$

Fig. 25

Mean performance by D. Randomized