Local but not global graph theoretic measures of semantic networks generalize across tasks

Abstract

“Dogs” are connected to “cats” in our minds, and “backyard” to “outdoors.” Does the structure of this semantic knowledge differ across people? Network-based approaches are a popular representational scheme for thinking about how relations between different concepts are organized. Recent research uses graph theoretic analyses to examine individual differences in semantic networks for simple concepts and how they relate to other higher-level cognitive processes, such as creativity. However, it remains ambiguous whether individual differences captured via network analyses reflect true differences in measures of the structure of semantic knowledge, or differences in how people strategically approach semantic relatedness tasks. To test this, we examine the reliability of local and global metrics of semantic networks for simple concepts across different semantic relatedness tasks. In four experiments, we find that both weighted and unweighted graph theoretic representations reliably capture individual differences in local measures of semantic networks (e.g., how related pot is to pan versus lion). In contrast, we find that metrics of global structural properties of semantic networks, such as the average clustering coefficient and shortest path length, are less robust across tasks and may not provide reliable individual difference measures of how people represent simple concepts. We discuss the implications of these results and offer recommendations for researchers who seek to apply graph theoretic analyses in the study of individual differences in semantic memory.

Appendices

Appendices

Appendix 1: Technical description of the adaptive multi-arrangement task and algorithm

The adaptive multi-arrangement task requires participants to arrange items in an arena based on their semantic relatedness. On subsequent trials, the algorithm samples two items for which there is already some evidence. This ensures that dissimilarity matrices are aligned across trials. Once a pair of items is sampled, additional items are sampled if they improve “trial efficiency.” Trial efficiency is the ratio between “utility benefit” and “trial cost.” Trial cost is defined as the additional cost of evaluating a given pair of items and is simply the number of items sampled for a given trial (n) raised to a power X, that is, ${n}^{X}$. We used an exponent of 1.2 under the assumption that the time it takes items is super-linear but sub-quadratic. Trial benefit is the additional utility gained if a given pair of items is included on a trial. In this context, trial benefit is the sum of evidence utility, which is calculated using the exponential saturation function $1-{e}^{-w*d}$, where $w$ is the current evidence weight for a given word pair, which is simply the onscreen distance of that item squared. This definition of evidence utility assumes that the dissimilarity signal-to-noise ratio is proportional to the onscreen distances, such that smaller distances have a smaller signal-to-noise ratio. The evidence utility exponent $d$ was set to 10, which is the default value used by Kriegeskorte and Mur (2012). For this formula, evidence utility is arbitrarily close to 1 as $w$ approaches .5. For this reason, .5 is used as a criterion for terminating the algorithm; that is, once each item pair has an evidence value of .5, or times out (after 35 minutes), the experiment ends.

Since the algorithm “zooms in” on subclusters of items on different trials, a scaling factor needs to be defined that rescales and combines distances of each arrangement in a way that ignores the on-screen distance for that specific arrangement. This is implemented iteratively. A reference dissimilarity matrix is used to calculate the rescaling factor on each trial. For 20 words, the reference dissimilarity matrix can be seen as a vector of 190 values, and the rescaled matrix is this vector normalized. The values in this vector are the average of the onscreen distances weighted by their evidence utility obtained from previous trials. These values are used to rescale dissimilarity vectors for all item pairs obtained on the current and previous trials. Specifically, after the reference dissimilarity vector is normalized, entries from dissimilarity estimates from all trials are set to equal the values in the normalized reference vector. Then a new reference matrix is calculated using the evidence-weighted average of the rescaled distances. This is repeated iteratively until the root mean square of the deviations between the reference matrix from the previous and current iteration is arbitrarily close to 0.

Appendix 2: Example graph representations

Graphs

Each figure below shows binary graphs of two randomly sampled participants and the average data from Experiment 1a (Fig. 7) and 2a (Fig. 8), as well as the similarity adjacency matrices of two different sample participants and average data from Experiment 1a (Fig. 9) and 2a (Fig. 10).

Fig. 7

Example graphs from Experiment 1 of two participants (first two upper panels) and graph constructed from average data (lower panel)

Fig. 8

Example graphs from Experiment 2 of two participants (first two upper panels) and graph constructed from average data (lower panel)

Fig. 9

Heatmaps of adjacency matrices (0=similar, 1=similar) from Experiment 1 of two participants (first two upper panels) and average data (lower panel)

Fig. 10

Heatmaps of adjacency matrices (0=similar, 1=similar) from Experiment 2 of two participants (first two upper panels) and average data (lower panel)

Appendix 3: Secondary methodological contribution

Validation of the adaptive multi-arrangement task for detecting individual differences

A secondary methodological contribution of our research is that we are the first to show that the adaptive version of the spatial multi-arrangement task can be used to predict individual differences in semantic processing for words. Previous work by Kriegeskorte and Mur (2012) demonstrated that this task and algorithm can be used to recover the high-dimensional structure of similarity judgments for visual stimuli, and follow-up work by Charest et al. (2014) demonstrated that the task correlates with individual differences in neural representations of visual stimuli, e.g., real-world objects. Furthermore, recent work by Majewska et al. (2021) applied the adaptive spatial multi-arrangement task to a large-scale data set with verb stimuli and demonstrated that it can provide a fine-grained measure of subclasses of semantic concepts, although the authors did not examine its potential to capture individual differences. Finally, Richie et al. (2020) recently demonstrated that a non-adaptive version of the algorithm (Goldstone, 1994), which does not involve “zooming in” on clusters of objects, can be used to recover high-dimensional structures of words. Richie et al. (2020) also found that performance on this task correlates with performance on binary similarity judgment; however, they did not demonstrate that this measure is sensitive to individual differences and did not compare the robustness of different modeling approaches in their capacity to capture such individual differences. In short, our work contributes to a line of research on validating the adaptive version of the spatial multi-arrangement task. While prior work has shown its potential to recover the high-dimensional structure of similarity judgments and its relative efficiency, we show that it can also be used to recover individual differences in similarity judgments for concepts with different modeling approaches.

Appendix 4: Example of experiment instructions

Instructions for spatial multi-arrangement task

In this study you will complete two tasks. This is the first session and this task will take approximately 35–40 minutes to complete. The first part of the experiment is called the Word Arrangement task. It is explained below. The second part of the experiment will be described to you during the second session of the study. The Word Arrangement task requires you to arrange 20 words according to their similarity. Specifically, you will use the mouse to click on a word and drag it into a circular arena. You should use the relative distance between words to indicate how similar you think each words is relative to other words. In other words, similar objects are placed closer together; dissimilar objects are placed far apart. In the current context, the objects are blocks with words inside of them, and the distance from the center of two blocks represents their dissimilarity. If you were to place two blocks such that they completely overlap with one another, that would mean you consider the words to be identical. Consider this example...<Image of example arrangement> You will not be shown all of the 20 words at once, but will be shown subsets of the 20 (up to 10) words on each trial. Often, words will repeat across trials so that we can obtain similarity judgments between all words, and/or get more precise similarity judgements for specific words pairs. Thus, on some trials, you will see many words (up to 10), and on other trials you will see fewer words (as few as 3). It does not matter how many words you see; you should use all of the space available to you in the circle to arrange the words and communicate the dissimilarity between words given to you on a given trial. For instance, if you see 10 words, you should arrange words that are relatively similar to one another closer together, and words that are less similar further apart. As an example, consider the example array below. The words “Dracula,” “vampire,” and “cape” are relatively close to each other because this person considered these words to be more similar to one another than the others. On a different trial, however, the algorithm <b>zooms in</b> on the three words “Dracula,” “vampire,” and “cape.” This allows us to collect more precise measurements of your judgments of similarity. Therefore, if you see fewer (3 words), you should still use ALL of the space in the arena to precisely communicate the relative similarity of those three words. As an example, consider the example array below, which now just has the words “Dracula,” “vampire,” and “cape,” This person considered “Dracula” and “vampire” to be more similar to one another, so they are placed closer together, and are further apart from the word “cape.” Again, it is important to note that even if these words are similar to one another, all of the space in the circle is used to communicate the relative similarity between these three words. Finally, note from this example that it does not matter where on the circle you place words—that is, whether they are on the top, left, right, or bottom of the circle. What matters is the distance between each word, which is a measure of how similar you think those words are to each other. If you need to reset an arrangement, you can right-click the “START OVER” button on the bottom left-hand side of the screen. Once you are ready to advance, you should click the “NEXT TRIAL” button on the bottom right hand side of the screen. The program uses an adaptive algorithm that is based on the precision and consistency with which you make your judgments. Therefore, you will see words repeatedly on different trials. The more you think about your judgments on each trial, the faster the experiment will end. If you arrange words randomly on each trial, the algorithm will not reach criterion, and it will take longer to complete the task. Therefore, you should try to make your arrangements precise and consistent, rather than speeding through. This will ensure that the algorithm reaches criterion faster, and ends this task. Keep in mind that there is no wrong way to arrange the words. This is a method for measuring your subjective judgment of similarity between each of the words, so there is no wrong answer as long as you are not doing the arrangements randomly and follow the instructions given above.

Instructions for relatedness ranking task (100-point slider scale). Note that instructions for the Likert 6-point rating task are identical with the exception that they refer to the 6-point rather than 100-point scale

You are done with the first part of the experiment and ready to start the second part of the experiment. In this part of the task you will be asked to judge similarity between two words in a different way. This part of the experiment will last 20–25 minutes. Specifically, you will be shown a pair of words at a time, and asked to rate how similar you think the two words are on a scale from 1 (maximally different) to 100 (identical). To report on your similarity judgments, you will use a sliding scale with your mouse. Please think about each rating carefully and try to make your ratings as precise as possible using values of the scale that seem to best match your judgment. For instance, I may think that the words “bread” and “baguette” are extremely similar, so I would give them a rating of 90. I may think that “bread” and “butter” are similar, but less similar than baguette, so I would give them a rating of 75. I may think that “bread” and “knife” are somewhat similar, though less similar than “bread” and “butter” so I would give them rating of 60. Similarly, I may think that “bread” and “doctor” are extremely dissimilar, so I would give them a rating of 1. Note that these are just examples to illustrate how different values of the scale relate to different similarity judgments, and you should choose values that seem best to you.