Quaddles: A multidimensional 3-D object set with parametrically controlled and customizable features
Many studies of vision and cognition require novel three-dimensional object sets defined by a parametric feature space. Creating such sets and verifying that they are suitable for a given task, however, can be time-consuming and effortful. Here we present a new set of multidimensional objects, Quaddles, designed for studies of feature-based learning and attention, but adaptable for many research purposes. Quaddles have features that are all equally visible from any angle around the vertical axis and can be designed to be equally discriminable along feature dimensions; these objects do not show strong or consistent response biases, with a small number of quantified exceptions. They are available as two-dimensional images, rotating videos, and FBX object files suitable for use with any modern video game engine. We also provide scripts that can be used to generate hundreds of thousands of further Quaddles, as well as examples and tutorials for modifying Quaddles or creating completely new object sets from scratch, with the aim to speed up the development time of future novel-object studies.
KeywordsStimulus set Feature space Multidimensional objects
As the use of novel objects in research has become more commonplace, there has been a parallel rise in studies in which participants have engaged and interacted with complex, continually changing virtual environments. Such dynamic tasks, presented on traditional monitors or stereoscopic displays, enable the presentation of much richer stimuli and the collection of much richer data streams than more traditional, static tasks. Dynamic tasks have been used to investigate the processes underlying phenomena such as the mechanisms of spatial navigation in humans and other animals (Bohil, Alicea, & Biocca, 2011; Ekstrom et al., 2003; Weisberg, Schinazi, Newcombe, Shipley, & Epstein, 2014); multisensory integration in the determination of one’s own location (Ehrsson, 2007; Lenggenhager, Tadi, Metzinger, & Blanke, 2007); memory retrieval (Watrous, Tandon, Conner, Pieters, & Ekstrom, 2013); priority in attention, gaze, and memory (Aivar, Hayhoe, Chizk, & Mruczek, 2005; Jovancevic, Sullivan, & Hayhoe, 2006); the temporal organization of gaze in realistic tasks (Johnson, Sullivan, Hayhoe, & Ballard, 2014); subliminal cueing (Aranyi et al., 2014; Barral et al., 2014); and the development of brain–computer interfaces (Leeb et al., 2007).
Both dynamic tasks and novel 3-D object sets, then, have become standard tools in the cognitive science repertoire. We are not aware of any published work that combines the two streams, but we anticipate that this will rapidly become commonplace, as more researchers become aware of the power and flexibility these tools enable, without a corresponding sacrifice in experimental control. Our laboratory has begun running such studies, in which we examine attentional and oculomotor changes as participants learn about a novel object set in a dynamic environment (Watson, Voloh, Naghizadeh, Chen, & Womelsdorf, 2017). With so many sets of novel objects freely available (see Fig. 1), it came as a surprise that we could not find a multidimensional set that met our requirements. Instead, we had to design our own and test their suitability for our task, a much more difficult and time-consuming project than we had originally anticipated, and one that we hope to make substantially easier for future researchers.
In the present article we review this novel object set, named Quaddles in reference to the four feature dimensions that define the object space. In addition, we describe (and provide links to) tools that allow the creation of thousands of parametrically varied objects using preexisting features, which enable researchers to design custom features quickly and relatively easily. Finally, we present the results of a feature detection task showing how sensitivity, response bias, and detection efficiency to different feature values of a particular set of Quaddles can be quantified, establishing that, for the most part, the feature values are equally detectable and do not produce strong response biases.
An aesthetically pleasing appearance
Multiple feature dimensions, including nonshape dimensions
Multiple feature values along each feature dimension
Feature values that are roughly equally perceptible and do not show strong response biases
Features that can all be clearly and simultaneously viewed from any angle around the vertical axis
The ability to be exported to any commonly used image format (PNG, GIF, JPG, etc.)
The ability to be exported to any common video game engine (Unity 3D, Unreal Engine, etc.)
No previous set of novel objects we are aware of meets all these criteria. Very few of these sets have viewpoint-independent features that can all be viewed simultaneously from any angle around the object, and most have purely shape-based feature dimensions (see Fig. 1). Furthermore, we are only aware of one object set for which feature similarity has been quantified (Barry, De Rossi, & Hermans, 2014), and balancing perceptibility across different feature dimensions has only been attempted for much simpler stimuli (Kaldy, Blaser, & Leslie, 2006)—hence, the need for a new set of objects.
In the remainder of the article, we summarize the methods for creating Quaddles (more complete details can be found in the manual, hosted at http://accl.psy.vanderbilt.edu/resources/analysis-tools/), and present the results of a feature detection study quantifying sensitivity, response bias, and efficiency to the different feature values. In this task, participants were cued with two feature values prior to being shown a single Quaddle that contained only one of these values, and they had to report which of the two values was present. We calculated independent measures of sensitivity (d') and response bias (criterion) for each of the feature values, using standard signal detection measures (Macmillan & Creelman, 2005). Sensitivity on this task quantifies the ability to correctly report a feature as being present or absent, whereas response bias quantifies the likelihood of reporting a particular feature overall, either correctly or incorrectly. The desired results were indistinguishable sensitivities to the different feature values and bias scores that would cluster around 0, indicating no consistent biases across participants. We also calculated efficiency scores that combined accuracy and response time, controlling for speed–accuracy trade-offs (Smilek, Enns, Eastwood, & Merikle, 2006; Townsend & Ashby, 1983). Furthermore, we quantified the stability of the efficiency scores using consistency metrics, both across blocks within individual participants and between participants. To prefigure our results, this analysis demonstrated that feature value detection efficiency is fairly consistent within individual participants, but this consistency is substantially weaker across participants. Furthermore, these can be used as covariates in analyses of other tasks using Quaddles, thus controlling for any effects that differential sensitivities, strong response biases, or different efficiencies might produce on these tasks.
Quaddles were generated using Autodesk 3DS Max software. The primary set, used to generate all results presented in this article, is defined by four feature dimensions (body shape, branch angularity, pattern, and color), each of which can take on two possible feature values (e.g., body shape can be pyramidal or oblong), giving a total of eight feature values and 16 possible objects (Fig. 2). The scripts we provide online include options for incorporating further feature dimensions and values into stimulus sets, allowing the creation of hundreds of thousands more objects, but we do not describe these in detail here (full instructions are given in the manual).
Textures, which define the surface colors and patterns, are imported from PNG files created in a Matlab script. The neutral gray color is the same for all objects, whereas the other colors are chosen within the CIE L*c* h* space such that the L* and c* values (luminance and saturation, respectively) are held constant, but h* values (hue) vary by 15°, meaning that there is a small difference in hue between the two colors, but not in the other components of the colors. Textures are applied to object surfaces using standard UV mapping options: a cylindrical wrap for pyramidal bodies and a spherical wrap for oblong ones (different wraps were chosen because they resulted in smaller artifacts at the top and bottom of objects).
Quaddle bodies are initially generated as spheres and then molded into the desired body shapes using freeform deformation (Sederberg & Parry, 1986), in which a lattice of control points is added to the object and manipulated to create the desired shape. Thus, all body shapes are morphs of each other, allowing for intermediary shapes as desired. Each Quaddle has four arms, initially generated as straight cylinders and then morphed into the desired shape, thus also supporting intermediary values. The same is true of both the hues used to define object color, and their surface patterns. This means it is easy to create objects chosen from anywhere within the feature space defined by the four feature dimensions (Fig. 3). Given any two objects, one can also create videos of the morph between them, or even objects that morph in real time in a 3-D environment.
Object generation was automated using a 3DS Maxscript that creates and saves complete object sets. An optional function allows JPEG, PNG, or other image files to be generated of every object created from any distance, height, and rotation. Experimenters also have the option of saving videos of the objects rotating 360° from any perspective.
For illustration purposes, we generated two more feature values along each dimension, and generated partial morphs of the objects along each dimension (Fig. 3). We also generated a number of further objects with varied numbers of arms, amalgamated body shapes stacked on top of each other, and various new textures and shapes (Fig. 4). Making new Quaddles in this way is quite easy using simple modifications of our existing scripts, allowing the powerful and flexible generation of new object sets.
The York University Office of Research Ethics approved the present study as conforming to the standards of the Canadian Tri-Council Research Ethics guidelines (Certificate # 2016-214). Ten participants (mean age 28 years ± 3.8 SE; six male, four female) took part in the study. One was excluded from further analyses due to chance performance. They were seated approximately 60 cm from an LED monitor with a 60-Hz refresh rate, with heads unrestrained. The entire study, including an instructional tutorial, took approximately 1 h. The task was coded in the Unity game engine.
Participants were given approximately 5 min of training on a slower version of the task prior to starting, and they were instructed to respond as quickly and accurately as possible. A single block contained 512 trials, consisting of 32 trials for each of the 16 Quaddles. Each of the four feature values present on a given Quaddle was presented as a valid cue eight times, twice with each of the four feature values not found on the same Quaddle as the invalid cue, on both the left and the right side. Every 32 trials, each of the 16 Quaddles was shown twice, once with the valid cue on the right, and once on the left, but in all other respects the cues, Quaddles, and side of the valid cue were randomized. After a block, participants were given an optional break. Most participants ran through three blocks in approximately 60 min, but three participants were only able to finish two due to time constraints.
Several pilot versions of the study were run. After each, we adjusted the object feature values to try to eliminate any gross response biases. We present the results only for participants run using the final set of feature values, which had the most unbiased performance across feature dimensions.
One participant was excluded from analyses due to chance accuracy. For the remaining participants, all trials in which a given feature value was a valid cue were grouped together, as were all trials in which it was an invalid cue. This resulted in 16 groups of trials, within which the accuracy and response time on correct trials were averaged. There were substantial interindividual differences, with individual accuracies for particular feature values ranging from 54% to 99%, and correct response times ranging from 500 to 1,180 ms.
The mean sensitivity across all participants and all feature values was 2.59, which lay between Quartiles 1 and 2 for all feature values. The checkered and diamond patterns were clearly different from the other feature values, in that they had the two lowest median sensitivities, which were the only median sensitivities below the first quartiles of any of the other feature values, as well as third quartiles that were below two of the other values’ median sensitivities (orange color and pyramidal shape). All other median sensitivities were between the first and third quartiles for all other feature values, with a single exception (the oblong shape’s third quartile was below the median sensitivity of the orange color). Thus, participants were consistently less sensitive to pattern than to the other feature dimensions. Consistent with this, trials with either valid or invalid pattern cues had the lowest median response times (not shown), indicating that the values of the pattern dimension were particularly difficult to discriminate and that the lower sensitivity was not simply due to a speed–accuracy trade-off.
The response biases were clustered around 0, with a median bias for one feature value (diamond pattern) at exactly 0, three below 0, and four above. The absolute value of all medians was less than .20, with the exception of the checkered pattern, which had a bias of – .29. All feature values produced both positive and negative biases in different participants, with the exception of downward-bent arms, which had no bias or a positive bias for all participants. Thus, in general, feature values did not produce strong and consistent response biases, with the exceptions of checkered patterns and downward-bent arms.
There were 56 cue pairs (8×8 feature values, but the same value cannot be paired with itself), for which accuracies and response times were combined by dividing the response time on correctly answered trials in which these cues were presented by the mean accuracy on these trials (Smilek et al., 2006; Townsend & Ashby, 1983). Doing this corrects for speed–accuracy trade-offs in an intuitive way: When accuracy is perfect, efficiency will be identical to the mean response time, and as accuracy decreases, the combined score increases, making efficiency scores similar to response time in that smaller values indicate better performance. This assumes that response time and accuracy are linearly correlated, which was supported in the present data (r = – .53, p < .001). These scores were then transformed into z scores using the mean and standard deviation of the efficiency scores across all trials for each participant. These 56 scores were combined into 28 for each participant by averaging over the valid–invalid and invalid–valid trials for each feature value pair, which was also justified, on the basis of a strong correlation between the two sets of scores (r = .83, p < .001). Finally, these 28 feature value pair scores were combined into eight feature value scores, by averaging all trials in which a given feature value was presented as a cue with any of the other seven feature values. A score of 0 indicated the mean efficiency across all feature values for each participant, and 1 (or – 1) indicated a standard deviation away from this mean, with lower scores indicating greater efficiency.
These normalized efficiency scores are shown in Fig. 7. Their median values across all participants are close to 0, with no scores being more than 0.25 SDs away from the mean, indicating that most of the variance in response times and accuracies on the present task was due to factors other than response biases to particular feature values. All the 95% confidence intervals for these scores included 0, indicating little, if any, difference between the efficiencies for any feature values. However, once again the two surface pattern feature values have the two highest scores (.23 and .14), consistent with the lowered sensitivity scores to these values (Fig. 6).
As well as quantifying efficiency scores for particular feature values, we wanted to quantify how consistent these scores were, both between and within participants. This was accomplished using a tool from the content analysis field, Krippendorff’s alpha, or Kα (Hayes & Krippendorff, 2007; Krippendorff, 2011). Kα, which indicates the reliability of multiple sets of scores for a number of items, ranges between – 1 and 1, where 1 indicates perfect consistency, 0 indicates a completely random distribution of scores across sets, and – 1 indicates perfectly systematic disagreement (Krippendorff, 2008). Generally speaking, Kα is used to measure the consistency of questionnaires or other rating instruments, in which case a high value (e.g., .80 or higher) is desirable. However, in the present case, values approaching 0 indicated a lack of consistent bias toward particular feature values, as we desired for our object set.
To calculate Kα, the normalized efficiency scores were transformed to rank orders. For the between-participants Kα, this was done across all trials to give a single set of scores for each participant. Kα was calculated using a freely available Matlab script (Eggink, 2012), and standard errors and confidence intervals were calculated using a bootstrap method, as recommended by Zapf, Castell, Morawietz, and Karch (2016), save that we used 10,000 samples due to the small number of participants, and used bias-corrected and accelerated confidence intervals, which provide more accurate estimates of the true interval (DiCiccio & Efron, 1996). The resulting between-participant consistency was low (Kα = .14, SE = .17), and its 95% confidence interval included 0. For the within-participants Kα, we calculated a separate set of efficiency scores for each block performed by each participant, found the rank-ordering of these scores, then calculated a single across-block Kα for each participant using these rank orders and averaged these values across participants, using a 10,000-sample bootstrap to calculate standard errors and confidence intervals. This showed a substantial degree of within-participant consistency (Kα = .51, SE = .12). Thus, individuals have reasonably consistent efficiencies to the different feature values over time, although across individuals these efficiencies are much closer to randomly distributed (see Fig. 8).
This article presented the multidimensional set of Quaddle objects, suitable for use both in studies involving navigation through visually appealing virtual 3-D worlds, or for more static studies that require images or videos of multidimensional objects. The results of a simple feature discrimination task showed that participants are roughly equally sensitive to the feature values along each dimension (Fig. 6a), with the exception of the two pattern dimensions; that there are not strong and consistent response biases to most of these feature values (Fig. 6b); and that although individuals’ response efficiencies for particular feature values are somewhat consistent across time, there is little consistency across participants (Figs. 7 and 8). These results suggest that Quaddles can be used “off the shelf” in a wide array of tasks that require balanced feature values, and they point to modifications that could further improve this balance (e.g., by increasing the size of individual checkers or diamonds, or increasing the contrast between their dark and light elements, to increase the discriminability of the patterns). They can even be 3-D-printed, should an experiment be truly real-world, and can generate stereoscopic images for use with virtual reality or augmented reality experiments.
Limitations of the discrimination study
Although the results of our feature detection task are encouraging, it is important to acknowledge their limitations. First, there is substantial individual variability. For a study in which it is critical that each feature value be equally discriminable by all participants (for some arbitrarily small value of “equally”), it might be necessary to produce a much larger set of objects with many intermediate feature values, and run participants on a complex adaptive staircase task (see, e.g., Anderson & Johnson, 2006; Klein & Macmillan, 2001; Kujala & Lukka, 2006; Treutwein, 1999), resulting in a personalized set of objects for each participant. Producing such intermediate objects is possible with relatively simple modifications of our existing scripts. Developing such a staircase task would require careful consideration of the specific requirements of the experiment in question.
Our study presents objects at a single distance. In a study in which object distances vary, such as any involving navigation through a 3-D world, different features will become more or less discriminable at different distances. Equating discriminability across multiple distances would make for a much longer and more complex study than was feasible in our time frame.
Finally, our study presented objects within a single arena that does not change, save for the floor, which changes drastically across trials. This is simply because the study for which we developed these objects involves a single arena with floors that change across trials. The surround of an object can have powerful effects on feature discrimination, but our study does not control for these effects, as we reasoned that given the wide variety of floors we present, their effects would be essentially random. Once again, any experimenter for whom this is a critical concern will have to run another set of feature detection studies, modifying the objects and environment as needed.
We believe it is apparent that controlling for all factors that affect feature detection is not feasible, because one can only measure detection in a given experimental context. We have mentioned three factors that we did not control for, which likely interact in highly non-linear ways: individual preferences, object distance, and visual background. Even if their interactions are completely linear, controlling for all of them simultaneously would require a very complex task and a very large number of participants. Furthermore, there are certainly other relevant contingencies that we have not outlined here. Indeed, preliminary results from our laboratory suggest that in a dynamic object selection task in which the rewarded feature values are learned through trial and error, Quaddle shapes may have an advantage over the other feature dimensions (data not shown). Sensitivity, response bias, and efficiency scores need to be measured in their respective task environments, consistent with the insight that multiple factors determine whether an object guides attention in visual search tasks (Wolfe & Horowitz, 2017).
Experimenters will have to determine to what degree their particular task requires controlling for different factors that might affect feature value discrimination and detection, and design their objects and tasks accordingly. Alternatively, instead of controlling for such factors, one could simply quantify their differential effects, and use sensitivity, bias, or efficiency scores as covariates in statistical analyses, to be partialed out from the main effects of the respective studies.
Possibilities for further customization
Figures 2, 3, and 4 show several ways in which the basic Quaddle feature space can be manipulated, but there are many other ways in which researchers might change Quaddles for their own purposes. For example, they might wish to remove the vertical symmetry of some, or all, feature dimensions, so that manipulating or navigating around objects would be an important part of identifying them, as is the case with most of the objects shown in Fig. 1, as well as with many, if not most, real-world objects. Similarly, it might also be of interest to systematically vary the discriminability or salience of different feature dimensions, and to quantify this variance using a feature detection task. This enables the role of feature bias to be directly studied, as opposed to minimized as with the present object set. Such changes would require trivial modifications to the existing scripts. Aside from these examples, many other possibilities of course exist for future studies to implement additional changes.
With this article, we introduced a new object set, characterized its discriminability, and provided tools to facilitate its use in a wide range of possible future studies. This novel set of 3-D objects has normed, parametric features, suitable for a wide range of tasks; open online access to the examples and tools allows researchers to rapidly create custom object sets suitable for other studies. This pragmatic aspect resonates well with the spirit of recent toolkits for video game engines that have streamlined the development and running of dynamic experiments (Doucet, Gulli, & Martinez-Trujillo, 2016; Jangraw, Johri, Gribetz, & Sajda, 2014). The properties of Quaddles make them a suitable set of novel objects for future studies using more realistic and complex tasks, and the scripts we provide can significantly reduce the time necessary to develop different objects for such tasks.
This work was supported by Grant MOP 102482 from the Canadian Institutes of Health Research (to T.W.) and by the Natural Sciences and Engineering Research Council of Canada, Brain in Action CREATE-IRTG program (to M.R.W. and T.W.). The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of this article. The authors thank Hongying Wang for technical support, and Isabel Gauthier for comments on a draft version of the manuscript. The study described herein was approved by the York University Office of Research Ethics (Certificate # 2016-214).
- Barral, O., Aranyi, G., Kouider, S., Lindsay, A., Prins, H., Ahmed, I., . . . Cavazza, M. (2014). Covert persuasive technologies: bringing subliminal cues to human–computer interaction. In A. Spagnolli, L. Chittaro, & L. Gamberini (Eds.), Persuasive technology (pp. 1–12). Berlin, Germany: Springer International Publishing.Google Scholar
- Bennett, M., Vervoort, E., Boddez, Y., Hermans, D., & Baeyens, F. (2015). Perceptual and conceptual similarities facilitate the generalization of instructed fear. Journal of Behavior Therapy and Experimental Psychiatry, 48, 149–155. https://doi.org/10.1016/j.jbtep.2015.03.011 CrossRefPubMedGoogle Scholar
- Biederman, I., & Gerhardstein, P. C. (1993). Recognizing depth-rotated objects: Evidence and conditions for three-dimensional viewpoint invariance. Journal of Experimental Psychology: Human Perception and Performance, 19, 1162–1182. https://doi.org/10.1037/0096-15126.96.36.1992 CrossRefPubMedGoogle Scholar
- Eggink, J. (2012). kriAlpha [Matlab script]. Retrieved August 30, 2017 from https://www.mathworks.com/matlabcentral/fileexchange/36016-krippendorff-s-alpha
- Harris, J. (2015). Yadgits. Retrieved August 24, 2017, from http://wiki.cnbc.cmu.edu/Novel_Objects, last modified May 14, 2015
- Krippendorff, K. (2011). Computing Krippendorff’s alpha-reliability. Retrieved August 28, 2017, from http://repository.upenn.edu/asc_papers/43
- Leeb, R., Friedman, D., Müller-Putz, G. R., Scherer, R., Slater, M., & Pfurtscheller, G. (2007). Self-paced (asynchronous) BCI control of a wheelchair in virtual environments: A case study with a tetraplegic. Computational Intelligence and Neuroscience, 2007, 79642. https://doi.org/10.1155/2007/79642 CrossRefPubMedCentralGoogle Scholar
- Macmillan, N. A., & Creelman, D. C. (2005). Detection theory: A user's guide (2nd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
- Tarr, M. J., Bülthoff, H. H., Zabinski, M., & Blanz, V. (1997). To what extent do unique parts influence recognition across changes in viewpoint. Psychological Science, 8, 282–289. https://doi.org/10.1111/j.1467-9280.1997.tb00439.x CrossRefGoogle Scholar
- Townsend, J. T., & Ashby, F. G. (1983). Stochastic modeling of elementary psychological processes. New York, NY: Cambridge University Press.Google Scholar
- Watson, M. R., Voloh, B., Naghizadeh, M., Chen, S., & Womelsdorf, T. (2017). Information sampling and object selection strategies demonstrate the learning and exploitation of feature relevance. In 2017 Neuroscience Meeting Planner (Program No. 801.30). Washington, DC: Society for Neuroscience.Google Scholar
- Weisberg, S. M., Schinazi, V. R., Newcombe, N. S., Shipley, T. F., & Epstein, R. A. (2014). Variations in cognitive maps: Understanding individual differences in navigation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40, 669–682. https://doi.org/10.1037/a0035261 CrossRefPubMedGoogle Scholar
- Williams, P. (1998). Representational organization of multiple exemplars of object categories (Working paper, University of Massachusetts at Boston). Retrieved August 23, 2017, from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.5.8336&rep=rep1&type=pdf
- Wong, A. C.-N., & Hayward, W. G. (2005). Constraints on view combination: Effects of self-occlusion and differences among familiar and novel views. Journal of Experimental Psychology: Human Perception and Performance, 31, 110–121. https://doi.org/10.1037/0096-15188.8.131.52 CrossRefPubMedGoogle Scholar