To provide behavioral researchers with the power of Unity and the convenience of programs such as PsychoPy, we created the Unity Experiment Framework (UXF). UXF is a software framework for the development of human behavior experiments with Unity and the main programming language it uses, C#. UXF takes common programming concepts and features that are widely used and are often reimplemented for each experiment, and implements them in a generic fashion (Table 1). This gives researchers the tools to create their experimental software without the need to redevelop this common set of features. UXF aims specifically to solve this problem, and it deliberately excludes any kind of stimulus presentation system, with the view that Unity (and its large asset-developing community) can provide all the necessary means to implement any kind of stimulus or interaction system for an experiment. In summary, UXF provides the “nuts and bolts” that work behind the scenes of an experiment developed within Unity.
UXF provides a set of high-level objects that directly map onto how we describe experiments. The goal is to make the experiment code more readable and avoid the temptation for inelegant if–else statements in the code as complexity increases. Sessions, blocks, and trials are the “objects” that can be represented within our code. The creation of a session, block, or trial automatically generates the properties we would expect them to have—for example, each block has a block number, and each trial has a trial number. These numbers are automatically generated as positive integers based on the order in which objects were created. Trials contain functionality such as “begin” and “end,” which will perform useful tasks implicitly in the background, such as recording the timestamp when the trial began or ended. Trials and blocks can be created programmatically, meaning that UXF can support any type of experiment structure, including staircase or adaptive procedures.
Measuring dependent variables
While the trial is ongoing, at any point researchers can add any observations to the results of the trial, which will be added to the behavioral .CSV output data file at the end of the session. Additionally, a variable can be continuously logged over time at the same rate as the display refresh frequency (90 Hz in most currently available commercial VR HMDs). The main use case will be the position and rotation of any object in Unity, which can be automatically recorded on a per-trial basis, saving a single .CSV file for each trial of the session. This allows for easy cross-referencing with behavioral data. All data files (behavioral and continuous) are stored in a directory structure organized by experiment > participant > session number.
Setting independent variables
Settings can be used to attach the values of an independent variable to an experiment, session, block, or trial. Settings have a cascading effect, whereby one can apply a setting to the whole session, a block, or a single trial. When attempting to access a setting, if the setting has not been assigned in the trial, UXF will attempt to access the setting in the block. If the setting has not been assigned in the block, UXF will search in the session (Fig. 2). This allows users to very easily implement features common to experiments, such as “10% of trials contain a different stimulus.” In this case, one could assign a “stimulus” setting for the whole session, but then assign 10% of the trials to have a different value for a “stimulus” setting.
Settings are also a useful feature to allow for changing experimental parameters without modifying the source code. A simple text file (.JSON format) can be placed in the experiment directory that will be read at the start of a session, and its settings will be applied to that session. This system speeds up the iteration time during the process of designing the experiment; the experimenter can change settings from this file and see their immediate effect, without changing any of the code itself. The system also allows multiple versions of the same experiment (e.g., different experimental manipulations) to be maintained within a single codebase using multiple settings files. One of these settings profiles can be selected by the experimenter on launching the experiment task.
Experimenter user interface
UXF includes an (optional) experimenter user interface (UI; Fig. 3) to allow selection of a settings profile and inputting of additional participant information, such as demographics. The information the experimenter wishes to collect is fully customizable. The UI includes support for a “participant list” system, whereby participant demographic information is stored in its own .CSV file. As new participants perform the experiment, their demographic information is stored in the list. This allows participant information to be more easily shared between sessions, or even separate experiments—instead of having to input the information each time, the experimenter can easily select any existing participant found in the participant list via a drop-down menu.
Below is an example of the C# code used to generate a simple two-block, ten-trial experiment in which the participant is presented with a number x and must input the doubled value (2x).
Elsewhere in our project, we must define what happens when we begin the trial (such as making the value of xx appear for the participant) and mechanisms to retrieve the participant’s response for the trial (i.e., the participant’s calculated value of 2x). These mechanisms are to be created with standard Unity features for making objects appear in the scene, collecting user response via keyboard input, and so forth. The resulting .CSV behavioral data file would be automatically generated and saved (Table 2). A typical structure of a task developed with UXF is shown in Fig. 4.
Multithreading file input/output (I/O)
Continuous measurement of variables requires that large amounts of data be collected over the course of the experiment. When using a VR HMD, it is essential to maintain a high frame rate and keep stutters to a minimum in order to minimize the risk of inducing sickness or discomfort in the participant. Handling of tasks such as reading and writing to a file may take several milliseconds or more, depending on the operating system’s background work. Constant data collection (particularly when tracking the movement of many objects in the scene) and writing these data to file therefore poses a risk of dropping the frame rate below acceptable levels. The solution is to create a multithreaded application that allows the virtual environment to continue to be updated while data are being written to files simultaneously in a separate thread. Designing a stable multithreaded application imparts additional technical requirements on the researcher. UXF abstracts file I/O away from the developer, performing these tasks automatically, with a multithreaded architecture working behind the scenes. Additionally, the architecture contains a queuing system, where UXF queues up all data tasks and writes the files one by one, even halting the closing of the program in order to finish emptying the queue, if necessary.
UXF is a standalone, generic project, so it does not put any large design constraints on developers using it. This means that UXF does not have to be used in a traditional lab-based setting, with researchers interacting directly with participants; it can also be used for data collection opportunities outside the lab, by embedding experiments within games or apps that a user can partake in at their discretion. Data are then sent to a web server, from which they can later be downloaded and analyzed by researchers (Fig. 5). Recently these cloud-based experiments have become a viable method of performing experiments on a large scale.
UXF can be used in cloud-based experiments (Fig. 5) using two independent pieces of software that accompany UXF:
UXF S3 Uploader allows all files that are saved by UXF (behavioral data, continuous data, and logs) to be additionally uploaded to a location in Amazon’s Simple Storage Service, as set up by a researcher. This utilizes the existing UXF functionality of setting up actions to be executed after a file has been written, so a developer could potentially implement uploading the files to any other storage service.
UXF Web Settings replaces the default UXF functionality of selecting experiment settings via a user interface, allowing settings instead to be accessed automatically from a web URL by the software itself. This allows a deployed experiment (e.g., via an app store, or simply by transferring an executable file) to be remotely altered by the researcher, without any modification to the source code. Settings files are stored in .json format and would usually be of very small file size, so they can be hosted online cheaply and easily.
A developer can implement neither, either, or both of these extras, depending on the needs of the research. For lab-based experiments, neither is required. For experiments without any need to modify settings afterward, but with the requirement of securely backing up data in the cloud, the first option can be used. If a researcher wants to remotely modify settings but has physical access to the devices to retrieve data, the second option can be used. For a fully cloud-based experiment without direct researcher contact with the participant, both optional functionalities can be used. This has been successfully tried and tested, in the context of a museum exhibition in which visitors could take part in VR experiments, with the recorded data being uploaded to the internet. Both UXF S3 Uploader and UXF Web Settings are available as open-source Unity packages.