Gorilla in our midst: An online behavioral experiment builder

Behavioral researchers are increasingly conducting their studies online, to gain access to large and diverse samples that would be difficult to get in a laboratory environment. However, there are technical access barriers to building experiments online, and web browsers can present problems for consistent timing—an important issue with reaction-time-sensitive measures. For example, to ensure accuracy and test–retest reliability in presentation and response recording, experimenters need a working knowledge of programming languages such as JavaScript. We review some of the previous and current tools for online behavioral research, as well as how well they address the issues of usability and timing. We then present the Gorilla Experiment Builder (gorilla.sc), a fully tooled experiment authoring and deployment platform, designed to resolve many timing issues and make reliable online experimentation open and accessible to a wider range of technical abilities. To demonstrate the platform’s aptitude for accessible, reliable, and scalable research, we administered a task with a range of participant groups (primary school children and adults), settings (without supervision, at home, and under supervision, in both schools and public engagement events), equipment (participant’s own computer, computer supplied by the researcher), and connection types (personal internet connection, mobile phone 3G/4G). We used a simplified flanker task taken from the attentional network task (Rueda, Posner, & Rothbart, 2004). We replicated the “conflict network” effect in all these populations, demonstrating the platform’s capability to run reaction-time-sensitive experiments. Unresolved limitations of running experiments online are then discussed, along with potential solutions and some future features of the platform. Electronic supplementary material The online version of this article (10.3758/s13428-019-01237-x) contains supplementary material, which is available to authorized users.

The Flanker task is a fairly simple task to program on Gorilla. Out of the five tabs of the Task Builder (Task Structure, Spreadsheet, Stimuli, Manipulations, Script), we only need to use to first three ones.
The "Task Structure" (highlighted in red below) tab allows us to define the different sections within the task, and the different screens within each section.
On the left, we can see the different sections. There is a first set of instructions ("Instructions 1"), followed by a very simple demonstration of the stimuli ("Example 1"). The instructions are fully developed in "Instructions 2" and followed by some practice trials ("Practice_Trials").
1 Supplementary Material: Gorilla.sc: An online behavioral experiment builder 2 On the right, we can see more specifically how the screen is designed for the practice trials. Five images zones have been defined. Their name is written in green: the central fish is the "Target", the flanker fish the "Distractors". The rectangular zone with an orange label indicates the response modality.
Participants have to press a button saying whether the central fish is pointing to the right ("m") or to the left ("z"). They will see the instructions that are written in orange on the screen: "Press the matching key". The squared zone in between the green and orange zones is used to provide feedback. We will not develop it here since it is only used during practice, and not during the main task.
Crucially, the Screen set up only has to be defined once. The images that will populate it from trial to trial are specified in the Spreadsheet.
The "Display" columns mirrors the sections that have been created in our "Task Structure". Let's focus on the "Practice_Trials" display. In the "Target" and "Distractor" columns, we indicate which image will be uploaded for each trial. On the first Practice Trial, for example (Row 5), the Target is a fish pointing to the left, and the Distractors are also pointing to the left: it is actually a Congruent trial, uploaded in the "Stimuli" menu. structure", the "Spreadsheet" and the "Stimuli" tabs, we need to put it in an experiment.

Putting the task into an experiment
Gorilla uses what we call an "Experiment tree". The tree specifies which tasks are used within an experiment, as well as their order. Here, the experiment (Called "Flanker Fish") is pretty simple, because we only use the Flanker Task.
3 Supplementary Material: Gorilla.sc: An online behavioral experiment builder 4 It is shown in blue, and it is surrounded by a "Start node", and a "Finish node", specifying the beginning and the end of the experiment. Many more functionalities are available in the "Experiment tree", allowing researchers to counterbalance the order of presentation for an experiment that contains several tasks, for example.
The "Recruitment tab" allows us to generate a link to share the experiment online, and to select how many participants we would like to recruit, along with any specific technical requirement (device types, browser types of connection speed that the participants would need to have). The "Participants" tab references all the participants that joined the experiment, and the "Data" tab allows us to download our data.