Introduction

One important paradigm of investigation in social psychology and behavioral economics is the interactive study in which the behavior of two or more persons jointly determines the consequences for the participants and the collective. Interactive studies can be contrasted with non-interactive studies mainly used in fields such as cognitive psychology, in which consequences of behavior are not influenced by the behavior of other persons and are only dependent on the person’s own behavior and some predefined algorithm (e.g., determining which answer is right or wrong) or chance. The high and still increasing interest in interactive studies (for reviews see Chaudhuri, 2011; Dawes, 1980; Komorita & Parks, 1995; Sally, 1995; Van Lange, Joireman, Parks, & Van Dijk, 2013) is due to several reasons. Some of the most severe problems society faces today – such as environmental protection and depletion of natural resources – depend on coordinated interactive behavior between persons. Such real-world situations are often structured as a social dilemma in which incentives for the collective and the individual agent diverge (Hardin, 1968). The prototypical examples for an interactive problem structure are prisoners’ dilemmas and public good games. In both dilemmas, persons have to simultaneously make strategic decisions about whether to choose the option that maximizes their self-interest (i.e., to defect) or to choose the option that maximizes the outcome for the collective (i.e., to cooperate), if the others in the collective cooperate as well (Rapoport & Chammah, 1965). A good understanding of the mechanisms underlying cognitive and affective processes in strategic decision making is consequentially of crucial importance for society.

Running interactive studies comes with the pragmatic complication that many individuals participate simultaneously and that their responses have to be anonymously matched in groups of varying size. While in early work on strategic decision making researchers had to rely on effortful and error-prone manual matching or had to use deception (i.e., misinforming participants that they interact with each other although they do not), nowadays research is mainly conducted using software that is specifically developed for this purpose.

One of the most prominent tools for user-generated experiments in this domain is the Zurich Toolbox for ready-made economic experiments (z-Tree). z-Tree, which is developed and maintained by Urs Fischbacher (2007), is probably the most mature and powerful experiment system in this domain yet and continues to be very popular, especially among experimental economists, and this immense popularity is documented in an impressive citation count (Thomson Reuter Web of Knowledge: 1,125 citations for the document as of October, 2014). It offers a graphic approach to programming which appeals to many experimental scientists who either lack sophisticated programming skills or prefer the convenience of an intuitive programming interface. As a core advantage over all-purpose tools (e.g., Java), z-Tree provides easy routines for matching agents and transferring information between them, which is essential for any kind of interactive studies. Technically, the system consists of the programs z-Tree, which contains both the server and the experimenter interface, and z-Leaf, which is the client interface for multiple participants that can interact with each other.

Despite its popularity, z-Tree also has its limitations. Since z-Tree has been developed to be used in economic experiments, it has limited capabilities to record process measuresFootnote 1 due to the fact that economists are typically less interested in cognitive processes than psychologists. Specifically, it lacks support for precise response-time measurement as well as mouse- and eye-tracking, which are of great interest for many behavioral researchers.Footnote 2 Furthermore, the architecture, which requires the manual installation of client software, makes non-laboratory set-ups such as large-scale interactive internet experiments and studies conducted using workers from Amazon’s Mechanical Turk (Mason & Suri, 2012) very difficult or in many cases even impossible to conduct. Finally, z-Tree is provided only for the Windows operating system and is based on a closed-source code which makes the system hard to modify or expand upon without explicit support.

Hence, aside from many advantages, z-Tree’s limited capacity for investigating cognitive processes in a fine-grained manner (e.g., by recoding eye-fixations that precede a choice) reduces the program’s usefulness for psychologists. Researchers that are, for instance, interested in running interactive z-Tree studies with eye-tracking need to rely on effortful and error-prone workarounds based on manual coding (e.g., Fiedler, Glöckner, & Nicklisch, 2012; Fiedler, Glöckner, Nicklisch, & Dickert, 2013).

One alternative is to rely on more general programs for developing and running non-interactive studies. These more general programs have usually been created for local experiments with individual participants that do not interact with each other (Table 1, first section). Programs such as NBS Presentation, E-Prime, or their non-commercial counterparts such as OpenSesame (Mathot, Schreij, & Theeuwes, 2012), PsychoPy (Peirce, 2007), PsyToolkit (Stoet, 2010) or others (e.g., Geller, Schleifer, Sederberg, Jacobs, & Kahana, 2007; von Bastian, Locher, & Ruflin, 2013) already have excellent capabilities for recording process measure. These programs could also possibly be expanded by including libraries that enhance the capabilities for matching, interconnecting, and simultaneously supervising participants in interactive studies, but standard extensions are not available yet and implementing them might be relatively effortful.

Table 1 Overview of programs to develop experiments
Full size table

A second alternative are programs that have been developed to run browser-based online experiments relying on Javascript (Table 1, second section). Tools such as jsPsych (de Leeuw, 2014) and WEXTOR (Reips & Neuhaus, 2002) have been designed for enabling researchers to develop browser-based studies that run on any platform and without any software installation. jsPsych, for example, is a JavaScript library for creating and running experiments in a browser, which allows building a wide range of experiments that can be run online. jsPsych provides the structure for a study and several plugins, that are ready-made templates for simple experimental tasks such as displaying instructions and collecting responses. The researcher can create a study by adapting a partially pre-specified javascript code that calls these plugins in the order they are needed for implementing the research design. Still, although being based on a generally very promising approach, to date neither jsPsych nor WEXTOR provide standard functionality to manage and supervise the interaction between many participants.

The Bonn eXperiment System (BoXS) presented in this paper provides an alternative that has been developed from scratch. It aims to combine core advantages of z-Tree and the established non-commercial software in that it contains full capabilities for programming interactive studies and possibilities to record process data in an easy to use, platform-independent open-source programing tool. At the same time, BoXS also allows for implementing standard questionnaires and non-strategic decision-making studies with persons taking part at the laboratory or over the internet according to standard requirements for web research (e.g., Birnbaum, 2004; Reips, 2002; Reips & Birnbaum, 2011).

Key features of the Bonn Experiment System

BoXS is a platform for conducting experiments that is both powerful and easy to use. The architecture of BoXS is based on open software and supports both laboratory and internet experiments. BoXS allows non-interactive experiments and questionnaires as well as interactive experiments including the recording of process measures such as mouse-tracking and eye-tracking. A comprehensive and up-to-date documentation as well as tutorials and examples are provided to guide and assist experimenters new to BoXS.

Platform independence and flexibility

The BoXS client, which is the part of the system the participants interact with, is implemented as a Java applet. Unlike software compiled for a specific system, Java applets run on Windows, Linux, and MacOS, and do not need to be installed prior to being used, which vastly expands the scope of how experiments using the BoXS can be conducted. Without the need to distribute and install client software, laboratory experiments can be set up easier. Furthermore, BoXS also allows participants to attend experiments from home without the need for a local infrastructure for the experimenter and the “autorun” feature allows them to start experiments at any time. BoXS makes it easier to investigate interactions of participants that are otherwise difficult to reach simultaneously (e.g., intercultural research, forensic research, research on minorities, etc.) and it allows for conducting interactive web-based experiments with participants from different locations, as is for example required in studies involving workers from Amazon’s Mechanical Turk. Finally, BoXS can also be used as a powerful teaching tool as students can use it instantly to create and conduct improvised classroom experiments.Footnote 3

Open-source issues/availability

BoXS is released as open-source software (OSS) in accordance with the terms of the GNU Public License (GPL). This license grants every user the right to download, examine, and modify the software in any way as long as the resulting derivative work is attributed to the original author and is itself released as OSS under the GPL.

Releasing the BoXS as OSS offers two major advantages for the users. Many experiments include the recording of sensitive information such as individual preferences, attitudes, and personal information (i.e., income, health). This can raise questions about the system’s reliability in handling and protecting this data. In BoXS the experimenter can examine and retrace the software routines herself in order to verify their correctness and appropriateness.

The second major advantage of releasing BoXS as OSS is that it enables experimenters to expand upon and adapt BoXS. If the need for specific extensions arises, for example an element in the user interface that supports a specific type of hardware device, the experimenter can implement such a change herself without relying on the support of the system’s main developers. Furthermore, upon creating such an extension, the experimenter can submit and publish this extension to make it available for other researchers in need of a similar functionality.

Feature set

The basic feature set of BoXS allows experimenters to implement most popular economic and many psychological experiments.

The display method can present basic text, static and animated images, as well as tables and enumerations (Fig. 1). This functionality can be enhanced by using web services like Google Image Chart,Footnote 4 which allow the experimenter to include complex graphs and diagrams based on experiment data.

Fig. 1
figure 1

Examples of a display method

Full size image

The available user interface components include text fields, buttons, radio buttons, check boxes, and slider bars (cf. Reips & Funke, 2008). It is possible to either use two-step input procedures in which data for each page are entered and confirmed with a button press or one-step input procedures in which the program directly reacts on hitting pre-defined keys. The experimenter can define arbitrary restrictions on the participant’s input (e.g., ensuring that the person’s age is above 10 years), and provide specific error messages to assist participants in correcting mistakes. The elements of the user interface can be laid out both automatically and manually, in which case the experimenter can specify the exact position of each element.

BoXS provides an “input history” and also supports mouse-tracking and eye-tracking. The “input history” records every input submitted by each participant with a time-stamp, which allows the experimenter to detect instances in which choices were revised and to analyze the time participants spent pondering over each item contained in one display. An integrated mouse-tracking routine records every movement of each participant’s mouse. Finally, eye-tracker support including a basic calibration procedure and the recording of the tracking data during the experiment is available.Footnote 5 All three data sources are automatically synchronized with the respective participants’ system clocks and can therefore be jointly analyzed after the experiment.

On the basic programming level BoXS provides both basic algebra (+, -, /, *, % (modulo)), logical operators (<, >, <=, >=, ==, !=), common mathematical functions (exp, log, sin, cos, tan, round), uniform and gaussian random number generators, as well as methods for conditional (if) and repeated execution (for, while). The data within a BoXS experiment is stored in local, group-specific, or global variables that can be accessed and modified at any point in the experiment.

Finally, four different kinds of matching of individuals for interactive studies are provided, including alphabetical matching, stranger matching, perfect stranger matching, and manual matching, which can be used to implement arbitrary matching patterns. For more detailed information and example programs please refer to the online documentation.Footnote 6

Documentation and getting started

Experiments for BoXS are written in a custom programming language (BoXSPL), which is designed to be easy to learn and master. Novice users of BoXS can get started by watching a tutorial video and examining examples that demonstrate how basic experiments like questionnaires or public good games can be implemented. An up-to-date online documentation that also provides detailed information on each feature is maintained on the official homepage. The “frequently asked questions”-section (FAQ) of the website and a mailing list provide further support for novice and advanced users.

What is inside BoXS?

From a technical perspective, BoXS consists of a server process, which manages participants and experiments, and a client applet, which provides the graphic user interface (GUI). The server is a central process to which the experimenter and all participants connect. The server parses and executes programs, handles the necessary communication between participants, and collects and relays all data generated by the experiment to the experimenter. The strict separation of experimenter client and server offers the advantage that a crash on an experimenter's computer does not affect running experiments. Equally, neither the crash of a participant’s computer nor participants accidentally closing the browser affects the running program. Participants can usually reconnect to and resume the experiment by restarting their web browser (or computer) and reopening the given experiment address. They then can regularly continue the study at the point they left it. The default BoXS server is located in Bonn (Germany) and can be used without further set-up. Experimenters are, however, free to set up their own custom servers, which may be advisable for environments in which no sufficiently reliable internet connection is available.

The client applet connects to the server and provides a GUI for participants and experimenters (Fig. 2). Based on the respective login information, the server decides whether the client applet takes on the role of an experimenter client or a participant client. The experimenter client provides an interface that consists of a code editor, in which the experimenter can program and manage the experiment code. The experimenter client also allows the experimenter to track the status of all connected participants as well as all variables generated in the running experiment. The appearance of the subject client is based on the running experiment and can include text, graphics, and input elements (i.e., radio buttons, slider bars).

Fig. 2
figure 2

Experimenter client (left) and participant client (right)

Full size image

Multiple instances of applets can be executed alongside (for example, one experimenter client and two subject clients) which is useful for implementing and testing new experiments. If the connection between any client and the server is disrupted, the running experiment can usually be resumed without further issues by simply restarting the respective applet. A detailed explanation of the inner workings of the BoXS would exceed the scope of this paper. For more information, please refer to Seithe (2012). The latest version of the source code and files for custom server installations of BoXS can be downloaded from the official webpage (see footnote 6) and are additionally provided at https://github.com/plattenschieber/BoXSServer

Applications and examples

Previous versions of BoXS have already been successfully used for local experiments investigating strategic decision making in social dilemmas within one laboratory (e.g., Glöckner & Hilbig, 2012) but also for experiments including interactions between laboratories in different cities (Dorrough, Glöckner, Hellmann, & Ebert, 2015). In the currently released version 1.0, some further features have been implemented that allow BoXS to be applied to a wider range of research questions in psychology and to make handling easier.

How easy it is to realize various experiments with a few lines of code in BoXS will be demonstrated in the following three examples. In the first two examples we present a basic version of the program in the main text and an advanced version in the appendix. The basic versions already include all functional parts; the advanced versions demonstrate how additional useful features can be added to the programs. Due to formatting restrictions, the printed versions of the provided example programs contain unintended line breaks. Executable versions are available at https://github.com/plattenschieber/BoXSServer/tree/master/examples

Example 1: risky-choice study

The first example involves a classic paradigm from individual decision making. Participants are repeatedly asked to choose between risky prospects. The provided risky choice program makes use of the Google Charts API to present gambles (Fig. 3). In the basic version provided below, ten randomly generated choices between two-outcome gambles are presented. The generated charts are displayed until one gamble is selected by pressing a key. The decision time for each task is measured. The first part (lines 2–8) generates the instruction screen, the main part (lines 9–53) presents ten randomly generated gambles in a loop, and the last part (lines 54–55) finishes the experiment.Footnote 7 In the main part, probabilities and outcomes of gambles are generated and stored in variables (lines 11–19), graphic gamble presentation is prepared (lines 23–25), and gambles are presented to the participants until one of two keys is pressed (lines 27–34). Finally, variables and gamble information are stored (lines 36–45) and the program waits for a key response before continuing to the last screen (lines 47–52).

Fig. 3
figure 3

Display in risky-choice experiment in a basic (top) and an extended (bottom) version

Full size image

// **** BEGIN Code Example 1a ****

while(_lastkey ! = 32)

{

display(" < CENTER > <H2 > Choice Study </H2 > </CENTER > <br > <br > In the following you are asked to select the gamble you prefer. <br > Each gamble consists of two outcomes which realize with some probability that is visually displayed")

display(" < br > <br > <H4 > Please press space to continue</H4 > ")

recordKeys()

waitTime(50)

}

for (i = 0; i < 11; i = i + 1)

{

// randomly generate gambles and store in variables to shorten the google chart api call

Lp1 = round(randomUniform()*100)/100

Lo1 = randomUniformInteger(-10,10)

Lp2 = 1-Lp1

Lo2 = randomUniformInteger(-10,10)

Rp1 = round(randomUniform()*100)/100

Ro1 = randomUniformInteger(-10,10)

Rp2 = 1-Rp1

Ro2 = randomUniformInteger(-10,10)

// measure the time when gamble is shown

beginTime = time

// collect images from google api

// gather players decision

while (_lastkey ! = 67 && _lastkey ! = 77)

{

// present pies to player

display("Gamble A: " + tdPic1 + "Gamble B: " + tdPic2 )

display(" < br > <H4 > Please press 'C' to select Gamble A and 'M' to select Gamble B </H4 > ")

recordKeys()

waitTime(50)

}

// save gambles, choice and time

results[i][0] = _lastkey

results[i][1] = time - beginTime

results[i][2] = Lp1

results[i][3] = Lo1

results[i][4] = Lp2

results[i][5] = Lo2

results[i][6] = Rp1

results[i][7] = Ro1

results[i][8] = Rp2

results[i][9] = Ro2

while(_lastkey ! = 32)

{

display(" < br > <br > Please press Space to continue")

recordKeys()

waitTime(50)

}

}

display(" < br > <br > Thank you very much for taking part in the study!")

waitForExperimenter("Finish")

// **** END Code Example 1a ****

The full-fletched version of Example 1b in the appendix uses predefined gambles that are hard coded into an array structure, which is then accessed randomly via an index array “shuffle.” It furthermore uses a table display and adds the selection of one gamble at the end which is played out and incentivized (see Fig. 3, lower part).

Example 2: Repeated prisoners’ dilemma with eye-tracking

The second example shows a basic version of a prisoners’ dilemma in which participants play ten rounds and dyads are reshuffled every round (Fig. 4). Using the eye-tracking feature, which currently supports the eye-trackers provided by SensoMotoric Instruments (SMI), consists of five simple commands: (1) initializing the connection: eyetrackerInitialise ("localhost",4444,5555), (2) performing the calibration: eyetrackerCalibrate(), (3) starting the recording: eyetrackerStart(120, filename), (4) sending triggers to indicate events in BoXS: eyetrackerTrigger("Trigger Name") (see extended Example 2b), and (5) stopping the eyetracker: eyetrackerStop(). Eye-tracking data can be recorded for all participants by removing the comment markers (//) in front of the eye-tracker commands.

Fig. 4
figure 4

Display in the Prisoners’ Dilemma (Example 2a)

Full size image

// **** BEGIN Code Example 2a ****

for (gameCounter = 1; gameCounter < =10; gameCounter = gameCounter + 1)

{

matchStranger(A,B)

i = i + 1

if (i==1)

{

display(" < H2 > Prisoners Dilemma Game </H2 > <br > In this study in 10 subsequent rounds you and a randomly chosen other person will simultaneously decide whether to cooperate or to defect. < br > ")

display("If you both decide to cooperate, you will both earn 200 point. If you both decide to defect you will both earn 100 points. <br > ")

display("If one of you decides to defect and the other to cooperate, the defecting person receives 300 points and the cooperating person receives 50 points. <br > ")

display("In each round you are randomly matched with a new person.")

display("The total points your earned are paid to you according to the following exchange rate: 500 point = 1 Euro.")

wait()

//eyetrackerInitialise("localhost",4444,5555)

display("The calibration begins after pushing the wait button < br > Please hold still and follow the points.")

wait()

//eyetrackerCalibrate()

display("Thank you - the calibration was successful")

//eyetrackerStart(60,"PrisonersDilemma_" + username + "_" + gameCounter + ".dat")

wait()

}

clear()

choice(action,"Cooperate", "Defect")

waitForPlayers("Continue…")

// calculate Payoffs

if (A.action == "Cooperate" && B.action == "Cooperate")

{

A.win = 200

B.win = 200

}

if (A.action == "Cooperate" && B.action == "Defect")

{

A.win = 50

B.win = 300

}

if (A.action == "Defect" && B.action == "Cooperate")

{

A.win = 300

B.win = 50

}

if (A.action == "Defect" && B.action == "Defect")

{

A.win = 100

B.win = 100

}

display("You choose " + action + " < br > Your opponent choose " + opponent.action + " < br > Your roundwin equals " + win)

// save win and actions into the global array 'vp'

totalwin = totalwin + win

*.*.vp[username][1][i] = action

*.*.vp[username][2][i] = win

wait()

if (i==10)

{

display("You won " + totalwin + " Points. This equals " + totalwin/200 + "EUR.")

// eyetrackerStop()

waitForExperimenter("FINISH")

}

matchDone()

}

// **** END Code Example 2a ****

An extended example with an optimized display that also allows eye-tracking for a single player (e.g., in case one has only eye-trackers in the laboratory) and involves the sending of detailed time triggers is provided in the appendix as Example 2b.

Example 3: Repeated public good with mood induction and eye-tracking or mouse-tracking

Example 3 shows a public good game in which four persons can contribute to maximize their shared profit (Fig. 5). The example includes a mood induction by video (converted to gif), full eye-tracker usage (if comment markers in front of eyetracker commands are removed), usage of mouse-tracking (if comment markers in front of ‘enableMouseTracking()’ are removed), a graphic visualization of the contributions and reshuffling of groups after every round (stranger matching) in only 43 lines of code (without comments and newlines).

Fig. 5
figure 5

Display of the public good experiment from Code Example 3; left: mood induction by video, middle: contribution screen, right: feedback about contributions of others

Full size image

The interaction goes over ten rounds implemented by a loop over gameCounter. Participants are randomly re-matched every round in new groups of four (lines 5 and 48). Before the first round starts, several issues are taken care of (lines 7–23): the eye-tracker is turned on (lines 10–11) and calibrated (lines 15–17), the instruction is shown (lines 12–14), and the mood induction is carried out by playing a video (lines 20–22). Participants’ contributions to the public good are recorded and checked (lines 27–30) and graphic feedback concerning the contributions of all four players is presented (lines 31–32). The round payoff and the total payoff for all participants are calculated (line 34–35) and displayed (lines 40–41) and all variables are stored (lines 36–39). After the last round, the final payoff is displayed and the eye-tracker is stopped (lines 42–47).

// **** BEGIN Code Example 3 ****

for(gameCounter = 1; gameCounter < =10; gameCounter = gameCounter + 1)

{

// match 4 players each round

matchStranger(A,B,C,D)

i = i + 1

if (i==1)

{

// enable mouse tracking

// enableMouseTracking()

// eyetrackerInitialise("localhost",4444,5555)

display(" < H2 > Public Good Example</H2 > <br > Please read the paper instructions in front of you before starting the experiment.")

display("The calibration begins after pushing the wait button. < br > Please hold still and follow the points.")

wait()

//eyetrackerCalibrate()

display("Thank you - the calibration was successful")

//eyetrackerStart(60,"PublicGoodGame" + username + "_" + i + ".dat")

wait()

// mood induction video, showed for 10 sec.

display("Please watch the following movie!")

display(" < img src = '' > ")

waitTime(10000)

}

beginTime = time

clear()

// read in contribution and check for consistency

display(" < br > Please choose the amount you would like to contribute:")

inputNumber(z)

assert(z > =0 && z < =10 && z==round(z))

waitForPlayers("Continue…")

display("You are player " + role + ". You contributed < b > " + z + " Talers</b>. <br > The total amount donated is < b > " + sum(z) + " Talers</b > .")

// Calculate payoff and save decision, decision time in ms and payoff

payoff = 10 - z + 0.4*sum(z)

totalPayoff = totalPayoff + payoff

results[i][0] = z

results[i][1] = time - begintime

results[i][2] = payoff

results[i][3] = totalPayoff

display("Your payoff is < b > " + payoff + " Talers</b > ")

wait()

if(i==10)

{

display("Your total payoff is < b > " + totalPayoff + " Talers</b > ")

// eyetrackerStop()

waitForExperimenter("FINISH")

}

matchDone()

}

// **** END Code Example 3

Validation of time measurement and presentation timing

To assure exact measurements of response times as well as synchronization with external devices such as eye-trackers, BoXS reads time directly from the system clock of the client PC. Measurement errors and delays in stimulus presentation due to slow connections with the server can therefore be avoided.

We used the time measure of a high precision eye-tracker from Sensomotoric Instruments SMI RED250 to validate time measurement in BoXS. Specifically, we wrote a program that sends time stamps from BoXS to the eye-tracker and to record the timing of these stamps in BoXS as well. The times recorded by the eyetracker and in BoXS on average deviated by 0.2 ms and the maximum difference observed was 4.2 ms. Hence, time measurement in BoXS can be considered to be very precise and valid. At the same time, this validation shows that measurement in BoXS and in external eye-trackers is very well synchronized.

In a second validation study, we investigated the precision of stimulus presentation in BoXS. Stimulus presentation is naturally limited by the refresh rate of the monitor in that stimuli cannot be presented shorter than the time which the monitor needs to overwrite the previous display. Many of the programs for local experiments with individual participants such as PsychoPy or DirectRT have the ability to control stimuli on the level of single display frames. BoXS does not have this capability, so that lower precision in timing can be expected. In the validation study we presented stimuli (numbers vs. pictures from IAPS (Lang, Bradley, & Cuthbert, 2005)) for durations of 20, 50, 100, 200, 500, and 1,000 ms each 200 times using the waitTime function of BoXS. We measured the (validated) time after each presentation and calculated time differences to infer presentation time. The differences were on average 1.6 ms larger than the programmed durations (SD = 1 ms) with a range of deviations between 0 and 16 ms. Note that our measurement includes time for stimulus presentation plus processing some lines of code so that a positive deviation could be expected. The time for presenting IAPS pictures was 0.16 ms longer than the time for presenting numbers, thus complexity of the stimuli did not make a noticeable difference for presentation time. Hence, overall the waitTime function can be considered to work very precisely for timing stimuli.

Although the second validation study shows that stimulus timing in BoXS is very precise this does not necessarily mean that the true presentation duration is precise as well since the stimuli might be timed for presentation but still not be shown for the correct time on the screen. Most importantly, for very short presentation times the interplay between hardware and software could lead to the fact that stimuli are not presented at all. We conducted a third validation study in which we presented stimuli for 50, 80, 100, and 200 ms with an intertrial interval of 1,000 ms on a laptop with a 50-Hz display. Visual inspection showed that only very few stimuli were actually presented at 50 ms, some stimuli were omitted for 80 ms, and essentially all stimuli were shown for 100 ms and beyond. The presentation time was noticeably longer for 200 ms as compared to 100 ms, although the precise actual presentation duration on-screen could not be measured further. Given this result, BoXS cannot be used for priming below 100 ms, such as subliminal priming. Furthermore, taking 50 ms as the threshold for the time necessary to present stimuli on the screen, arguably screen presentation time could be considered to be precise ±50 ms.

Challenges and limitations

Complex applications and interactivity

One further challenge for web-based experimental systems such as BoXS is that they should be able to implement designs that require fairly intensive, real-time interaction between many persons. Could, for example, a fully functional double auction with 16 participants be programmed in this software? And can BoXS handle second-by-second interaction between players in real time? In general, the BoXS server can handle such complex interactions with many subjects at the same time, including quasi real-time cross-subject interactions. While the BoXS was already successfully tested in experiments involving many simultaneous users, it is hard to specify the system’s exact limits as they rely on numerous factors including the performance of the involved computers, the latency of the network, and the type and complexity of the experiment. Hence, sufficient pre-testing of such complex experimental programs is required.

Furthermore, it would be useful for some kinds of interactive studies if a web-based experimental system could have the functionality to handle interactions in which participants use input devices (e.g., sliders), which instantly causes real-time changes to a graph. As introduced above, in BoXS graphs and diagrams can be included in an experiment and shown to the subjects by using the Google Charts API. This API can create different types of diagrams based on data provided by the program. Examples for this would be a pie chart visualizing risky prospects (Fig. 3) or the contributions in a public goods game (Fig. 5) or a line graph showing the change in contributions in a repeated trust game. While this approach does provide many useful diagram types, it is limited by the fact that the subjects cannot interact with these diagrams. Compared to the BoXS, z-Tree does provide a greater selection of interactive user interface elements, and if these are necessary in an experiment it might be advisable to implement the study using z-Tree.

Mobile devices

For some types of studies (e.g., tracking of behavioral changes over longer periods) it becomes increasingly important that studies can also be run on mobile devices such as “smartphones” and tablets. As introduced above, the BoXS client requires the Java browser plugin in order to work properly. While this plugin can be easily installed on most computers, it is unfortunately available for neither iOS nor Android, which are the most popular smartphone- and tablet-operating systems. Hence, in its current implementation BoXS does not allow implementation of such studies since it does not run on most smartphones and tablets. For such studies Javascript-based experimental systems such as jsPsych might be an alternative, since Javascript is usually available on smartphones and tablets too (although they do not allow implementation of interactions between persons yet).

In the long run and given sufficient support, the BoXS could be extended to support such mobile devices. Even experiments in which both desktop and smartphone users participate at the same time might be possible. From a technical perspective there are two approaches implementing mobile device support. The first approach would be to program platform-specific clients for mobile-operating systems. Given its large user base and the fact that it is itself based on Java, the Android system seems to be a particularly appealing choice for this approach. Programming such a platform-specific client would allow most of the BoXS features to work on mobile devices. In addition, it would be possible to create device-specific user interface elements which could adapt to smaller screen sizes and other device-specific properties. The alternative approach would be to write a purely HTML-based client for simple experiments which would work on PCs and mobile devices alike. The advantage of this approach would be that it would work on all mobile operating systems. Unfortunately, the set of features which can be implemented using pure HTML is limited. Especially advanced features like time-based experiments or experiments involving real-time interaction may be impossible to implement using this approach. One potential solution would be to also include elements of Javascript, potentially combining positive features of jsPsych and BoXS.

Discussion

Bonn eXperimental System (BoXS) allows experimenters to develop and run interactive and non-interactive studies in and outside the laboratory. It is particularly useful for conducting research with persons at different locations interacting over the internet, which is for example necessary in studies involving participants from Amazon’s Mechanical Turk. BoXS combines core features of z-Tree and specialized programs for developing non-interactive studies in that it provides simple routines for connecting, matching, and supervising participants and recording process measures. BoXS is platform-independent and completely browser-based to allow for large-scale interactive experiments around the world. The source-code of BoXS is provided under an open license which allows experimenters to extend the program if necessary and to verify data handling for data protection reasons. Finally, BoXS provides a relatively simple programing language which is easy to learn and to apply.

BoXS has been successfully used in recent research and could provide a viable tool for the growing number of researchers interested in cognitive processes and interactive behavior. BoXS can also be applied for conducting simple non-interactive studies. However, one has to acknowledge that the tailoring of BoXS for allowing browser-based interactive studies comes at a cost. Some functionalities concerning data manipulation and data input (i.e., loading stimuli files and saving data files) are currently limited due to Java’s inherently strict security policies. Workarounds, such as hard coding of stimuli (see Example 1b, appendix), can be implemented, but they might not be very elegant. Other programs such as PsychoPy and OpenSesame might be preferable to BoXS for running non-interactive cognitive studies that involve comprehensive data manipulation. Furthermore, we could show in validation studies that time measurement in BoXS is very precise and synchronize with other external devices such as eye-trackers by relying on the same high precision system time at the user client. The internal timing of presentation durations is precise as well, but it was found that the interplay between software and hardware makes priming with stimuli presentation durations below 100 ms basically impossible in BoXS. At 100 ms and above we estimate display time to be precise ± 50 ms, which should be sufficient for many applications. Also BoXS has the functionality to implement complex real-time interactions of many individuals, although it has to be acknowledged that it has not been intensely tested for such purposes yet. Two further important limitations are that BoXS in the current version cannot handle real-time changes in interactive graphs and that it does not run on most mobile devices since it requires Java.

The currently released version 1.0 of BoXS contains many crucial features that allow investigating cognitive processes in interactive studies. BoXS is aimed to make conducting such research easier. BoXS will allow answering research questions that were hard to address with previously established tools such as hypotheses concerning changes in attention and arousal in the course of making interactive decisions as well as many others. BoXS is also constantly improved and extended to include additional features and support for external devices of other manufacturers to allow broadening the range of experimental applications and making the program attractive for more potential users. One important project for future developments is also the extension of the BoXS to run on mobile devices. We hope that BoXS can contribute to the cumulative development of knowledge by supporting researchers with implementing their innovative research ideas also concerning complex topics involving cognition and interactive behavior.