Software and hardware requirements
CFS toolbox requires the installation of MATLAB or GNU/Octave and PTB-3 (Brainard 1997; Kleiner et al. 2007; Pelli 1997). PTB-3 is a set of functions for experimental psychology research that runs on multiple platforms (Windows, Mac, Linux) and that makes the presentation of accurately controlled visual and auditory stimuli easy.
CFS toolbox has been tested under Windows 7 with MATLAB R2014a and Linux Ubuntu 15.04 with GNU/Octave 4.0. The toolbox versions for both environments are available from the project page. CFS toolbox draws and plays visual and auditory stimuli using the functions of ScreenFootnote 4 and PsychPortAudioFootnote 5 in PTB-3. Screen is a function for the precise control of video display. PsychPortAudio contains a set of parameters for working with sounds. The functionality requires a suitable graphic and audio cards and computer. It is recommended that the user refers to the PTB-3 documentationFootnote 6 for up-to-date details.
In our laboratory setup, visual stimuli are viewed through a mirror stereoscope that presents the stimuli in the right half of the display exclusively to the right eye and the stimuli in the left half of the display exclusively to the left eye (see Fig. 1). Some scholars (Hesselmann et al. 2016; Moors et al. 2016b) have used a setup in which two displays that are set opposite to each other, project stimulus to the left and right eye respectively via two mirrors. More details of our mirror stereoscope setup can be found from the toolbox project page.
The program architecture is described in Fig. 2. The primary interface to CFS toolbox are stimulus parameters (“stimulus_parameters.csv”) and trial parameters (“trial_parameters.csv”) files. Toolbox reads the parameter files and runs the experiment according to the parameters. Stimulus parameters file contains parameters for the whole experimental setup (see Table 1). Trial parameters file contains a sorted list of definitions for each individual trial (see Table 2). The parameter values in both files can be changed, although the structure of the files must be preserved when saving.
Furthermore, the structure of the CFS toolbox director and the file names must be preserved (see Fig. 3). For example, there are directors for visual and auditory stimuli and borders and infographs files. The MATLAB m-files and csv parameter files are located in the root of the toolbox (‘/CFS/’). When the all trials defined in trial parameters file have been run, the result file is saved to the result director (‘/CFS/Results’). The structure of result file is presented in Section “Result file”.
Stimulus parameters file
The stimulus parameters define the location and size of the experiment window and the options for all elements of the experiment. All elements, such as border frames, noise pattern, and visual target, are drawn inside the experiment window. That is, the coordinate space of the experiment window is used for defining the locations and sizes of the elements. The location and size of the experiment window are defined using the coordinate space of the display device.
Figure 4 shows an example experiment window containing the elements of visual target, noise pattern, fixation crosses, and border frames. In the actual experiment, mirror stereoscope projects the left half of the experiment window (i.e., visual target or noise pattern) to the left eye and the right half to the right eye. Figure 5 shows a simulated perception of Fig. 4 assuming that the experiment window is watched through a mirror stereoscope. That is, the figure visualizes perception after the visual target located in the upper left half of the left side (i.e., presented to the left eye) of the experiment window is detected.
Table 1 lists the parameters of stimulus parameters file that can be changed in order to modify the experiment setup. For example, the experiment window’s location and size are set by the parameter #2a. The values of parameter #2a are the coordinates for the left-up (x1,y1) and right-down (x2,y2) corners of the experiment window. It is recommended that the user sets the coordinates to fulfill the whole screen area of display. In this way, distracting light sources from the borders of display screen can be minimized.
The coordinate options for visual targets and the coordinate values for noise patterns and border frames are set by the parameters #3c, #4i, and #5a, respectively. CFS toolbox enables the user to set eight different options for a visual target stimulus. For instance, the targets can be displayed in the four corners of the two frames (eight options), or up or down in one frame only (two options). The principle is that the eight coordinate options are recorded in stimulus parameters file from which the trial specific values are then selected for each trial according to trial parameters file.
The border frames are used for framing the left and right active area of the experiment window; that is, the areas for the left eye and for the right eye. The border frame image (‘border.png’) is read from the director ‘/CFS/Borders_and_info’. The type of border frames can be changed by changing the image file from the director while preserving the naming of that file. Different noise types and noise presentation parameters are defined in more detail in Section “Noise”.
After the user has set or changed the coordinate values of visual target, noise, or border frames, it is recommended to run the function “test_settings.m”. The function draws a mock-up of the experiment window that can be used for checking that all experimental elements are correctly located.
A new property of CFS toolbox, compared to many previous CFS research settings, is the possibility to include timed audio stimulus for experimentation; an audio stimulus can be displayed before, at the same time, or after the visual target onset. We see that the possibility to utilize audio stimuli can create new CFS research branches. For example, audio can be utilized in triggering some novel multimodal effects. The timing options for audio stimulus are set by the parameter #6. The timing of visual and audio stimuli is defined in more detail in Section “Stimulus timing”.
Trial parameters file
Each row of trial parameters file defines the presentation of visual and auditory stimuli for one trial. The number of rows defines the number of trials. That is, if 20 rows have been filled, then the experiment will run 20 trials. The order of trials is randomized for each observer by the parameter #5 of stimulus parameters file.
In summary, each trial is defined by four parameters: (1) the location of visual target, (2) the type of visual target (image file), (3) the timing of audio stimulus, and (4) the type of audio stimulus (audio file). The parameters are described in Table 2. The location of visual target is set by integer value from 1 to 8. Actual coordinate locations for different integer values are coded in the parameter #3c of stimulus parameters file. For example, value 1 can be coded to mean the top-left location of the right-side frame in the experiment window, value 2 the top-right location of the left-side frame, and so on.
In this version of the toolbox, there is an option to use two different visual target types in experimentation. The toolbox reads the visual targets from the director ‘/CFS/visual_stimuli’. The value 1 of the parameter #2 means that the visual target is the image file named as ‘visual_stimulus_1.png’, and the value 2 means that the target is the image file ‘visual_stimulus_2.png’. Visual target images can be changed just by changing the image files from the director while preserving the naming of images.
Audio stimulus timing and type are set by the parameters #3 and #4. The timing of the audio stimulus in each trial relative to the visual target is set by integer values from 1 to 4. Actual timing values for audio stimulus relative to visual targets are coded in the parameter #6 of stimulus parameters file. For example, value 1 can be coded to mean that the audio stimulus is presented 0.5 s before the visual target, value 2 can mean that audio and visual targets are presented at the same time, and so on. The timing principle of audio and visual targets is explained in more detail in section “Stimulus timing”.
The audio stimulus type for a trial is set by the parameter #4. In this version of the toolbox, there is an option to use two different audio stimulus types in experimentation. Audio stimulus files are read from the director ‘/CFS/audio_stimuli’. The value 1 of the parameter #4 means that the audio file named as ‘auditory_stimulus_1.wav’ is selected for the trial, and value 2 means that ‘auditory_stimulus_2.wav’ is selected. As with the visual target files, audio stimuli can be changed by changing the audio files from the director while preserving naming of the files.
The timing settings for noise and target stimuli should be carefully designed when building a new CFS experiment. Figure 6 shows an example CFS trial as a function of time line. In the figure, the timings of target stimuli (right eye location in this trial and audio) and noise (left eye location in this trial) have been visualized. For example, the interval of t2 − t4 describes the time between the noise stimulus onset (left eye) and the consequent target stimulus onset (right eye) that follows. This discrepancy between the stimulus timings is used to ensure the dominance of the noise mask at target initiation.
The length of the interval of t2 − t4 is controlled by the parameters of noise start timing (#4g) and random interval (#4h). The noise start timing is a constant delay time for all trials of the experiment. That is, what is the minimum time before visual target forming starts after the noise onset. The random interval is a random additive delay added to the noise start timing. For example, if the value of random interval is 1, then additive delay is something between 0 and 1 s, and the total delay between the noise and visual target consists of the constant noise start timing plus the random delay of 0–1 s that varies from trial to trial during the experiment.
The interval of t4 − t5 is the duration of a visual target reaching its full contrast state. CFS toolbox enables ramping up the targets so that the contrast increases slowly from low to full. This feature is used to avoid the abrupt onset of target stimuli. The duration of transition from low to full contrast is controlled by the parameters of stimulus refresh rate (#3a) and stimulus fade rate (#3b). The stimulus refresh rate defines how many times the stimulus contrast is updated per second. The stimulus fade rate defines how much the contrast increases at each update. For example, if the stimulus refresh rate, r
r = 75f
s, and the stimulus fade rate, f
r = 0.0133, then the interval t4 − t5 is 1.0 s. That is, it takes 1 s to form a full-contrast visual target after the target triggering. If f
r = 1, then the full contrast state is reached immediately after the triggering.
In the example trial presented in Fig. 6, the audio stimulus onset takes place at the time point of t3. That is, the audio file is played before the visual target onset with a duration defined as t3 − t4. For each trial, the duration of t3 − t4 is defined by the parameter #4 of trial parameters file. The principle is that CFS toolbox enables displaying audio files at different time points (as compared to the visual target) for different trials of the experiment.
A trial ends when the observer presses the arrow key in the keyboard or when the value given for the time out is reached. The value of the time out is set by the parameter #7 in stimulus parameters file. Two arrow keys (up, down, left or right) that observers use to respond to the stimuli should be set by the parameter #9. The next trial starts when the observer presses the space key.
The properties of the dynamically changing noise pattern, presented to the other eye than the visual target, are selected by the parameters #4a-#4h of stimulus parameters file. Noise type is selected by the parameter #4a. This version of toolbox contains the functions to form the noise types “grayscale”, “color”, “pepper”, and “dead leaves”. Figure 7 presents the grayscale, pepper, and color noise. The parameters of these noise types are the block size and intensity range. In Fig. 7, the block size of the noise images is 15 pixels. The native size of the noise images produced by CFS toolbox is 500 × 500 pixels. The spatial frequency of the example noise images is 33 units per image width (500 pixels / 15 pixels = 33). That is, the spatial frequency content of the noise images is controlled by the block size parameter. The intensity range is the range from which the intensity values of single blocks are randomly selected. The intensity ranges for the grayscale and pepper noise examples in Fig. 7 are 0–220 and 0–255, respectively.
Figure 8 presents dead leaves noise examples. CFS toolbox uses the code ofFootnote 7 to form the dead leaves noise images. The spatial structure of the dead leaves noise differs from the grayscale, pepper, and color noise, as the dead leaves noise consists of a series of overlapping elements. In this version of toolbox, the form of the elements can be a disk or square. The dead leaves noise with square elements corresponds to the so-called Mondrian patterns that are typically used in CFS experiments (Tsuchiya et al. 2006). The second parameter of the dead leaves noise is the sigma, defined by the parameter #4b. Sigma controls the repartition of the size of the basic shape: s
a → 0 gives a more uniform repartition of shape and s
a = 3 gives a nearly scale invariant image. In Fig. 8, the elements of noise targets are square (up) and disk (down). The sigma value is 1 in the left-side examples and 3 in the right-side examples.
In addition to noise type and noise type-specific parameters, the number of different noise images (n) and presentation frequency (f) should be defined. The dynamically changing noise mask (video stream) is produced by preparing n random noise images that are streamed in consecutive order. For example, when n = 60 and f = 30, CFS toolbox displays a sequence of 60 noise images with a frequency of 30 images per second. That is, the same noise video sequence is re-played every 2 s. The value n is set by the parameter #4d and f by the parameter #4c of stimulus parameters file.
After all trials defined in the trial parameters file have been run, an experiment-specific result file is saved to the result director (‘CFS/Results’). Result files are named as ‘seed-ID-date-dd.mm.yyyy-hh.mm.ss.txt’, in which ‘ID’ is the participant’s identification number, ‘dd.mm.yyyy’ is the date, and ‘hh.mm.ss’ is the exact time when the experiment was finished.
Figure 9 shows example rows from a result file. The first row of the file shows that the trial parameters were selected from the row 9 of trial parameters file (column: trial_rand). That is, the order of the trials was randomized. The visual target of the first trial was the image file ‘visual_stimulus_1.png’ (column: visual_stimulus_file), which was triggered 1.5177 s after the noise onset (column: stimulus_start). The value of stimulus_starts equals to the interval of t2 − t4 in Fig. 6. The position of the visual target was the integer value 1 coded in stimulus parameters file.
The audio file of the first trial was ‘auditory_stimulus _1.wav’ (column: audio_file). In this trial, the audio was initiated at the same time with the visual target (column: audio_start = 0.0000). The answer column shows that the observer pressed the “DOWN” button of the keyboard. The response time was 0.5798 second (column: ansdur). The response time is the duration from the visual stimulus onset to the response button press. It thus tells how long did it take for the observer to detect the visual target.
The function displays the location and size of the experiment window and all elements inside it (Fig. 4). This enables the user to ensure that all targets are displayed correctly (e.g., in the correct locations on the monitor relative to the mirror stereoscope). It is recommended to run this function before running a new experiment.
Once the experiment has been specified in the parameter files, it can be run from MATLAB command window by calling the function run_experiment. This function reads the parameter files and runs the experiment. The experiment window and visual and audio targets are presented using the function draw_stimulus. The noise mask is created using the function create_noise_basic (grayscale, pepper or color noise) or create_noise_dead_leaves (dead leaves noise).
After the experiment is finished, result data are output to a text file and saved to the folder “Results” using the participant’s ID code and the time stamp of the experiment in the file name.