Introduction

The ways in which researchers have been able to observe participants have changed significantly over the years. When Barker and Wright (1951) turned the town of Oskaloosa in Kansas into their field laboratory and studied a boy’s life via continuous observation, there were no more advanced technologies than cameras and microphones to capture phenomena as they occurred in their natural settings. Continuous field observation was costly and burdensome, and a careful process was required to distill the full mass of information to its essence. Researchers thus turned to the idea of randomly sampling from the stream of experience. At the beginning of electronically supported experience-sampling research, participants had to carry pagers, and when those beeped, they wrote down details of their current experience. Smartphones have made it possible for participants to record their responses directly, making smartphone-based tasks equivalent to computer-based tasks. The question of whether smartphones can be used to bring experience sampling and other typically computer-based tasks from the laboratory to the field has already been answered positively in this journal with, e.g., supporting empirical evidence from our own laboratory (Stieger, Lewetz, & Reips, 2018).

Experience-sampling methodology (ESM) encompasses a range of methods based on prompting participants to respond to a survey or a task at various time points, which could be random (Ruwaard, Kooistra, & Thong, 2018). In the health sciences, this approach is known as ecological momentary assessment (EMA; Shiffman, Stone, & Hufford, 2008). The main idea is to assess peoples’ experiences or behaviors while they are having those experiences or exhibiting those behaviors, thus minimizing the potential for retrospective memory distortion (Schwarz & Sudman, 2012). A further advantage of ESM is increased external validity, as thoughts and behaviors are assessed in their natural environments (Stone, Shiffman, & DeVries, 1999).

Repeated assessments of varying intensity also enable a more detailed analysis of trends and processes than does cross-sectional research. Measurement intensities can be tailored to specific research questions and proposed processes. In addition to random interval sampling, with reminders sent by investigators, participants can also be asked to complete a survey when a specific event occurs—known as event-based sampling. For example, in a smoking cessation study by Shiffman et al. (2008), participants were asked to complete the survey every time they smoked a cigarette.

Numerous research questions can be answered using ESM data, such as individual differences, the influences of context and time, and interactions between personality and the environment (Shiffman et al., 2008). ESM is applicable not only in the health sciences but also in understanding human reactions to their environments: people’s thoughts and feelings can be used to create a representation of their environment, like a city. Research on urban emotions has examined people’s perceptions of and emotional reactions to city structures (Kohn et al., 2018; also see Klein & Reips, 2017) using data collected while they moved through them. Participants in Kohn et al.’s study (2018) reported their positive and negative emotions linked to locations in a park area. This approach can support the design and construction of optimized living places, where people are more satisfied with their surroundings. The present authors, for example, have been trying to cross-validate ESM ratings and enhance research about environments and users’ reactions to them by including data such as temperature, longitude, latitude, altitude, wind speed, rainfall, etc., from external databases to predict fluctuations in subjective ESM ratings (Stieger & Reips, 2019).

There is now a wide range of applications—commonly referred to as apps—supporting experience-sampling research (Conner, 2015). One approach developed in the late 1980s is based on Short Message Service (SMS) technology. Commercial companies like Twilio, Message Media, SurveySignal, or Poll Everywhere use SMS as part of their services (e.g., market research). This approach requires participants to send SMS messages when prompted (SMS-to-SMS technique). Since SMS must usually be paid for, the cost of messages is usually included in participants’ remuneration fees. Because messages are limited to 160 characters, commercial companies often send several questions in succession to elicit responses. To circumvent this limitation, some researchers gave participants paper-based forms with a coding scheme that allowed them to respond concisely with codes and abbreviations (Andrews, Bennett, & Drennan, 2011). Another way to overcome the limitations of SMS length is to provide a web link within the SMS which leads to an online survey. Hofmann and Patel (2015), for example, developed the web-based SurveySignal application, which links messages to online surveys.

An alternative approach is using a standalone application installed on a preprogrammed handheld device or the user’s smartphone (e.g., personal data assistants [PDA] vs. local application approach, in Hofmann & Patel, 2015). Participants using their smartphones must download and install an app to receive notifications, but as Hofmann and Patel (2015) noted, some participants may be wary of this procedure. Mistrust can be reduced if the app is distributed through official stores (Google Play or App Store), which indicates that the app has been reviewed and approved for the general public. Installing apps has since become common practice, and most smartphone users are experienced with the procedure.

Many of the proprietary standalone apps or web services for sending SMS have costs, either via a subscription model or a fee per participant or measurement. For example, the SurveySignal subscription plan costs USD 50 per month and is limited to 1000 “signals” (equivalent to 500 SMS messages); MovisensXS provides a standalone Android app and recommends an 8000-credit bundle (EUR 500) suited for a five-day survey involving 50 participants. Because Samply does not send SMS, but instead operates through a standalone smartphone app, there are no costs associated with notifications. Samply is free for both researchers and participants. Its open-source code is available via GitHub (https://github.com/Yury-Shevchenko/samply), and the web application for researchers is accessible at https://samply.js.org. We believe that the open-source approach will benefit Samply’s long-term sustainability (Bonaccorsi & Rossi, 2003).

Some standalone apps only work on one of the mobile platforms, such as only with iOS (ESm Capture, iDialogPad) or only with Android (MovisensXS, SampleMe, Aware). Our first approach also suffered from this limitation (Reips, Stieger, Heinrichs, & de Spindler, 2016), but the Samply Research mobile app is available on both major platforms and can be downloaded from Google Play for Android and App Store for iOS mobile operating systems. The original code (written with the React Native JavaScript library) is rendered in the native code on both platforms, so the application displays the same user interface as native applications. Because we maintain one codebase in JavaScript, the application is functionally identical on Android and iOS, with the only difference being the user interface’s platform-specific design.

Thirdly, even though experience-sampling software is available on both platforms, researchers are often limited to predefined question types. For example, ESm Capture only allows researchers to choose between binary, tripart, multiple choice, slider, and open-ended questions. The LifeData service has more options, but that app’s user interface limits customization. Instead, we have connected a mobile app to a mobile web browser by linking native push notifications to web links that are opened in the mobile browser. As a result, researchers can link browser-based studies to notifications sent by the Samply Research app. Many of today’s survey and experiment-building platforms (e.g., Qualtrics, SurveyMonkey, lab.js) are mobile-friendly. Responsive design adapts the user interface to the device. Another advantage of linking a web study to a mobile app is that no additional software needs to be installed on the participant’s smartphone: installing the Samply Research mobile app links the user to the web browser. Because our push notifications are sent using Internet protocols via Wi-Fi or cable and enable browser-based studies, our approach’s only prerequisite is the Internet connectivity of participants’ smartphones or other devices—SMS reception is not required, and devices that are not connected to a mobile service provider can also be used if they have an Internet connection. As the mobile InternetFootnote 1 continues to expand (via phone providers or Wi-Fi networks), we believe that this approach is feasible.

The following work describes the methodological challenges of conducting an experience-sampling study and outlines the solutions already programmed in Samply (see Table 1). More technical details can be found on the project web page (https://samply.js.org). We subsequently provide a step-by-step tutorial for researchers interested in conducting a study using Samply, and finally, we present the results of two validation studies whose goals were to (1) improve the web app’s design and usability for researchers, and (2) validate the data collected through the Samply Research mobile app (i.e., participants’ interactions with notifications) by using an external survey’s results as a criterion.

Table 1 Overview of tasks and solutions implemented in Samply
Full size table

Methodological challenges

Notifications content

Notifications may cause disruption if they require attention at inopportune moments. Previous studies have shown that the sender–recipient relationship and the type, complexity and level of completion of the participant’s task interrupted by the notification all influence the response time and the perceived level of disruption (Mehrotra, Pejovic, Vermeulen, Hendley, & Musolesi, 2016). The notification’s priority level may also decrease the response time, and the higher the perceived level of disruption, the higher the probability of non-response.

Thus, the notification’s priority, suggested in the title and message, and the researcher–participant relationship can help to reduce the risk of notification dismissal. Samply’s notifications contain a title and a short message inviting the participant to click on it to complete a survey. In contrast to a text message that is limited to 160 characters, the Samply notification can contain an embedded web link of any length and unseen by the user. The notification’s three components of title, message, and study URL are customizable and can be changed in Samply’s administration menu by the researcher. The available length of the title and message displayed depends on the screen size and text language. Longer texts can appear trimmed on the smartphone screen (e.g., “Dear user, welcome to the survey on …”). In general, it is good to have a distinctive notification title, which helps participants quickly recognize that the notification is important. Further studies will be required to determine which notification-content features improve participants’ experience and reduce non-response.

The effect of the mobile device

Van Berkel, Ferreira, and Kostakos (2017) found that about half of the studies in their meta-analysis provided participants with a mobile device, whereas participants in the other half used their own devices; the authors concluded that the use of personal devices in research was increasing. This trend is likely to continue as the number of personal mobile devices is growing. There are some disadvantages to providing participants with devices: they may have to learn how to use the new device, so some training may be required before the data collection, and they may feel uncomfortable about being observed by a dedicated device which they are unaccustomed to. Using homogeneous equipment across the study, however, gives researchers more control over how notifications and survey questions are displayed on the device (Meers, Dejonckheere, Kalokerinos, Rummens, & Kuppens, 2020).

The Samply Research native mobile app supports both strategies (providing a smartphone or having participants use personal ones) on both the Android and iOS mobile operating systems, which together accounted for 99.32% of market share worldwide in May 2020Footnote 2.

Should the personal-device strategy be chosen, one way to ensure that the survey works on different users’ devices is to conduct a compatibility test (Hofmann & Patel, 2015). Participants are asked to respond to a notification, which takes them to a screening test where they provide information about their mobile device and practice answering questions in formats typical of those that will be used in the study. For example, they may be asked to select one option from a set of radio buttons, move a slider to set a specific value, or enter specified text in a text field. If their answers are correct, participants are given a password to proceed with the study.

Notifications schedule

The other necessary component of an experience-sampling study app is the types of notification schedules it supports. In their meta-analysis, Van Berkel et al. (2017) found that the three most commonly used schedule types were randomized time points during the day (e.g., three times a day), time points at specific intervals (e.g., every hour), and event-dependent time points. Samply supports the first two types of schedule based on time points and intervals. Event-dependent notifications, as defined by the participant (e.g., smoking a cigarette), can be implemented via the Samply Research mobile app’s interface. The participant can be instructed to use the app to answer survey questions each time an event occurs (smoking a cigarette) . Notifications based on events detected by the mobile app using mobile sensors (GPS, gyroscope, etc.) are planned to be implemented in the future. For example, we are currently working on geofencing technology that will allow participants to be sent a notification when they enter or exit a specified location such as their home, office, or a public area (e.g., Klein & Reips, 2017; Nguyen et al., 2017).

Samply also records users’ interactions with their notifications, so it is possible to customize their notification schedule based on user response. For example, if compliance is low, researchers can implement a contingency plan such as contacting participants to help them with technical problems or switching to a paper-based version of the study (Thomas & Azmitia, 2016).

Analytics

One line of research involves interruption management, i.e., analyzing people’s interactions with mobile notifications and exploring more efficient ways of interrupting users with notifications (Pielot et al., 2017). Switching between the phone’s vibrate, ring, and silent modes is one way that people often manage notifications. People rarely tend to adjust notification settings for individual apps, but usually prefer to change general settings (Pielot et al., 2017). Previous research found that 50% of interactions with notifications occurred in the first 30 seconds, and users’ most important notifications involve communication with other people (i.e., messengers) or provide information about the user’s context (i.e., calendar events) (Shirazi et al., 2014).

Ideally, information on users’ interactions with notifications should be available to researchers conducting experience-sampling studies. Samply automatically records the moments when notifications are sent from the server and when participants tap on them—information that can help researchers design better strategies for approaching participants.

One major practical question for a researcher is whether a notification should be withdrawn if it remains unanswered for some time. The push notifications used in Samply are displayed on the smartphone until the user interacts with them. Depending on the device, they may be hidden in the notification bar or shown in the background. It is up to the researcher to decide how to handle the data from participants (or their devices) who delay their responses. Samply’s analytics provide the data to make this decision: by comparing “sent” and “tapped” events in the notification history, the researcher can determine the length of time between the notification being displayed and the reaction to it.

Data protection

Experience-sampling studies may contain personal data deserving protection, such as participants’ geolocation data. Because Samply does not collect survey data itself, only providing a venue for notifying participants, it is the researcher’s responsibility to ensure data confidentiality.

Participants must sign up to use the Samply Research mobile app with their email address. In this way, Samply prevents (to a certain degree) multiple participations in a single study on the same device. The email addresses are not shared with researchers and are only used for user authentication. Participants are not required to provide any demographic or sensitive information to use the app.

When a participant allows the app to send them notifications, a unique anonymous link is stored on the Samply server. This link is used to deliver notifications through iOS and Android services, and it contains no personal information about the user. When the participant leaves the study, the link is deleted from the server. Samply users have the option of contacting the researcher to request the deletion of their data (i.e., the log of interactions with notifications).

Technological overview

Samply for researchers is a website at https://samply.uni-konstanz.de/ (or short samply.uni.kn) programmed in Node.js. The server side of the website is an Express.js server connected with a mongoDB database. Notification management is carried out through cron, a time-based job scheduler running on the server.

Samply for participants is a native mobile app written in the React Native JavaScript library and rendered with native code on Android and iOS. The mobile app uses push notification technology to deliver notifications.

Samply: A step-by-step tutorial

Researcher workflow

Samply is a web-based application that can be opened in any web browser (https://samply.uni.kn). In a nutshell, once a study notification plan has been created, a link is sent to the participants or is published online. Participants download the Samply Research mobile app, follow the link on their smartphones, sign up for the study, and start to receive notifications.

The key steps to using the web app as a researcher are:

  1. 1.

    Signing up

  2. 2.

    Creating a new study

  3. 3.

    Setting up notifications

  4. 4.

    Inviting participants

  5. 5.

    Monitoring participation

Signing up as a researcher

Signing up requires the researcher to enter an email address and create a password (Fig. 1). These will be used to log in to the account. If the password is forgotten, it can be reset via a link sent to the researcher by email.

Fig. 1
figure 1

Screenshot of the Samply page showing the sign-up form for researchers

Full size image

Creating a new study

After the researcher signs up, they will be prompted to create a new study. They can write a study description to give their participants more information about what is expected of them in the study. The description will be displayed alongside the study’s name to help potential participants decide whether they wish to participate. A consent form text can also be shown to participants when they join the study. As an option, participants can be asked to enter a participation code or a username at this stage. Should a researcher wish to share access to the study with colleagues, their email addresses can be entered into the designated field (or left blank, as it can be done later). Colleagues should already be registered as researchers in Samply if they wish to have access to a study.

After a new study has been added, a tab for it will be displayed on the web page (Fig. 2). The tab contains the study’s name and status along with two buttons to edit and delete it. The toggle switch at the bottom is grey when it is turned off—meaning that the study is not currently publicly accessible, i.e., not displayed in the list of studies in the mobile application. However, participants can still be invited using a direct link. When the study is ready to be opened to the public, the switch can be toggled to activate it. If a study has invited colleagues or registered participants, this information will also be displayed on the study tab.

Fig. 2
figure 2

Screenshot of the page with the overview of the new study

Full size image

Setting up notifications

Setting up the notifications plan requires defining the notification content (a title, message, and study URL) and deciding on the schedule. It is recommended that one keeps the title and message shorter than 30 characters each (including spaces) to make them compatible with all potential devices and screen sizes. The title can contain the study’s name (e.g., “Life style”) or the name of its current phase (“Life style – Day 1”). The message can include an instruction or what is expected from the participant, e.g., “Please complete a 1-minute survey.” Another strategy is placing a survey question in the notification title or message. For example, Pielot et al. (2017) put “How do you feel?” in the title and “Click to respond” in the message body.

The URL may contain the participant’s ID: the placeholder “%PARTICIPANT_CODE%” inside the web link will be replaced with each user’s Samply ID (e.g., https://examplesurvey.com/?id=%PARTICIPANT_CODE%). This can be used to track survey participation if the URL query parameters are captured and saved in the survey.

If the user interface is followed step-by-step, a few choices have to be made in order to define the notifications schedule.

  1. 1.

    Choose participants

    Notifications can be sent either to all current and future participants or to a particular participant chosen from the menu under “Current participants.”

  2. 2.

    Choose time(s)

    A specific time point or time points can be selected by entering the hour and minute in the input field. Alternatively, a time window can be defined (e.g., from 9 a.m. to 6 p.m.), during which a time point or time points can be randomly drawn. If several participants are selected, notification times will be randomized for each one.

  3. 3.

    Choose date(s)

    A specific date can be selected in the format of day, month, and year. Alternatively, if a notification needs to be repeated, other options can be chosen. Notifications can be sent every day or at a range of intervals (every 2nd, 3rd, 4th, etc. day). Alternatively, the day(s) of the week or day(s) of the month can be selected.

  4. 4.

    Choose month(s)

    Any specific month(s) can be chosen for notifications.

  5. 5.

    Choose when to start

    Notifications can begin to be sent at specific time points defined by the time and date. Alternatively, a time point can be picked relative to the current moment or the moment of participant registration. In the latter case, the starting point will be different for each participant and will be determined by the time when a participant joins the study via the mobile app.

  6. 6.

    Choose when to stop

    The options here are similar to choosing when to start notifications. Different combinations of starting and stopping points can be created; e.g., notifications can start from the moment each participant registers but stop at one specific time point, which will be the same for everybody.

When a notification has been added, it appears in the list of scheduled notifications, which can be controlled and cancelled at any time by clicking either the delete icon in the table or the “Delete all notifications” button.

Inviting participants

The “Invitations” page incorporates two ways of inviting participants to the study: a direct link to the study and using the search function inside the app. The direct link works even if the study is not publicly available in the list of studies. To use it, participants must already have the Samply Research app installed on their smartphone, and they should open the link in it too. When they open the app for the first time, a question about notifications appears automatically on the screen, and they must allow those notifications. Participants will also need to create a new account. Login information is stored securely on the Samply server (which is the University of Konstanz’s server, in Germany) and is only used to authenticate users.

After creating an account, participants are automatically redirected to the page within the app describing the study. When they click on the “Join the study” button, the consent form text appears, and on tapping the “Yes, I agree” button, they join the study and allow notifications from it. The application can then be closed.

Researchers can also add additional variables to the direct link using the query parameters; e.g., to create an individual link for a particular participant, the link can be modified by appending “&code=123” to the end of it. The value of the variable “code” (in this case it is 123) will be recorded and shown in that particular participant’s data on the Samply website.

If, for some reason, participants cannot use the direct link, they can find the study in the list of studies within the app by using the search function. The study must be public, however. After logging in to the app, participants must click on the “Studies” tab and can then use the search field to find the study. The study’s name, author, and creation date are displayed in the list of studies. Tapping on the study transfers the participant to the study page.

Monitoring participation

There are two ways of monitoring participation: via the list of participants (“Participants” menu) and the history of notifications sent (“History” menu). The list of participants displays them in order of registration to the study (Fig. 3). Together with their unique ID, the time at which they joined the study, and their username, the table provides information from the direct link query (e.g., code). The “Log” column displays the links to the logs of the notifications sent to a particular participant. Next to this, the “New notification” column has links to the Notifications menu where the participant’s notifications can be scheduled. A participant can also be deleted by clicking the basket icon on the right-hand side of the table.

Fig. 3
figure 3

Screenshot of a page with an overview of a participant’s notifications

Full size image

The “History” menu displays the time points for the following events of each notification:

  1. 1.

    When the notification was sent from the server.

  2. 2.

    When the user opened the online study by tapping on the notification.

  3. 3.

    When the user opened the notification within the mobile application.

  4. 4.

    When the user archived the notification message within the mobile application.

The title, message, and study URL for each notification are shown. If the user dismisses a notification, no event is triggered or recorded. Clicking on the user ID reveals the notification events filtered for that user. To further inspect and analyze their interactions with notifications, the history of all the participants’ events can be downloaded in the CSV format.

Participant workflow

Once a participant has downloaded and installed the app on their smartphone, the study can be opened with the direct link shared by a researcher or found through the mobile app interface in the “Studies” tab. After joining the study, the participant can view it on their personal dashboard. The participant can later unsubscribe from the study by clicking the “Leave the study” button on the study’s page. All past notifications can be found in the “History” tab. The table displays the title, message, and study URL together with the time the notification was sent. A study link can be opened from the “History” tab by tapping on the notification. Participants can also customize the app by setting the times at which it is allowed to send them notifications. This information is displayed to the researcher on the page listing participants.

Next, we describe two validation studies. Study 1 aimed at improving the web app’s design and usability for researchers. Study 2 validated the data collected through the Samply Research mobile app by using an external survey’s results as a criterion.

Empirical study 1

Method

This first study’s goal was to test and improve the Samply website’s usability for researchers, and it was conducted from April to September 2020 with experience sampling method and usability experts. The experts evaluated the website’s functionality for researchers, particularly signing up, creating a study, and the notification scheduling interface.

Participants

We involved five usability experts and five researchers with expertise in experience-sampling methodologies, and we are grateful for their opinions on the website and app (see Acknowledgments).

Procedure

The usability study was conducted in two waves. Wave one (3 experts) took place from April to June, and wave two (7 experts) from June to September 2020. Experts were asked to contribute to the development of a free tool that would benefit the larger research community; they received no financial incentive to participate. Those who agreed received an instruction file with tasks to complete (see Appendix A) and installed the screencast software. The tasks were to sign up for a researcher account, create a study, schedule five different types of notifications, and edit the profile information. Participants could complete their tasks at their own pace, and they recorded screencasts as they completed them. Afterwards, they uploaded their screencast to the Dropbox link provided and completed the online usability survey.

The usability study was conducted in two waves to improve the website’s design and functionality based on the feedback received. Wave one participants were asked to participate in the usability study, and each one’s screen recordings and survey responses were analyzed. Based on wave one’s results, the website’s design and the user interface were substantially revised (see Table 4 for the revisions). Wave two participants were provided with the same new version of the website.

Materials

Instructions

The materials given to participants included explanations of the experience-sampling method and the Samply app, and step-by-step instructions on how to perform the usability test (see Appendix A). The instructions also contained a description of the screen recording software and the link to the online usability survey.

Usability survey

The usability survey was based on the summative evaluation version of the IsoMetrics usability inventory (Gediga, Hamborg, & Düntsch, 1999). The IsoMetrics statements are usually grouped into seven scales that were derived using the principles of the ISO 9241-10 standard. We used 69 statements after removing items that were not relevant to the Samply website (e.g., “The software lets me change the names of commands, objects, and actions to suit my personal vocabulary.”) (see Table 5 for the statements). Participants were asked to rate each statement on a separate screen using a visual analogue scale from 0 (“Predominantly disagree”) to 100 (“Predominantly agree”) (see Fig. 4).

Fig. 4
figure 4

Screenshot of a page from the usability survey

Full size image

Participants were also asked whether they had encountered any errors or problems and how the app could be improved. The online survey was created in the lab.js experiment builder (https://lab.js.org, Henninger, Shevchenko, Mertens, Kieslich, & Hilbig, 2020) and hosted on the Open Lab platformFootnote 3 (https://open-lab.online).

Screen recording

Instructions on how to use QuickTime Player for Mac and Captura Portable for Windows were provided in the task description for participants (see Appendix A). The recorded videos were uploaded to the first author’s Dropbox account using a public “upload file” link.

Analysis

The evaluation’s goal was to detect any usability problems within the app and improve its interface and design. The analysis of the screen recordings and the comments in the usability survey focused on three questions: (1) Were there any critical problems that interrupted the workflow? (2) Were there any usability problems that delayed the workflow? (3) What possible improvements could enhance the user experience?

The usability survey’s results were aggregated for each of the seven scales and are presented separately for wave one and two, as substantial changes were introduced between the two waves. Survey scales and items with a mean below 61 were considered to have a potential usability problem. This cutoff value was chosen in accordance with the four-point cutoff in the original IsoMetrics usability inventory. The analysis and visualization were done with R software, version 4.0.2 (R Core Team, 2020) and the ggplot2 package (Wickham, 2016). Data and materials are available at OSF (https://osf.io/u57aw/).

Results

Video analysis and revisions to the web application

Video analysis of wave one identified 11 critical problems and seven usability problems (see Table 2 for a summary and Table 4 for full information on the problems and app revisions). Eight of the critical problems were related to the use of the cron format in scheduling notifications, and the other three involved the display of scheduled notifications. Usability problems were due to the lack of an option to enter values using a keyboard, to missing or incorrect display of information, and to confusing error messages.

Table 2 Critical problems, usability problems, and potential improvements identified in waves one and two
Full size table

During wave one, we revised the web application after both the first and second participant, fixing critical problems that had previously obstructed the user flow. Major changes were made after the third participant. First, cron-formatted strings were removed from the user interface so that researchers would not get confused by this new terminology. Instead, researchers were presented with step-by-step questions and choices between different scheduling options. For example, when the researcher chose the time of notification in the new interface, they had to select either a specific time point or a time window. Thus, the majority of critical problems, which might have prevented scheduling notifications, were eliminated. Second, we integrated critical information into the display of scheduled notifications. Each scheduled notification showed the settings chosen for each question asked during the planning stage. In this way, researchers could now spot mistakes (e.g., wrong choice of dates). Third, each usability problem was addressed by corresponding actions (see Table 4).

No critical problems were found in wave two, and the usability problems and potential improvements were addressed after the last participant.

Survey

On average, experts chose the “No opinion” option for 15 (SD = 11) out of 69 items. The statement with the highest non-response rate was “I can adjust the presentation of results (on the screen, to printer, plotter, etc.) to my various work requirements,” which received a “No opinion” answer five times. Five statements (e.g., “I can call up specific explanations for the use of the system if necessary”) remained unanswered four times each.

The mean scores for the seven IsoMetrics scales in waves one and two were all above the cutoff value of 61, except for the Error tolerance scale in wave one (M = 41.6, SD = 31.5). The means by scale and wave are displayed in Fig. 5. In wave one, the average rating across all scales was 71.9 (Med = 74.7, SD = 14.3), ranging from 34.3 to 89.3. In wave two, the average score was 72.6 (Med = 71.4, SD = 14.8), ranging from 33 to 100.

Fig. 5
figure 5

Average values of the usability study’s scales. Note. This figure shows the average values of the seven IsoMetrics scales for the groups of experts in waves one (n = 3) and two (n = 7). Error bars display one standard deviation above and below the mean. Small circles represent experts’ scores for each scale.

Full size image

Some of the statements scoring below the cutoff value of 61, such as statement 35 (“When selecting menu items, I can speed things up by directly entering a letter or a command code”), could not have their underlying problem improved due to the website’s format. Other problems, however, such as statement 48 (“When I attempt to perform a destructive operation (e.g., data deletion, etc.), I am always asked to confirm the action.”), were addressed in the final revision of the program (see Table 4).

Discussion

All the participants in this usability study were able to create an experience-sampling research study and use the scheduling tool for notifications. Wave one revealed critical errors that prevented some notifications from being saved correctly, but these were corrected in the subsequent app revision. The main improvement was changing the format from a cron-based string to a number of simple selection steps that created a schedule. We made the scheduler more flexible and improved error feedback, user control, and visibility of the system’s current state. All the participants in wave two were able to build the notification schedules specified in their instructions. There were no critical problems hindering system functionality. After wave two, usability problems were addressed, and potential improvements were implemented.

Data from surveying our expert participants confirmed the improved user experience between waves one and two, particularly in the application’s error tolerance. There remained some variability in the experts’ ratings, possibly due to different personal experiences with other online tools. Therefore, to improve the website’s general usability, we also added the documentation explaining each step of the researcher workflow. The notification planning page was enhanced with instructions and example schedules. In this way, the application now meets the needs of researchers with different experiences: it supports experienced users with error feedback and instructs newcomers with hints and clues. Finally, the experts’ comments in the usability survey were a source of new ideas and app enhancements.

Samply comprises two interlinked applications: the website for researchers and the mobile application for participants. The first study focused on improving the website’s usability for scheduling and planning notifications. In the second study, we evaluated the mobile application’s functionality for delivering notifications.

Empirical study 2

Method

The second study’s goal was to validate the Samply Research mobile application using the participation data from an external survey. The study was conducted in June and July 2020 in two student projects carried out during the experimental practicum at the University of Konstanz, Germany.

Because the study’s main goal was validation of the mobile app, we will omit a description of the underlying results obtained in the two student projects and instead focus on their participants’ response and dropout rates. The first project studied the effects of gratitude on well-being, whereas the second project examined the role of time management techniques in increasing the efficiency of studying. Participants were recruited via the University of Konstanz’s participant management system, SonaSystems, and through personal contacts with students.

Materials

The Samply Research mobile application

The Samply Research mobile app was available for free download from Google Play or the App Store. Participants in both projects completed the first browser-based survey online, receiving instructions on how to download and use the app. In order to match participants’ IDs in the Gratitude project, a random four-digit participant number was generated and displayed in the first survey together with the instructions for entering it in the mobile app. In the Time Management project, participants were asked to create and enter their own personal identification number (e.g., by using the initials) in both the online survey and the mobile app.

After the participants had installed the application and joined the projects, the scheduled notifications began. When participants tapped a notification in their smartphones, they were redirected to the first page of the mobile survey. The web link contained the participants’ Samply ID, so that each ID was recorded inside the survey and could later be matched with Samply’s data.

Web surveys and notification schedules

The projects’ web surveys were constructed in the lab.js experiment builder (https://lab.js.org, Henninger et al., 2020) and hosted on the Open Lab platform (https://open-lab.online).

The Gratitude project’s schedule sent notifications once a day for 14 consecutive days at a random time between 6 p.m. and 8 p.m. The notification contained a link to the project’s web survey that asked participants to list things they felt grateful for that day in one condition and things that had happened to them during the day in the control condition. Each daily survey took several minutes to complete.

The Time Management project’s schedule sent notifications once a day for 13 consecutive days at a random time between 6 p.m. and 8 p.m. The daily survey asked two questions, one on well-being and one rating personal studying progress.

The data and materials are available at OSF (https://osf.io/u57aw/), and both student projects were preregistered.

Results

The Samply Research mobile app was installed by 76 participants, 44 of whom participated in the Gratitude project, 26 of whom joined the Time Management project, and 6 of whom participated in both. Thus, the Gratitude project collected data from 50 participants, and the Time Management project gathered data from 32. Eight participants failed to complete their daily surveys (NGr = 3 and NTM = 5), so the final samples for the analysis consisted of 74 participants (NGr = 47 and NTM = 27) (see Table 3 for the participation rate).

Table 3 Student project’s participation rates
Full size table

Compliance rate

The overall compliance rate, defined as the percentage of notifications that were followed by participation in a survey, was 65%, i.e., 657 out of 1011 notifications were followed (Med = 76.9, SD = 28.4).

To analyze the effects of the project and the participation day on the compliance rate, we applied a multiple linear regression model, using the participation rate (0–100%) as a dependent variable and project, participation day, and their interaction as predictors (see Fig. 6 for the compliance rate). The model explained a significant proportion of variance in the compliance rate, R2 = .58, R2adjusted = .52, F(3, 23) = 10.37, p < .001. The main effect—which project was being participated in—was significant, indicating that the compliance rate in the Gratitude project, M = 69.8, Med = 71.3, SD = 10.7, was significantly higher than in the Time Management project, M = 56.1, Med = 57.1, SD = 9.4; b = −0.22, t(23) = −3.71, p = .001; unstandardized coefficients are reported. The main effect of participation day was also significant, showing that the compliance rate decreased by about 2% a day, b = −0.02, t(23) = −3.70, p = .01. There was no significant interaction between the project and the day, indicating that the decrease in participation over time was not significantly different between the two projects, b = 0.01, t(23) = 1.71, p = .10.

Fig. 6
figure 6

The compliance rates over the course of the two student projects. Note. The percentage of notifications (0–100%) that were followed by survey participation each day. The Gratitude project lasted 14 days and the Time Management project lasted 13 days.

Full size image

Response time

Study 2’s aim was to evaluate the Samply Research mobile app using participation data from student project surveys. Besides the compliance rate, another informative measure of an app’s usage is response time, which is defined as the time between when the notification is sent and when the survey web page is opened. Overall, the median response time across both projects was 52 minutes 34 seconds (in hours, Med = 0.9, M = 2.9, SD = 5.0). The response time was not significantly different between the Gratitude project, Med = 0.8, M = 2.7, SD = 4.6, and the Time Management project, Med = 1.1, M = 3.4, SD = 5.4, as confirmed by a two-sided Wilcoxon rank-sum test, W = 45346, p = .38. Response times did not change significantly over the course of the projects, as indicated by the day’s lack of any significant effect on the response time in a linear regression model, b = −0.03, t(660) = −0.62, p = .54, model fit: R2 < .001, R2adjusted < .001, F(1, 660) = 0.38, p = .54 (see Fig. 7 for response times).

Fig. 7
figure 7

Response times across different participation days and projects. Note. Box plots display the response time medians (in hours) with the interquartile range (IQR) of the middle 50% of values. Whisker lengths are 1.5 × IQR. Outliers are represented by circles.

Full size image

Interaction with notifications

The mobile app offered two ways to interact with notifications: tapping the notification in the smartphone’s notification bar or inside the mobile application. We analyzed the proportions and response times of interaction events to understand differences in the use of these two options. Of the 657 notifications that resulted in a survey response, 33% (n = 219) were opened by tapping the notification and 62% (n = 406) were opened from the app interface. Interestingly, 2.5% (n = 16) of notifications were both tapped and opened in the app interface, and 2.5% (n = 16) of the notifications had missing information about the user interactionFootnote 4. Analyzing the same data by participants, we found that 39 participants (53%) both tapped and used the app to open the survey link, 11 (15%) only tapped, and 24 (32%) only used the app interface to open notifications.

The median time between when a notification was sent and when a participant tapped on it in the notification bar was 21 minutes 24 seconds (in hours, Med = 0.4, M = 1.9, SD = 4.1). The median time between when a notification was sent and when the survey link was opened in the mobile app was 1 hour 24 minutes 28 seconds (Med = 1.4, M = 6.8, SD = 25.6). The higher standard deviation in the latter case indicated the presence of outliers (in this case, participants who responded to the notification the next day).

Device effects

Based on the information recorded about their device when participants took part in the survey, 55% (n = 41) used Android smartphones and 45% (n = 34) used iOS. Comparing these participants’ percentage of notifications that were followed by participation in a survey (from 0 to 100%) using an independent two-sided t test, we found no significant differences between Android users (M = 66.3, SD = 29.9) and iOS users (M = 62.6, SD = 26.8), t(72.55) = 0.57, p = .57, d = −0.13, 95% CI [−0.59, 0.33]. There were also no significant differences in the individual median response times between groups of participants with Android, in hours Med = 0.9, M = 2.1, SD = 3.4, and iOS, Med = 1.8, M = 3.5, SD = 4.9, t(56.84) = −1.46, p = .15, d = 0.35, 95% CI [−0.11, 0.81].

Discussion

Compliance rate

The second study’s goal was to validate the Samply Research mobile app’s feasibility with the participants of two student experience-sampling projects. We also used data from these external surveys to measure response times and compliance rates. The average compliance rate in our validation study (65%) was lower than the 75% rate reported in a meta-analysis of studies with more than six daily notifications (Wen, Schneider, Stone, & Spruijt-Metz, 2017) and was closer to the average compliance rate of 70% (SD = 23%) calculated in the meta-analysis by Van Berkel et al. (2017).

Interestingly, compliance rates differed between the two student project studies, which might have been due to their subject matter (one related to time management), the interactions between researchers and participants, or the sample characteristics (see meta-analysis in Vachon, Viechtbauer, Rintala, & Myin-Germeys, 2019). The Gratitude project, with its higher compliance rate, may have established a stronger connection between participants’ efforts and the intrinsic reward of self-reporting, which is a motivational factor acknowledged in previous research (Van Berkel et al., 2017). We also observed that compliance decreased with a sharp initial drop and then during the study at a rate of 2% a day, which was in line with observations of comparatively large dropout rates in online studies (Frick, Bächtiger, & Reips, 2001; Galesic, 2006; Reips, 2002, 2007).

Response time

The distribution of response times was highly skewed, with a median of around 52 minutes and outliers who only responded to notifications the next day. This could be considered an exceptionally long time interval but can be explained in light of the student project framework. Both projects sent one notification per day, with instructions stating that participants should provide their evaluation of that day (therefore notifications were sent in the evening). The instructions did not emphasize that participants should respond as fast as possible, although this type of instruction might be important for studies using a different schedule (e.g., several times a day). Furthermore, we did not let notifications expire, so the surveys could be opened at any time after receiving the notification. We used the same web link for these recurring surveys, but other studies could use different links to better monitor the survey’s progress.

Response times did not differ significantly between the projects and did not change throughout their durations, suggesting that participants who stayed with the projects retained the same pattern of application usage.

Participants in both projects made use of the possibility to open notifications via the app interface. Displaying notifications in the app was the backup plan to ensure that participants who had accidentally deleted a notification (e.g., by swiping it to the left instead of tapping it) could still find the survey link in the app. We did not explicitly instruct participants to use this feature, but more users did so (62%) than by tapping a notification (33%). Response times were significantly different, with notifications being tapped on after an average of 21 minutes or being opened via the application after an average of 1 hour 24 minutes. This may indicate that in cases where participants delayed their response, they preferred to use the app to get the link to the survey later. However, it could also be that they missed a notification and found it later in the app. The fact that most participants (67%) tapped a notification at least once suggests that this notification method did work for them technically. However, because participants were using their own devices, we could not prevent them from turning off notifications or switching to a battery-saving mode. Such concerns are generic for all mobile apps and should be discussed between researchers and participants at the beginning of a study.

Device effects

The participants in both student projects used their own Android or iOS smartphone systems. The absence of any significant differences in compliance rates and response times between the operating systems indicated that the Samply Research mobile app operates similarly on either device.

General discussion

Limitations

Researchers should be aware of the Samply Research mobile app’s two main limitations, but they are common to any mobile app that sends push notifications. First, participants need an Internet connection to receive notifications and open web links in the mobile browser. Secondly, participants must allow notifications in their phone settings. Depending on the model, additional steps may be required to enable notifications, and settings may be general or application-specific. For example, using the battery-saving mode may prevent the reception of notifications in Huawei phones. Moreover, if a participant uses task killer, battery-saver, performance-booster, task-cleaner, or security-type apps, these can also temporarily block notifications. If researchers have the opportunity to meet with participants at the beginning of a study, we recommend checking these settings with the participant so as to decrease the risk of undelivered notifications. In cases involving an Internet-based study, the Samply website has trouble-shooting instructions, which researchers can send to participants when inviting them to participate in a new study.

Conclusion

Samply offers a number of advantages, including all the benefits of Internet services and open-source, web-based software. For researchers, the Samply platform makes organizing and conducting experience-sampling studies easier, without the need to program separate native mobile apps. Potential areas of use are Internet-based studies in which participants must respond more than once (e.g., diary studies, ecological momentary assessment). For participants, Samply is a convenient way to take part in research studies thanks to the convenience of using one’s own smartphone, e.g., as part of citizen science projects. The present paper provided a step-by-step tutorial on how to conduct a study using Samply, and it demonstrated the app’s functionality and usability via two empirical study projects organized by students. Our earlier usability study helped to significantly improve the Samply website’s ease of use for researchers, and our subsequent validation study confirmed the mobile app’s main functionalities. Further research should investigate the factors that influence participants’ responses to notifications, thus helping to develop more efficient strategies for interacting with them.