Keywords

1 Introduction

Using questionnaires for eliciting subjects’ opinions has a long term history. Generally, there are two main approaches for conducting such surveys: personal and self-administered. In the first case, one performs, e.g., a face-to-face structured interview or a telephone survey. In the latter option, data are gathered, e.g., by paper-and-pencil, online, or mail questionnaires. Various examples, detailed description along with advantages and disadvantages of specific modes of conducting surveys can be found, for instance, in the works of Yu and Cooper (1983), Tourangeau et al. (2000), Aldridge and Levine (2001), Bowling (2005), Burns et al. (2008), or Fink (2015). The present paper focuses on self-administered questionnaires that are very popular mostly due to low costs of applying them in comparison with interviews or telephone calls.

There has been an abundance of research on almost every aspect of conducting such studies including technical, physical, and psychological aspects. Many investigations showed differences between various types of self-administered questionnaires in their effectiveness and efficiency (Dillman 1999; Wright and Ogbuehi 2014). However, much less is known about subjects’ preferences towards specific survey designs and circumstances in which these experiments are carried out. Since subjective opinions of respondents may influence the questionnaire results (e.g., Wright and Ogbuehi 2014; Voutilainen 2016), thus, extending the knowledge in this respect is desired both from the methodological as well as practical point of view.

1.1 Assistant Presence

According to Bourque and Fielder (2003, p. 3) the self-administered questionnaires may be characterized on a continuum with one end including cases when the assistant, surveyor, or other supervisory person is present, and the opposite end where no such a supervision exists. There are, naturally, advantages and disadvantages of both extreme approaches. For example, the surveyor may increase the quality of the gathered data by answering participants’ questions or explain doubts (Bowling 2005).

Moreover, some general studies show that people prefer direct, face-to-face rather than other ways of communication (e.g., D’Ambra et al. 1998). On the other hand, costs related with self-administered questionnaires filled in without the supervision can be significantly smaller than in the former case (Dillman 1999). Though there have been some studies regarding the influence of the surveyor presence on the subjects’ responses, not much is known about participants’ preferences in this regard.

1.2 Survey Form

Recently, computer software supports more and more commonly the data collection process. Thus, specific aspects of conducting computer assisted surveys also draw researchers’ attention (e.g., Andreadis 2015; Toepoel and Ludtig 2015). The researchers analyzed advantages and drawbacks of traditional and electronic forms of questionnaires. They indicate, among other things, that electronic versions allow for better completeness of the gathered data as compared with the paper based form, however require at least basic computer literacy and access to some electronic devices.

Most of the research dealing with the questionnaire form focuses on their efficiency and effectiveness, still, it is also interesting what the subjects’ attitudes towards these two basic forms of questionnaires are. Recently, Wright and Ogbuehi (2014) in their paper about how data collection methodology influences response quality examined, inter alia, respondents’ fatigue, boredom, and satisfaction. Though they found no differences between paper and pencil and electronic questionnaires, their experimental design was not focused specifically on the questionnaire form and included many other factors that could have had an impact on the results.

1.3 Scale Type

While designing questionnaires, researchers have to decide what type of scales to use. There are number of possibilities, but among the most common there are Likert (LS) and the visual analogue (VA) scales. The Likert approach allows for selecting one of the specific, commonly odd number of response options, located between two end points, called usually “strongly agree” and “strongly disagree”.

The visual analogue scale, in turn, is a line ended with two points between which subjects specify their level of agreement to a statement by placing a mark. There are numerous studies examining various features of these scales (e.g., Lewis and Erdinç 2017), but again little is known about how respondents perceive them. Among few of them there are the works of Laerhoven et al. (2004), or lately, Voutilainen et al. (2016).

1.4 Research Objectives

The current research examines subjects’ preferences towards factors differentiating the conditions of conducting self-administered questionnaires. Particularly, three factors are investigated, that is, the assistant presence (present, absent), survey form (paper or electronic), and scale type (visual analogue or Likert scale). As far as we are aware the influence of these effects on subjects’ preferences have not been studied simultaneously.

2 Method

A pairwise comparison technique was employed to obtain the participants’ subjective opinions. Such an approach allows for acquiring better accuracy than, for example, direct ranking (Koczkodaj 1998). Calculations of the relative preferences were performed according to the Analytic Hierarchy Process (AHP) methodology (Saaty 1977; 1980; Barzilai 1997).

The use of conjoint methodology (Luce and Tukey 1964; Krantz and Tversky 1971; Green and Srinivasan, 1978; 1990; Green et al. 2001) allowed for assessing partial utilities of the examined factor levels and relative importances for the effects. The full factorial analysis of variance apart from verifying the statistical significance of the investigated effects provided, additionally, information on possible interactions between them. Such a multimethod approach gives a fuller picture of subjects’ preference structures (Grobelny and Michalski 2011).

2.1 Independent and Dependent Variables

The investigated variants were graphically presented as pictograms along with captions informing both about the factor and its level. The applied images with original Polish captions are demonstrated in Table 1.

Table 1. Experimental factors and their levels.
Full size table

2.2 Experimental Design and Procedure

A full factorial design involving the abovementioned effects resulted in eight experimental variants. Participants were asked which survey version they preferred the most. Each subject performed 28 pairwise comparisons of these eight conditions.

The Analytic Hierarchy Process methodology proposed by Saaty (1977; 1980) was used to calculate relative weights for every participant. Overall, the whole procedure took about five minutes, on average.

2.3 Apparatus

A custom made software (see Fig. 1) was applied to present stimuli, gather, store, and process the data. Left/right stimuli locations along with the order of comparisons were randomized by the software. Subjects expressed their opinions by selecting a radio button situated on this side where the preferred survey option appeared. Participants were given the opportunity to freely move forward and backward between comparisons and change their responses as needed. All the results were exported to the IBM SPSS Statistics 24 statistical software to perform a conjoint analysis.

Fig. 1.
figure 1

A graphic software interface for a sample pairwise comparison.

Full size image

2.4 Participants

A total of 61 subjects expressed their opinions towards examined variants. Most of them (70%) were at the age range of 20–35 years old, and the rest was older. There were 29 females (48%) and 32 males (52%).

3 Results

3.1 Ahp

Consistency Ratios.

All of the consistency ratios (CR) were lower than .1 - the threshold recommended by Saaty (1977; 1980). The lowest value equaled .00476 while the highest - .0281. Therefore, results from all subjects were taken into account in further analyses. The mean value of CR amounted to .0133 (median: .0129) with the standard deviation of .00544. One way analysis of variance was used to check whether the CRs depended on the gender. The results revealed no significant influence of the gender on the inconsistency level (F(1, 59) = 1.8, p = .18).

Basic Statistics.

The preferences obtained in a form of relative weights demonstrated that the highest score was attributed to the electronic version of the survey with the Likert scale, conducted without the presence of a survey assistant. The least preferable option was the paper based questionnaire including visual analogue scale, filled in while the survey assistant was present. The full hierarchy along with the computed weights and basic descriptive statistics are put together in Table 2 whereas the graphic illustration of the results are given in Fig. 2.

Table 2. Basic descriptive statistics.
Full size table
Fig. 2.
figure 2

Preference weights for all experimental conditions. Whiskers denote mean standard errors. (F(7, 480) = 8.9, p < .0001)

Full size image

Analysis of Variance.

First, a two way analysis of variance was used to check if the examined options influence mean preference weights and whether or not the gathered data depended on the subject gender. The results presented in Table 3 show significant impact of the examined variants irrespective of the participant gender.

Table 3. Two-way Anova results for Experimental conditions and Gender factors.
Full size table

Next, to get further insight into obtained preferences, a standard, three way analysis of variance was applied to verify if differences between factor means were statistically significant. The obtained results provided evidence for meaningful influence of all examined factors on mean preference weights. Interestingly, the interaction between Assistant presence and Survey form factors also occurred to be statistically significant. All other interactions were irrelevant. The detailed ANOVA results are given in Table 4.

Table 4. Three-way Anova results for Assistant presence, Survey form, and Scale type factors.
Full size table

The mean preference weights for all statistically significant effects are graphically illustrated in Figs. 3, 4, 5 and 6.

Fig. 3.
figure 3

The influence of Assistant presence on mean preference weights. F(1, 480) = 32, p < .0001, η2 = .062.

Full size image
Fig. 4.
figure 4

The influence of Survey form on mean preference weights. F(1, 480) = 6.8, p = .0096, η2 = . 014

Full size image
Fig. 5.
figure 5

The influence of Scale type on mean preference weights. F(1, 480) = 16, p < .0001, η2 = .033.

Full size image
Fig. 6.
figure 6

The influence of Survey form × Assistant presence interaction on mean preference weights. F(1, 480) = 7.1, p = .0080, η2 = .014.

Full size image

3.2 Conjoint

The conjoint analysis, performed in IBM SPSS Statistics 24, provided information on partial utilities of the examined factor levels. They are demonstrated in Table 5. Respondents assigned the higher utility to the option without an assistant than when the assistant is present while filling in the questionnaire. Subjects rated the electronic questionnaire better than its classic paper version counterpart, and preferred the Likert over the VA scale type.

Table 5. Conjoint results – partial utilities.
Full size table

Overall importances of the investigated effects amounted to 40.3% for the Assistant presence, the Survey form obtained 35.8% whereas the least significant was the Scale type – 23.9%. The R Pearson correlation between the observed and predicted preferences amounted to .94 while tau Kendall’s correlation equaled .786. Both parameters were statistically significant at α = .005.

4 Discussion and Conclusion

The presented study examined subjects’ preferences towards various conditions concerned with filling in questionnaires. The results, generally, proved that all three investigated factors, that is, Assistant presence, Survey form, and Scale type, significantly differentiated the respondents’ subjective ratings. Overall, the analyses revealed that study participants preferred answering survey questions alone than in the presence of an assistant. They liked more the Likert than the visual analogue type of scale. The electronic version of a questionnaire was better perceived than the paper form, however as the significant interaction shows, this is the case only when the assistant is absent. In the condition with present assistant, there is no significant difference in mean preference weights for electronic and paper options.

The conjoint analysis provided evidence that the Assistant presence is the most important factor shaping respondents preferences in this study. The second examined factor, Survey form, was less important by only 4.5 percentage points. The Scale type occurred to be considerably the least significant than the other two effects. The difference in percentages amounted to as much as 16.4% in comparison with Assistant presence, and 11.9% as compared with Survey form. These outcomes are consistent with magnitudes of partial utilities and only partially correspond with effect sizes from ANOVA where Scale type had higher eta-square than Survey form. Mean weights obtained by AHP suggest that the most preferred variant is the one without the participation of an assistant, prepared in an electronic form, with Likert scales.

The presence of assistant may be helpful in explaining the purpose of the questionnaire and answering respondents’ questions. Some previous studies (e.g., Yu and Cooper 1983) indicated that subjects prefer direct human interaction to other ways of providing information about the questionnaire. The current research, however, does not support it. The explanation of this finding may be related to the specific sample which consisted of mostly young people who probably completed questionnaires in an electronic form without any supervision much often than paper and pencil versions administered by assistants. Another possible reason for lower acceptance of the assistant presence while completing questionnaires is the reluctance of providing sensitive data in such circumstances. Researchers should be aware that perceived greater anonymity while assistants are not present may sometimes prompt subjects to provide fake data.

Studies of Yu and Cooper (1983) and Hox and Leeuw (1994) indicate that there are no differences in paper and electronic forms of questionnaires with regard to the response rate. Our findings may suggest something else. Since preferences in this regard significantly differ, they may influence the response rate as well. The reason may, again, be attributed to the young sample in the present research that have generally bigger experience with electronic devices and computer software than the participants taking part in previous studies. Current experiment respondents are also more accustomed to electronic questionnaires. Furthermore, electronic versions of questionnaires, especially those available on the Internet, may be attractive for respondents because they can freely choose the moment when to provide answers and how much time they can spend to complete them.

From the researcher’s perspective it is easier to process data as they are already collected in a digital form and ensure consistency and correctness of the information received (e.g., by validating data during the input of responses). The investigators have also the possibility of embedding audio, video files or some interactive objects into their questionnaires. On the other hand, there is a higher risk of obtaining a bigger number of incomplete data since such questionnaires can be easily interrupted. Moreover, one should not forget that some respondents may still have problems with accessing computers, the Internet or exhibit insufficient computer literacy necessary to complete electronic forms.

The Scale type factor influenced the subjects’ preferences the least significantly from among investigated factors. They rated the Likert better than the VA scale probably because the Likert scale restricts the number of possible response options to several items while VA allows for selecting any point on a line which may be perceived as a more demanding task. Such an explanation is consistent with the results presented by van Laerhoven et al. (2004) who examined scale type preferences in children’s questionnaires. They found that children preferred the Likert over the numeric and simple VA scales and find it easiest to complete while exhibiting similar reliability.

Another possible explanation may result from the fact that Likert type of scale is much more prevalent both in professional scientific questionnaires and in many amateur surveys that often appear on various websites.

As in any scientific research also here there are some limitations in interpreting the findings. First, numerous aspects of conducting self-administered questionnaires were not included in the present research. Usually, the number of factors and their levels needs to be limited to perform the experiment in reasonable time using available resources. In this investigation, the experimental design was additionally restricted due to the employment of the pairwise comparisons based methodology. The number of comparisons in AHP increases, approximately, with the square of the number of options examined.

Thus, a number of other studies on respondents’ preferences should involve other possible factors and their interactions. Some other follow-up experiments may be carried out to investigate in a greater detail subjects’ preferences towards different modes of questionnaire administration and combine them with completing real questionnaires. A qualitative study on the reasons for reluctance of the assistance presence while completing questionnaires would be very interesting. This effect should also be compared with preferences in different age groups. One should be cautious in generalizing the presented outcomes as the sample size was moderate and included mostly young, white people from a geographically restricted area and uniform social background.

Despite these limitations, the present, preliminary study provides some more insight into how people subjectively perceive various aspects of conducting self-administered questionnaires. Choosing appropriate conditions in this regard can possibly increase the response rate, quality of gathered data, and may have an influence on the overall peoples’ attitude towards participation in surveys.