1 Introduction

During times of crisis, such as pandemics, wars, and earthquakes, existing patterns of lower response may become more visible. In this regard, a variety of efforts to convert refusals are laudable attempts (Singer 2002; Brick and Tourangeau 2017; Smith et al. 2019), particularly when reasons for refusals are considered (e.g., privacy concerns and wasting of time). Theoretically, Groves et al. (2004) explained the factors that affect survey participation, including the use of incentives, pre-notification letters, and mode switching. However, accepting survey requests may not guarantee reliable and valid survey estimates as long as the respondents are not motivated to respond. In this regard, Haan and Ongena (2023) underlined that sensitivity and difficulty of questions may be a problem among survey participants. Similarly, the answers given by reluctant respondents may suffer from measurement and nonresponse errors, even though they are persuaded to participate in surveys.

Although interviewer effects in surveys are well-established (West and Blom 2017), little is known about the respondent side of the data collection. Interviewers, who have principal roles in surveys (e.g., contacting sample units, asking questions, and recording answers), are also responsible for maintaining the motivation of respondents during the interviews (Schaeffer et al. 2010; West and Blom 2017). Researchers have focused on the relationship between motivation and the resulting quality of estimates in recent years (Groves et al. 2004; Blom and Korbmacher 2013; Schaeffer et al. 2010), although a few studies touched on the determinants of respondent motivation in earlier times (Dijkstra 1987).

It is clear that individuals who are more motivated than others would contribute to getting high-quality survey data, as the previous works suggested. For instance, Knippenberg and Daamen (1996) found that motivated respondents provide more complete and accurate answers for public opinion surveys. Masuda et al. (2017) showed that respondents tend to choose the middle response option when they are not so willing to read items properly. Tourangeau and Yan (2007) discussed the close relationship between the accuracy of responses to sensitive questions and respondent motivation. In light of these works, the willingness to participate in surveys, a complete understanding of questions, and a lower burden for respondents are essential when collecting high-quality survey data. Moreover, such factors appear to have a relationship with the desired level of respondent motivation as well as the resulting quality of the survey data.

Respondent motivation has been associated with various factors such as interviewing techniques, the sensitivity level of questions, feelings towards survey completion, and attitudes towards social surveys (Dijkstra 1987; Tourangeau et al. 1997; Tourangeau and Yan 2007; Rogelberg et al. 2006). Rogelberg et al. (2006) also underlined several factors behind respondent attitudes toward surveys including item-nonresponse, the utility of instructions, and willingness to respond. Briefly, the operationalization of respondent motivation ranges from willingness to survey participation to involvement in the issue (Knippenberg and Daamen 1996; Wenemark et al. 2010). This study mainly utilizes information about the willingness and ability of respondents when formulating respondent motivation.

The main goal of the study is to investigate the potential impact of respondent motivation on the item-nonresponse for a set of questions designed with the split-ballot technique. Accordingly, this study also seeks to investigate the size and direction of the relationship between respondent motivation and the level of item missing. The main motivation for choosing split-ballot questions in such an explanatory process is to reduce item-nonresponse for split-ballot-designed data through respondent motivation. The split-ballot approach is commonly used in the ESS questionnaire when asking questions that can be affected by sex of the respondents (e.g., ideal ages for women/men, attitudes toward social norms among women/men). Thus, keeping respondent motivation at a high level may be much more important for split-ballot-designed questions relative to other questions. Furthermore, it is known that split-ballot-designed questions are prone to item-missing data (Axenfeld et al. 2022). Overall, the main reason for choosing split-ballot designed questions is to reduce item-missing data coming from these questions through respondent motivation.

The data comes from the 9th round of the ESS, which is a cross-national survey that collects rich information about social attitudes, beliefs, and behavior patterns across European nations. The selected questions within this work include hypothetical ages for particular situations as well as attitudes toward social norms. Such questions appear to be more susceptible to the respondent effect relative to socio-demographics or daily life practices. In the 9th round of the European Social Survey (ESS9), these questions were designed with the split-ballot technique, which randomly assigns selected respondents to male and female versions of the questions (Fig. 1). The split-ballot designed questions may be more prone to item-missing data while shortening the interview duration (Axenfeld et al. 2022).

Fig. 1
figure 1

An example of the split-ballot designed question in the ESS9

Full size image

The “don’t know” and “no response” answer choices were taken when measuring item-nonresponse for the selected set of questions. The respondent’s motivation was constructed based on the interviewer’s assessments of each respondent. Thus, the interviewer data set was also utilized in addition to individual data. This study presents results for 29 participating countries comparatively, focusing on the relationship between motivation and item-nonresponse. Particularly, five countries (France, the United Kingdom, Norway, the Netherlands, and Portugal) were selected for the multivariate part of the study. The country selection is based on the descriptive results that refer to stronger relationships between motivation and item-nonresponse. Individuals over 15 years of age constitute the unit of analysis for this work.

The suggested strategies within the study could be adopted by various surveys, targeting high-quality survey data by reducing item-nonresponse. Moreover, the findings related to interviewer and respondent characteristics, controlled in the multivariable models, would also give insights for future survey designs. Lastly, split-ballot questions were not examined methodologically so much, as Axenfeld et al. (2022) underlined.

1.1 Theoretical framework

The study objectives could be associated with some relevant theories in the field. Cannell et al. (1981) suggested that the quality of data can be improved with enhanced respondent motivation in surveys. In this regard, social theories that support enhanced respondent motivation would underpin the main arguments of the current study.

Influential factors on survey participation may also be considered as factors that could increase respondent motivation. According to Leverage-Salience Theory, each sample unit has different sources of motivation and weights when deciding to participate in surveys (Groves et al. 2000). Survey topic, sponsorship, community involvement, and interview duration are among the factors that affect survey participation of sample units. Particularly, community involvement can increase respondents’ sense of responsibility in a social survey setting, and make them more motivated during the interview. This situation may cause respondents to think carefully before answering questions and give accurate and reliable answers, not to allow providing item-missing data.

Moreover, the level of effort spent to answer questions could be evaluated as an indirect measurement of respondent motivation. In this sense, satisficing theory asserts that respondents tend to complete interviews using a minimum level of effort (Krosnick et al. 2022). The quality of data obtained from the interviews completed with satisficing behavior of respondents would also be poor, as expected. Respondents may would like to end the interview quickly and thus, cause high item-nonresponse rates.

2 Methodology

2.1 Data source

The data source is the ESS9, which is a biennial, cross-national, and large-scale survey carried out in 29 European countries in 2018. The target population of the ESS is individuals aged 15 and over who live in private households in Europe. The survey collects information about a wide range of topics, such as political attitudes, voting behaviors, trust in institutions and organizations, well-being, immigration, religion, discrimination, and social media use, in addition to household characteristics and socio-demographics (ESS 2018). The core sections in the questionnaire enable researchers to follow survey interests over time. Moreover, information needed to understand specific issues in depth is collected through rotating modules. Computer-assisted personal interviewing (CAPI) was used as the mode of data collection in the ESS9, while the use of self-administration techniques has begun since the 10th round.

This study also uses interviewer data, including basic socio-demographics, the presence of any person during the interview, and evaluations per respondent in addition to the main individual data. The interviewer data set provides information about the willingness and ability of respondents that could help to operationalize respondent motivation toward the interview. These two data sets were combined to conduct statistical analyses of the study, bringing motivation variables to the individual data. The unique respondent identification numbers and country codes were used to merge data sets, as suggested by the ESS (2023).

The sampling design of the ESS is based on a probability sampling approach, which gives all sample units a chance to be selected. The survey has also adopted a complex sample design strategy, referring to multi-stage, stratified, cluster sampling for each participating country. Therefore, this study presents results by adjusting the complex sample design features of the ESS9. The stratum and cluster variables provided in the data set allow such analyses. The weight variable was also used when performing statistical analyses to obtain accurate estimates for the target population. The analysis weight variable considers the factors for post-stratification, sample design, and population distribution (ESS 2023). In other words, the weighted analyses within the study adjusted the inconsistencies due to sampling design, non-response, and population distribution.

Both descriptive and multivariable analyses were conducted to present results for all participating countries. Particularly, the regression models were also constructed separately for France, the United Kingdom, Norway, the Netherlands, and Portugal. The main reason behind the country selection comes from the stronger associations detected between respondent motivation and item-nonresponse levels in the descriptive part. Statistical analyses were carried out on 42,440 individuals in total: 2,204 cases in the United Kingdom, 2,010 cases in France, 1,673 in the Netherlands, 1,406 in Norway, and 1,055 in Portugal.

2.2 Variables

2.2.1 Respondent motivation

Table 1 shows the study variables to introduce ‘respondent motivation’. Interviewers assessed the factors related to the willingness and ability of respondents to participate in the interview through a five-point response scale. Clarification of questions refers to the degree of any need for an explanation when asking questions. The answering questions to the best of the respondent’s ability and the respondent’s reluctance to answer represent the respondent’s level of interest in responding. The understanding of the questions represents the degree of the respondent’s ability to understand questions properly.

Table 1 Variables for respondent motivation
Full size table

The understanding of the questions and answering to the best of the respondent’s ability could be evaluated within the ability while the respondent’s reluctance to answer could be evaluated within the willingness when constructing respondent motivation. On the other hand, clarification of questions was evaluated within the ability and willingness due to its potential relationship with both dimensions (Table 1).

2.2.2 Item-nonresponse

Table 2 presents a set of hypothetical questions and attitudes toward social norms in order to be used for item-level nonresponse analysis. All of the questions here were designed with the split-ballot technique, which assigns respondents to female and male versions of questions randomly. The main aim of the questions is to reveal gender differences in social norms for particular issues. The ‘refused to answer/refusal’, ‘don’t know’, and ‘no answer’ response choices are available in the data in addition to the different numeric responses given for hypothetical ages and 5-point scales for attitudes toward social norms.

Table 2 Variables for item-nonresponse analysis
Full size table

2.2.3 Model covariates

Table 3 gives the potential predictors that have an impact on the item-nonresponse level of respondents. Both respondent and interviewer characteristics were controlled in the regression models, although the key study variable is respondent motivation. The motivation score per respondent (calculated through variables in Table 1) was accumulated into three categories: ‘low’ (4–11 points), ‘middle’ (12–17 points), and ‘high’ (18–20 points). The age of respondents refers to 10-year age groups while the interviewer’s age was recoded into ‘17–25’, ‘26–31’, ‘32–37’, ‘38–43’, ‘44–60’, and ‘61 and older’ age groups. The total years of education were classified into ‘Less than 11 years’, ‘11–14 years’, and ‘More than 14 years’ educational levels.

Table 3 Predictors for item-nonresponse analysis
Full size table

2.3 Measurement of respondent motivation and item-nonresponse

The motivation score per respondent was calculated by excluding “refused to answer/refusals”, “don’t know”, and “no answer” response options. A few of the motivation-related items have negative meanings, while the remaining ones have positive meanings based on the 5-point scale that varies from 1-never to 5-very often. Therefore, the directions of the points were set in the right way before creating a score per respondent. Hence, the motivation scores that range from 4 to 20 points represent a degree from the lowest to the highest level of motivation. At this point, it is assumed that the four factors used to operationalize respondent motivation have equal weights.

The motivation score per respondent (ms) is calculated as below

$$ms=\sum_{i=1}^{4}{z}_{i}, {z}_{i}=1,\dots ,5$$

where zi is the point scale given by the respondent for the i-th item, ranging from 1 to 5. Thus,

$$ms\in [\mathrm{4,20}]$$

The item-nonresponse for each selected question was calculated based on the “don’t know”, “no answer”, and “refused to answer/refusals” response choices.

$$y_{i} = \left\{ {\begin{array}{*{20}l} {1,} \hfill & {don^{\prime}t \,know/no\, response/refused\, for \,i - th\,item} \hfill \\ {0,} \hfill & {any\, response \,for\, i - th \,item} \hfill \\ \end{array} } \right. i = 1, \ldots , 20$$

The yi represents the item-nonresponse indicator for the i-th item.

The proportion of item-nonresponse per respondent was calculated according to the following formulation:

$$i-nr=\frac{\sum_{i=1}^{20}{y}_{i}}{20}$$

where i − nr is the abbreviation of the proportion of item-nonresponse. Thus,

$$i-nr\in [\mathrm{0,1}]$$

2.4 Statistical analyses

2.4.1 Descriptive analysis

The percentage distribution of item-nonresponse levels was examined, as an outcome variable. Moreover, a correlation analysis was conducted to understand the relationship between respondent motivation and item-nonresponse descriptively. The main reason behind the selection of correlation analysis is to understand the relationships between these continuous variables based on linear associations. Accordingly, the null hypothesis, which implies no association between the motivation and item-nonresponse was tested. The null hypothesis is as follows:

$${H}_{0}: r=0$$

where r is the measure of association between the motivation and item-nonresponse (Kleinbaum et al. 2013).

$$w\circ r=\frac{\sum_{j=1}^{n}({X}_{j}-\overline{X })({Y}_{j}-\overline{Y })}{{\left[\sum_{i=1}^{n}{\left({X}_{j}-\overline{X }\right)}^{2}\sum_{j=1}^{n}{\left({Y}_{i}-\overline{Y }\right)}^{2}\right]}^{1/2}}$$

where Xj refers to motivation score for j-th respondent and Yj refers to the proportion of item non-response for j-th respondent, i − nr. The \(\overline{X }\) is the mean score of motivation, the \(\overline{Y }\) is the mean of i − nr, n is the number of cases, and w is the analysis weight factor for the ESS9 population.

The correlation analysis enabled us to interpret the significance, direction, and size of the relevant relationships. Thus, the correlation coefficients (e.g., Pearson correlation, Kendall’s tau, and Spearman’s rho) were estimated for each country within the study.

2.4.2 Multivariable analysis

Multivariable analyses include multiple linear regression modeling, utilizing the Taylor series linearization (TSL) estimation technique. This technique is appropriate for exploring the linear relationships between continuous outcome variable, number of item-nonresponse here, and a set of covariates. Moreover, bivariate relationships between the outcome variable and predictors were evaluated based on the design-adjusted Wald tests. The complex sample design of the ESS9 was accounted for in these analyses by introducing the stratum, cluster, and weight variables. The model results include estimated regression coefficients for predictors as well as their standard errors, and significance values. The significant effects were determined according to p < 0.001, p < 0.01, p < 0.05, and p < 0.1 levels within the study.

The stepwise approach was adopted during the model construction process to reveal the additive effect of predictors on the dependent variable. In this sense, the first model (Model 1) includes respondent motivation only. The second model (Model 2) includes respondent characteristics in addition to respondent motivation. Lastly, interviewer characteristics were added to the final model (Model 3).

The final models were also constructed for five subpopulations for France, the United Kingdom, Norway, the Netherlands, and Portugal in particular, in addition to the overall model. All predictors, including motivation, in Table 3 were entered into the following regression equation:

$$w\circ Y=w\circ \left[\alpha +\beta X+\sum_{i=1}^{3}{\delta }_{i}{R}_{i}+\sum_{j=1}^{2}{\rho }_{j}{I}_{j}\right]$$

where Y is the proportion of item-nonresponse and X is the respondent’s motivation. Here, the β parameter refers to the effect of motivation on the level of item-nonresponse estimated by the model. The Ri denotes respondent characteristics, while Ij denotes interviewer characteristics controlled in the model. The δi and ρj refers to the estimated parameters of respondent and interviewer characteristics, respectively. Lastly, α is the constant term of the equation, while w is the analysis weight factor.

Graphical diagnostics were also performed to ensure that regression estimates are robust in addition to multi-collinearity analysis. Variance inflation factors (VIF) for the final model ranged from 1.09 to 1.30, implying no multi-collinearity problem among predictors. Graphical diagnostics comparing residuals with fitted values and leverage show that regression models are fit well. These results confirm that we can safely interpret own effects of each predictor on the outcome.

Statistical analyses were conducted in RStudio-Version 1.4.1717 using the required packages and functions (e.g., svydesign, svyglm, regTermTest).

3 Results

3.1 Descriptive results

3.1.1 Item-nonresponse analysis

Descriptive analyses cover the examinations for item-nonresponse levels for split-ballot designed questions and the relationships between item-nonresponse and respondent motivation scores at the country level.

Table 4 presents the weighted percentage distribution of the calculated item-nonresponse levels i − nr as well as several individuals. The overall results showed that most of the respondents (71%) answered all of the selected questions, referring to no item-nonresponse. 13% of respondents did not answer one of the questions, leading to a 0.05 level of item-nonresponse while 5% of those did not answer two questions resulting in a 0.10 level of item-nonresponse. About 15% of the respondents provided “don’t know”, “no answer”, or “refused to answer/refusals” responses for three or more items. The levels of item-nonresponse vary across survey countries.

Table 4 Percentage distribution of item-nonresponse levels
Full size table

3.1.2 Correlation analysis

The correlation analysis results show that there is a significant, negative, and moderate relationship between respondent motivation and item-nonresponse level in total (Fig. 2, Table 5). The significant relationship was also confirmed by the Wald test (Wald-F test statistic: 155.01, p < 0.01). The results pointed out that the level of item-nonresponse for split-ballot questions reduces while the respondent motivation score increases. The estimated coefficients (Pearson’s correlation, Kendall’s tau, and Spearman’s rho) confirmed similar relationships for almost all participating countries (Fig. 2).

Fig. 2
figure 2

The estimated correlation coefficients for the association between respondent motivation and the proportion of item-nonresponse

Full size image
Table 5 Estimated correlation coefficients for selected countries
Full size table

The strongest relationships between respondent motivation and item-nonresponse were found in France, the United Kingdom, Norway, the Netherlands, and Portugal (Table 5). Belgium, Switzerland, Germany, Denmark, and Italy are the other countries where significant moderate relationships were also estimated (correlation coefficients range from − 0.20 to − 0.24). However, the results for Austria, Cyprus, Estonia, Iceland, Lithuania, Slovakia, and Montenegro were insignificant, probably originating from the lower sample sizes as opposed to other countries.

3.2 Multivariable results

3.2.1 Multiple regression analysis

Table 6 shows the estimated effects of the multiple linear regression model overall, indicating the negative impact of respondent motivation on the proportion of item-level nonresponse. Additionally, the table includes results for the Wald-F test to understand the significance of bivariate relationships between predictors and the level of item-nonresponse.

Table 6 Estimated model effects
Full size table

In line with the descriptive results, the item-level nonresponse significantly declines with the increasing level of respondent motivation. The first model (Model 1) results suggested that the item-nonresponse is about 2.5 points lower among highly motivated respondents relative to lowly motivated ones. The results for Model 2 (including respondent characteristics) and Model 3 (including interviewer characteristics) are similar, indicating a relatively reduced impact of interviewer characteristics on the item-nonresponse.

The final model (Model 3) results suggested that the level of item-nonresponse is approximately 1.8 points lower among highly motivated respondents relative to lowly motivated respondents when the other predictors are held constant. Moreover, middle-level motivated respondents have about 1.3 points less item-nonresponse compared to highly motivated ones (Table 6).

The controlling factors in the models were the socio-demographics of respondents and interviewers. The overall results also pointed out that females, educated respondents, and adults appeared to be more advantageous in terms of getting fewer item-nonresponse. Respondents who have more than 14 years of education have significantly 0.24 points less item-nonresponse compared to respondents with less than 14 years of education, keeping other covariates fixed. Respondents aged between 35 and 64 years have significantly less item-nonresponse compared to individuals between 15 and 24 years of age. The elderly population (65 years and older) has a slightly higher level of item-nonresponse compared to the youngest age group, referring to challenges with surveying older people (Wagner et al. 2019; Schanze 2021).

Although the estimate for the interviewer’s sex was insignificant, the interviewer’s age has a significant effect on the level of item-nonresponse. Interviewers aged between 38 and 43 years have a 0.39-point higher level of item-nonresponse compared to interviewers aged 17–25 years. This may result from the greater effort spent by younger interviewers when recording answers (Table 6).

The model effects constructed for France, the United Kingdom, Norway, the Netherlands, and Portugal are estimated in the same way as the results for the overall model (Table 7). Particularly, the item-nonresponse level is significantly 3.25 points lower among highly motivated individuals relative to lowly motivated respondents in France. Similarly, middle-level motivated respondents had 2.65 fewer points of item-nonresponse compared to lowly motivated ones. The significant model effects for respondent motivation were also found in Portugal. The effect of having high respondent motivation decreased the level of item-nonresponse by an average of 1.65 points relative to having a low level of motivation in Portugal. The results confirmed the study expectations at the beginning, claiming lower item-nonresponse owing to the high level of motivation.

Table 7 Estimated model effects for selected countries
Full size table

Although the estimated effects of respondent age were found to be insignificant in the selected countries, respondent sex had a significant effect on the level of item-nonresponse in France and Norway. Female respondents had 0.29 and 0.18 points less item-nonresponse than males in Norway and France, respectively. They may be more cooperative than males in surveys. The high educational level of respondents significantly affected the level of item-nonresponse in the Netherlands and the United Kingdom, as well as the overall model. The effect of having more than 14 years of education decreased the level of item-nonresponse by an average of 0.54 points relative to having less than 11 years of education when other predictors were fixed (Table 7).

The effects of interviewer age vary across countries, implying no strong inferences for the impact of interviewer age on the level of item-nonresponse. In Norway, female interviewers led to 0.38 points fewer item-nonresponse on average than male interviewers.

4 Practical and theoretical implications

The study has several practical implications that could be followed during the survey process. In light of the study findings on respondent motivation, this study suggests that interviewers should keep the respondent’s motivation at a high level during the interview. This would be helpful to collect better-quality survey data. Thus, all efforts spent on ensuring an accurate understanding of questions, providing willingness to answer, and assisting reluctant respondents would be beneficial in practice. Additionally, interviewer training instructors should clearly explain to interviewers the effect of respondent motivation on response quality, particularly for split-ballot-designed questions. In this regard, taking a short break from the interview may be a good strategy if an interviewer observes that the respondent is becoming less interested in the interview. Hence, interviewers can keep respondent’s attention after the break, helping to collect high-quality survey data.

Improving data quality by reducing the item-nonresponse could be possible with certain design-related decisions. For instance, sending pre-notification letters to sample units and using incentives when cooperating might be evaluated by survey teams. Moreover, questions designed with the split-ballot technique in the ESS may be more prone to item-level nonresponse compared to core questions. Therefore, alerts for such questions should be designed in the questionnaire according to the mode of data collection in the ESS (switching from interviewer-assisted to self-administered in recent times) (ESS 2022).

The inverse relationship between respondent motivation and proportion of item-nonresponse refers to respondents’ unwillingness to respond and their trying to finalize the interview quickly under the impact of reduced motivation. This may be explained by the satisficing theory which refers to a complete survey using a minimum level of effort (Krosnick et al. 2002). Conversely, highly motivated respondents may answer questions by doing their best ability (e.g., reading items completely, thinking for a while before answering, and giving valid and reliable answers) without any item-missing. This finding also may be associated with community involvement, which is an influential factor for participating in the study, as asserted by the Leverage-Salience theory (Groves et al. 2000).

5 Conclusion and discussion

This study seeks to answer the question, “What is the impact of respondent motivation on the item-nonresponse level for the split-ballot designed questions in the ESS9?”. In this sense, the study hypothesis is that the level of item-nonresponse for split-ballot designed questions could be decreased through the high level of respondent motivation.

Although a couple of studies have focused on the effect of respondent motivation on the resulting quality of survey data, no study has touched on the item-nonresponse for split-ballot-designed data. Yet, this particular set of data may suffer from data quality problems frequently. Among rare studies, Axenfeld et al. (2022) employed various module-building approaches to improve the quality of data collected using split-ballot design. Consequently, the explanatory process between respondent motivation and item-nonresponse for split-ballot-designed data would give insights when developing motivation-related strategies to improve the quality of these survey data.

In this study, a set of questions including hypothetical and too-old/young ages for specific situations (e.g., becoming an adult, retiring permanently, getting married, becoming a father and mother) and attitudes toward social norms (e.g., child preference, living with a partner without marriage, working of individuals with children) were selected for item-nonresponse analysis. Respondents may tend to leave these split-ballot-designed questions unanswered. The negative impact of low respondent motivation, particularly on sensitive items, was also emphasized by social researchers (Tourangeau and Yan 2007). The respondent motivation for this study was constructed through the interviewer’s observations regarding the respondent’s willingness, and ability, as established in various forms previously (Knippenberg and Daamen 1996; Wenemark et al. 2010; Rogelberg et al. 2006).

The results of the correlation analysis suggested that there is a significant, moderate, and negative relationship between respondent motivation and level of item-nonresponse for most of the participating countries. In other words, study findings put forward that as respondent motivation increases the level of item-nonresponse decreases, leading to high-quality data. The multivariable models constructed for selected countries where stronger relationships were estimated also produced similar results.

The overall results showed that highly motivated respondents have 1.81 points less item-nonresponse compared to lowly motivated respondents when respondent and interviewer characteristics are held constant. France, the Netherlands, and Portugal are the countries where significant model effects were estimated (3.25, 2.02, and 1.65 points, respectively). This main finding may be associated with respondents’ unwillingness to respond and their trying to finalize the interview quickly under the impact of reduced motivation. This may be explained by the satisficing theory which refers to a complete survey using a minimum level of effort, too (Krosnick et al. 2002). Conversely, highly motivated respondents may answer questions by doing their best ability (e.g., reading items completely, thinking for a while before answering, and giving valid and reliable answers) without any item-missing. This finding also may be associated with community involvement, which is an influential factor for participating in the study, as asserted by the Leverage-Salience theory (Groves et al. 2000).

The study findings also suggested the importance of interviewer evaluations to measure respondent motivation. It seems that their observations during the interviews would help to operationalize respondent motivation based on various dimensions (e.g., willingness and ability). The overall model also estimated significant results for the respondent’s education, age, and sex, which are the control covariates. Individuals who are older than 35 years of age appear to be more willing to respond without any item-missing compared to younger respondents. Female respondents have significantly less item-nonresponse relative to males. The positive impact of more years of education on reducing item-nonresponse was also estimated for models. These results can be attributed to fruitful interviews conducted with females, highly educated people, and adults. Similar results on the relationships between socio-demographics and data quality can be found in several studies, even though the significance values of the results are controversial (Kwak and Radler 2002; Krosnick et al. 2002; Matthijsse et al. 2015; Silber et al. 2021).

6 Limitations and future research

This study has two limitations. The first limitation is related to the operationalization of the respondent motivation which is based on the ability and willingness of respondents. Survey design-related factors such as survey topic, incentive use, survey organization, and sending pre-notification letters might be used when constructing the motivation. Moreover, surveys should consider the different interviewer observations that could affect motivation (interview environment, observable response reliability, interaction with the respondent, etc.) as well as their design in the questionnaire. The utilization of interviewer observations, as a type of paradata would be insightful for such examinations (West 2013). Additionally, future research should consider survey design-related factors when constructing the respondent motivation.

The second limitation is related to the data source when formulating respondent motivation within the study. The respondent motivation was formulated through interviewer observations towards respondents such as understanding of questions and reluctance when answering questions. These evaluations may not be objective and comparable for all interviewers. Instead, respondents’ own thoughts may be fruitful when assessing their motivation during the interview. Thus, future surveys should consider designing a separate form where respondents can answer questions about their feelings, thoughts, feedback, and motivation level.

This study also calls for future studies that will focus on the identification of respondent motivation and the resulting quality of data with different quality indicators (e.g., tendency to round numerical answer, primacy and recency effects) rather than item-nonresponse.