The qualitative pretest interview for questionnaire development: outline of programme and practice

Abstract

Good survey research depends on asking the right questions; it is the only way to ensure that the information collected from respondents is suitable for providing good answers to our research questions. The article discusses and advocates a comprehensive consideration of qualitative-interpretive methodology in open forms of pretesting for the evaluation of draft survey questionnaires. We outline an approach we call Qualitative Pretest Interview (QPI). It transfers the idea of negotiated common understanding in everyday communication to the clarification of meaning in draft survey questions and similar stimuli. The QPI involves ascribing interview partners the role of co-experts in this process and employing methodically integrated communication strategies. This paper focusses on how QPIs are conducted. Using an example interview, we illustrate how the particular way of qualitative pretest interviewing aims at a dialogic clarification of meaning in order to reach intersubjective understanding between participant and interviewer. In the process, we gain detailed insights into how and why a certain questionnaire might not work as intended, and ideally how this might be alleviated. QPIs pursue similar goals as Cognitive Interviews but rely more systematically on qualitative-interpretive methodology.

Introduction

Surveys require researchers to have a precise idea of the theoretical constructs they want to investigate and of their presumed connectedness. Furthermore, it is necessary to measure these constructs validly by means of a questionnaire. In other words, good survey research depends on asking the right questions; it is the only way to ensure that the information collected from respondents is suitable for providing good answers to our research questions. In this article, we will essentially deal with a specific issue of valid measurement: How can we formulate questions in such a way that they are understood adequately and produce answers that we can consider valid? The paper is intended as a contribution to a pragmatic-professional consideration and integration of qualitative-interpretive perspectives in the preparation of standardised surveys.

Based on the works of Tourangeau and colleagues on “Cognitive Aspects of Survey methodology” (CASM), in-depth evaluation of questionnaires is often structured along the lines of the “Response Process Model” (Tourangeau et al. 1984, 2000). It describes the respondents’ way of processing and answering questions as a four-step procedure: comprehension of the intended meaning; retrieval of an answer from memory; judging the answer e.g. whether memory is complete; and formulating a response that fits the format of the question. Each of these steps poses its own potential source of measurement error, e.g. lack of comprehension, problems remembering an answer, or socially desirable answering. There is by now extensive literature providing suggestions on how to alleviate these problems when designing standardised questionnaires (e.g., Bradburn et al. 2004; Faulbaum et al. 2009; Groves et al. 2009; Gobo and Mauceri 2014; Porst 2014; Schnell 2012; Wolf et al. 2016).

Since the challenges faced in designing a questionnaire and the resulting potential sources of measurement error depend to a large degree on the particular study at hand, general recommendations are not sufficient to ensure high quality. It is therefore advisable to make extensive use of pretest-procedures to evaluate and improve questionnaires before the main fieldwork (cf. e.g., Bradburn et al. 2004; Schnell 2012). Some common approaches include the inspection of non-response patterns or unexpected distributions across response categories quantitatively, or the use of even more complex approaches like the testing of psychometric properties.

While these techniques help to identify problematic questions, it is often more useful to employ open forms of pretesting in order to get a deeper understanding of how and why certain questions might not work as originally intended. These open forms of pretesting are particularly relevant as soon as a general concept of the questionnaire, regarding e.g. topics and concepts, has been decided upon and the next step consists in developing and refining individual questions. Already in the 1940s there was a debate about the potential of open forms of interaction between interviewer and respondent for providing more valid answers to research questions (Beatty 1995). Today, standardised procedures dominate primary data collection due to efficiency issues and concerns about interviewer bias. At the same time, the use of open techniques and the benefits of reactivity for the validation of survey questionnaires are largely undisputed. Since the 1980s cognitive techniques have been an important part of the process of developing, testing, reviewing, and evaluating standardised questionnaires and individual survey questions. With their origin in the tradition of cognitive psychology (Ericsson and Simon 1980) they help identify, together with participants, problems with the questionnaire and correct them if necessary.

Over the last decades, Cognitive Interviewing (CI) as developed by Gordon Willis (e.g., Willis 2005, 2015) has emerged as the main approach to conducting in-depth evaluations of survey questionnaires. The original challenge of the design of the CI consisted in reducing response and measurement errors by “identifying and correcting problems with survey questions” through, as (Beatty and Willis 2007, p. 287) define the CI in practical terms, “the administration of draft survey questions while collecting additional verbal information about the survey responses, which is used to evaluate the quality of the response or to help determine whether the question is generating the information that its author intends.” Over the past 30 years the CI has been elaborated continuously as a method and refined into what has become one of the most effective tools for improving survey research.

The starting point of our contribution to advancing this very effort is the observation of a somewhat ambiguous relationship of the CI to qualitative-interpretive research (Bethmann et al. 2019; Buschle et al. 2020). Over the course of the past decades its authors and advocates moved the CI ever closer to the purposes and sphere of qualitative research—from the mere proposition that cognitive interviewing is “designed to help researchers understand how respondents interpret questions” (Beatty 1995, p. 1052) to the more recent textbook about the analysis of CIs published in the Series in Understanding Qualitative Research of Oxford University Press (Willis 2015). The textbook—“in seeking a theory of method to apply to the cognitive interview as an alternative to cognitive theory” (ibid., p. 29)—suggests rudimentary links to qualitative research theory. This is an important step in the ongoing development of the CI that is corroborated by other contributions (Miller 2014; Collins 2015). Yet this ‘qualification’ of the CI is done ex post, for the origin of the method as a practical solution to reducing measurement error is rooted in debates about cognitive approaches in psychology and survey research in the 1980s (e.g., Willis et al. 1991; Willis 2005, chapter 3). At the current point in its development a few decades later the CI appears as a methodological hybrid, at least in conceptual terms: “cognitive interviewing can be viewed as either a psychological procedure, or as a special case of qualitative interviewing” (Willis 2015, p. 16).

Against this background, we take sides and advocate the more comprehensive consideration of qualitative-interpretive aspects characterising the process of exchanging meaning in interview encounters. Our starting point is the notion of intersubjective understanding as the conceptual pillar of interpretive interview research. While we do not contest the practical objectives addressed with CI-techniques our initial formulation of the underlying challenge is, in contrast, rather conceptual. Our point of departure is social-scientific and sociological. Thus, our argument and conclusions arrive at labelling the method in a way that does not reproduce traditions and connotations associated with the term ‘cognitive’. We propose to call it Qualitative Pretest Interview (QPI), a genuinely qualitative technique of interviewing for the purpose of improving survey questions and their delivery.

To put it more succinctly, there are large and obvious overlaps between the CI and the QPI methodology, namely employing a “think-aloud” approach and encouraging interview partners to say everything that comes to their minds when reading the question. Furthermore, both approaches underline, that the aim of the pretest interview is not to collect data to answer a substantive research question, but rather that the interviewer and the participant work together to identify sources of difficulty and vagueness in survey items. That being said, there are a few important distinctions between the proposed QPI and the traditional CI. Firstly, the QPI emphasizes the participant more explicitly as an equal partner in the discussion, resulting in specific interviewing techniques, e.g. a more extensive briefing sequence to establish this work relation. It also has a stronger focus on discussing the question’s intended meaning, including a more thorough deliberation of potential improvements to the questions already vis-à-vis the participant. These results can then be used as a qualified input for further discussion of revisions to the survey instrument among the researchers.

Compared to the above definition of the CI our conceptual rather than practical starting point is reflected in our development of the working definition of the QPI. It is provided in the following chapter 2 together with an outline of the methodological programme of the QPI and its key communication strategies. Chapter 3 illustrates and synthesises the practical implementation of these programmatic aspects of the QPI based on a concrete example from pretesting an add-on questionnaire to the German sub-study of the Survey of Health, Ageing and Retirement in Europe (SHARE, http://share-eric.eu). In addition to exemplifying the joint production of understanding with regard to one particular item, we discuss the important steps of briefing and debriefing interview partners for the particular challenge of participating in a QPI. The conclusion reviews the argument and connects it to survey methodological practice.

Outline of the methodological background and programme of the QPI

The QPI has its roots in methodology debates that provide particular perspectives on the issue of producing mutual understanding in research encounters. They informed the development of our approach to pretest interviewing and are discussed in the following.

Indexical expressions and intersubjective understanding

Already more than four decades ago, Kohli (1978) contributed a fundamental discussion of differences and similarities between unstructured and standardised interviews. He argues that situativity, interactivity and reactivity characterise both open and closed interview techniques. The validity of all kinds of interview data crucially depends on the equal understanding of the meaning communicated in questions and answers, pre-formulated or not, by all participants—i.e. at least researchers, interviewers, and respondents. In the direction of standardised procedures, Kohli (1978, p. 9; own translation) finally writes: “Communication always means intersubjective understanding (albeit to varying degrees).” According to their origin in the French term ‘entrevue’ interviews are arranged encounters for verbal exchange and understanding based on mutual expectations (see also Kvale and Brinkmann 2009, pp. 1–2). Participants in all types of interviews face the challenge of understanding inevitably vague and ‘indexical expressions’ that, according to Garfinkel (1984/1967), are constitutive of human communication and mutual understanding (Schutz 1967/1932). Following Garfinkel’s ethnomethodological approach, expressions are potentially ambiguous references to meanings that are best understood in their context of origin. Thus, intersubjective understanding (Fremdverstehen) of meaning requires the consideration of these origins and contexts of expressions.

The various approaches in the social sciences subscribe to different ways of dealing with this vagueness and indexicality described by Garfinkel (Przyborski and Wohlrab-Sahr 2008, pp. 28–31). Qualitative-interpretive approaches recognise that expressions (within everyday frames of interpretation) always refer to everyday meanings that are specific for the very context of their articulation. Accordingly, social scientists can only reconstruct these meanings (within scientific-theoretical frames of interpretation) in a valid and appropriate way if some of the original context of expression is preserved during data collection. Ideally, the very context that is unfolded by the participants themselves constitutes the background against which expressions are interpreted, for this is the context, in which these expressions originally ‘made sense’. On the contrary, and in line with its very own programme, standardised research tries to predefine the context of collecting responses and to eliminate the risks (and chances) of achieving an embedded understanding of expressions and ‘answers’. It claims to neutralise the indexicality of expressions (i.e. their potentially unclear reference) by attempting to standardise the semantic content. In practice, standardisation should ensure that all participants within an artificial-scientific context (i.e. the standardised situation, procedure, and contents of data collection) understand formulations of questions and answers in the same way (ibid.).

Fabricated versus negotiated common understanding

Standardised survey interviewing must rely on fabricated common understanding for the sake of measurement throughout the whole chain of participants involved in meaning making and taking like, for instance, in secondary analysis (Suchman and Jordan 1990). Qualitative interviews, on the other hand, generate what could be called negotiated common understanding. They do not attempt to bypass the requirement of intersubjective understanding by artificially neutralising indexicality or by re-contextualising expressions in a scientific perspective. Rather, qualitative-interpretive approaches suggest various strategies to capture expressions within context, thus preserving their indexical quality. In other words, in qualitative-interpretive approaches, the inevitable process of intersubjective understanding is “methodically controlled in so far as the difference between the frames of interpretation of the researchers and those of the researched is systematically accounted for” (cf. Przyborski and Wohlrab-Sahr 2008, p. 31, own translation).

The qualitative interview is one of those methods that provide the opportunity to clarify the meaning of indexical expressions together with our research subjects. It can be characterised as “a fundamentally asymmetrical form of communication, which, however, is always co-produced and maintained by both participants, because, for example, during the interview situation both cannot help but find out what the other 'actually wants' (what his actual interests are, how he sees the situation, how he assesses his counterpart, etc.)” (Honer et al. 2006, p. 95f, emphases in the original, own translation). Despite this asymmetry—i.e. it is initiated (and usually compensated for) by the researcher with certain knowledge interests—the qualitative interview is not a role-playing game where one person asks questions and the other provides answers. As discussed above, in order to achieve intersubjective understanding as qualitative researchers we need to decipher the relation between indexical expressions and their original context. The genuine advantage of qualitative interviews consists in the fact that we have the chance do that under the guidance of our interview partners.

The goal of generating negotiated common understanding requires both participants to meet each other as partners in an encounter where researchers consider interviewees as experts with specific knowledge that is otherwise not accessible. This knowledge can be substantial. Alternatively, and of importance for our discussion of pretest interviews, it can be semantic in the sense that it clarifies the links between utterances and their context of origin. This helps us to comprehend how the respondent’s expressions and intended implications should be understood (i.e. how he or she wants it to be interpreted) in the context of what has been said—in contrast to how it could be understood in terms of possible interpretations researchers can come up with.¹ In order for both participants to be able to contribute equally to the challenge of handling potentially ambiguous expressions the research event of the qualitative interview needs to revert to the taken-for-granted rules of everyday communication. Deviations from these familiar and unquestioned rules, which are necessary due to the specific nature and dynamic of research encounters in general and pretest interviews in particular, require explanation that is delivered in the frame of extensive briefing at the beginning of the conversation.

Towards a definition of the QPI

The programme and practice of the QPI follows elaborate methods of qualitative interviewing that translate these requirements of intersubjective understanding into principles and techniques for research communication. Within the diverse landscape of qualitative interviewing methods originating in the particular German methodology discourse in the tradition of intersubjective understanding (Fremdverstehen; Schutz 1967/1932; Cicourel 1964), the Problem-centred Interview (PCI) stands out in that it programmatically promotes active understanding (in addition to active listening). The PCI (Witzel 1982; Witzel and Reiter 2012; Reiter and Witzel 2019) conceptualises the situation of data collection explicitly as a research encounter for the discursive-dialogical clarification of (implied) meanings. Researchers and respondents exchange views in everyday language on topics (‘problems’) that interest the former scientifically and the latter from a practical perspective (Witzel and Reiter 2012). In view of the ambiguity and indexicality of expressions, the PCI proposes to make good use of the distinct interpretive competence of the interview partners. They are not just respondents providing information that is collected and later analysed. The PCI recognises their participation, presence, and readiness to clarify meaning and involves interview partners as temporary research assistants in the co-construction of interpretations and comprehension.

The QPI inherits these aspects of the PCI’s programme of joint production of understanding. It is a method of pretest interviewing for quality improvement of questionnaires by conceptualising the clarification of comprehension as a social process of negotiating meaning through intersubjective understanding. It involves interview partners as co-experts for this process in a dialogic clarification, in everyday language, of manifest and implied meanings and understanding of formulations and expressions used in draft survey questions and other standardised stimuli.

Communication strategies—active listening and active understanding

The QPI adopts from the PCI the idea of applying two sets of methodologically integrated communication strategies of general and specific exploration for collecting information and for directly negotiating implied meaning (Bethmann et al. 2019; Buschle et al. 2020; Witzel and Reiter 2012; Reiter and Witzel 2019). Together they constitute an integrated portfolio of techniques that allow the interviewer, depending on the course of the conversation, to change back and forth between active listening and active understanding. A first set of communication strategies of general exploration are dedicated to generating self-perpetuating accounts; the respondent’s point of view should be able to unfold gradually and mainly inductively. In terms of methodology and in line with the above discussion, these communication strategies of general exploration pursue the goal of eliciting contextualised expressions—i.e. expressions that are best interpreted from within their context of origin. For example, in addition to having each survey question first read out by the interview partner this set of strategies combines open and ad-hoc questions with comparative probing and repeated prompts and invitations to specify what was said (see next chapter). The interviewer’s attention is occupied mostly with active listening and making up his or her mind about what is being said. Analytically, these communication strategies of general exploration follow exactly the purpose of providing the ‘material’ basis for the subsequent specific exploration and stepwise reduction of the ambiguity of communication that Garfinkel (1984/1967, p. 41) calls “the occasionality of expressions, the specific vagueness of references.”

The second set of communication strategies of specific exploration introduce a deductive moment into the conversation by identifying interconnections between statements, by clarifying possible misunderstandings and pre-interpretations, and by validating attributions of meaning. In this way, the conversation becomes much more dialogic. The stronger involvement of the interviewers reflects their effort to engage in active understanding of what is being said and of establishing intersubjective understanding. Comprehension questions are used to clarify evasive and hidden answers in an unobtrusive manner. Mirroring is a prime strategy for achieving communicative validation of ideas and hypotheses about possible improvements of formulations that emerge during the conversation. Thereby, interviewers summarise and rephrase statements of the participants, and in this way put their re-contextualised interpretations of what has been said up for discussion.

Experienced interviewers pursue the QPI’s goal of intersubjective understanding by deliberately combining these communication strategies. The flow of the QPI dialogue is then characterised by shifting back and forth between the two reference systems of the interviewer (i.e. research) and the interview partner (i.e. everyday world). It is the task of the interviewer to make sure that this shift is performed by both parties. Experience in previous studies shows that interview partners appreciate these dialogical elements (Witzel and Reiter 2012). They feel taken seriously by the active negotiation, appropriation, and critical consideration of their contributions as research participants. With the appropriate briefing interview partners will not be irritated even by confrontations (e.g. with contradictions) that go beyond mere clarification or when interviewers doubt, question or even reject their suggested amendments. The purpose of confrontations and critical discussions of contradictory aspects is solely the joint identification of the appropriate, comprehensible and, in this sense, valid wording of questions and answers. And that is a step that can have features of a struggle beyond negotiation. Yet once this purpose is explicated and its necessity ratified by the interview partners, they should be able to understand that opposition and even conflict over interpretations and suggestions are productive and benefit clarification. Interviewers need to be trained and equipped with the required qualifications and communication skills (e.g., Roulston et al. 2003; Kvale 2007; Mann 2016). They need to be prepared to deal with alternative perspectives, insinuations of meaning and provocations in a way that a confrontational situation, despite its asymmetrical orientation, does not resemble an interrogation. This includes the appropriate briefing of the participants, the creation and maintenance of a trusting atmosphere, and the careful and flexible handling of the dynamics of difficult communication situations.

Practical aspects of the QPI

The following chapter is dedicated to practical examples of conducting QPIs. They illustrate how the interplay of the programmatic aspects of QPI methodology (c.f. chapter 2) translates into practice. In this paper our focus is on the three key steps of (1) briefing the interview partner before or at the beginning of the conversation, (2) the use of communication strategies, and (3) the final step of debriefing and concluding the research encounter. The commonalities and differences between the QPI and the CI mentioned in the introduction become apparent here.

The following examples were collected in the frame of a university seminar at the Ludwig-Maximilians-University of Munich/Germany in the winter semester 2019/2020. All excerpts are taken from the same interview and were translated into English by the authors. The aim of the seminar was to discuss different techniques of pretesting in the context of questionnaire development. The course introduced students to the procedure and to different communication techniques. One of their tasks was to conduct QPIs to test an already developed paper and pencil drop-off questionnaire,² which was part of the German sub-study in Wave 8 of the Survey of Health, Ageing and Retirement in Europe (SHARE, http://share-eric.eu) investigating the problem of income non-response. The students had detailed information about the background and development of the questionnaire and the task was to test it in its entirety. The questionnaire consists of seven printed pages (incl. instructions on the first two pages) and 31 items with separate instructions. A block of nine items, which were irrelevant in terms of contents for this step, was excluded; thus, altogether 22 items had to be tested.

In general, the length of QPIs varies because they are always unique encounters resulting from the confrontation of two individuals and their priorities and perspectives. There is no default requirement to discuss all aspects of the questionnaire in equal detail, and the process may be faster in case the wording or contents of some items do not require clarification. Nevertheless, researchers can obviously also decide in advance, that certain items need to be tested more thoroughly than others.

Briefing and opening the conversation

Like any conversation, QPIs are ideally embedded within a pleasant social situation based on trust. However, as a research encounter they are different from everyday conversations in many ways (due to the required recording alone). Some of the special features of the QPI need to be introduced accordingly in the frame of a separate briefing before the interview, and the opening of the conversation follows certain conventions (incl. the compliance with interview norms such as a proper welcome and expressing thanks for participation, etc.).

Researchers need to explain what the interview is about and how it will proceed. Some of the information included in the cover letter for the recruitment are repeated orally, like project description (background, purpose and utilisation of the interview), and research-ethical aspects (e.g. voluntary participation, data protection, declaration of consent). In addition, some conversational instructions should be included (e.g. the possibility of interrupting the interview etc.) and open questions clarified. Furthermore, some basics regarding situation, duration, participation, recording etc. need to be addressed. All these preparatory steps set the stage for the research conversation. In the case of the QPI they also clarify the goal of the interview and establish the interview partners as co-experts for achieving this goal and to underline their key role in improving draft survey questions and exploring and struggling for alternative formulations. This process can vary in terms of length as it depends on both participants. In the case of our example, it took about four minutes.

Goal of the QPI

First of all, it is important to familiarise the interview partner with the peculiarities of this encounter. As described above, the QPI is a discursive exchange based on a working relationship at eye level. It follows rules of engagement that are different from those suggested by the question–answer scheme that may usually be associated with a (survey) ‘interview’. The common goal of the pretest situation is the improvement of a draft survey question or similar stimuli. We need to emphasise that the QPI is not about collecting and analysing answers to the substantive survey questions that could possibly be provided to certain questions. To put it differently: we are not interested in what answer participants give to the survey question, but rather why they answer the way they do. Even if interviewees do actually provide answers to the questions, they are not analysed in terms of substance. The purpose of the exchange is the mutual clarification of the meaning of the contents (e.g. items) and of formal aspects (e.g. design) of the draft survey questionnaire (including cover letter, instructions, formulations, response categories and scales etc.). The following excerpt illustrates how the interviewer (I) introduces purpose and procedure to the interview partner (IP) in the context of briefing³:

I: The whole thing that we are having here now is not a classical interview, that you may know from television, or that you may have done yourself at some point, that is, I ask questions and you answer. Instead, you can imagine this as a conversation between the two of us [IP: Ok.]. Because in a nutshell it's not about you answering these questions from the questionnaire and I sit and nod, but it's really about us looking at this sheet together, step by step, page by page, question by question, and so on... [Interview 1, 00:21-00:51]

Interview partners as co-experts

Another purpose of the briefing is to underline that it is a joint process and that both participants are working together on an unfinished document. This includes asking critical questions on both sides, discussing and arguing, and making suggestions for improvement:

...and I depend on your support, that means, first I would like you to say everything that you notice, [IP: Uhm.] to talk about everything that comes to your mind when you read each of the questions, when you look at the individual pages. And I depend on that, that’s why I say support, uhm, that you are welcome to put your finger on it and say: ‘The term is not clear to me now.’ [IP: Ok.] ... [Interview 1, 00:52-01:20]

The QPI constitutes an unusual situation for both participants. The briefing provides an opportunity for creating a space for close collaboration with a previously unknown person and for talking freely about substantial issues as well as problematic formulations. On the one side, researchers need to put aside their research knowledge and distance themselves from the ambition of having constructed a somehow ‘perfect’ questionnaire. They must dismiss the idea of non-reactivity and need to be aware of the weak spots of the draft questionnaire and disclose them. Interview partners, on the other side, need to realise that they are in a position of temporary research assistance. Their interpretive competence is equally important in this situation, and it is essentially their contribution that will advance the improvement of the questionnaire. This clarification of roles and status are key to a productive QPI and should therefore be done very thoughtfully and depending on the circumstances. These aspects are illustrated in the following excerpt that continues where the previous one ended:

I: ... background is that we are quite aware of the fact that we sometimes have such a ‘researcher speak’, which is perhaps not at all used in everyday life in this way.

IP: Ok, am I everyday speak or am I researcher speak?

I: Uhm, for me it is important that you just tell me how you understand it.

IP: Ok, so how I understand it now.

I: Exactly. And that we just look at it together, maybe even ‘How could we formulate something differently?’ [IP: Uhm.] Because we are currently in the middle of the development phase with this drop-off questionnaire, so we are in the middle of it, we still have the chance to adjust some things, ask questions differently or maybe use different terms [IP: Ok.]... [Interview 1, 01:20-02:04]

General exploration and thinking aloud

The briefing should pay particular attention to the introduction of the communication strategy of thinking aloud that is also used in forms of cognitive interviewing (Bethmann et al. 2019; Buschle et al. 2020).⁴ The QPI exploits the idea of thinking aloud that is used as a strategy of general exploration at the beginning of the review of each item. The discussion of each draft question starts by handing over to the interview partners as co-experts. They are invited to articulate what comes to mind while the interviewer assumes the role of the active listener that develops ideas for follow-up probing. Thinking aloud may not be an obvious thing to do for interview partners. Freely articulating one’s flow of thoughts can be unusual and exhausting and needs some preparation and introduction. For instance, in this example the interviewer adds: “What makes this situation so special is the fact that I am actually encouraging you to express everything that comes to your mind.” Our experiences indicate that the best way of getting used to this form of exchange and of handling the interview partners’ possible uncertainties consists in trying it out together—“This is not as simple as that. [IP: laughs.] And it means that we will still practise it together now. [IP: Uhm, good.]”.

Anticipation of unusual interventions

The briefing should also indicate how the interviewer intends to participate in the conversation, which part he or she will play, and what kind of interventions can be expected—e.g. “I will interrupt you from time to time to follow up on things.” Addressing spontaneous interruptions and inquiries in the frame of briefings is particularly important due to their significance for the goal of clarification. Interruptions create pauses and produce space for reflection; they may even take the form of confrontations and (polite!) controversies when contradictions need explanation. As part of the briefing, objections and confrontations are introduced as forms of constructive irritation and invitations to join the struggle for clarification. They demonstrate that in this unusual encounter called QPI it is socially desirable not to settle for easy compromises but to be ready, if necessary, to contest and dispute each other’s proposals. Both parties are welcome to do that. With the proper briefing, the QPI can be a method of negotiating controversial aspects as long as the conversation remains at the factual level. Productive interruptions and ad-hoc questions also have the positive side-effect of demonstrating that this kind of conversation breaks with the question–answer-scheme of conventional interviews. In this respect QPIs resemble dialogic everyday conversations where the thoughts, contributions, questions, doubts etc. of two participants are more closely intertwined than in conventional research communication.

The following section illustrates the practical steps of general and specific exploration in QPIs as well as the interviewer’s shift between active listening and active understanding.

Communication in QPIs—an example

The following segment is taken from the same interview quoted in the previous section. The interview has a total length of about 50 min. The example refers to item 11 on page 4 of the questionnaire. The item is part of a set of items dealing with the question of how respondents handle their own privacy protection. 30 min and 55 s into the interview, the general exploration of item 11 starts with the interviewee first reading the question out loud and expressing any thoughts that come to his mind.

Reading the item out loud & thinking aloud

This first step of reading out loud is the same for every item.

IP: How concerned are your relatives, friends or your partner that you provide too much personal information (.) to people you don’t know very well? (.) So, now I don't know what it is that relatives, friends or partners should be worried about. (.) Is it simply about my, are they worried that I make my own information available to others that I use smartphones. That’s how I understand that. [I: Yes.] Or, uh, is it about (.) that they (.) are concerned that the information I have about the relatives. (.) So, [I: Ah.] yes. [Interview 1, 30:55–31:43]

After reading out the item the interview partner follows the instruction of thinking aloud and immediately expresses his confusion about its precise meaning. The short pause within indicates that it is related to the term ‘information’. He leans towards one interpretation but offers a second one that is also conceivable. His uninterrupted thoughts conclude with a Coda (“(.) So, yes.”). This is when the interviewer realises that there is a problem in the formulation (“Ah.”). It is her turn to take over and she shifts from active listening to a dialogue of active understanding.

Stepwise clarification and active understanding

In the following interview excerpt, the researcher first responds to the interviewee’s thinking aloud with an explicit affirmative statement (“totally exciting”). While this is unusual in qualitative interviews because it might introduce bias towards social desirability (in terms of contents), it serves here as a strong motivational intervention. This is followed by the disclosure of the intention behind the question (“To explain what we had in mind…”) initiating a stepwise dialogical process of clarifying understanding at eye level by using different communication strategies.

I: That is totally exciting. To explain what we had in mind: that too much of your own, of course personal information from you, so to speak, that you report what you do and that then your relatives, friends or so, that they are worried that you pass on information.

IP: From myself?

I: Exactly. But of course, this is also a point that they might be worried that, uh, [IP: Yes.] by you passing on information…

IP: that something about them will also come out.

I: That also something about them, that also something about them would be found out. OK. That means, when we now say that we are actually interested in your information [IP: Uhm.], that is, that friends are concerned about you [IP: Uhm.], would it be possible for us to say: “How concerned are your relatives, friends or your partner that too much personal information about you, even more so, own personal information (.) or (.) personal information about oneself…”

IP: Yes, about oneself, that would be so with me (..) personal information about oneself.

I: About oneself?

IP: About oneself, exactly.

I: Exactly, so that we specify this once again. That would be one possibility [IP: Yes, totally.], so that you would know what we want? [IP: Totally, totally.] Yes, ok.

IP: And that's what it's about, isn't it?

I: Yes exactly, that would be the point. [Interview 1, 31:44–32:58]

The interviewer’s disclosure of the intended purpose of the questionnaire item is a first substantial contribution that actively initiates the process of clarifying meaning. It provides the interview partner with the opportunity to understand and agree with the interviewer’s proposal (“From myself?”). By paraphrasing and mirroring the interview partner’s second interpretation (“…they might be worried that … by you passing on information…”) the interviewer both acknowledges it and asks him to confirm that she got it right. This kind of response of the researcher inviting validation is a step of active understanding. The interview partner confirms the researcher’s statement and even completes her sentence (“… that something about them will also come out”). In turn, she repeats it in similar words and concludes this first part of negotiating agreement (“OK”).

Negotiating indexicality and joint adjustment of the wording

Once common ground is established, they can move on to the next step of working together on an adjustment of the wording of the item. The interviewer’s suggestion (“personal information about oneself”) is accepted and verbally ratified repeatedly by the interview partner; he can “totally” agree that such a specification would help to reduce the ambiguity of the original wording. This segment illustrates the significance of listening to each other: the full appreciation of the perspective of the interviewer, and at times its redundant confirmation (like in this example), is equally important for building consensus. This capability of the two interlocutors to agree on an interpretation is crucial for the next part of the sequence:

IP: Yes, and then it's just that, it's such a mishmash, then I would just say "rather not like that", four. [i.e. the answer category 4; authors]

I: Mishmash, because?

IP: Yes, maybe my mother is totally worried and, uh, all my friends are just not worried, that’s just how it was now.

I: Ok, so because now we simply packed all [I: Yes.] contacts [I: Yes, all.] into this question [I: Uhm.], so that it is difficult to name exactly [I: Exactly.] who is worried. [I: Yes.] That means, if we wanted to, so if we were interested in what your partner or friends were saying and if we wanted to separate that, we would have to list them separately. [I: Uhm.] Do I understand that correctly?

IP: Totally. Exactly like that.

I: Ok. [Interview 1, 32:58-33:37]

The interview partner addresses another flaw (“it's such a mishmash”) that he notices as he tries to select an appropriate answer category. The problem is that relatives, friends, and partner are all lumped together in the item. Even if one of these three categories would deserve the highest rating of 1 (“maybe my mother is totally worried”) he would still have to mark the second lowest rating of 4 because all the others would not be concerned (“all my friends are just not worried”).

This sequence underlines the interview partner’s commitment to his status as diligent co-expert: of his own accord he addresses another inconsistency by continuing to think aloud. The interviewer shifts from active listening to active understanding by interposing a simple detailing question that explores the contextuality of the term “mishmash” (“Mishmash, because?”). This question triggers an explication, which is then followed by a longer sequence of active understanding through paraphrasing and mirroring accompanied by several affirmations of the interview partner. In this way the indexicality of what he has said is gradually dissolved and replaced by genuine understanding. Again, the interviewer initiates the conclusion of this process by suggesting an adjustment of the item based on a compromise between research interest (“if we were interested in…”) and the now resolved irritation of the interview partner (“Exactly like that.”).

However, the interview partner is not quite finished yet. He once again assumes his responsibility as co-expert and submits his very own alternative formulation of the item.

IP: Yes. Otherwise, there might perhaps be the option to simplify it by saying "environment" or so.

I: Uh, "your personal environment", something like that? [IP: Uhm.] Okay. (...) Thank you very much. [IP: Uhm.] [Interview 1, 33:38–33:51]⁵

The researcher adapts his proposal and seeks one final agreement for her suggested clarification of understanding. At this point, the discussion of this item is finalised because all the suggestions and adjustments of the wording are fully acceptable for both sides. If this were not the case, the debate would continue. In the example it could then, hypothetically, evolve along the suggested notion of “environment” that is, in fact, just as indistinct as a list of potentially important members of this ‘environment’. Instead of concluding the negotiation, the interviewer could have initiated further clarification by confronting the interview partner with this contradiction. An open probe would be appropriate to obtain the interview partner’s notion of ‘environment’. In case he sticks with such a vague term, she could question its usefulness for improving the item. She could finally indicate her inclination to reject the suggestion and would provide arguments underlining her decision with the aim of finding out whether she would be able to convince the interview partner as co-expert. In any case, each proposal made by the interview partner enters the further discussion and questionnaire development among the researchers as a qualified contribution.

Summary

This short interview sequence demonstrates that with all these instructive shifts between the researcher’s and the interviewee’s reference systems (of research and everyday knowledge) intersubjective understanding can be accomplished in a very effective and highly indexical way. It requires careful briefing of the interview partner and an interviewer that is able to handle the idiosyncrasies of an open conversation and channel them towards negotiated agreement. In summary, these examples show different strategies of clarifying meaning and establishing mutual understanding in QPIs:

• (1) Active listening to the interview partner’s thinking aloud is the most subtle form of general exploration with the least risk of introducing reactivity to the conversation. With its foundation in thorough briefing it encourages most authentic expressions and preserves their indexicality. They constitute the basis for active understanding.

• (2) Active understanding through specific exploration is dedicated to the exploration and negotiation of contextual meanings and interpretations of these expressions. It is done through the strategic and repeated use of summaries, by paraphrasing and mirroring (of contradictions, if necessary) and by carefully employing interpretive confrontations that emphasise the QPI’s purpose of clarifying misunderstandings.

• (3) The joint discussion and adjustment of the wording of draft survey questions and items contributes significantly to the active clarification of understanding. Concrete suggestions by the interview partner are indicators for his or her degree of involvement and for the overall progress of the specific exploration of interpretations.

• (4) The identification of the interviewee's point of view provides valuable information about the range of possible interpretations and for determining appropriate alternative formulations. By adjusting sampling and analysis, the characteristic features of particular target groups can be established as well as possible divergences in their understanding of the relations between question, answer, and conceptual background.

The choice and use of certain communication strategies depends on the researcher’s interest in specific aspects of the draft questionnaire or other survey material (e.g., cover letter, instructions, items, answer categories), the various possible goals in the process of intersubjective understanding as well as the kind of course the conversation should take. Thus, researchers need to consider in advance the potential and advantages of certain kinds of communication and probing strategies with regard to active understanding and active listening. It is worthwhile figuring out for which parts of the questionnaire or which types of draft questions these strategies might be most suitable. All this can be documented in an interview guide. However, as the example indicates, interviewers need to remain flexible and should not just work through the interview guide without considering the previous part of the conversation (Hopf 1978). They need to remain open to the perspective of the interview partner for true conversations can be full of surprising turns and opportunities.

Interviewers must be trained carefully in general conversation techniques as well as the particular QPI strategies. Ideally, in addition to their QPI training they have some practical experience as well as comprehensive training in qualitative methods in general and beyond interviewing. The latter can help to understand exactly why it is so important for intersubjective understanding in QPIs to reflect both their own and the interview partner’s possible interpretations: it is the necessary precondition for improving the quality of questionnaires through an exchange and alignment of everyday knowledge and research knowledge. This article does not cover the analysis of QPIs; we need to postpone this to a different occasion. Depending on purpose, time, and resources the same data can be analysed in many different ways (e.g., Flick 2014). This applies also to the QPI, which can, in principle, equally utilise the full potential of available approaches.

Debriefing

The exit sequence is about concluding the social encounter appropriately, by recognising their status as co-experts. In the procedure described so far, the importance of uncovering the interviewees’ point of view and using their interpretive competence as co-experts was repeatedly emphasised. This aspect becomes relevant once again in the frame of debriefing.

In addition to expressing one’s thanks for their time, interest and commitment, the final appreciation of their participation should address the importance of their contribution (e.g. practical perspective, specific expertise and background) as well as the information provided (e.g. substance of their comments and significance for the improvement of the questionnaire). One part of this closing ritual consists in inviting the interview partners to address issues that have remained open so far or that are still missing. Their answers can hold valuable remarks like in the next example. Following the concluding recapitulation of the whole questionnaire, the interview partner emphasises, after a long pause for thought of six seconds, the importance of the briefing for this unusual QPI encounter in one of his final comments:

IP: It was a lot of fun. I found it very interesting, because (……) yes, I actually found it, I found it much more interesting to say everything that comes to my head. [I: laughs] Completely crazy this situation. [Interview 1, 47:35–47:52]

The final sequence illustrates that debriefing can be used for a summary and reflection of the background and purpose of the conversation. It highlights the researcher’s take-away conclusions and emphasises that the contribution of the interviewee will lead to adjustments in the questionnaire:

I: Now I would like to thank you very much for being ready to participate here. As you have noticed in many places, there were a few [IP: laughs] points where I noticed, “OK, here we could differentiate a bit more, there we could put things differently”, [IP: Uhm.] or maybe some things are not so clear yet. I can tell you now from my side, there were some things I was worried about, “Oops, I hope that's clear”, where I noticed, “OK, that's totally clear, you're really good with that.” [IP: Uhm, ok.] That was also very nice. [IP: Cool.] But above all there were some points where I thought we could make some improvements. [IP: Uhm.] That's very valuable for us. [IP: Yes.] Many, many thanks for being willing to do this. [Interview 1, 48:04–48:44].

Finally, debriefing addresses the next steps and informs, for instance, about how comments will be processed and considered, and what will likely need to be adjusted. Interview partners are also informed about how they could participate further and, if interested, who to contact for further information.

Summary and conclusion

The discussion about the degree of openness of the interaction between interviewers and interview partners has existed since at least the 1940s (Beatty 1995); more than 75 years after Lazarsfeld's (1944) first conciliatory intervention it is still ongoing (e.g., West et al. 2018). Both parties involved in this debate provide reasonable arguments that are highly relevant for interview practice. The proponents of a strong standardisation on the one side emphasise the necessity of using an efficient and coherent protocol of data collection to generate adequate data for the analysis of statistical distributions. Advocates of open interviewing on the other side argue that the validity of the answers of participants depends essentially on a shared understanding of questions. They consider the possibility of reflecting on this understanding and clarifying it in the interview interaction a key element in reducing bias and error in surveys (Suchman and Jordan 1990). At the same time, more intensive interaction implies the danger of interviewer influence and thus the distortion of answers. Ongoing comparative research into the techniques of “conversational” and “standardised” interviews (e.g., Schober and Conrad 1997) sheds light on the trade-off between the two main sources of measurement error—i.e. wrong understanding of the question by the respondents and distorted answers due to the influence of the interviewer.

One area, in which the use of open interviewing techniques does not cause much controversy, is questionnaire development. It is possible to increase the validity of answers by preponing the open exchange about the meaning of questions and moving it from the primary survey to the process of questionnaire development. The “cognitive turn” in survey research in the 1980s resulted in a systematisation of open pretesting techniques, especially through the development of methods that are now subsumed under the label of ‘Cognitive Interviews’ (Willis 2005, 2015). These methods were developed to identify and, ideally, remedy problems in the respondents’ understanding already in the process of designing the wording of questions.

This paper discusses and advocates a comprehensive consideration of qualitative-interpretive aspects in open forms of pretesting survey questionnaires. More specifically, we start out from the notion of intersubjective understanding as the conceptual pillar of interpretive interview research. Borrowing from the well-developed methodology and techniques of Problem-centred Interviews (PCI) (Witzel and Reiter 2012; Reiter and Witzel 2019; Bethmann et al. 2019), we propose an approach we call Qualitative Pretest Interviews (QPI), which focuses on a joint production of understanding. The QPI can be described as a method of pretest interviewing that involves interview partners as co-experts in a joint discursive clarification of meaning in draft survey questions and other standardised stimuli.

The QPI shares its practical objectives with the Cognitive Interview (CI) developed by Willis (e.g., Willis 2005, 2015), in aiming at reducing response and measurement errors due to problematic survey questions. While the QPI also takes an open approach to pretest interviewing it is conceptually rooted in qualitative-interpretive methodology, as opposed to the CI’s heritage of cognitive psychology. In a sense, we accept Willis’ (2015) invitation to elaborate the qualitative potential of pretest interviewing by fleshing it out in the QPI’s methodology.

In the course of the paper, we outline the QPI’s background in interpretive interview research and elaborate on the value of the concept of intersubjective understanding for the purpose of evaluating survey questions. Namely, it helps to identify and explicate ambiguous meanings of survey stimuli between participants, interviewers and (ideally) researchers by discussing them in their contexts of origin. Eventually, this dialog should lead to shared understanding of meaning and in turn yield suggestions for improvement of the questionnaire.

Regarding practical interviewing techniques, we borrow from the PCI and its particular qualitative interviewing methodology developed for the joint clarification of meanings related to topics, which, like in the QPI, have been identified in advance. Translating the concept of intersubjective understanding into actionable communication strategies, the QPI promotes strategies of active listening for general exploration and strategies of active understanding for specific exploration. We further differentiate this approach and show its practical application in an example interview discussing items in a paper-and-pencil questionnaire on income non-response developed as an add-on for the Survey of Health, Ageing and Retirement in Europe (SHARE, http://share-eric.eu). We explain the concepts of general and specific exploration for the discursive-dialogical clarification of (implied) meaning, using straight-forward examples like encouragement to think aloud and paraphrasing by the interviewer, as well as more complex sequences of making and discussing suggestions to improve the questionnaire. We also point out the importance of briefing and debriefing in order to create a relaxed and professional atmosphere, as well as a relationship between participant and interviewer that is friendly and appreciative, but also sober enough to debate disagreements about the questionnaire critically. In order for a QPI to work, it is vital that the participant’s role of a co-expert is thoroughly understood and accepted by the participant him- or herself as well as the interviewer.

While rooting open pretesting techniques in qualitative-interpretive methodology, as we have done with the QPI, seems to be a fruitful approach, this paper can only give a first impression of what has to be further developed into a proper framework of the methodology and practice of QPIs in the future. Apart from further refining the interviewing techniques themselves, a second equally significant topic we were not able to cover here, is the discussion of methods to analyse the rich and complex material collected during the QPI. Also, a thorough comparison of priorities, procedures, and results with other pretesting methods like the CI needs to be postponed to another publication.

The task of pretesting standardised instruments of data collection with QPIs is at the intersection of quantitative and qualitative research; and only people or teams that are familiar with both approaches will really be able to accomplish it. One thing that becomes clear when looking at the implementation of QPIs in practical questionnaire pretesting is the substantial demand it poses on the training of interviewers. The techniques of QPIs are rather involved, and interviewers would benefit from comprehensive prior training in qualitative interviewing methods. In contrast to other forms of pretests, QPIs cannot be standardised very well as they rely heavily on the interviewer being able to react flexibly during the interaction with the participant. This leads to the need for a specific set of skills, rather than a simple list of prescriptions, which have to be learned through a substantial amount of practice.

QPIs allow for a very thorough investigation of how and why survey questions might not be understood and answered as intended. In our view, employing the methods described above provides a good way of making use of the potential that lies within the direct social interaction during an open pretest situation. By recognising the participants as co-experts, it is possible to tap into their knowledge and ability to discuss and interpret the meanings of questionnaire items as they understand them themselves or as is brought forward by the interviewer. Depending on the specific survey project, this might be very helpful indeed to mitigate important risks to survey quality.

At the same time, QPIs are very resource intensive, as they require well-trained interviewers and a considerable amount of time and money for conducting and processing the interviews, as well as their analysis. In direct comparison with CIs, we would assume that QPIs would require even more training and experience on the side of the interviewers, especially regarding qualitative interviewing techniques. The same will probably hold true for the analysis of the interview material, as the conversations will presumably be more complex due to the higher degree of dialogic interaction. For the same reasons it is likely harder to ensure high quality for a QPI. One could argue that the QPI allows for more degrees of freedom in interview and analysis, putting the result more at risk if done inaptly. The traditional CI, while still one of the most involved pretesting approaches available, might be—at least to some extent—easier to standardise than the QPI, reducing its margin for error. Or, to put it differently: the QPI puts more responsibility for the end result on the well-trained interviewer, rather than on the process. Whether these costs and quality risks are outweighed by the benefits of a potentially higher quality of the questionnaire needs to be decided for each survey project individually.

Finally, even open forms of pretesting will always have their limitations. Even the most elaborate procedure will not result in question wordings that produce equally valid answers for all different, heterogeneous groups of respondents. Conducting pretests with a variety of different types of respondents seems indicated. Still there will be cases where we are only able to find that some respondents may understand a specific question wording, while others do not. Standardisation across all groups of respondents will not at all be possible in some cases, and even open pretest procedures such as the proposed QPI will not be able to resolve these contradictions. Nevertheless, they can provide a nuanced basis for identifying group-differences in the comprehension of questions and for recognising the fundamental necessity of reflecting the concepts in the survey. Sometimes, QPIs might even be instrumental in determining that the study would benefit from an altogether more open approach to interviewing (e.g., Conversational Interviewing, cf. Schober and Conrad 1997). With regard to the decision whether to spend a considerable amount of time and money on thorough pretesting at all, be that QPIs, CIs, or other, maybe less elaborate strategies, we would like to close the discussion with a reminder put forward by Sudman and Bradburn already in 1982 (p. 283): “It is even more important for researchers with limited resources to pilot-test their questionnaires before spending all their money. If you do not have the resources to pilot-test your questionnaire, don’t do the study.”