The following chapter is dedicated to practical examples of conducting QPIs. They illustrate how the interplay of the programmatic aspects of QPI methodology (c.f. chapter 2) translates into practice. In this paper our focus is on the three key steps of (1) briefing the interview partner before or at the beginning of the conversation, (2) the use of communication strategies, and (3) the final step of debriefing and concluding the research encounter. The commonalities and differences between the QPI and the CI mentioned in the introduction become apparent here.
The following examples were collected in the frame of a university seminar at the Ludwig-Maximilians-University of Munich/Germany in the winter semester 2019/2020. All excerpts are taken from the same interview and were translated into English by the authors. The aim of the seminar was to discuss different techniques of pretesting in the context of questionnaire development. The course introduced students to the procedure and to different communication techniques. One of their tasks was to conduct QPIs to test an already developed paper and pencil drop-off questionnaire,Footnote 2 which was part of the German sub-study in Wave 8 of the Survey of Health, Ageing and Retirement in Europe (SHARE, http://share-eric.eu) investigating the problem of income non-response. The students had detailed information about the background and development of the questionnaire and the task was to test it in its entirety. The questionnaire consists of seven printed pages (incl. instructions on the first two pages) and 31 items with separate instructions. A block of nine items, which were irrelevant in terms of contents for this step, was excluded; thus, altogether 22 items had to be tested.
In general, the length of QPIs varies because they are always unique encounters resulting from the confrontation of two individuals and their priorities and perspectives. There is no default requirement to discuss all aspects of the questionnaire in equal detail, and the process may be faster in case the wording or contents of some items do not require clarification. Nevertheless, researchers can obviously also decide in advance, that certain items need to be tested more thoroughly than others.
Briefing and opening the conversation
Like any conversation, QPIs are ideally embedded within a pleasant social situation based on trust. However, as a research encounter they are different from everyday conversations in many ways (due to the required recording alone). Some of the special features of the QPI need to be introduced accordingly in the frame of a separate briefing before the interview, and the opening of the conversation follows certain conventions (incl. the compliance with interview norms such as a proper welcome and expressing thanks for participation, etc.).
Researchers need to explain what the interview is about and how it will proceed. Some of the information included in the cover letter for the recruitment are repeated orally, like project description (background, purpose and utilisation of the interview), and research-ethical aspects (e.g. voluntary participation, data protection, declaration of consent). In addition, some conversational instructions should be included (e.g. the possibility of interrupting the interview etc.) and open questions clarified. Furthermore, some basics regarding situation, duration, participation, recording etc. need to be addressed. All these preparatory steps set the stage for the research conversation. In the case of the QPI they also clarify the goal of the interview and establish the interview partners as co-experts for achieving this goal and to underline their key role in improving draft survey questions and exploring and struggling for alternative formulations. This process can vary in terms of length as it depends on both participants. In the case of our example, it took about four minutes.
Goal of the QPI
First of all, it is important to familiarise the interview partner with the peculiarities of this encounter. As described above, the QPI is a discursive exchange based on a working relationship at eye level. It follows rules of engagement that are different from those suggested by the question–answer scheme that may usually be associated with a (survey) ‘interview’. The common goal of the pretest situation is the improvement of a draft survey question or similar stimuli. We need to emphasise that the QPI is not about collecting and analysing answers to the substantive survey questions that could possibly be provided to certain questions. To put it differently: we are not interested in what answer participants give to the survey question, but rather why they answer the way they do. Even if interviewees do actually provide answers to the questions, they are not analysed in terms of substance. The purpose of the exchange is the mutual clarification of the meaning of the contents (e.g. items) and of formal aspects (e.g. design) of the draft survey questionnaire (including cover letter, instructions, formulations, response categories and scales etc.). The following excerpt illustrates how the interviewer (I) introduces purpose and procedure to the interview partner (IP) in the context of briefingFootnote 3:
I: The whole thing that we are having here now is not a classical interview, that you may know from television, or that you may have done yourself at some point, that is, I ask questions and you answer. Instead, you can imagine this as a conversation between the two of us [IP: Ok.]. Because in a nutshell it's not about you answering these questions from the questionnaire and I sit and nod, but it's really about us looking at this sheet together, step by step, page by page, question by question, and so on... [Interview 1, 00:21-00:51]
Interview partners as co-experts
Another purpose of the briefing is to underline that it is a joint process and that both participants are working together on an unfinished document. This includes asking critical questions on both sides, discussing and arguing, and making suggestions for improvement:
...and I depend on your support, that means, first I would like you to say everything that you notice, [IP: Uhm.] to talk about everything that comes to your mind when you read each of the questions, when you look at the individual pages. And I depend on that, that’s why I say support, uhm, that you are welcome to put your finger on it and say: ‘The term is not clear to me now.’ [IP: Ok.] ... [Interview 1, 00:52-01:20]
The QPI constitutes an unusual situation for both participants. The briefing provides an opportunity for creating a space for close collaboration with a previously unknown person and for talking freely about substantial issues as well as problematic formulations. On the one side, researchers need to put aside their research knowledge and distance themselves from the ambition of having constructed a somehow ‘perfect’ questionnaire. They must dismiss the idea of non-reactivity and need to be aware of the weak spots of the draft questionnaire and disclose them. Interview partners, on the other side, need to realise that they are in a position of temporary research assistance. Their interpretive competence is equally important in this situation, and it is essentially their contribution that will advance the improvement of the questionnaire. This clarification of roles and status are key to a productive QPI and should therefore be done very thoughtfully and depending on the circumstances. These aspects are illustrated in the following excerpt that continues where the previous one ended:
I: ... background is that we are quite aware of the fact that we sometimes have such a ‘researcher speak’, which is perhaps not at all used in everyday life in this way.
IP: Ok, am I everyday speak or am I researcher speak?
I: Uhm, for me it is important that you just tell me how you understand it.
IP: Ok, so how I understand it now.
I: Exactly. And that we just look at it together, maybe even ‘How could we formulate something differently?’ [IP: Uhm.] Because we are currently in the middle of the development phase with this drop-off questionnaire, so we are in the middle of it, we still have the chance to adjust some things, ask questions differently or maybe use different terms [IP: Ok.]... [Interview 1, 01:20-02:04]
General exploration and thinking aloud
The briefing should pay particular attention to the introduction of the communication strategy of thinking aloud that is also used in forms of cognitive interviewing (Bethmann et al. 2019; Buschle et al. 2020).Footnote 4 The QPI exploits the idea of thinking aloud that is used as a strategy of general exploration at the beginning of the review of each item. The discussion of each draft question starts by handing over to the interview partners as co-experts. They are invited to articulate what comes to mind while the interviewer assumes the role of the active listener that develops ideas for follow-up probing. Thinking aloud may not be an obvious thing to do for interview partners. Freely articulating one’s flow of thoughts can be unusual and exhausting and needs some preparation and introduction. For instance, in this example the interviewer adds: “What makes this situation so special is the fact that I am actually encouraging you to express everything that comes to your mind.” Our experiences indicate that the best way of getting used to this form of exchange and of handling the interview partners’ possible uncertainties consists in trying it out together—“This is not as simple as that. [IP: laughs.] And it means that we will still practise it together now. [IP: Uhm, good.]”.
Anticipation of unusual interventions
The briefing should also indicate how the interviewer intends to participate in the conversation, which part he or she will play, and what kind of interventions can be expected—e.g. “I will interrupt you from time to time to follow up on things.” Addressing spontaneous interruptions and inquiries in the frame of briefings is particularly important due to their significance for the goal of clarification. Interruptions create pauses and produce space for reflection; they may even take the form of confrontations and (polite!) controversies when contradictions need explanation. As part of the briefing, objections and confrontations are introduced as forms of constructive irritation and invitations to join the struggle for clarification. They demonstrate that in this unusual encounter called QPI it is socially desirable not to settle for easy compromises but to be ready, if necessary, to contest and dispute each other’s proposals. Both parties are welcome to do that. With the proper briefing, the QPI can be a method of negotiating controversial aspects as long as the conversation remains at the factual level. Productive interruptions and ad-hoc questions also have the positive side-effect of demonstrating that this kind of conversation breaks with the question–answer-scheme of conventional interviews. In this respect QPIs resemble dialogic everyday conversations where the thoughts, contributions, questions, doubts etc. of two participants are more closely intertwined than in conventional research communication.
The following section illustrates the practical steps of general and specific exploration in QPIs as well as the interviewer’s shift between active listening and active understanding.
Communication in QPIs—an example
The following segment is taken from the same interview quoted in the previous section. The interview has a total length of about 50 min. The example refers to item 11 on page 4 of the questionnaire. The item is part of a set of items dealing with the question of how respondents handle their own privacy protection. 30 min and 55 s into the interview, the general exploration of item 11 starts with the interviewee first reading the question out loud and expressing any thoughts that come to his mind.
Reading the item out loud & thinking aloud
This first step of reading out loud is the same for every item.
IP: How concerned are your relatives, friends or your partner that you provide too much personal information (.) to people you don’t know very well? (.) So, now I don't know what it is that relatives, friends or partners should be worried about. (.) Is it simply about my, are they worried that I make my own information available to others that I use smartphones. That’s how I understand that. [I: Yes.] Or, uh, is it about (.) that they (.) are concerned that the information I have about the relatives. (.) So, [I: Ah.] yes. [Interview 1, 30:55–31:43]
After reading out the item the interview partner follows the instruction of thinking aloud and immediately expresses his confusion about its precise meaning. The short pause within indicates that it is related to the term ‘information’. He leans towards one interpretation but offers a second one that is also conceivable. His uninterrupted thoughts conclude with a Coda (“(.) So, yes.”). This is when the interviewer realises that there is a problem in the formulation (“Ah.”). It is her turn to take over and she shifts from active listening to a dialogue of active understanding.
Stepwise clarification and active understanding
In the following interview excerpt, the researcher first responds to the interviewee’s thinking aloud with an explicit affirmative statement (“totally exciting”). While this is unusual in qualitative interviews because it might introduce bias towards social desirability (in terms of contents), it serves here as a strong motivational intervention. This is followed by the disclosure of the intention behind the question (“To explain what we had in mind…”) initiating a stepwise dialogical process of clarifying understanding at eye level by using different communication strategies.
I: That is totally exciting. To explain what we had in mind: that too much of your own, of course personal information from you, so to speak, that you report what you do and that then your relatives, friends or so, that they are worried that you pass on information.
IP: From myself?
I: Exactly. But of course, this is also a point that they might be worried that, uh, [IP: Yes.] by you passing on information…
IP: that something about them will also come out.
I: That also something about them, that also something about them would be found out. OK. That means, when we now say that we are actually interested in your information [IP: Uhm.], that is, that friends are concerned about you [IP: Uhm.], would it be possible for us to say: “How concerned are your relatives, friends or your partner that too much personal information about you, even more so, own personal information (.) or (.) personal information about oneself…”
IP: Yes, about oneself, that would be so with me (..) personal information about oneself.
I: About oneself?
IP: About oneself, exactly.
I: Exactly, so that we specify this once again. That would be one possibility [IP: Yes, totally.], so that you would know what we want? [IP: Totally, totally.] Yes, ok.
IP: And that's what it's about, isn't it?
I: Yes exactly, that would be the point. [Interview 1, 31:44–32:58]
The interviewer’s disclosure of the intended purpose of the questionnaire item is a first substantial contribution that actively initiates the process of clarifying meaning. It provides the interview partner with the opportunity to understand and agree with the interviewer’s proposal (“From myself?”). By paraphrasing and mirroring the interview partner’s second interpretation (“…they might be worried that … by you passing on information…”) the interviewer both acknowledges it and asks him to confirm that she got it right. This kind of response of the researcher inviting validation is a step of active understanding. The interview partner confirms the researcher’s statement and even completes her sentence (“… that something about them will also come out”). In turn, she repeats it in similar words and concludes this first part of negotiating agreement (“OK”).
Negotiating indexicality and joint adjustment of the wording
Once common ground is established, they can move on to the next step of working together on an adjustment of the wording of the item. The interviewer’s suggestion (“personal information about oneself”) is accepted and verbally ratified repeatedly by the interview partner; he can “totally” agree that such a specification would help to reduce the ambiguity of the original wording. This segment illustrates the significance of listening to each other: the full appreciation of the perspective of the interviewer, and at times its redundant confirmation (like in this example), is equally important for building consensus. This capability of the two interlocutors to agree on an interpretation is crucial for the next part of the sequence:
IP: Yes, and then it's just that, it's such a mishmash, then I would just say "rather not like that", four. [i.e. the answer category 4; authors]
I: Mishmash, because?
IP: Yes, maybe my mother is totally worried and, uh, all my friends are just not worried, that’s just how it was now.
I: Ok, so because now we simply packed all [I: Yes.] contacts [I: Yes, all.] into this question [I: Uhm.], so that it is difficult to name exactly [I: Exactly.] who is worried. [I: Yes.] That means, if we wanted to, so if we were interested in what your partner or friends were saying and if we wanted to separate that, we would have to list them separately. [I: Uhm.] Do I understand that correctly?
IP: Totally. Exactly like that.
I: Ok. [Interview 1, 32:58-33:37]
The interview partner addresses another flaw (“it's such a mishmash”) that he notices as he tries to select an appropriate answer category. The problem is that relatives, friends, and partner are all lumped together in the item. Even if one of these three categories would deserve the highest rating of 1 (“maybe my mother is totally worried”) he would still have to mark the second lowest rating of 4 because all the others would not be concerned (“all my friends are just not worried”).
This sequence underlines the interview partner’s commitment to his status as diligent co-expert: of his own accord he addresses another inconsistency by continuing to think aloud. The interviewer shifts from active listening to active understanding by interposing a simple detailing question that explores the contextuality of the term “mishmash” (“Mishmash, because?”). This question triggers an explication, which is then followed by a longer sequence of active understanding through paraphrasing and mirroring accompanied by several affirmations of the interview partner. In this way the indexicality of what he has said is gradually dissolved and replaced by genuine understanding. Again, the interviewer initiates the conclusion of this process by suggesting an adjustment of the item based on a compromise between research interest (“if we were interested in…”) and the now resolved irritation of the interview partner (“Exactly like that.”).
However, the interview partner is not quite finished yet. He once again assumes his responsibility as co-expert and submits his very own alternative formulation of the item.
IP: Yes. Otherwise, there might perhaps be the option to simplify it by saying "environment" or so.
I: Uh, "your personal environment", something like that? [IP: Uhm.] Okay. (...) Thank you very much. [IP: Uhm.] [Interview 1, 33:38–33:51]Footnote 5
The researcher adapts his proposal and seeks one final agreement for her suggested clarification of understanding. At this point, the discussion of this item is finalised because all the suggestions and adjustments of the wording are fully acceptable for both sides. If this were not the case, the debate would continue. In the example it could then, hypothetically, evolve along the suggested notion of “environment” that is, in fact, just as indistinct as a list of potentially important members of this ‘environment’. Instead of concluding the negotiation, the interviewer could have initiated further clarification by confronting the interview partner with this contradiction. An open probe would be appropriate to obtain the interview partner’s notion of ‘environment’. In case he sticks with such a vague term, she could question its usefulness for improving the item. She could finally indicate her inclination to reject the suggestion and would provide arguments underlining her decision with the aim of finding out whether she would be able to convince the interview partner as co-expert. In any case, each proposal made by the interview partner enters the further discussion and questionnaire development among the researchers as a qualified contribution.
This short interview sequence demonstrates that with all these instructive shifts between the researcher’s and the interviewee’s reference systems (of research and everyday knowledge) intersubjective understanding can be accomplished in a very effective and highly indexical way. It requires careful briefing of the interview partner and an interviewer that is able to handle the idiosyncrasies of an open conversation and channel them towards negotiated agreement. In summary, these examples show different strategies of clarifying meaning and establishing mutual understanding in QPIs:
Active listening to the interview partner’s thinking aloud is the most subtle form of general exploration with the least risk of introducing reactivity to the conversation. With its foundation in thorough briefing it encourages most authentic expressions and preserves their indexicality. They constitute the basis for active understanding.
Active understanding through specific exploration is dedicated to the exploration and negotiation of contextual meanings and interpretations of these expressions. It is done through the strategic and repeated use of summaries, by paraphrasing and mirroring (of contradictions, if necessary) and by carefully employing interpretive confrontations that emphasise the QPI’s purpose of clarifying misunderstandings.
The joint discussion and adjustment of the wording of draft survey questions and items contributes significantly to the active clarification of understanding. Concrete suggestions by the interview partner are indicators for his or her degree of involvement and for the overall progress of the specific exploration of interpretations.
The identification of the interviewee's point of view provides valuable information about the range of possible interpretations and for determining appropriate alternative formulations. By adjusting sampling and analysis, the characteristic features of particular target groups can be established as well as possible divergences in their understanding of the relations between question, answer, and conceptual background.
The choice and use of certain communication strategies depends on the researcher’s interest in specific aspects of the draft questionnaire or other survey material (e.g., cover letter, instructions, items, answer categories), the various possible goals in the process of intersubjective understanding as well as the kind of course the conversation should take. Thus, researchers need to consider in advance the potential and advantages of certain kinds of communication and probing strategies with regard to active understanding and active listening. It is worthwhile figuring out for which parts of the questionnaire or which types of draft questions these strategies might be most suitable. All this can be documented in an interview guide. However, as the example indicates, interviewers need to remain flexible and should not just work through the interview guide without considering the previous part of the conversation (Hopf 1978). They need to remain open to the perspective of the interview partner for true conversations can be full of surprising turns and opportunities.
Interviewers must be trained carefully in general conversation techniques as well as the particular QPI strategies. Ideally, in addition to their QPI training they have some practical experience as well as comprehensive training in qualitative methods in general and beyond interviewing. The latter can help to understand exactly why it is so important for intersubjective understanding in QPIs to reflect both their own and the interview partner’s possible interpretations: it is the necessary precondition for improving the quality of questionnaires through an exchange and alignment of everyday knowledge and research knowledge. This article does not cover the analysis of QPIs; we need to postpone this to a different occasion. Depending on purpose, time, and resources the same data can be analysed in many different ways (e.g., Flick 2014). This applies also to the QPI, which can, in principle, equally utilise the full potential of available approaches.
The exit sequence is about concluding the social encounter appropriately, by recognising their status as co-experts. In the procedure described so far, the importance of uncovering the interviewees’ point of view and using their interpretive competence as co-experts was repeatedly emphasised. This aspect becomes relevant once again in the frame of debriefing.
In addition to expressing one’s thanks for their time, interest and commitment, the final appreciation of their participation should address the importance of their contribution (e.g. practical perspective, specific expertise and background) as well as the information provided (e.g. substance of their comments and significance for the improvement of the questionnaire). One part of this closing ritual consists in inviting the interview partners to address issues that have remained open so far or that are still missing. Their answers can hold valuable remarks like in the next example. Following the concluding recapitulation of the whole questionnaire, the interview partner emphasises, after a long pause for thought of six seconds, the importance of the briefing for this unusual QPI encounter in one of his final comments:
IP: It was a lot of fun. I found it very interesting, because (……) yes, I actually found it, I found it much more interesting to say everything that comes to my head. [I: laughs] Completely crazy this situation. [Interview 1, 47:35–47:52]
The final sequence illustrates that debriefing can be used for a summary and reflection of the background and purpose of the conversation. It highlights the researcher’s take-away conclusions and emphasises that the contribution of the interviewee will lead to adjustments in the questionnaire:
I: Now I would like to thank you very much for being ready to participate here. As you have noticed in many places, there were a few [IP: laughs] points where I noticed, “OK, here we could differentiate a bit more, there we could put things differently”, [IP: Uhm.] or maybe some things are not so clear yet. I can tell you now from my side, there were some things I was worried about, “Oops, I hope that's clear”, where I noticed, “OK, that's totally clear, you're really good with that.” [IP: Uhm, ok.] That was also very nice. [IP: Cool.] But above all there were some points where I thought we could make some improvements. [IP: Uhm.] That's very valuable for us. [IP: Yes.] Many, many thanks for being willing to do this. [Interview 1, 48:04–48:44].
Finally, debriefing addresses the next steps and informs, for instance, about how comments will be processed and considered, and what will likely need to be adjusted. Interview partners are also informed about how they could participate further and, if interested, who to contact for further information.