Item comparability in cross-national surveys: results from asking probing questions in cross-national web surveys about attitudes towards civil disobedience
- 1.1k Downloads
This article focuses on assessing item comparability in cross-national surveys by asking probing questions in Web surveys. The “civil disobedience” item from the “rights in a democracy” scale of the International Social Survey Program (ISSP) serves as a substantive case study. Identical Web surveys were fielded in Canada (English-speaking), Denmark, Germany, Hungary, Spain, and the U.S. A category-selection and a comprehension probe, respectively, were incorporated into the Web surveys after the closed-ended “civil disobedience” item. Responses to the category selection-probe reveal that notably in Germany, Hungary, and Spain the detachment of politicians from the people and their lack of responsiveness is deplored. Responses to the comprehension probe show that mainly in the U.S. and Canada violence and/or destruction are associated with civil disobedience. These results suggest reasons for the peculiar statistical results found for the “civil disobedience” item in the ISSP study. On the whole, Web probing proves to be a valuable tool for identifying interpretation differences and potential bias in cross-national survey research.
KeywordsProbing Web surveys Mixed method Comparability Cross-national survey research
This work was supported by the German Research Foundation (DFG) as part of the PPSM Priority Programme on Survey Methodology (SPP 1292) [project # 574187].
This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.
- Behr, D., Kaczmirek, L., Bandilla, W., Braun, M.: Asking probing questions in web surveys: which factors have an impact on the quality of responses? Soc. Sci. Comput. Rev. (2012). doi: 10.1177/0894439311435305
- Behr, D., Braun, M., Kaczmirek, L., Bandilla, W.: Testing the validity of gender ideology items by implementing probing questions in web surveys. Field Methods (Forthcoming)Google Scholar
- Bolzendahl, C., Coffe, H.: A gender gap in citizenship norms? The importance of political, civil and social rights and responsibilities. Br. J. Sociol. 60(4), 763–791. http://escholarship.org/uc/item/71x598x2 (2008). Accessed 24 April 2012
- Childs J., Goerman P.: Bilingual questionnaire evaluation and development through mixed pretesting methods: the case of the U.S. census nonresponse followup instrument. J. Off. Stat. 26, 535–557 (2010)Google Scholar
- DeMaio T.J., Rothgeb J.M.: Cognitive interviewing techniques: in the lab and in the field. In: Schwarz, N., Sudman, S. (eds) Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research, pp. 177–195. Jossey-Bass, San Francisco (1996)Google Scholar
- Donovan, T., Denemark, D., Bowler, S.: Trust in government. The United States in comparative perspective. http://faculty.wwu.edu/donovat/trust.pdf (2008). Accessed 24 April 2012
- Fitzgerald R., Widdop S., Gray M., Collins D.: Identifying sources of error in cross-national questionnaires: application of an error source typology to cognitive interview data. J. Off. Stat. 27, 569–599 (2011)Google Scholar
- Harkness J.A., Edwards B., Hansen S.E., Miller D.R., Villar A.: Designing questionnaires for multipopulation research. In: Harkness, J.A., Braun, M., Edwards, B., Johnson, T.P., Lyberg, L., Mohler, P.P., Pennell, B.-E., Smith, T.W. (eds) Survey Methods in Multinational, Multiregional, and Multicultural Contexts., pp. 33–57. Wiley, Hoboken (2010)CrossRefGoogle Scholar
- ISSP Research Group: International social survey programme 2004: citizenship (ISSP 2004). Source questionnaire. GESIS Data Archive, Cologne (2004a)Google Scholar
- ISSP Research Group: International social survey programme 2004: citizenship (ISSP 2004). Data file Vers. 1.2.0. GESIS Data Archive, Cologne. doi: 10.4232/1.10078 (2004b)
- Lee, J.: Conducting cognitive interviews in cross-national settings. Assessment (2012). doi: 10.1177/1073191112436671
- Morren, M., Gelissen, J., Vermunt, J. K.: Exploring the response process of culturally differing survey respondents with a response style: a sequential mixed-methods study. Field Methods (Forthcoming)Google Scholar
- Prüfer, P., Rexroth, M.: Kognitive Interviews [cognitive interviews]. ZUMA How-to-Reihe 15. http://www.gesis.org/fileadmin/upload/forschung/publikationen/gesis_reihen/howto/How_to15PP_MR.pdf?download=true (2005). Accessed 24 April 2012
- Schwarz N.: Cognition and Communication: Judgmental Biases, Research Methods, and the Logic of Conversation. Erlbaum, Mahwah (1996)Google Scholar
- Thrasher J.F., Quah A.C.K., Dominick G., Borland R., Driezen P., Awang R., Omar M., Hosking W., Sirirassamee B., Boado M.: Using cognitive interviewing and behavioral coding to determine measurement equivalence across linguistic and cultural groups: an example from the international tobacco control policy evaluation project. Field Methods 23, 439–460 (2011)CrossRefGoogle Scholar
- Willis G.B.: Cognitive Interviewing: A Tool for Improving Questionnaire Design. Sage, Thousand Oaks (2005)Google Scholar