Abstract
The collection of demographic data in developing and, increasingly, developed countries often requires the translation of a survey instrument. This article addresses the implications for data and analysis of two of the most common modes of translation. The first, the officially sanctioned—though not empirically verified—method, involves the pre-fieldwork production of a standardized translation of the template questionnaire into all or most languages in which interviews are expected to be conducted. The second, rarely acknowledged in the literature but quite common in the field, occurs where there is a mismatch between the language of the questionnaire available to the interviewer and the language in which the actual interview is conducted. In this case, it is up to the interviewer to translate from the language of the questionnaire to the language of the interview. Using the 1998 Kenya DHS, in which 23% of interviews were translated in this non-standardized manner, we explore the effects of the two translation modes on three indicators of measurement error and on estimated multivariate relations. In general we find that the effects of non-standardized translation on univariate statistics—including higher-order variance structures—are rather moderate. The effects become magnified, however, when multivariate analysis is used. This suggests that the advantages of—and also costs associated with—standardized translation depend on the ultimate purposes of data collection.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
All references in this section to the number of local languages into which questionnaires were translated are adapted from DHS country-specific final reports.
We make no assumption that a standardized translation automatically generates full “stimulus equivalence” across a sample of respondents. On the contrary, some level of role-restricted interviewer error—signaling differences among interviewers—can be identified on even the most standardized approaches (Fowler and Mangione 1990). The question we raise here is therefore about the aggregate level of this type of error, and others, across these two translation modes.
We suspect that the careful matching of language-specific teams to areas in which that language was dominant is the main reason that translators—another “under the radar” deviation from standard survey practice (since it inserts “3rd party effects”)—were relatively uncommon in the KDHS. Specifically, in these KDHS data, translators were used either “sometimes” or “all the time” in only 2.1 percent of interviews. Not surprisingly given this low frequency, controlling for “translator” has no effect on our analytic results. Consequently, we do not refer to it hereon.
See http://kenya.pop.upenn.edu for more on the KDICP (including access to data).
Note that across sampling clusters, women interviewed later (e.g., in month 4 of fieldwork) differ from those interviewed in earlier months in terms of geographic location. But they do not differ in terms of individual-level characteristics (they do within cluster, but that has no bearing on this analysis). This is because DHS field methods emphasize constant movement from one sample cluster to another. In other words, teams do not wait until month 4 in order to find the missing folk in villages visited in months 1, 2, and 3. Rather, they complete each cluster before moving on to the next.
Throughout this section we draw heavily on Snijders and Bosker (1999). All multilevel analyses described in this paper were implemented in Stata 9 using the “xtmixed” command (do-files available from the authors upon request). Two details about model specification are worth noting here. First, all models were estimated using restricted maximum likelihood (REML) over maximum likelihood (ML) since the latter is more sensitive to loss of degrees of freedom when dealing with a small number of groups (see Snijders and Bosker 2003(1999), p. 56). As level-3 specification is set to the district level and there are 33 districts, this is of concern, especially in final models that have a relatively large number of regression parameters. Second, no assumptions were made about the structure of the covariance matrix. Rather, all variances and covariances were distinctly estimated.
Two factors drove this second set of specifications. The first was that it would fit more neatly with the KDHS (multistage) cluster sampling approach. The second was that, given the non-experimental nature of the data, we were concerned that some of the variation in x 2ijk might unwittingly capture differences between clusters. This would occur, for example, if there was greater heterogeneity in ethnicity (with implications for reported attitudes and behavior) across clusters than within them, since differences of this sort would not be adequately controlled for by coefficients like month of fieldwork. We therefore considered the identification of cluster-level variance to represent a more conservative estimation strategy vis-à-vis the identification of translation effects on systematic differences in response values.
We also ran two other complete series of models. In the first we allowed variance associated with non-standardized translation to vary only at level-2 (restricting level-3 variance to one term). In the second, we allowed variance associated with non-standardized translation to vary only at level-3 (restricting level-2 variance to one term). Results of all these models are available upon request. We have omitted them for brevity and because model 5 yields the most comprehensive information.
One possible reason for the clustering on household-related variables is that in the African context in particular, households are considerably less bounded and, therefore, more difficult to define, than in most other contexts. In addition, types of household structure vary from one area to another. It is not difficult to envisage how, in the absence of a standardized questionnaire, these two factors would tend to increase measurement error (Sara Randall, personal communication; see Coast et al. 2007).
The .01 significance can easily be improved to .001 by dropping some of the non-performing interaction terms—results available from the authors—but for comparability we use the full model.
References
Aquilino, W. S. (1994). Interview mode effects in surveys of drug and alcohol use. Public Opinion Quarterly, 58, 210–240.
Axinn, W. G., Fricke, T. E., & Thornton, A. (1991). The microdemographic community-study approach: Improving survey data by integrating the ethnographic method. Sociological Methods & Research, 20, 187–217. doi:10.1177/0049124191020002001.
Babbie, E. (2006). The practice of social research (11th ed.). Belmont, CA: Wadsworth Publishing.
Becker, S., Feyistan, K., & Makinwa-Adebusoye, P. (1995). The effect of sex of interviewers on the quality of data in a Nigerian family planning questionnaire. Studies in Family Planning, 26, 233–240. doi:10.2307/2137848.
Bernard, H. R. (2000). Social research methods: Qualitative and quantitative approaches. Thousand Oaks, CA: Sage Publications.
Chen, A., Liu, Z., & Ennis, C. D. (1997). Universality and uniqueness of teacher educational value orientations: A cross-cultural comparison between USA and China. Journal of Research and Development in Education, 30, 135–143.
Coast, E.E., Randall, S., Leone, T., & Bishop, B. (2007). The commodity chain of the household: From survey design to policy planning. Presented at the 2007 meetings of the Union of African Population Science (UAPS), Arusha, Tanzania, December 10–14.
Converse, J. M., & Presser, S. (1986). Survey questions: Handcrafting the standardized questionnaire. Thousand Oaks, CA: Sage Publications, Sage Series of Quantitative Applications in the Social Sciences, No. 63.
Cordell, D. D., Gregory, J. W., & Piché, V. (1996). Hoe and wage: A social history of a circular migration system in West Africa. Boulder, CO: Westview Press.
Creswell, J. W. (2002). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Thousand Oaks, CA: Sage Publications.
Delavande, A., & Kohler, H.-P.(2007). Subjective expectations in the context of HIV/AIDS in Malawi. University of Pennsylvania, Population Aging Research Center, PARC Working Paper 07-06.
Dijkstra, W. (1987). Interviewing style and respondent behavior: an experimental study of the survey-interview. Sociological Methods & Research, 16, 309–334. doi:10.1177/0049124187016002006.
Fowler, F. J., & Mangione, T. W. (1990). Standardized survey interviewing: minimizing interviewer-related error. Newbury Park, CA: Sage Publications.
Groves, R. M. (1989). Survey errors and survey costs. New York: John Wiley & Sons.
Helleringer, S., & Kohler, H.-P. (2005). Social networks, risk perceptions and changing attitudes towards HIV/AIDS: New evidence from a longitudinal study using fixed-effect estimation. Population Studies, 59(3), 265–282. doi:10.1080/00324720500212230.
Maynard, D. W., & Schaeffer, N. C. (2002). Standardization and its discontents. In D. W. Maynard, H. Houtkoop-Steenstra, N. C. Schaeffer & H. van der Zouwen (Eds.), Standardization and tacit knowledge: Interaction and practice in the survey interview (pp. 3–45). New York: Wiley.
Mensch, B. S., Hewett, P. C., & Erulkar, A. S. (2003). The reporting of sensitive behavior among adolescents: a methodological experiment in Kenya. Demography, 40, 247–268. doi:10.1353/dem.2003.0017.
Mitchell, R. E. (1965). Survey materials collected in the developing countries: Sampling, measurement and interviewing obstacles in international comparisons. International Social Science Journal, 17, 665–685.
Overton, J., & van Dierman, P. (2003). Using quantitative techniques. In R. Scheyvens & D. Storey (Eds.), Development fieldwork: A practical guide (pp. 37–56). Thousand Oaks, CA: Sage Publications.
Plummer, M. L., Wight, D., Ross, D., Balira, R., et al. (2004). Asking semi-literate adolescents about sexual behaviour: the validity of assisted self-completion questionnaire (ASCQ) data in rural Tanzania. Tropical Medicine & International Health, 9, 737–754. doi:10.1111/j.1365-3156.2004.01254.x.
Sana, M., & Weinreb, A. A. (2008). Insiders, outsiders, and the editing of inconsistent survey data. Sociological Methods & Research, 36, 515–541. doi:10.1177/0049124107313857.
Sapsford, R., & Jupp, V. (2006). Data collection and analysis (2nd ed.). Thousand Oaks, CA: Sage Publications.
Schaeffer, N. C., & Presser, S. (2003). The science of asking questions. Annual Review of Sociology, 29, 65–88. doi:10.1146/annurev.soc.29.110702.110112.
Schober, M., & Conrad, F. G. (1997). Does conversational interviewing reduce survey measurement error? Public Opinion Quarterly, 61, 576–602. doi:10.1086/297818.
Snijders, T.A.B., & Bosker, R.J. (2003(1999)). Multilevel analysis: an introduction to basic and advanced multilevel modeling. Sage Publications.
Suchman, L., & Jordan, B. (1990). Interactional troubles in face-to-face survey interviews. Journal of the American Statistical Association, 85, 232–253. doi:10.2307/2289550.
Ware, H. (1977). Language problems in demographic field work in Africa: The case of the Cameroon Fertility Survey. London: International Statistical Institute & World Fertility Survey, Scientific Report No. 2.
Weinreb, A. A. (2006). The limitations of stranger-interviewers in rural Kenya. American Sociological Review, 71(6), 1014–1039.
Zeller, R. A., & Carmines, E. G. (1980). Measurement in the social sciences: The link between theory and data. Cambridge: Cambridge University Press.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Weinreb, A.A., Sana, M. The Effects of Questionnaire Translation on Demographic Data and Analysis. Popul Res Policy Rev 28, 429–454 (2009). https://doi.org/10.1007/s11113-008-9106-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11113-008-9106-5