Abstract
To address declining response rates and rising data-collection costs, survey methodologists have devised new techniques for using process data (“paradata”) to address nonresponse by altering the survey design dynamically during data collection. We investigate the substantive consequences of responsive survey design—tools that use paradata to improve the representative qualities of surveys and control costs. By improving representation of reluctant respondents, responsive design can change our understanding of the topic being studied. Using the National Survey of Family Growth Cycle 6, we illustrate how responsive survey design can shape both demographic estimates and models of demographic behaviors based on survey data. By juxtaposing measures from regular and responsive data collection phases, we document how special efforts to interview reluctant respondents may affect demographic estimates. Results demonstrate the potential of responsive survey design to change the quality of demographic research based on survey data.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Many earlier studies also used CAPI (e.g., the NSFG Cycle 5 in 1995), and some earlier studies used analyses of paradata and responsive design on a large scale.
In fact, in many ways, responsive design is similar to common practices in telephone survey call centers of using information about the respondent’s circumstances to maximize the efficiency of phone calls (Groves et al. 1988). It is also similar to the common practice of having experienced study managers devise intelligent strategies for the focus of interviewer effort in large- and small-scale face-to-face surveys. The key advantages of responsive design are the ability to simultaneously consider many different factors using statistical models and concrete documentation forced by explicit modeling that can be replicated. These tools provide a powerful means for systematic intervention.
For example, one analytic process involves estimating key statistics (e.g., the mean number of live births) and plotting the cumulative estimates of these statistics by the number of call attempts. During the course of the data-collection period, these plots are monitored to see whether estimates of key statistics begin to show stability after a certain number of call attempts. At this point, additional call attempts would produce values for these statistics yielding the same substantive conclusions. This information is used to determine the maximum number of call attempts to be made on future cases. For additional examples of background analytic processes using paradata, see Groves et al. (2005).
Of course, some differences that appear statistically significant may be the result of random chance rather than systematic processes. Further advances in hypotheses regarding the mechanisms producing specific differences will be needed to adjudicate this possibility.
Note that more sophisticated approaches to estimation of the relationship between religious service attendance and sexual partnerships use longitudinal data, not cross-sectional data as in the NSFG. This is because there is known reciprocal causation between religious service attendance and sexual partnerships, in which religious service attendance affects subsequent sexual partnering behaviors but sexual partnering behaviors also affect subsequent religious service attendance (Thornton et al. 1992). We do not use such an approach here because our aim is to evaluate how responsive survey design may affect model estimates in typical uses of NSFG data, and NSFG data are used by some analysts for this purpose.
In fact, when the size of the responsive phase sample is small relative to the size of the main phase sample, estimates based on the pooled sample will rarely differ greatly from estimates based on the main phase only. However, as we argue later, responsive design is becoming a more common feature of survey data collection and is likely to shape a higher proportion of cases in the future, so that these differences are likely to become an increasingly common feature of demographic analyses based on survey data.
We also examined models of ever biologically fathered a child or gave birth to a child, estimated using both logistic regression and modeling the dependent variable as the hazard of first birth. This was the only case where, at least among women, there were no significant differences in coefficient estimates between the main phase sample and the responsive phase sample. Among men, the effect of being foreign-born is markedly stronger among the responsive phase sample than among the main phase sample (an odds ratio of 4.93 among the responsive phase sample compared with an insignificant effect among the main phase sample).
References
Barber, J. S. (2000). Intergenerational influences on the entry into parenthood: Mothers’ preferences for family and nonfamily behavior. Social Forces, 79, 319–348.
Barber, J. S. (2001). Ideational influences on the transition to parenthood: Attitudes toward childbearing and competing alternatives. Social Psychology Quarterly, 64, 101–127.
Bloom, D. E., & Bennett, N. G. (1990). Modeling American marriage patterns. Journal of the American Statistical Association, 85, 1009–1017.
Botman, S. L., & Thornberry, O. T. (1992). Survey design features correlates of nonresponse. Proceedings of the Section on Survey Research Methods of the American Statistical Association (pp. 309–314).
Cheung, G.-Q. (2007, September). Mixed-mode sample management system: An early glance. Presented at International Blaise Users’ Conference, Annapolis, MD.
Couper, M. P. (1998). Measuring survey quality in a CASIC environment. Proceedings of the Survey Research Methods Section of the American Statistical Association (pp. 41–49).
Couper, M. P., & Lyberg, L. E. (2005, April). The use of paradata in survey research. Presented at the Proceedings of the 55th Session of the International Statistical Institute, Sydney, Australia.
Darroch, J. E., Landry, D. J., & Oslak, S. (1999). Age differences between sexual partners in the United States. Family Planning Perspectives, 31, 160–167.
de Leeuw, E., & de Heer, W. (2002). Trends in household survey nonresponse: A longitudinal and international comparison. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 41–54). New York: Wiley.
Finer, L. B., & Henshaw, S. K. (2006). Disparities in rates of unintended pregnancy in the United States, 1994 and 2001. Perspectives on Sexual and Reproductive Health, 38, 90–96.
Goyder, J. (1985). Face-to-face interviews and mailed questionnaires: The net difference in response rate. Public Opinion Quarterly, 49, 234–252.
Groves, R. M. (1989). Survey errors and survey costs. New York: Wiley.
Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70, 646–675.
Groves, R. M., & Couper, M. P. (1998). Nonresponse in household interview surveys. New York: Wiley.
Groves, R. M., & Heeringa, S. G. (2006). Responsive design for household surveys: Tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society: Series A (Statistics in Society), 169, 439–457.
Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72, 167–189.
Groves, R. M., Biemer, P. P., Lyberg, L. E., Massey, J. T., Nicholls, W. L., II, & Waksberg, J. (1988). Telephone survey methodology. New York: Wiley.
Groves, R. M., Singer, E., & Corning, A. (2000). Leverage-saliency theory of survey participation: Description and an illustration. Public Opinion Quarterly, 64, 299–308.
Groves, R. M., Presser, S., & Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68, 2–31.
Groves, R. M., Benson, G., Mosher, W. D., Rosenbaum, J., Granda, P., Axinn, W.G.,…Chandra, A. (2005). Plan and operation of Cycle 6 of the National Survey of Family Growth. Vital and Health Statistics, 1(42). Hyattsville, MD: National Center for Health Statistics.
Groves, R. M., Mosher, W. G., Lepkowski, J. M., & Kirgis, N. G. (2009). Planning and development of the continuous National Survey of Family Growth. Vital and Health Statistics, 1(48). Hyattsville, MD: National Center for Health Statistics.
Guzzo, K. B., & Furstenberg, F. F., Jr. (2007). Multipartnered fertility among american men. Demography, 44, 583–601.
Hapuarachchi, P., March, M., & Wronski, A. (1997). Using statistical methods applicable to autocorrelated processes to analyze survey process quality data. In L. Lyberg, P. Biemer, M. Collins, E. de Leeuw, C. Dippo, N. Schwarz, & D. Trewin (Eds.), Survey measurement and process quality (pp. 589–600). New York: Wiley.
Hayford, S. R., & Morgan, S. P. (2008). Religiosity and fertility in the United States: The role of fertility intentions. Social Forces, 86, 1163–1188.
Keeter, S., Miller, C., Kohut, A., Groves, R. M., & Presser, S. (2000). Consequences of reducing nonresponse in a national telephone survey. Public Opinion Quarterly, 64, 125–148.
Laumann, E. O., Gagnon, J. H., Michael, R. T., & Michaels, S. (1994). The social organization of sexuality: Sexual practices in the United States. Chicago, IL: University of Chicago Press.
Lepkowski, J. M., Mosher, W. D., Davis, K. E., Groves, R. M., van Hoewyk, J., & Willem, J. (2006). National Survey of Family Growth, Cycle 6: Sample design, weighting, imputation, and variance estimation. Vital and Health Statistics, 2(142). Hyattsville, MD: National Center for Health Statistics.
Manlove, J., Terry-Humen, E., & Ikramullah, E. (2006). Young teenagers and older sexual partners: Correlates and consequences for males and females. Perspectives on Sexual and Reproductive Health, 38, 197–207.
Manlove, J., Logan, C., Ikramullah, E., & Holcombe, E. (2008). Factors associated with multiple-partner fertility among fathers. Journal of Marriage and Family, 70, 536–548.
Rindfuss, R. R., Morgan, S. P., & Swicegood, C. G. (1988). First births in America: Changes in the timing of parenthood. Berkeley: University of California Press.
Scheuren, F. (2001, January). Macro and micro paradata for survey assessment. Presented at the United Nations Work Session on Statistical Metadata, Washington, DC.
Singer, E. (2002). The use of incentives to reduce nonresponse in household surveys. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 163–178). New York: Wiley.
Smith, T. W. (1992). Discrepancies between men and women in reporting number of sexual partners: A summary from four countries. Social Biology, 39, 203–211.
Thornton, A., Axinn, W. G., & Hill, D. H. (1992). Reciprocal effects of religiosity, cohabitation, and marriage. The American Journal of Sociology, 98, 628–651.
Thornton, A., Axinn, W. G., & Xie, Y. (2007). Marriage and cohabitation. Chicago, IL: University of Chicago Press.
Tourangeau, R., Rips, L. J., & Rasinski, K. A. (2000). The psychology of survey response. Cambridge, UK: Cambridge University Press.
Traugott, M. W., Groves, R. M., & Lepkowski, J. M. (1987). Using dual frame designs to reduce nonresponse in telephone surveys. Public Opinion Quarterly, 51, 522–539.
Zhang, L. (2008). Religious affiliation, religiosity, and male and female fertility. Demographic Research, 18, article 8, 233–262. doi:10.4054/DemRes.2008.18.8
Acknowledgments
The authors wish to thank the NSFG staff at both the National Center for Health Statistics and the University of Michigan for all their effort to produce the NSFG data with the most innovative survey tools available. We also wish to thank Paul Schulz and Sarah Brauner-Otto for their contributions to previous versions of this paper, and Mick Couper, Peter Granda, Nicole Kirgis, Jim Lepkowski, and James Wagner for their helpful comments regarding the analysis of responsive design effects. The responsibility for all errors remains with the authors. An earlier version of this paper was presented at the 2009 annual meetings of the Population Association of America.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Axinn, W.G., Link, C.F. & Groves, R.M. Responsive Survey Design, Demographic Data Collection, and Models of Demographic Behavior. Demography 48, 1127–1149 (2011). https://doi.org/10.1007/s13524-011-0044-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13524-011-0044-1