Abstract
We address two long-standing survey research problems: measuring complicated concepts, such as political freedom and efficacy, that researchers define best with reference to examples; and what to do when respondents interpret identical questions in different ways. Scholars have long addressed these problems with approaches to reduce incomparability, such as writing more concrete questions—with uneven success. Our alternative is to measure directly response category incomparability and to correct for it. We measure incomparability via respondents’ assessments, on the same scale as the self-assessments to be corrected, of hypothetical individuals described in short vignettes. Because the actual (but not necessarily reported) levels of the vignettes are invariant over respondents, variability in vignette answers reveals incomparability. Our corrections require either simple recodes or a statistical model designed to save survey administration costs. With analysis, simulations, and cross-national surveys, we show how response incomparability can drastically mislead survey researchers and how our approach can alleviate this problem.
The current article is a reprint of King/Murray/Salomon/Tandon (2004), authorized by the authors.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Aldrich, John H./ McKelvey Richard D (1977): A Method of Scaling with Applications to the 1968 and 1972 Presidential Elections. In: American Political Science Review 71(March): 111–30.
Alt, James/ Sarlvik, Bo/ Crewe, Ivor (1976): Individual Differences Scaling and Group Attitude Structures: British Party Imagery in 1974. In: Quality and Quantity 10(October): 297–320.
Baum, Lawrence (1988): Measuring Policy Change in the U.S. Supreme Court. In: American Political Science Review 82(September): 905–12.
Brady, Henry E. (1985): The Perils of Survey Research: Inter-Personally Incomparable Responses. In: Political Methodology 11(June): 269–90.
Brady, Henry E. (1989): Factor and Ideal Point Analysis for Interpersonally Incomparable Data. In: Psychometrika 542(June): 181–202.
Cantril, Hadley. 1965. The Pattern of Human Concerns. New Brunswick.
Caroll, J. D./ Chang, J. J. (1970): Analysis of Individual Differences in Multidimensional Scaling. In: Psychometrika 35(September): 283–319.
Cheung, Gordon W./ Rensvold, Roger B. (2000): Assessing Extreme and Acquiescence Response Sets in Cross-Cultural Research Using Structural Equations Modeling (with Comments). In: Journal of Cross-Cultural Psychology 31(March): 187–212.
Clarkson, Douglas B. (2000): A Random Effects Individual Difference Multidimensional Scaling Model. In: Computational Statistics and Data Analysis 32(January): 337–47.
Clinton, Joshua/ Jackman, Simon/ Rivers, Douglas (2002): The Statistical Analysis of Roll Call Data. Unpublished manuscript. Stanford University.
Gelman, Andrew/ King, Gary (1994): A Unified Method of Evaluating Electoral Systems and Redistricting Plans. In: American Journal of Political Science 38(June): 514–54.
Green, Donald P./ Gerber, Alan (2001): Reclaiming the Experimental Tradition in Political Science. In Political Science: State of the Discipline, III. (Milner Helen/Katznelson, Ira). Washington, DC: APSA.
Groot, Wim/ Maassen van den Brink, Henriette (1999): Job Satisfaction and Preference Drift. In: Economics Letters 63(June): 363–67.
Groseclose, Tim/ Levitt, Steven D./ Snyder, James (1999): Comparing Interest Group Scores Across Time and Chambers: Adjusted ADA Scores for the U.S. Congress. In: American Political Science Review 93(March): 33–50.
Heckman, James/ Snyder, James (1997): Linear Probabilty Models of the Demand for Attributes with an Empirical Application to Estimating the Preferences of Legislators. In: Rand Journal of Economics 28(Special Issue): 142–89.
Holland, Paul W./ Wainer, Howard (eds.) (1993): Differential Item Functioning. Hillsdale.
Johnson, Timothy P. (1998): Approaches to Equivalence in Cross-Cultural and Cross-National Survey Research. ZUMA Nachrichten Spezial 3: 1–40.
Johnson, Valen E./ Albert, James H. (1999): Ordinal Data Modeling. New York.
Kahneman, Daniel/ Schkade, David/ Sunstein, Cass R. (1998): Shared Outrage and Erratic Awards: The Psychology of Punitive Damages. In: Journal of Risk and Uncertainty 16(April): 49–86.
Kinder, Donald R./ Palfrey, Thomas R. (eds.) (1993): Experimental Foundations of Political Science. Ann Arbor.
King, Gary (1997): A Solution to the Ecological Inference Problem. Reconstructing Individual Behavior from Aggregate Data. Princeton.
King, Gary/ Honaker, James/ Joseph, Anne/ Scheve, Kenneth (2001): Analyzing Incomplete Political Science Data: An Alternative Algorithm for Multiple Imputation. In: American Political Science Review 95(March): 49–69.
Lewis, Jeffrey B. (2001): Estimating Voter Preference Distributions from Individual-Level Voting Data. In: Political Analysis 9(Summer): 275–97.
Linden, Wim Van Der/ Hambleton, Ronald K. (eds.) (1997): Handbook of Modern Item Response Theory. New York.
Londregan, John (2000): Estimating Legislator’s Preferred Points. In: Political Analysis 8(Winter): 21–34.
Martin, Elizabeth A./ Campanelli, Pamela C./ Fay, Robert E. (1991): An Application of Rasch Analysis to Questionnaire Design: Using Vignettes to Study the Meaning of’ Work’ in the Current Population Survey.“ In: The Statistician 40(September): 265–76.
Mead, A. (1992): Review of the Development of Multidimensional Scaling Methods. In: The Statistician 41 (April): 27–39.
Palfrey, Thomas R./ Poole, Keith T. (1987): The Relationship between Information, Ideology, and Voter Behavior. In: American Journal of Political Science 31(September): 511–30.
Piquero, Alex R./ Macintosh, Randall (2002): The Validity of a Self-Reported Delinquency Scale: Comparisons across Gender, Age, Race, and Place of Residence. In: Sociological Methods and Research 30(May): 492–529.
Poole, Keith T. (1998): Recovering a Basic Space from a Set of Issue Scales. In: American Journal of Political Science 42(September): 954–993.
Poole, Keith/ Daniels, R. Steven (1985): Ideology, Party, and Voting in the U.S. Congress, 1959-1980. In: American Political Science Review 79(June): 373–99.
Poole, Keith,/ Rosenthal, Howard (1991): Patterns of Congressional Voting. In: American Journal of Political Science 35(February): 228–78.
Przeworski, Adam/ Teune, Henry (1966-67): Equivalence in Cross-National Research. In: Public Opinion Quarterly 30(Winter): 551–68.
Rossi, P. H./ Nock, S. L. (eds.) (1983): Measuring Social Judgements: The Factorial Survey Approach. Beverly Hills.
Sen, Amartya (2002): Health: Perception versus Observation. In: British Medical Journal 324(April 13): 860–61.
Shealy, R./ Stout, W. (1993): A Model-Based Standardization Approach That Separates True Bias/DIF from Group Ability Differences and Detects Test Bias/DIF as Well as Item Bias/DIF. In: Psychometrika 58(June): 159–94.
Sniderman, Paul M./ Grob, Douglas B. (1996): Innovations in Experimental Design in Attitude Surveys. In: Annual Review of Sociology 22(August): 377–99.
Stewart, Anita L./ Napoles-Springer, Anna (2000): Health-Related Quality of Life Assessments in Diverse Population Groups in the United States. In: Medical Care 38(September): II–102–II–124.
Suchman, L./ Jordan, B. (1990): Interactional Troubles in Face to Face Survey Interviews (with Comments and Rejoinder). In: Journal of the American Statistical Association 85(March): 232–53.
Thissen, David/ Steinberg, Lynn/ Wainer, Howard (1993): Detection of Differential Item Functioning Using the Parameters of the Item Response Models. In: (Holland, Paul H./ Wainer, Howard) (eds.): Differential Item Functioning.
Torgerson, Warren S. (1958): Theory and Methods of Scaling. New York.
Wolfe, Rory/ Firth, David (2002): Modelling Subjective Use of an Ordinal Reponse Scale in a Many Period Crossover Experiment. In: Applied Statistics 51(April): 245–55.
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2009 VS Verlag für Sozialwissenschaften | GWV Fachverlage GmbH, Wiesbaden
About this chapter
Cite this chapter
King, G., Murray, C.J.L., Salomon, J.A., Tandon, A. (2009). Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research. In: Pickel, S., Pickel, G., Lauth, HJ., Jahn, D. (eds) Methoden der vergleichenden Politik- und Sozialwissenschaft. VS Verlag für Sozialwissenschaften. https://doi.org/10.1007/978-3-531-91826-6_16
Download citation
DOI: https://doi.org/10.1007/978-3-531-91826-6_16
Publisher Name: VS Verlag für Sozialwissenschaften
Print ISBN: 978-3-531-16194-5
Online ISBN: 978-3-531-91826-6
eBook Packages: Humanities, Social Science (German Language)