We use the case of welfare recipients to validate conjoint experiments as a measure of stereotype content. Stereotypes are politically consequential, but their content can be difficult to measure. The conjoint measure of stereotype content, in which respondents see profiles describing hypothetical persons and rate these persons’ degree of belonging to the target group, offers several advantages over existing measures. However, no existing work evaluates the validity of this new measure. We evaluate this measurement technique using the case of welfare recipients. Stereotypes of welfare recipients are politically important and extensively studied, providing strong a priori expectations for portions of the stereotype, especially race, gender, and “deservingness.” At the same time, scholars disagree about the importance of another attribute with important political implications: immigration status. We find that aggregate stereotypes, measured via a conjoint experiment, match the strong a priori expectations: white Americans see welfare recipients as black, female, and violating the norms of work ethic. Individual-level stereotypes also predict welfare policy support—even when other demographic and ideological factors are accounted for. We also find that immigration status is not a part of the welfare recipient stereotype for most Americans, but support for welfare is lower among those who do stereotype welfare recipients as undocumented immigrants. Finally, we suggest an improvement in the conjoint task instructions. Overall, we confirm that conjoint experiments provide a valid measure of stereotypes.
This is a preview of subscription content,to check access.
Access this article
Similar content being viewed by others
While some work validates the use of conjoint experiments to measure preferences (Hainmueller et al., 2015), none examines its validity as a measure of stereotypes.
Notably, these authors do not use the term, though their designs measure stereotypes: clusters of attributes associated with social groups.
For example, most PhDs are Democrats, but the typical Democrat does not have a PhD.
It is worth noting that these stereotypes differ from the average characteristics of actual recipients of means-tested government programs; only 25% of recipients are black, and most are also employed, with 85% of recipient families having at least one employed adult (Foster & Rojas, 2013). Recipient families are more likely to be headed by a single woman, but there appears to be no relationship between welfare receipt and increased birth rate for women of child-bearing age (for a review, see Kearney, 2004).
We excluded two respondents who failed attention checks and 33 respondents who used the exact same rating scores for all profiles in the conjoint experiment.
For example, the racial resentment scale includes the statement “It’s really a matter of some people not trying hard enough; if blacks would only try harder they could be just as well off as whites.”
Due to randomization, two respondents who never saw a profile with no children and were excluded from the analysis.
Respondents can complete this number of conjoint tasks without evidence of satisficing (Bansak et al., 2018). See Appendix B in Supplementary Material for results broken down by earlier and later rounds.
When completing the conjoint experiment, 55 respondents ended up rating less than 30 profiles (the lowest number of rated profiles was 27 for three respondents). These respondents were kept in the analysis.
Replication materials for all presented analyses are available at: https://doi.org/10.7910/DVN/6ECD1D.
Appendix C in Supplementary Material reports the AMCEs for nonwhite respondents. Having children and employment-seeking appear to play a smaller role in black and Hispanic respondents’ stereotypes of welfare recipients, though our samples of these groups are small.
In the case of employment, this is a misperception, as welfare recipients are often employed.
Appendix E in Supplementary Material (Figure S3) investigates whether race interacts with other attributes. No notable interaction effects found.
We re-estimate these effects accounting for uncertainty in the estimated IMCEs (Figure S4 in Appendix F, Supplementary Material). The substantive interpretation of the results is unchanged, though the estimated effect of FIRE on “Black” IMCE in the typicality condition is no longer significant.
IMCEs for “Black” and “Hispanic” attribute values are still insignificant when we control only for demographics and partisanship but not for FIRE and individualism (see Figure S6 in Appendix G, Supplementary Material).
We thank an anonymous reviewer for pointing out that this result could be a function of our use of measure of welfare support that includes several questions that might implicate the deservingness of welfare recipients. To evaluate this possibility, Figure S7 (Appendix H, Supplementary Material) replicates the analysis in Fig. 4 using the two items from the welfare support scale that are least connected to recipients’ deservingness (“The high cost of welfare…” and “When people can’t support themselves…”). The coefficient of “deservingness” indeed decreases. This can indicate sensitivity of the results to how support for welfare is measures, but also lower reliability of a two-item scale.
We re-estimate these effects accounting for uncertainty in the estimated IMCEs (Figure S5 in Appendix F, Supplementary Material). The substantive interpretation of the results is unchanged, though the estimated effect for “Black” IMCE in the typicality condition without controls is no longer significant.
Compared to welfare recipient data referenced in footnote 5 (Forster & Rojas, 2013; Kearney, 2004), respondents accurately assess mothers and unmarried individuals being more typical. But they are inaccurate in recipient race, employment status, and number of children, as well as in assessing non-citizen immigrants as being as typical as citizen. However, typicality scores are difficult to compare with objective data on welfare recipients, therefore our results do not allow claiming presence of certain (in)correct stereotypes in the public.
Racial stereotypes can impact support for welfare via racial conservatism and partisanship (although reverse causality is also possible; see Goren, 2021), and thus our multivariate model may underestimate the effect of “Black” IMCE. We are grateful to an anonymous reviewer for suggesting this possibility.
We are grateful to anonymous reviewers for suggesting these possibilities.
Importantly, we do not explore whether naturalized immigrants are stereotyped as welfare recipients, which could be a welfare stereotype that some Americans hold.
Ahler, D. J., & Sood, G. (2018). The Parties in Our Heads: Misperceptions about Party Composition and Their Consequences. Journal of Politics, 80(3), 964–981
Amundson, K., Zajicek, A. M., & Kerr, B. (2015). A Social Metamorphosis: Constructing Drug Addicts From the Poor. Sociological Spectrum, 35(5), 442–464
Appelbaum, L. (2001). The Influence of Perceived Deservingness on Policy Decisions Regarding Aid to the Poor. Political Psychology, 22(3), 419–442
Banks, A. (2014). Anger and Racial Politics: The Emotional Foundation of Racial Attitudes in America. New York: Cambridge University Press
Bansak, K., Hainmueller, J., & Hopkins, D. J., & T. Yamamoto (2018). The Number of Choice Tasks and Survey Satisficing in Conjoint Experiments. Political Analysis, 26(1), 112–119
Bodenhausen, G. V., Kang, S. K., & Peery, D. (2012). Social Categorization and the Perception of Social Groups. In S. T. Fiske & C. N. Macrae (Eds.), The SAGE Handbook of Social Cognition (pp. 311–329). Los Angeles: SAGE
Bridgman, A., Merkley, E., Loewen, P. J., & Owen, T., & Ruths, D. (2021). All in This Together? A Preregistered Report on Deservingness of Government Aid during the COVID-19 Pandemic. Journal of Experimental Political Science. https://doi.org/10.1017/XPS.2021.10. Advance online publication
Brosius, H. B. (2003). Exemplars in the News: A Theory of the Effects of Political Communication. In J. Bryant, D. R. Roskos-Ewoldsen, & J. Cantor (Eds.), Communication and Emotion: Essays in Honor of Dolf Zillmann (pp. 179–194). Mahwah, NJ: Routledge
Brown, H. (2013). The New Racial Politics of Welfare: Ethno-Racial Diversity, Immigration, and Welfare Discourse Variation. Social Service Review, 87(3), 586–612
Brown, J. (2021). "Not Sending Us Their Best": Media Framings of Immigrants as the "Parasitic Poor." In S. L. Borden (Ed.), The Routledge Companion to Media and Poverty (pp. 277–287). New York: Routledge
Brown-Iannuzzi, J. L., Dotsch, R., Cooley, E., & Payne, B. K. (2017). The Relationship between Mental Representations of Welfare Recipients and Attitudes toward Welfare. Psychological Science, 28(1), 92–103
Brown-Iannuzzi, J. L., Cooley, E., & McKee, S. E., & Hyden, C. (2019). Wealthy Whites and Poor Blacks: Implicit Associations between Racial Groups and Wealth Predict Explicit Opposition toward Helping the Poor. Journal of Experimental Social Psychology, 82, 26–34
Calavita, K. (1996). The New Politics of Immigration: "Balanced-Budget Conservatism" and the Symbolism of Proposition 187. Social Problems, 43(3), 284–305
Claassen, R. L., Djupe, P. A., Lewis, A. R., & Neiheisel, J. R. (2021). Which Party Represents My Group? The Group Foundations of Partisan Choice and Polarization. Political Behavior, 43, 615–636
Clawson, R. A., & Trice, R (2000). Poverty as We Know It: Media Portrayals of the Poor. Public Opinion Quarterly, 64(1), 53–64
Clawson, R. A., & Jett, J. (2019). The Media Whiteness of Social Security and Medicare. Politics Groups and Identities, 7(1), 207–218
Clifford, S., & Jerit, J. (2016). Cheating on Political Knowledge Questions in Online Surveys: As Assessment of the Problem and Solutions. Public Opinion Quarterly, 80(4), 858–887
Coppock, A., & McClellan, O. A. (2019). Validating the Demographic, Political, Psychological, and Experimental Results Obtained from a New Source of Online Survey Respondents. Research and Politics, 6(1), https://doi.org/10.1177/2053168018822174
DeSante, C. D. (2013). Working Twice as Hard to Get Half as Far: Race, Work Ethic, and America’s Deserving Poor. American Journal of Political Science, 57(2), 342–356
DeSante, C. D., & Candis, W. S. (2020). Fear, Institutionalized Racism, and Empathy: The Underlying Dimensions of Whites’ Racial Attitudes. PS: Political Science and Politics, 53(4), 639–645
Eagly, A. H., & Mladinic, A. (1989). Gender Stereotypes and Attitudes Toward Women and Men. Personality and Social Psychology Bulletin, 15(4), 543–558
Ellis, C., & Faricy, C. (2020). Race, "Deservingness," and Social Spending Attitudes: The Role of Policy Delivery Mechanism. Political Behavior, 42, 819–843
Entman, R. M., & Rojecki, A. (2000). The Black Image in the White Mind: Media and Race in America. Chicago: University of Chicago Press
Fang, A. H., & Huber, G. A. (2020). Perceptions of Deservingness and the Politicization of Social Insurance: Evidence From Disability Insurance in the United States. American Politics Research, 48(5), 543–559
Federico, C. (2006). Race, Education, and Individualism Revisited. Journal of Politics, 68(3), 600–610
Feldman, S., Huddy, L., & Wronski, J., & Lown, P. (2020). The Interplay of Empathy and Individualism in Support for Social Welfare Policies. Political Psychology, 41(2), 343–362
Fiske, S. T., Amy, J. C., Cuddy, P., Glick, & Xu, J. (2002). A Model of (Often Mixed) Stereotype Content: Competence and Warmth Respectively Follow From Perceived Status and Competition. Journal of Personality and Social Psychology, 82(6), 878–902
Flores, R. D., & Schachter, A. (2018). Who Are the “Illegals”? The Social Construction of Illegality in the United States. American Sociological Review, 83(5), 839–868
Foster, A., & Rojas, A. (2013). Program Participation and Spending Patterns of Families Receiving Government Means-Tested Assistance. U.S. Bureau of Labor Statistics. https://www.bls.gov/opub/btn/volume-2/pdf/spending-patterns-of-families-receiving-means-tested-government-assistance.pdf
Garand, J. C., Xu, P., & Davis, B. C. (2017). Immigration Attitudes and Support for the Welfare State in the American Mass Public. American Journal of Political Science, 61(1), 146–162
Garcia-Marques, Leonel, A., Sofia, C., Santos, & Mackie, D. M. (2006). Stereotypes: Static Abstractions or Dynamic Knowledge Structures? Journal of Personality and Social Psychology, 91(5), 814–831
Gilens, M. (1999). Why Americans Hate Welfare: Race, Media, and the Politics of Antipoverty Policy. Chicago: University of Chicago Press
Goggin, S. N., Henderson, J. A., & Theodoridis, A. G. (2020). What Goes with Red and Blue? Mapping Partisan and Ideological Associations in the Minds of Voters. Political Behavior, 42, 985–1013
Goren, P. (2003). Race, Sophistication, and White Opinion on Government Spending. Political Behavior, 25(3), 201–220
Goren, P. (2008). The Two Faces of Government Spending. Political Research Quarterly, 61(1), 147–157
Goren, P. (2021). Pliable Prejudice: The Case of Welfare. American Journal of Political Science. https://doi.org/10.1111/ajps.12652. Advance online publication
Green, J., & McElwee, S. (2019). The Differential Effects of Economic Conditions and Racial Attitudes in the Election of Donald Trump. Perspectives on Politics, 17(2), 358–379
Gross, K., & Wronski, J. (2021). Helping the Homeless: The Role of Empathy, Race and Deservingness in Motivating Policy Support and Charitable Giving. Political Behavior, 43, 585–613
Hancock, A. M. (2004). The Politics of Disgust: The Public Identity of the Welfare Queen. New York: NYU Press
Hainmueller, J., Hangartner, D., & Yamamoto, T. (2015). Validating Vignette and Conjoint Survey Experiments against Real-World Behavior. Proceedings of the National Academy of Sciences 112 (8): 2395–400
Hainmueller, J., Hopkins, D. J., & Yamamoto, T. (2014). Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments. Political Analysis, 22(1), 1–30
Heit, E., & Nicholson, S. P. (2010). The Opposite of Republican: Polarization and Political Categorization. Cognitive Science, 34(8), 1503–1516
Horiuchi, Y., Markovich, Z., & Yamamoto, T. (2021). Does Conjoint Analysis Mitigate Social Desirability Bias? Political Analysis. https://doi.org/10.1017/pan.2021.30. Advance online publication
Jardina, A., Kalmoe, N., & Gross, K. (2020). Disavowing White Identity: How Social Disgust Can Change Social Identities. Political Psychology, 42(4), 619–636
Kinder, D. R., & Kam, C. D. (2010). Us against Them: Ethnocentric Foundations of American Opinion. Chicago: University of Chicago Press
Katz, D., & Braly, K. W. (1933). Racial Stereotypes of One Hundred College Students. Journal of Abnormal and Social Psychology, 28(3), 280–290
Krimmel, K., & Rader, K. (2021). Racial Unfairness and Fiscal Politics. American Politics Research, 49(2), 143–156
Landy, D., Guay, B., & Marghetis, T. (2018). Bias and Ignorance in Demographic Perception. Psychonomic Bulletin & Review, 25(5), 1606–1618
Levy, M. E. (2021). Once Racialized, Now “Immigrationized”? Explaining the Immigration-Welfare Link in American Public Opinion. Journal of Politics, 83(4), 1275–1291
Lippmann, W. (1922). Public Opinion. New York: Macmillan, Harcourt, Brace
Mason, L., & Wronski, J. (2018). One Tribe to Bind Them All: How Our Social Group Attachments Strengthen Partisanship. Advances in Political Psychology, 39, 257–277
Peffley, M., Hurwitz, J., & Sniderman, P. M. (1997). Racial Stereotypes and Whites’ Political Views of Blacks in the Context of Welfare and Crime. American Journal of Political Science, 41(1), 30–60
Petersen, M. B. (2012). Social Welfare as Small-Scale Help: Evolutionary Psychology and the Deservingness Heuristic. American Journal of Political Science, 56(1), 1–16
Pierson-Balik, D. A. (2003). Race, Class, and Gender in Punitive Welfare Reform: Social Eugenics and Welfare Policy. Race Gender & Class, 10(1), 11–30
Rabinowitz, J. L., Sears, D. O., Sidanius, J., & Krosnick, J. A. (2009). Why Do White Americans Oppose Race-Targeted Policies? Clarifying the Impact of Symbolic Racism. Political Psychology, 30(5), 805–828
Rothschild, J. E., Howat, A. J., Shafranek, R. M., & Busby, E. C. (2019). Pigeonholing Partisans: Stereotypes of Party Supporters and Partisan Polarization. Political Behavior, 41(2), 423–443
Kearney, M. S. (2004). Is There an Effect of Incremental Welfare Benefits on Fertility Behavior? A Look at the Family Cap. Journal of Human Resources, 39(2), 295–325
Schneider, A., & Ingram, H. (1993). Social Construction of Target Populations: Implications for Politics and Policy. American Political Science Review, 87(2), 334–347
Schneider, M. C., & Bos, A. L. (2011). An Exploration of the Content of Stereotypes of Black Politicians. Political Psychology, 32(2), 205–233
Shapiro, R. Y., Kelly, D., Patterson, J., Russell, & Young, J. T. (1987). Public Assistance. Public Opinion Quarterly, 51(1), 120–130
Sherman, J. W. (1996). Development and Mental Representation of Stereotypes. Journal of Personality and Social Psychology, 70(6), 1126–1141
Smith, R. M., & King, D. (2021). White Protectionism in America. Perspectives on Politics, 19(2), 460–478
Stangor, C., & Schaller, M. (2000). Stereotypes as Individual and Collective Representations. In C. Stangor (Ed.), Stereotypes and Prejudice: Essential Readings (pp. 64–82). Philadelphia, PA: Psychology Press
Valentino, N. A., Neuner, F. G., & Vandenbroek, L. M. (2018). The Changing Norms of Racial Political Rhetoric and the End of Racial Priming. Journal of Politics, 80(3), 757–771
Weaver, R. K. (2000). Ending Welfare as We Know It. Washington, DC: Brookings Institution Press.
Winter, N. J. G. (2010). Masculine Republicans and Feminine Democrats: Gender and Americans’ Explicit and Implicit Images of the Political Parties. Political Behavior, 32(4), 587–618
Zhirkov, K. (2021a). Estimating and Using Individual Marginal Component Effects from Conjoint Experiments. Political Analysis, 30(2), 236–249
Zhirkov, K. (2021b). Perceptions of Stereotypically Immigrant Groups as Darker-Skinned and Politics of Immigration in the United States and Britain. Politics, Groups, and Identities. https://doi.org/10.1080/21565503.2021.1992285. Advance online publication
Zhirkov, K. (2021c). Who Are “the Immigrants”? Beliefs about Immigrant Populations and Anti-Immigration Attitudes in the United States and Britain. Social Science Quarterly, 102(1), 228–237
Earlier versions of this project were presented at the Annual Conference of the Southern Political Science Association as well as workshops at the University of Minnesota and Vanderbilt University. We are grateful to Chris Federico, Paul Goren, Cindy Kam, Steve Utych, three anonymous reviewers, and Geoffrey Layman, the editor of Political Behavior, for their helpful comments. This research was funded by a Seed Grant for Social Science Research from the University of Minnesota College of Liberal Arts.
Compliance with Ethical Standards
This research was determined exempt by the University of Minnesota Institutional Review Board on March 30, 2020, Study Number 00009407. The procedures used in this study adhere to the tenets of the Declaration of Helsinki.
Conflict of Interest
The authors have no relevant financial or non-financial interests to disclose.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic Supplementary Material
Below is the link to the electronic supplementary material.
About this article
Cite this article
Myers, C.D., Zhirkov, K. & Lunz Trujillo, K. Who Is “On Welfare”? Validating the Use of Conjoint Experiments to Measure Stereotype Content. Polit Behav (2022). https://doi.org/10.1007/s11109-022-09815-0