Skip to main content
SpringerLink
Log in

Who Is “On Welfare”? Validating the Use of Conjoint Experiments to Measure Stereotype Content

Political Behavior Aims and scope Submit manuscript

Cite this article

Abstract

We use the case of welfare recipients to validate conjoint experiments as a measure of stereotype content. Stereotypes are politically consequential, but their content can be difficult to measure. The conjoint measure of stereotype content, in which respondents see profiles describing hypothetical persons and rate these persons’ degree of belonging to the target group, offers several advantages over existing measures. However, no existing work evaluates the validity of this new measure. We evaluate this measurement technique using the case of welfare recipients. Stereotypes of welfare recipients are politically important and extensively studied, providing strong a priori expectations for portions of the stereotype, especially race, gender, and “deservingness.” At the same time, scholars disagree about the importance of another attribute with important political implications: immigration status. We find that aggregate stereotypes, measured via a conjoint experiment, match the strong a priori expectations: white Americans see welfare recipients as black, female, and violating the norms of work ethic. Individual-level stereotypes also predict welfare policy support—even when other demographic and ideological factors are accounted for. We also find that immigration status is not a part of the welfare recipient stereotype for most Americans, but support for welfare is lower among those who do stereotype welfare recipients as undocumented immigrants. Finally, we suggest an improvement in the conjoint task instructions. Overall, we confirm that conjoint experiments provide a valid measure of stereotypes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. While some work validates the use of conjoint experiments to measure preferences (Hainmueller et al., 2015), none examines its validity as a measure of stereotypes.

  2. Notably, these authors do not use the term, though their designs measure stereotypes: clusters of attributes associated with social groups.

  3. For example, most PhDs are Democrats, but the typical Democrat does not have a PhD.

  4. https://www.nytimes.com/2019/08/14/us/immigration-public-charge-welfare.html.

  5. It is worth noting that these stereotypes differ from the average characteristics of actual recipients of means-tested government programs; only 25% of recipients are black, and most are also employed, with 85% of recipient families having at least one employed adult (Foster & Rojas, 2013). Recipient families are more likely to be headed by a single woman, but there appears to be no relationship between welfare receipt and increased birth rate for women of child-bearing age (for a review, see Kearney, 2004).

  6. We excluded two respondents who failed attention checks and 33 respondents who used the exact same rating scores for all profiles in the conjoint experiment.

  7. For example, the racial resentment scale includes the statement “It’s really a matter of some people not trying hard enough; if blacks would only try harder they could be just as well off as whites.”

  8. Due to randomization, two respondents who never saw a profile with no children and were excluded from the analysis.

  9. Respondents can complete this number of conjoint tasks without evidence of satisficing (Bansak et al., 2018). See Appendix B in Supplementary Material for results broken down by earlier and later rounds.

  10. When completing the conjoint experiment, 55 respondents ended up rating less than 30 profiles (the lowest number of rated profiles was 27 for three respondents). These respondents were kept in the analysis.

  11. Replication materials for all presented analyses are available at: https://doi.org/10.7910/DVN/6ECD1D.

  12. Appendix C in Supplementary Material reports the AMCEs for nonwhite respondents. Having children and employment-seeking appear to play a smaller role in black and Hispanic respondents’ stereotypes of welfare recipients, though our samples of these groups are small.

  13. In the case of employment, this is a misperception, as welfare recipients are often employed.

  14. Appendix E in Supplementary Material (Figure S3) investigates whether race interacts with other attributes. No notable interaction effects found.

  15. We re-estimate these effects accounting for uncertainty in the estimated IMCEs (Figure S4 in Appendix F, Supplementary Material). The substantive interpretation of the results is unchanged, though the estimated effect of FIRE on “Black” IMCE in the typicality condition is no longer significant.

  16. IMCEs for “Black” and “Hispanic” attribute values are still insignificant when we control only for demographics and partisanship but not for FIRE and individualism (see Figure S6 in Appendix G, Supplementary Material).

  17. We thank an anonymous reviewer for pointing out that this result could be a function of our use of measure of welfare support that includes several questions that might implicate the deservingness of welfare recipients. To evaluate this possibility, Figure S7 (Appendix H, Supplementary Material) replicates the analysis in Fig. 4 using the two items from the welfare support scale that are least connected to recipients’ deservingness (“The high cost of welfare…” and “When people can’t support themselves…”). The coefficient of “deservingness” indeed decreases. This can indicate sensitivity of the results to how support for welfare is measures, but also lower reliability of a two-item scale.

  18. We re-estimate these effects accounting for uncertainty in the estimated IMCEs (Figure S5 in Appendix F, Supplementary Material). The substantive interpretation of the results is unchanged, though the estimated effect for “Black” IMCE in the typicality condition without controls is no longer significant.

  19. Compared to welfare recipient data referenced in footnote 5 (Forster & Rojas, 2013; Kearney, 2004), respondents accurately assess mothers and unmarried individuals being more typical. But they are inaccurate in recipient race, employment status, and number of children, as well as in assessing non-citizen immigrants as being as typical as citizen. However, typicality scores are difficult to compare with objective data on welfare recipients, therefore our results do not allow claiming presence of certain (in)correct stereotypes in the public.

  20. Racial stereotypes can impact support for welfare via racial conservatism and partisanship (although reverse causality is also possible; see Goren, 2021), and thus our multivariate model may underestimate the effect of “Black” IMCE. We are grateful to an anonymous reviewer for suggesting this possibility.

  21. We are grateful to anonymous reviewers for suggesting these possibilities.

  22. Importantly, we do not explore whether naturalized immigrants are stereotyped as welfare recipients, which could be a welfare stereotype that some Americans hold.

References

  • Ahler, D. J., & Sood, G. (2018). The Parties in Our Heads: Misperceptions about Party Composition and Their Consequences. Journal of Politics, 80(3), 964–981

    Article  Google Scholar 

  • Amundson, K., Zajicek, A. M., & Kerr, B. (2015). A Social Metamorphosis: Constructing Drug Addicts From the Poor. Sociological Spectrum, 35(5), 442–464

    Article  Google Scholar 

  • Appelbaum, L. (2001). The Influence of Perceived Deservingness on Policy Decisions Regarding Aid to the Poor. Political Psychology, 22(3), 419–442

    Article  Google Scholar 

  • Banks, A. (2014). Anger and Racial Politics: The Emotional Foundation of Racial Attitudes in America. New York: Cambridge University Press

    Book  Google Scholar 

  • Bansak, K., Hainmueller, J., & Hopkins, D. J., & T. Yamamoto (2018). The Number of Choice Tasks and Survey Satisficing in Conjoint Experiments. Political Analysis, 26(1), 112–119

    Article  Google Scholar 

  • Bodenhausen, G. V., Kang, S. K., & Peery, D. (2012). Social Categorization and the Perception of Social Groups. In S. T. Fiske & C. N. Macrae (Eds.), The SAGE Handbook of Social Cognition (pp. 311–329). Los Angeles: SAGE

  • Bridgman, A., Merkley, E., Loewen, P. J., & Owen, T., & Ruths, D. (2021). All in This Together? A Preregistered Report on Deservingness of Government Aid during the COVID-19 Pandemic. Journal of Experimental Political Science. https://doi.org/10.1017/XPS.2021.10. Advance online publication

    Article  Google Scholar 

  • Brosius, H. B. (2003). Exemplars in the News: A Theory of the Effects of Political Communication. In J. Bryant, D. R. Roskos-Ewoldsen, & J. Cantor (Eds.), Communication and Emotion: Essays in Honor of Dolf Zillmann (pp. 179–194). Mahwah, NJ: Routledge

  • Brown, H. (2013). The New Racial Politics of Welfare: Ethno-Racial Diversity, Immigration, and Welfare Discourse Variation. Social Service Review, 87(3), 586–612

    Article  Google Scholar 

  • Brown, J. (2021). "Not Sending Us Their Best": Media Framings of Immigrants as the "Parasitic Poor." In S. L. Borden (Ed.), The Routledge Companion to Media and Poverty (pp. 277–287). New York: Routledge

  • Brown-Iannuzzi, J. L., Dotsch, R., Cooley, E., & Payne, B. K. (2017). The Relationship between Mental Representations of Welfare Recipients and Attitudes toward Welfare. Psychological Science, 28(1), 92–103

    Article  Google Scholar 

  • Brown-Iannuzzi, J. L., Cooley, E., & McKee, S. E., & Hyden, C. (2019). Wealthy Whites and Poor Blacks: Implicit Associations between Racial Groups and Wealth Predict Explicit Opposition toward Helping the Poor. Journal of Experimental Social Psychology, 82, 26–34

    Article  Google Scholar 

  • Calavita, K. (1996). The New Politics of Immigration: "Balanced-Budget Conservatism" and the Symbolism of Proposition 187. Social Problems, 43(3), 284–305

    Article  Google Scholar 

  • Claassen, R. L., Djupe, P. A., Lewis, A. R., & Neiheisel, J. R. (2021). Which Party Represents My Group? The Group Foundations of Partisan Choice and Polarization. Political Behavior, 43, 615–636

    Article  Google Scholar 

  • Clawson, R. A., & Trice, R (2000). Poverty as We Know It: Media Portrayals of the Poor. Public Opinion Quarterly, 64(1), 53–64

    Article  Google Scholar 

  • Clawson, R. A., & Jett, J. (2019). The Media Whiteness of Social Security and Medicare. Politics Groups and Identities, 7(1), 207–218

    Article  Google Scholar 

  • Clifford, S., & Jerit, J. (2016). Cheating on Political Knowledge Questions in Online Surveys: As Assessment of the Problem and Solutions. Public Opinion Quarterly, 80(4), 858–887

    Article  Google Scholar 

  • Coppock, A., & McClellan, O. A. (2019). Validating the Demographic, Political, Psychological, and Experimental Results Obtained from a New Source of Online Survey Respondents. Research and Politics, 6(1), https://doi.org/10.1177/2053168018822174

  • DeSante, C. D. (2013). Working Twice as Hard to Get Half as Far: Race, Work Ethic, and America’s Deserving Poor. American Journal of Political Science, 57(2), 342–356

    Article  Google Scholar 

  • DeSante, C. D., & Candis, W. S. (2020). Fear, Institutionalized Racism, and Empathy: The Underlying Dimensions of Whites’ Racial Attitudes. PS: Political Science and Politics, 53(4), 639–645

    Google Scholar 

  • Eagly, A. H., & Mladinic, A. (1989). Gender Stereotypes and Attitudes Toward Women and Men. Personality and Social Psychology Bulletin, 15(4), 543–558

    Article  Google Scholar 

  • Ellis, C., & Faricy, C. (2020). Race, "Deservingness," and Social Spending Attitudes: The Role of Policy Delivery Mechanism. Political Behavior, 42, 819–843

    Article  Google Scholar 

  • Entman, R. M., & Rojecki, A. (2000). The Black Image in the White Mind: Media and Race in America. Chicago: University of Chicago Press

    Google Scholar 

  • Fang, A. H., & Huber, G. A. (2020). Perceptions of Deservingness and the Politicization of Social Insurance: Evidence From Disability Insurance in the United States. American Politics Research, 48(5), 543–559

    Article  Google Scholar 

  • Federico, C. (2006). Race, Education, and Individualism Revisited. Journal of Politics, 68(3), 600–610

    Article  Google Scholar 

  • Feldman, S., Huddy, L., & Wronski, J., & Lown, P. (2020). The Interplay of Empathy and Individualism in Support for Social Welfare Policies. Political Psychology, 41(2), 343–362

    Article  Google Scholar 

  • Fiske, S. T., Amy, J. C., Cuddy, P., Glick, & Xu, J. (2002). A Model of (Often Mixed) Stereotype Content: Competence and Warmth Respectively Follow From Perceived Status and Competition. Journal of Personality and Social Psychology, 82(6), 878–902

    Article  Google Scholar 

  • Flores, R. D., & Schachter, A. (2018). Who Are the “Illegals”? The Social Construction of Illegality in the United States. American Sociological Review, 83(5), 839–868

    Article  Google Scholar 

  • Foster, A., & Rojas, A. (2013). Program Participation and Spending Patterns of Families Receiving Government Means-Tested Assistance. U.S. Bureau of Labor Statistics. https://www.bls.gov/opub/btn/volume-2/pdf/spending-patterns-of-families-receiving-means-tested-government-assistance.pdf

  • Garand, J. C., Xu, P., & Davis, B. C. (2017). Immigration Attitudes and Support for the Welfare State in the American Mass Public. American Journal of Political Science, 61(1), 146–162

    Article  Google Scholar 

  • Garcia-Marques, Leonel, A., Sofia, C., Santos, & Mackie, D. M. (2006). Stereotypes: Static Abstractions or Dynamic Knowledge Structures? Journal of Personality and Social Psychology, 91(5), 814–831

    Article  Google Scholar 

  • Gilens, M. (1999). Why Americans Hate Welfare: Race, Media, and the Politics of Antipoverty Policy. Chicago: University of Chicago Press

    Book  Google Scholar 

  • Goggin, S. N., Henderson, J. A., & Theodoridis, A. G. (2020). What Goes with Red and Blue? Mapping Partisan and Ideological Associations in the Minds of Voters. Political Behavior, 42, 985–1013

    Article  Google Scholar 

  • Goren, P. (2003). Race, Sophistication, and White Opinion on Government Spending. Political Behavior, 25(3), 201–220

    Article  Google Scholar 

  • Goren, P. (2008). The Two Faces of Government Spending. Political Research Quarterly, 61(1), 147–157

    Article  Google Scholar 

  • Goren, P. (2021). Pliable Prejudice: The Case of Welfare. American Journal of Political Science. https://doi.org/10.1111/ajps.12652. Advance online publication

    Article  Google Scholar 

  • Green, J., & McElwee, S. (2019). The Differential Effects of Economic Conditions and Racial Attitudes in the Election of Donald Trump. Perspectives on Politics, 17(2), 358–379

    Article  Google Scholar 

  • Gross, K., & Wronski, J. (2021). Helping the Homeless: The Role of Empathy, Race and Deservingness in Motivating Policy Support and Charitable Giving. Political Behavior, 43, 585–613

    Article  Google Scholar 

  • Hancock, A. M. (2004). The Politics of Disgust: The Public Identity of the Welfare Queen. New York: NYU Press

    Google Scholar 

  • Hainmueller, J., Hangartner, D., & Yamamoto, T. (2015). Validating Vignette and Conjoint Survey Experiments against Real-World Behavior. Proceedings of the National Academy of Sciences 112 (8): 2395–400

  • Hainmueller, J., Hopkins, D. J., & Yamamoto, T. (2014). Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments. Political Analysis, 22(1), 1–30

    Article  Google Scholar 

  • Heit, E., & Nicholson, S. P. (2010). The Opposite of Republican: Polarization and Political Categorization. Cognitive Science, 34(8), 1503–1516

    Article  Google Scholar 

  • Horiuchi, Y., Markovich, Z., & Yamamoto, T. (2021). Does Conjoint Analysis Mitigate Social Desirability Bias? Political Analysis. https://doi.org/10.1017/pan.2021.30. Advance online publication

  • Jardina, A., Kalmoe, N., & Gross, K. (2020). Disavowing White Identity: How Social Disgust Can Change Social Identities. Political Psychology, 42(4), 619–636

    Article  Google Scholar 

  • Kinder, D. R., & Kam, C. D. (2010). Us against Them: Ethnocentric Foundations of American Opinion. Chicago: University of Chicago Press

    Google Scholar 

  • Katz, D., & Braly, K. W. (1933). Racial Stereotypes of One Hundred College Students. Journal of Abnormal and Social Psychology, 28(3), 280–290

    Article  Google Scholar 

  • Krimmel, K., & Rader, K. (2021). Racial Unfairness and Fiscal Politics. American Politics Research, 49(2), 143–156

    Article  Google Scholar 

  • Landy, D., Guay, B., & Marghetis, T. (2018). Bias and Ignorance in Demographic Perception. Psychonomic Bulletin & Review, 25(5), 1606–1618

    Article  Google Scholar 

  • Levy, M. E. (2021). Once Racialized, Now “Immigrationized”? Explaining the Immigration-Welfare Link in American Public Opinion. Journal of Politics, 83(4), 1275–1291

    Article  Google Scholar 

  • Lippmann, W. (1922). Public Opinion. New York: Macmillan, Harcourt, Brace

    Google Scholar 

  • Mason, L., & Wronski, J. (2018). One Tribe to Bind Them All: How Our Social Group Attachments Strengthen Partisanship. Advances in Political Psychology, 39, 257–277

    Article  Google Scholar 

  • Peffley, M., Hurwitz, J., & Sniderman, P. M. (1997). Racial Stereotypes and Whites’ Political Views of Blacks in the Context of Welfare and Crime. American Journal of Political Science, 41(1), 30–60

    Article  Google Scholar 

  • Petersen, M. B. (2012). Social Welfare as Small-Scale Help: Evolutionary Psychology and the Deservingness Heuristic. American Journal of Political Science, 56(1), 1–16

    Article  Google Scholar 

  • Pierson-Balik, D. A. (2003). Race, Class, and Gender in Punitive Welfare Reform: Social Eugenics and Welfare Policy. Race Gender & Class, 10(1), 11–30

    Google Scholar 

  • Rabinowitz, J. L., Sears, D. O., Sidanius, J., & Krosnick, J. A. (2009). Why Do White Americans Oppose Race-Targeted Policies? Clarifying the Impact of Symbolic Racism. Political Psychology, 30(5), 805–828

    Article  Google Scholar 

  • Rothschild, J. E., Howat, A. J., Shafranek, R. M., & Busby, E. C. (2019). Pigeonholing Partisans: Stereotypes of Party Supporters and Partisan Polarization. Political Behavior, 41(2), 423–443

    Article  Google Scholar 

  • Kearney, M. S. (2004). Is There an Effect of Incremental Welfare Benefits on Fertility Behavior? A Look at the Family Cap. Journal of Human Resources, 39(2), 295–325

    Article  Google Scholar 

  • Schneider, A., & Ingram, H. (1993). Social Construction of Target Populations: Implications for Politics and Policy. American Political Science Review, 87(2), 334–347

    Article  Google Scholar 

  • Schneider, M. C., & Bos, A. L. (2011). An Exploration of the Content of Stereotypes of Black Politicians. Political Psychology, 32(2), 205–233

    Article  Google Scholar 

  • Shapiro, R. Y., Kelly, D., Patterson, J., Russell, & Young, J. T. (1987). Public Assistance. Public Opinion Quarterly, 51(1), 120–130

    Article  Google Scholar 

  • Sherman, J. W. (1996). Development and Mental Representation of Stereotypes. Journal of Personality and Social Psychology, 70(6), 1126–1141

    Article  Google Scholar 

  • Smith, R. M., & King, D. (2021). White Protectionism in America. Perspectives on Politics, 19(2), 460–478

    Article  Google Scholar 

  • Stangor, C., & Schaller, M. (2000). Stereotypes as Individual and Collective Representations. In C. Stangor (Ed.), Stereotypes and Prejudice: Essential Readings (pp. 64–82). Philadelphia, PA: Psychology Press

  • Valentino, N. A., Neuner, F. G., & Vandenbroek, L. M. (2018). The Changing Norms of Racial Political Rhetoric and the End of Racial Priming. Journal of Politics, 80(3), 757–771

    Article  Google Scholar 

  • Weaver, R. K. (2000). Ending Welfare as We Know It. Washington, DC: Brookings Institution Press.

    Google Scholar 

  • Winter, N. J. G. (2010). Masculine Republicans and Feminine Democrats: Gender and Americans’ Explicit and Implicit Images of the Political Parties. Political Behavior, 32(4), 587–618

    Article  Google Scholar 

  • Zhirkov, K. (2021a). Estimating and Using Individual Marginal Component Effects from Conjoint Experiments. Political Analysis, 30(2), 236–249

    Article  Google Scholar 

  • Zhirkov, K. (2021b). Perceptions of Stereotypically Immigrant Groups as Darker-Skinned and Politics of Immigration in the United States and Britain. Politics, Groups, and Identities. https://doi.org/10.1080/21565503.2021.1992285. Advance online publication

  • Zhirkov, K. (2021c). Who Are “the Immigrants”? Beliefs about Immigrant Populations and Anti-Immigration Attitudes in the United States and Britain. Social Science Quarterly, 102(1), 228–237

    Article  Google Scholar 

Download references

Acknowledgements

Earlier versions of this project were presented at the Annual Conference of the Southern Political Science Association as well as workshops at the University of Minnesota and Vanderbilt University. We are grateful to Chris Federico, Paul Goren, Cindy Kam, Steve Utych, three anonymous reviewers, and Geoffrey Layman, the editor of Political Behavior, for their helpful comments. This research was funded by a Seed Grant for Social Science Research from the University of Minnesota College of Liberal Arts.

Author information

Authors and Affiliations

  1. University of Minnesota, Minneapolis, United States

    C. Daniel Myers

  2. University of Virginia, Charlottesville, United States

    Kirill Zhirkov

  3. Northeastern University, Boston, United States

    Kristin Lunz Trujillo

  4. Harvard University, Cambridge, United States

    Kristin Lunz Trujillo

Authors
  1. C. Daniel Myers
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kirill Zhirkov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kristin Lunz Trujillo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to C. Daniel Myers.

Ethics declarations

Compliance with Ethical Standards

This research was determined exempt by the University of Minnesota Institutional Review Board on March 30, 2020, Study Number 00009407. The procedures used in this study adhere to the tenets of the Declaration of Helsinki.

Conflict of Interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic Supplementary Material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Myers, C.D., Zhirkov, K. & Lunz Trujillo, K. Who Is “On Welfare”? Validating the Use of Conjoint Experiments to Measure Stereotype Content. Polit Behav (2022). https://doi.org/10.1007/s11109-022-09815-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11109-022-09815-0

Keywords

search

Navigation