Skip to main content
SpringerLink
Log in

Judging the “Vapid and Hollow Charade”: Citizen Evaluations and the Candor of U.S. Supreme Court Nominees

  • Original Paper
  • Published:
Political Behavior Aims and scope Submit manuscript

Abstract

Supreme Court confirmation hearings have been famously called a “vapid and hollow charade” by Elena Kagan. Indeed, perceptions of nominees’ refusal to answer questions about pending cases, prominent political issues, or give any hint of their ideological leanings have become a cornerstone of the modern confirmation process. We investigate the extent to which this reticence to speak of their ideological views, or candor, influences how individuals evaluate the nominee. To this end, we present the results of a survey experiment which examines how support for a hypothetical Supreme Court nominee is affected by information, especially when a nominee is presented to be very forthright or very reticent in answering ideological questions during the confirmation hearings. We find that while partisan compatibility with the president is the main determinant of support for a nominee, nominees who refuse to answer ideological questions can bolster support from respondents who would not support them on partisan grounds. We supplement these findings with observational state-level support data from real nominees over the last 40 years.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. See also Collins and Ringhand (2013), who argue that the confirmation hearings can act as a democratic forum of sorts and that the constitutional issues discussed during these hearings—whether nominees answer questions or not—provide a barometer of the social change the public is ready to accept.

  2. It is worth noting that this dynamic may be a relatively recent one. Farganis and Wedeking (2014) demonstrates that partisanship, in particular, has only been a prominent feature in senators’ votes on nominees since the early 1980s.

  3. While nominees can reveal a wide variety of types of information, we are concerned with ideological candor, both because it is the subject of debate among court observers (i.e., the Ginsburg Rule) and because we believe it is distinct from other types of candor because it references aspects of job performance and future rulings.

  4. Replication data for the experimental and observational results we present below can be found on the Political Behavior Dataverse page doi:10.7910/DVN/NPQ3FB.

  5. The Appendix presents the text of our manipulations and survey instrument. Mechanical Turk workers who opted in to the study were given a link to the survey. All questions outside of the dynamic process tracing environment (DPTE) were programmed using Qualtrics Research Suite.

  6. Our final sample included 631 respondents. Twenty eight of our original respondents claimed to be complete independents/moderates and were thus excluded. The rest of the attrition in our analysis results from missing answers on the dependent variable, between 25 and 50 per question.

  7. Subjects were paid one dollar for their participation. The survey took approximately 10 min to complete.

  8. For interested readers, we present the analyses replicated with the individual items from the index in the Appendix.

  9. This ratio ranges from a low of 8.3% for Judge Bork, meaning that of all his responses categorized as forthcoming, not forthcoming, and qualified, only 8.3% of these were qualified or not forthcoming, to a high of 31.6% for Justice Ginsburg, who was notoriously reticent in her confirmation hearings.

  10. See Kastellec 2010, pp. 771–773 for full estimation details.

  11. The data cover the confirmations of Justices Alito, Breyer, Ginsburg, O’Connor, Rehnquist (Chief Justice), Roberts, Sotomayor, Souter, and Thomas, as well as the failed confirmation of Judge Bork.

  12. Our models also contain state-level controls for state partisanship, ideology, and a revised version of the Stimson (1999) mood measure created by Enns and Koch (2015) which disaggregates mood by state. These variables are included to account for the possibility that Democrats or symbolic or operational liberals are more or less likely to support nominees than Republicans or symbolic or operational conservatives. Only the state-level partisanship control obtains marginal significance, suggesting that partisan or ideological direction exerts little influence over nominee support.

References

  • Anduiza, E., Gallego, A., & Muñoz, J. (2013). Turning a blind eye experimental evidence of partisan bias in attitudes toward corruption. Comparative Political Studies, 46(12), 1664–1692.

    Article  Google Scholar 

  • Badas, A. (2016). The public’s motivated response to Supreme Court decision-making. Justice System Journal, 37(4), 318–330.

    Article  Google Scholar 

  • Bagozzi, R., Gurhan-Canli, Z., & Priester, J. (2002). The social psychology of consumer behavior. Buckingham: Open University Press.

    Google Scholar 

  • Baker, R., Blumberg, S. J., Brick, M., Couper, M. P., Courtright, M., Michael Dennis, J., et al. (2010). Research synthesis: AAPOR report on online panels. Public Opinion Quarterly, 74(4), 711–781.

    Article  Google Scholar 

  • Bartels, L. (2002). Beyond the running tally: Partisan bias in political perceptions. Political Behavior, 24(2), 117–150.

    Article  Google Scholar 

  • Bartels, B. L., & Johnston, C. D. (2013). On the ideological foundations of Supreme Court legitimacy in the American public. American Journal of Political Science, 57(1), 184–199.

    Article  Google Scholar 

  • Berinsky, A., Huber, G. A., & Lenz, G. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20(3), 351–368.

    Article  Google Scholar 

  • Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5.

    Article  Google Scholar 

  • Cacioppo, J. T., & Petty, R. E. (1989). Effects of message repetition on argument processing, recall, and persuasion. Basic and Applied Psychology, 10(1), 3–12.

    Article  Google Scholar 

  • Caldeira, G. A., & Wright, J. R. (1998). Lobbying for justice: Organized interests, Supreme Court nominations, and the United States Senate. American Journal of Political Science, 42(2), 499–523.

    Article  Google Scholar 

  • Cameron, C. M., Cover, A. D., & Segal, J. A. (1990). Senate voting on Supreme Court nominees: A neoinstitutional model. American Political Science Review, 84(2), 525–534.

    Article  Google Scholar 

  • Campbell, A., Converse, P. E., Miller, W. E., & Stokes, D. E. (1960). The American voter. New York: Wiley.

    Google Scholar 

  • Chen, P. G., Appleby, J., Borgida, E., Callaghan, T., Ekstrom, P., Farhart, C., et al. (2014). The Minnesota Multi-Investigator 2012 Presidential Election Panel study. Analyses of Social Issues and Public Policy, 14(1), 78–104.

    Article  Google Scholar 

  • Chen, P. G., & Goren, P. N. (2016). Operational ideology and party identification: A dynamic model of individual-level change in partisan and ideological predispositions. Political Research Quarterly, 69(4), 703–715.

    Article  Google Scholar 

  • Christiansen, D., & Glick, D. M. (2015). Chief Justice Roberts’ health care decision disrobed: The microfoundations of the Supreme Court’s legitimacy. American Journal of Political Science, 59(2), 403–418.

    Article  Google Scholar 

  • Collins, P. M, Jr., & Ringhand, L. A. (2013). Supreme Court confirmation hearings and constitutional change. New York: Cambridge University Press.

    Book  Google Scholar 

  • Combs, D. J. Y., & Keller, P. S. (2010). Politicians and trustworthiness: Acting contrary to self-interest enhances trustworthiness. Basic and Applied Psychology, 32(4), 328–339.

    Article  Google Scholar 

  • Comiskey, M. (1994). The usefulness of Senate confirmation hearings for judicial nominees: The case of Ruth Bader Ginsburg. PS: Political Science and Politics, 27(2), 224–227.

    Google Scholar 

  • Converse, P. E. (1964). The nature of belief systems in the mass publics. In D. E. Apter (Ed.), Ideology and discontent (pp. 206–261). New York: Free Press.

    Google Scholar 

  • Craig, M. A., & Richeson, J. A. (2014). Not in my backyard! Authoritarianism, social dominance orientation, and support for strict immigration policies at home and abroad. Political Psychology, 35(3), 417–429.

    Article  Google Scholar 

  • Crawford, J. T., Brady, J. L., Pilanski, J. M., & Erny, H. (2013). Differential effects of right-wing authoritarianism and social dominance orientation on political candidate support: The moderating role of message framing. Journal of Social and Political Psychology, 1(1), 5–28.

    Article  Google Scholar 

  • Crawford, J. T., & Pilanski, J. M. (2014). The differential effects of right-wing authoritarianism and social dominance orientation on political intolerance. Political Psychology, 35(4), 557–576.

    Article  Google Scholar 

  • Dancey, L., & Sheagley, G. (2013). Heuristics behaving badly: Party cues and voter knowledge. American Journal of Political Science, 57(2), 312–325.

    Article  Google Scholar 

  • Delli Carpini, M. X., & Keeter, S. (1991). Stability and change in the U.S. public’s knowledge of politics. Public Opinion Quarterly, 55(4), 583–612.

    Article  Google Scholar 

  • Enns, P. K., & Koch, J. (2015). State policy mood: The importance of over-time dynamics. State Politics and Policy Quarterly, 15(4), 436–446.

    Article  Google Scholar 

  • Epstein, L., Lindsädt, R., Segal, J. A., & Westerland, C. (2006). The changing dynamics of Senate voting on Supreme Court nominees. Journal of Politics, 68(2), 296–307.

    Article  Google Scholar 

  • Farganis, D., & Wedeking, J. P. (2011). “No Hints, No Forecasts, No Previews”: An empirical analysis of Supreme Court nominee candor from Harlan to Kagan. Law and Society Review, 45(3), 525–559.

    Article  Google Scholar 

  • Farganis, D., & Wedeking, J. P. (2014). Supreme Court confirmation hearings in the U.S. Senate: Reconsidering the charade. Ann Arbor, MI: University of Michigan Press.

    Book  Google Scholar 

  • Gaziano, T. F., & Meese III, E. (2005). The Ginsburg rule. http://www.heritage.org/research/commentary/2005/07/the-ginsburg-rule.

  • Gibson, J. L. (2010). Expecting justice and hoping for empathy. Law and Courts Newsletter, 20(3), 21–25.

    Google Scholar 

  • Gibson, J. L., & Caldeira, G. A. (1992). The etiology of public support for the Supreme Court. American Journal of Political Science, 36(3), 635–664.

    Article  Google Scholar 

  • Gibson, J. L., & Caldeira, G. A. (2009a). Citizens, courts, and confirmations. Princeton, NJ: Princeton University Press.

    Book  Google Scholar 

  • Gibson, J. L., & Caldeira, G. A. (2009b). Confirmation politics and the legitimacy of the U.S. Supreme Court: Institutional loyalty, positivity bias, and the Alito nomination. American Journal of Political Science, 53(1), 139–155.

    Article  Google Scholar 

  • Gimpel, J. G., & Ringel, L. S. (1995). Understanding Court nominee evaluation and approval: Mass opinion in the Bork and Thomas cases. Political Behavior, 17(2), 135–153.

    Article  Google Scholar 

  • Goren, P. N. (2002). Character weakness, partisan bias, and presidential evaluation. American Journal of Political Science, 46(3), 627–641.

    Article  Google Scholar 

  • Goren, P. N. (2005). Party identification and core political values. American Journal of Political Science, 49(4), 881–896.

    Article  Google Scholar 

  • Goren, P. N. (2007). Character weakness, partisan bias, and presidential evaluation: Modifications and extensions. Political Behavior, 29(3), 305–325.

    Article  Google Scholar 

  • Guliuzza, F, I. I. I., Reagan, D. J., & Barrett, D. M. (1992). Character, competency, and constitutionalism: Did the Bork nomination represent a fundamental shift in confirmation criteria? Marquette Law Review, 75(2), 409–437.

    Google Scholar 

  • Hetherington, M. J., & Rudolph, T. J. (2008). Priming, performance, and the dynamics of political trust. Journal of Politics, 70(2), 498–512.

    Article  Google Scholar 

  • Hibbing, J. R., & Theiss-Morse, E. (Eds.). (2001). What is it about government that Americans dislike?. Cambridge: Cambridge University Press.

    Google Scholar 

  • Hopkins, D. (2015). The upside of accents: Language, inter-group difference, and attitudes toward immigration. British Journal of Political Science, 45(3), 531–557.

    Article  Google Scholar 

  • Huff, C., & Tingley, D. (2015). Who are these people? Evaluating the demographic characteristics and political preferences of MTurk survey respondents. Research and Politics, 2(3), 1–12.

    Article  Google Scholar 

  • Johnson, T. R., & Roberts, J. M. (2004). Presidential capital and the Supreme Court confirmation process. Journal of Politics, 66(3), 663–683.

    Article  Google Scholar 

  • Kagan, E. (1995). Confirmation messes, old and new. The University of Chicago Law Review, 62(2), 919–942.

    Article  Google Scholar 

  • Kastellec, J. P., Lax, J. R., & Phillips, J. H. (2010). Public opinion and Senate confirmation of Supreme Court nominees. Journal of Politics, 72(3), 767–784.

    Article  Google Scholar 

  • Kelly, D. E. (2010). Predicting public approval of Supreme Court nominees: Examining factors influencing mass public opinion of stealth nominees in the post-Bork era. Political Science, Iowa State University.

  • Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108(3), 335–358.

    Article  Google Scholar 

  • Lau, R. R., & Redlawsk, D. P. (2006). How voters decide. New York: Cambridge University Press.

    Book  Google Scholar 

  • Lewis-Beck, M., Jacoby, W. G., Norpoth, H., & Weisberg, H. (2008). The American voter revisited. Ann Arbor, MI: University of Michigan Press.

    Book  Google Scholar 

  • Lodge, M., & Taber, C. S. (2013). The rationalizing voter. New York: Cambridge University Press.

    Book  Google Scholar 

  • Lord, C. G., Ross, L., & Lepper, M. R. (1979). Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37(11), 2098–2109.

    Article  Google Scholar 

  • Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44(1), 1–23.

    Article  Google Scholar 

  • Munro, G. D., & Ditto, P. H. (1997). Biased assimilation, attitude polarization, and affect in reactions to stereotype-relevant scientific information. Personality and Social Psychology Bulletin, 23(6), 636–653.

    Article  Google Scholar 

  • NBC News/Wall Street Journal. (2005). Poll conducted September 9–September 12, 2005 and based on 1,013 telephone interviews.

  • Nemacheck, C. L. (2008). Strategic selection: Presidential nomination of Supreme Court Justices from Herbert Hoover through George W. Bush. Charlottesville, VA: University of Virginia Press.

    Google Scholar 

  • Overby, L. M., Henschen, B. M., Walsh, M. H., & Strauss, J. (1992). Courting constituents? An analysis of the Senate confirmation vote on Justice Clarence Thomas. American Political Science Review, 86(4), 997–1003.

    Article  Google Scholar 

  • Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23(3), 184–188.

    Article  Google Scholar 

  • Petty, R. E., & Cacioppo, J. T. (1986). Communication and persuasion: Central and peripheral routes to attitude change. New York: Springer.

    Book  Google Scholar 

  • Petty, R. E., & Cacioppo, J. T. (1996). Attitudes and persuasion: Classic and contemporary approaches. Boulder, CO: Westview Press.

    Google Scholar 

  • Ruckman, P. S, Jr. (1993). The Supreme Court, critical nominations, and the Senate confirmation process. Journal of Politics, 55(3), 793–805.

    Article  Google Scholar 

  • Segal, J. (1987). Senate confirmation of Supreme Court Justices: Partisan and institutional politics. Journal of Politics, 49(4), 998–1015.

    Article  Google Scholar 

  • Staff. (2016). Donald Trump says the law is settled on gay marriage but not on abortion. The Economist, November 15, 2016.

  • Stimson, J. A. (1999). Public opinion in America: Moods, cycles and swings (2nd ed.). Boulder, CO: Westview Press.

    Google Scholar 

  • Taber, C. S., Cann, D., & Kucsova, S. (2009). The motivated processing of political arguments. Political Behavior, 31(2), 137–155.

    Article  Google Scholar 

  • Taber, C. S., & Lodge, M. (2006). Motivated skepticism in the evaluation of political beliefs. American Journal of Political Science, 50(3), 755–769.

    Article  Google Scholar 

  • Wedeking, J. P., & Farganis, D. (2010). The candor factor: Does nominee evasiveness affect Judiciary Committee support for Supreme Court nominees? Hofstra Law Review, 39(2), 329–368.

    Google Scholar 

  • Zaller, J. R. (1991). Information, values, and opinion. American Political Science Review, 85(4), 1215–1237.

    Article  Google Scholar 

  • Zaller, J. R. (1992). The nature and origins of mass opinion. Cambridge: Cambridge University Press.

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Calgary, Calgary, Canada

    Philip G. Chen

  2. Loyola University, Chicago, USA

    Amanda C. Bryan

Authors
  1. Philip G. Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amanda C. Bryan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amanda C. Bryan.

Appendix

Appendix

Vignettes

These are the short biographical vignettes the respondents read before entering the DPTE. The differences, given the different conditions are bolded.

James/Mary Walker is a nominee for the Supreme Court. He was nominated by Benjamin Halston, the current Democratic/Republican president. He enjoys the support of the Current Chair of the Senate Judiciary Committee, a Democratic/Republican senator from Kansas.

Reticent Nominee Manipulation

WASHINGTON-DC: Today the Senate Judiciary Committee began its third day of confirmation hearings with Judge James/Mary Walker, President Halston’s nominee to the Supreme Court. Questions today centered on Judge Walker’s judicial philosophy. Many senators asked specific questions about how the judge would rule on important issues that may face the Court in the coming years including issues of gun rights, free speech, detainment of enemy combatants, and the rights of corporations. Keeping with a long-standing tradition, Judge Walker was reticent to answer any questions on cases that could potentially be heard by the Court in the future. While some have praised this decision as respecting important norms of judicial independence, others criticized Judge Walker’s answers today as evasive or even disingenuous. Senator Peter Kimble (I-ME), a member of the Judiciary Committee remarked this afternoon, “how are we supposed to know whether we support him/her if he/she won’t even answer the most basic questions”. Below is a transcript of an exchange today between Kimble and Judge Walker where Kimble pressed Walker on his/her views on gun control.

SENATOR KIMBLE: Judge Walker, your record on the right of the people to bear arms is a bit sparse. You have decided few cases on the issue and I know my constituents are particularly concerned about whether you believe the Second Amendment protects the right to own and carry a firearm. Could you say a bit more about your views?

JUDGE WALKER: Actually, Senator, I know there are several Second Amendment cases in the pipeline that could hit the Court’s docket in the next several years and I am concerned that any answer I give to that question would be a violation of my ethical responsibility to remain impartial in that case. With that in mind, I respectfully decline to say anything more.

SENATOR KIMBLE: You cannot even tell us how you would approach the case? How far you think the Second Amendment extends?

JUDGE WALKER: No Senator, I’m afraid I cannot. It just wouldn’t be ethical.

SENATOR KIMBLE: Can you at least say whether you would follow the established Court precedent that protects the right to a firearm?

JUDGE WALKER: I can say I believe in the Court’s tradition of respecting established precedent. But no, I cannot say whether I, or the rest of the Court would apply any past case to any hypothetical future case.

SENATOR KIMBLE: Well, I guess if you can’t even answer a most basic question like that one, I don’t have anything more. Mr. Chairman, I yield the remainder of my time.

Forthcoming/Conservative Nominee Manipulation

WASHINGTON-DC: Today the Senate Judiciary Committee began its third day of confirmation hearings with Judge James/Mary Walker, President Halston’s nominee to the Supreme Court. Questions today centered on Judge Walker’s judicial philosophy. Many senators asked specific questions about how the judge would rule on important issues that may face the Court in the coming years including issues of gun rights, free speech, detainment of enemy combatants, and the rights of corporations. Breaking with a long-standing tradition, Judge Walker was quite forthcoming about his/her views on cases that could potentially be heard by the Court in the future. While some have praised this decision as refreshingly honest and forthright, others criticized Judge Walker’s answers today as unethical and violating established traditions set to protect the Court’s independence. Senator Peter Kimble (I-ME), a member of the Judiciary Committee remarked this afternoon, “I really appreciated that Judge Walker was willing to express his/her views, even when people on either side may not agree with them.” Below is a transcript of an exchange today between Kimble and Judge Walker where Kimble pressed Walker on his/her views on gun control.

SENATOR KIMBLE: Judge Walker, your record on the right of the people to bear arms is a bit sparse. You have decided few cases on the issue and I know my constituents are particularly concerned about whether you believe the Second Amendment protects the right to own and carry a firearm. Could you say a bit more about your views?

JUDGE WALKER: Of course, Senator. I have long believed the Second Amendment protects a right of the people to keep and bear arms. Any reasonable reading of the Amendment makes clear the Founders intended gun ownership to be a fundamental right.

SENATOR KIMBLE: So how far do you think the Second Amendment extends? May Congress place any limits on gun ownership or possession?

JUDGE WALKER: Well, I’m sure there are probably reasonable limits. And the Court would decide those on a case-by-case basis. But I believe any decision must start from the premise that owning a gun is a fundamental right and we should approach limitations on that right with the highest level of scrutiny.

SENATOR KIMBLE: So you would follow the established Court precedent that protects the right to a firearm?

JUDGE WALKER: I believe in the Court’s tradition of respecting established precedent. And yes, I believe the Court was right in deciding those cases and I would absolutely adhere to that precedent in future cases.

Forthcoming/Liberal Nominee Manipulation

WASHINGTON-DC: Today the Senate Judiciary Committee began its third day of confirmation hearings with Judge James/Mary Walker, President Halston’s nominee to the Supreme Court. Questions today centered on Judge Walker’s judicial philosophy. Many senators asked specific questions about how the judge would rule on important issues that may face the Court in the coming years including issues of gun rights, free speech, detainment of enemy combatants, and the rights of corporations. Breaking with a long-standing tradition, Judge Walker was quite forthcoming about his/her views on cases that could potentially be heard by the Court in the future. While some have praised this decision as refreshingly honest and forthright, others criticized Judge Walker’s answers today as violating established traditions set to protect the Court?s independence. Senator Peter Kimble (I-ME), a Member of the Judiciary Committee remarked this afternoon, “I really appreciated that Judge Walker was willing to express his/her views, even when people on either side may not agree with them.” Below is a transcript of an exchange today between Kimble and Judge Walker where Kimble pressed Walker on his/her views on gun control.

SENATOR KIMBLE: Judge Walker, your record on the right of the people to bear arms is a bit sparse. You have decided few cases on the issue and I know my constituents are particularly concerned about whether you believe the Second Amendment protects the right to own and carry a firearm. Could you say a bit more about your views?

JUDGE WALKER: Of course, Senator. I have long believed the Second Amendment was intended to establish a militia and nothing more. It does not protect the right to own or possess a firearm. Any reasonable reading of the Amendment makes clear the Founders never intended gun ownership to be a fundamental right.

SENATOR KIMBLE: So how far do you think the Second Amendment extends? May Congress place any limits on gun ownership or possession?

JUDGE WALKER: Absolutely. And the Court would decide those on a case-by-case basis. But I believe any decision must start from the premise that owning a gun is a not fundamental right. As such, government restrictions on the privilege of gun ownership are not only constitutional but also probably make for a safer society.

SENATOR KIMBLE: So you would not follow the established Court precedent that protects the right to a firearm?

JUDGE WALKER: While I believe in the Court’s tradition of respecting established precedent, the Court does not get it right every time. And when the Court errs, as it did in these cases, it is the responsibility of future justices to correct the error by overturning that ruling.

Dependent Variables

Feeling Thermometer

Please rate your feelings about the nominee using this slider. 0 indicates you feel very negative about the nominee, 100 indicates you feel very positive about the nominee, and 50 means you feel neither negative nor positive about the nominee.

Support

Based on what you know about the nominee, [James/Mary] Walker, do you currently support or oppose the nominee?

  • Support

  • Oppose

  • Neither Support nor Oppose

    • How strongly or weakly do you [support/oppose] the nominee?

    • Strongly Support/Oppose

    • Moderately Support/Oppose

    • Weakly Support/Oppose

Qualification

How qualified do you believe [NOMINEE NAME] is to serve on the U.S. Supreme Court?

  • Completely Qualified

  • Highly Qualified

  • Moderately Qualified

  • Somewhat Qualified

  • Barely Qualified

  • Not Qualified At All

Mechanical Turk and Sample Specifics

While Mechanical Turk is not a nationally representative sample, the convenience sample is useful for experiments in political science. Indeed, the non-representative nature of online opt-in surveys essentially eliminates the ability to generalize results to the population without additional data drawn from representative samples. In particular, the Baker et al. (2010) recommendations urge researchers to avoid panels (like MTurk) “when one of the research objectives is to accurately estimate population values.” We do not disagree with this conclusion, and urge readers to understand our experimental results as working towards uncovering the mechanisms underlying the evaluation of nominees rather than understanding the behavior of the general population. Importantly, Baker et al. (2010) speak to the usefulness of opt-in panels for uncovering underlying public opinion, but do not address the use of these panels for experiments. Thus, while the external validity of experiments conducted on MTurk is relatively low, the internal validity of the study is affected only by the experimental design, not the sample composition.

Additionally, Mechanical Turk has been used extensively in both political science and other behavioral disciplines to answer questions experimentally. These investigations have shown Mechanical Turk samples to be more representative than traditional student or convenience samples and the quality of the data appears to be quite high (Berinsky et al. 2012; Buhrmester et al. 2011; Mason and Suri 2012; Paolacci and Chandler 2014). Furthermore, scholars have used the platform to investigate a number of different psychological and attitudinal traits, including racial resentment and authoritarianism (Craig and Richeson 2014; Crawford et al. 2013; Crawford and Pilanski 2014; Hopkins 2015). Thus, although the subject pool is more liberal, Democratic, and younger than the general population, carefully constructed experiments will not necessarily suffer for these sample deficiencies.

Despite the widespread usage of Mechanical Turk for social science experiments, we would be remiss if we did not address a key demographic issue with these samples. Political knowledge or sophistication is consistently much higher among Mechanical Turk workers than it is in the general population (Berinsky et al. 2012; Huff and Tingley 2015). We do not deny that this bias exists in the Mechanical Turk samples, but we have taken care to minimize its impact. First, by utilizing an experimental approach, differences between conditions are unaffected by sophistication biases in the sample generally. Second, although we believe the tendency of sophisticated individuals to rely on abstract values (Delli Carpini and Keeter 1991; Zaller 1992) makes them more likely to rely on partisanship and less likely to consider new information in their evaluation of nominees, we are sensitive to work that shows sophisticated individuals are more prone to motivated reasoning than those with lower levels of sophistication (Taber and Lodge 2006). Therefore, we again caution readers to interpret our experimental findings primarily as a comparison between conditions, as well as to evaluate the experimental results in concert with our observational results.

Our sample, which consisted of 727 U.S. adults, closely matched Mechanical Turk samples reported in other findings, suggesting that our sample was representative of Mechanical Turk workers (Chen et al. 2014). Our sample was well-educated (62% held at least a 2-year degree), white (83%), Democratic (62% Democratic, 24% Republican, 14% Independent), liberal (54% liberal, 23% conservative, 23% moderate), and young (mean age of 36.4).

Readers are likely to note the noticeable bias towards Democrats and liberals in our sample. While this is typical of Mechanical Turk samples, it does underscore the need to utilize experimental methods with Mechanical Turk. We also note that, although we would like to be able to analyze the independent effects of candor on Democrats/liberals and Republicans/conservatives, the non-representative nature of Mechanical Turk prevents this. Thus, there is some anecdotal evidence to suggest conservatives may be more acceptive of ideological justices than liberals (indeed, President Trump promised a pro-life litmus test for all his Supreme Court nominees, to the satisfaction of the conservative base Staff 2016), we are unfortunately unable to test this proposition with these data.

While the partisan and ideological composition of the sample may at first appear concerning, recall that our experiment utilized prior responses to block individuals into pro- or counter-attitudinal conditions. Thus, one needs not believe that the partisans or ideologues in the sample are representative of the broader population; instead, we only need to believe that the influence of pro- or counter-attitudinal information is similar among Mechanical Turk workers as it is in the general population. Prior research strongly suggests that psychological mechanisms work similarly in both the general and Mechanical Turk populations.

Robustness Check: Single Item Dependent Variables

As noted, we rely on an aggregated index of nominee support in the paper analysis. Nonetheless, we note that the results are not unique to this construction of the dependent variable. If we use the three component pieces of this dependent variable as single-item indicators of support, we see a similar pattern, presented in Table 3.

As Table 3 shows, the same pattern of pro- and counter-attitudinal nominee evaluation occurs for all three variables. In addition, for the feeling thermometer and nominee support variables, we see a significant interaction term between the reticent condition and the counter-attitudinal condition, just as we expected with H2. Only for the question regarding the nominee’s qualifications is the interaction term insignificant. Nonetheless, we are confident in the reliability of our index scale, based on the reliability coefficients and principal-factor analysis reported in the paper.

Table 3 Support single-item results
Full size table

Robustness Check: Experimental Manipulation Attention Check

We are sensitive to concerns that respondents may not have paid close attention to the experimental manipulation in the DPTE. To assuage these concerns, we can re-run the models with a sample constricted by the amount of time spent reading the experimental manipulation. As Table 4 shows, when comparing the full sample (column 1) to a sample of respondents who spent at least 30 s reading the manipulation (column 2), the results are nearly identical. As we restrict the sample to those who spent less than 30 s with the manipulation (column 3), we see that the interaction term becomes insignificant. If we further restrict the sample to those who spent less than 15 s reading the manipulation (column 4), we see that the interaction term remains insignificant.

Table 4 Sample comparison, nominee support index
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, P.G., Bryan, A.C. Judging the “Vapid and Hollow Charade”: Citizen Evaluations and the Candor of U.S. Supreme Court Nominees. Polit Behav 40, 495–520 (2018). https://doi.org/10.1007/s11109-017-9411-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11109-017-9411-y

Keywords

Search

Navigation