Skip to main content

Using Q methodology to augment evaluation of public diplomacy programs

Abstract

The evaluation of public diplomacy programs presents complicated challenges. Discernment of impact is complicated by statistical and practical issues: the nature of individualized personal experiences; the large number of factors that can influence an individual’s response to any experience; the long time horizon required for impact to develop; the influence of politics on defining desired outcomes; and a longrunning debate within the discipline over the proper objectives of exchange programs (mutual understanding for its own sake or the pursuit of foreign policy agendas). Researcher asked current and former participants in the Hubert H. Humphrey Exchange Program at Syracuse University for opinions on the outcomes, benefits, and attributes they expect of exchange programs, and used Q methodology, a scientific method for the study of subjectivity, to discern and describe differing perspectives. Results obtained revealed distinct differences in the opinion patterns of different groups of participants, including identifying participants who valued more agenda- and policy-driven objectives. Demographic information obtained was insufficient to identify the drivers of those groups and additional research, including expansion of the respondent pool and analysis of individual participants, is needed to refine the precise drivers.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. Baltrinic, E.R., M. Jencius, and S.R. Brown. 2017. Excellent Teaching as Viewed by International Teaching Elites. Operant Subjectivity 38 (3/4): 16–28.

    Google Scholar 

  2. Banasick, S. 2019. Ken-Q Analysis (Version 1.0.6) [Software]. https://shawnbanasick.github.io/ken-q-analysis/https://doi.org/10.5281/zenodo.1300201. Accessed Jan 2020.

  3. Broderick, F., T. Deignan, and H. Combes. 2017. An Exploratory Study into the Traumatic Impact of Advanced Cancer among Patients and Partners. Operant Subjectivity 39 (3/4): 1–24.

    Google Scholar 

  4. Brown, S.R. 1980. Political subjectivity: Applications of Q Methodology in Political Science. New Haven, CT, USA: Yale University Press.

    Google Scholar 

  5. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2000. Evaluation of the Hubert H. Humphrey Fellowship Program, Executive Summary. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  6. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2001a. Congress-Bundestag Youth Exchange Program (CBYXX), Executive Summary Briefing. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  7. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2001b. Evaluation Summary: Junior Faculty Development Program. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  8. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2002a. Outcome Assessment of the U.S. Fulbright Scholarship Program Executive Report. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  9. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2002b. Outcome Assessment of International Professional Exchange Programs in Philadelphia, Pennsylvania Executive Summary. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  10. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2003a. Outcome Assessment of the Institute for Representative Government (IRG) Program, Executive Summary. Washington, D.C. Retrieved between December 2020 and January 2021 from https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  11. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2003b. Assessment of the Future Leaders Exchange (FLEX) Program. Washington, D.C. Retrieved between December 2020 and January 2021 from https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  12. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2003c. FREEDOM Support Act Undergraduate Program, One-Page Summary. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  13. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2003d. Evaluation of the English Language Fellows Program, Executive Summary. Washington, D.C. Retrieved between December 2020 and January 2021 from https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  14. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2004. Evaluation of the FREEDOM Support Act Educational Partnerships Program, Executive Report. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  15. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2005. Outcome Assessment of the Visiting Fulbright Student Program Executive Summary. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  16. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2006. International Visitor Leadership Program Outcome Assessment. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  17. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2009. Evaluation of the Youth Exchange and Study Program. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  18. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2012. Evaluation of ECA’s Professional Journalism and Media Exchange Programs: International Visitor Leadership Program; Edward R. Murrow Program For Journalists; Citizen Exchange Program. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  19. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2015a. Evaluation of ECA’s English Language Programs: E-Teacher Scholarship Program. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  20. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2015b. Evaluation of ECA’s English Language Programs: English Language Specialist Program. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  21. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2016. English Access Microscholarship Program Evaluation Report. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  22. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2017a. African Women’s Entrepreneurship Program (AWEP-IVLP) Evaluation Report. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  23. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2017b. Evaluation of the Fulbright Foreign Student Program: Impact on STEM Participants. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  24. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2018. Hubert H. Humphrey Fellowship Program Evaluation Report. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  25. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2019. Alumni Engagement Innovation Fund Evaluation Report. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  26. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2019. Evaluation of Twelve Projects Supported by the U.S. Department of State Ambassadors Fund for Cultural Preservation. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  27. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2020a. Evaluation Report: International Writing Program. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  28. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2020b. Monitoring Data for ECA Framework Indicator Book, Version 2. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  29. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2020c. Evaluation of the Mandela Washington Fellowship 2014–2018. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  30. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2020d. Evaluation Report: Mandela Washington Fellowship of Young African Leaders. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  31. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2020e. Professional Fellows Program FY2012-FY1017 Evaluation Report. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  32. Bureau of Educational and Cultural Affairs, U.S. Department of State. 2020f. Evaluation Report: Sports Visitor Program. Washington, D.C. https://eca.state.gov/impact/evaluation-eca/evaluation-initiative/completed-evaluations. Accessed between Dec 2020 and Jan 2021.

  33. Cull, Nicholas J. 2019. Public Diplomacy: Foundations for Global Engagement in the Digital Age. Medfor, MA: Polity Press.

    Google Scholar 

  34. Cull, Nicholas J. 2008. Public Diplomacy: Taxonomies and Histories. Annals of the American Academy of Political and Social Science 616: 31–54.

    Article  Google Scholar 

  35. DeVore-Wedding, B., J. Thomas, and D. Montgomery. 2018. Determining Optimal Professional Development Formats: A Q Methodology Study of Science Teachers’ Preferences. Operant Subjectivity 40: 1–20.

    Google Scholar 

  36. Hart, J. 2013. Empire of Ideas: The Origins of Public Diplomacy and the Transformation of U.S. Foreign Policy. New York: Oxford University Press.

    Book  Google Scholar 

  37. Hedges, C.D. 2014. The Gender Factor of Survivor: A Q Method Approach. Operant Subjectivity 37 (1/2): 2–22.

    Google Scholar 

  38. Kinsey, D.F. 1991. Selecting a Winning Campaign Slogan. Operant Subjectivity 15 (1): 1–10.

    Google Scholar 

  39. Kinsey, D.F., and R.W. Taylor. 1982. Some Meanings of Political Cartoons. Operant Subjectivity 5 (3): 107–114.

    Google Scholar 

  40. McKeown, B., and Thomas, D.B. 2013. Q Methodology. Sage University Paper Series on Quantitative Applications in the Social Sciences, Vol. 07(66), 2nd ed.

  41. Pahlavi, P.C. 2007. Evaluating Public Diplomacy Programmes. Hague Journal of Diplomacy 2: 255–281.

    Article  Google Scholar 

  42. Pamment, J. 2014. Articulating Influence: Toward a Research Agenda for Interpreting the Evaluation of Soft Power, Public Diplomacy and Nation Brands. Public Relations Review 40: 50–59.

    Article  Google Scholar 

  43. Pike, Steven L., and D.F. Kinsey. 2021. Diplomatic Identity and Communication: Using Q Methodology to Assess Subjective Perceptions of Diplomatic Practitioners. Place Branding and Public Diplomacy. https://doi.org/10.1057/s41254-021-00226-2.

    Article  Google Scholar 

  44. Rhoads, J.C. 2017a. Foreword to the Special Issue: Q Methodology and the Single Case. Operant Subjectivity, 39(1–2): 1–100.

  45. Rhoads, J.C. 2017b. Investigating Political Types, Part I: A Study of “Phil”, the Political Agitator. Operant Subjectivity 39 (1–2): 58–69.

    Google Scholar 

  46. Rhoads, J.C. 2017c. Investigating Political Types, Part II: A Study of “Cole”, the Political Administrator. Operant Subjectivity 39 (1–2): 70–81.

    Google Scholar 

  47. Rhoads, J.C., and D.F. Kinsey. 2008. Democratic Campaign: Perceptions of the Obama-Clinton Nomination Struggle. Operant Subjectivity 31 (1): 100–123.

    Google Scholar 

  48. Rhoads, J.C., D.B. Thomas, and B.F. McKeown. 2017. Rationality vs. Rationale Among Trump Voters in 2016: What Were They Thinking? Operant Subjectivity 39 (3/4): 60–80.

    Google Scholar 

  49. Robinson, T., C. Callahan, and K. Evans. 2014. Why Do We Keep Going Back? A Q Method Analysis of Our Attraction to Horror Movies. Operant Subjectivity 37 (1/2): 41–57.

    Google Scholar 

  50. Scott-Smith, Giles. 2008. Some Thoughts on the Relevance of Exchange Programs within International Relations Theory. ANNALS, AAPSS, 616.

  51. Sevin, E. 2017. A Multilayered Approach to Public Diplomacy Evaluation: Pathways of Connection. Politics & Policy (Statesboro, Ga.) 45 (5): 879–901.

    Article  Google Scholar 

  52. Simpson, N.P., and R.C. Hill. 2020. A Theoretically Inspired Q Methodological Approach for the Evaluation of Public Participation in Environmental Impact Assessment. Operant Subjectivity 42: 1–32.

    Google Scholar 

  53. Snow, Nancy. 2010. Propaganda Inc: Selling America’s Culture to the World. New York: Seven Stories Press.

    Google Scholar 

  54. Sommerfeldt, E.J., and Buhmann, A. 2019. The status quo of evaluation in public diplomacy: insights from the U.S. State Department. Journal of Communication Management 23 (3): 198–212.

    Article  Google Scholar 

  55. Stephenson, W. 1953. The Study of Behavior: Q Technique and Its Methodology. Chicago, IL, USA: University of Chicago Press.

    Google Scholar 

  56. Stephenson, W. 1978. Concourse Theory of Communication. Communication 3: 21–40.

    Google Scholar 

  57. Stephenson, W. 1980. Consciring: A general theory for subjective communicability. In Communication yearbook 4, ed. D. Nimmo, 7–36. New Brunswick, NJ: Transaction.

    Google Scholar 

  58. VQMethod. (n.d.). VQMethod.com. http://www.vqmethod.com. Accessed Jan 2020.

  59. Zenor, J. 2014. Where are Those Good Ol’ Fashioned Values? Reception Analysis of the Offensive Humor on Family Guy. Operant Subjectivity 37 (1/2): 23–40.

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Steven L. Pike.

Ethics declarations

Conflict of interest

The author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Pike, S.L. Using Q methodology to augment evaluation of public diplomacy programs. Place Brand Public Dipl (2021). https://doi.org/10.1057/s41254-021-00229-z

Download citation

Keywords

  • Public diplomacy
  • Evaluation of programs
  • Evaluation methodology
  • Exchange programs
  • Q methodology