A multi-country, multi-sector replication challenge to the validity of the cultural tightness-looseness measure

  • Len J. TreviñoEmail author
  • Carolyn P. Egri
  • David A. Ralston
  • Irina Naoumova
  • Olivier Furrer
  • Yongjuan Li
  • Fidel León Darder
  • María Teresa de la Garza Carranza


In this study, we assess the internal and external validity of Gelfand et al.’s (2011) recently developed measure of cultural tightness-looseness (CTL). Our study is composed of six countries (China, Mexico, Netherlands Russia, Spain, U.S.) with three subsamples (business professionals, K-12 teachers, college students) per country. For these 18 subsamples, confirmatory factor analyses failed to support the unidimensional structure of the 6-item CTL measure. Exploratory factor analyses provided further evidence that the 6-item CTL measure does not have a unidimensional structure across cultures. Additionally, inter-rater agreement analyses did not support the use of aggregated scores to construct country-level scores for the CTL index. We also found that country rankings of CTL scores (in total and for subsamples) were substantively different from those reported by Gelfand et al. (2011). Further country-level correlation analyses yielded mixed support for the external validity of the CTL scores. We conclude with a commentary on the implications of our study for cross-cultural research.


Cultural tightness-looseness measure Validation study Measurement issues Cross-cultural research 



We would like to thank the Associate Editor, David Ahlstrom, and the two reviewers, whose comments were instrumental in the development of this article.


  1. Alesina, A., Devleeschauwer, A., Easterly, W., Kurlat, S., & Wacziarg, R. 2003. Fractionalization. Journal of Economic Growth, 8(2): 155–194.Google Scholar
  2. Au, K. Y. 1999. Intra-cultural variation: Evidence and implications for international Business. Journal of International Business Studies, 30(4): 799–812.Google Scholar
  3. Barro, R. J., & McCleary, R. M. 2003. Religion and economic growth across countries. American Sociological Review, 68(5): 760–781.Google Scholar
  4. Baumgartner, H., Weijters, B., & Pieters, R. 2018. Misresponse to survey questions: A conceptual framework and empirical test of the effects of reversals, negations, and polar opposite core concepts. Journal of Marketing Research, 55(6): 869–883.Google Scholar
  5. Bello, D., Leung, K., Radebaugh, L., Tung, R., & Van Witteloostuijn, A. 2009. From the editors: Student samples in international business research. Journal of International Business Studies, 40(3): 361–364.Google Scholar
  6. Bond, M. H., Au, K. L. A., Tong, K.-K., Murakami, F., Yamaguchi, S., Bierbrauer, G., Singelis, T. M., Boroer, M., Boen, F., Lambert, S. M., Ferreira, M. C., Noels, K. A., van Bavel, J., Safdar, S., Zhang, J., Chen, L., Solcova, I., Stetovska, I., Niit, T., Niit, K.-K., Hurme, H., Boling, M., Franchi, V., Magradze, G., Javakhishvili, N., Boehnke, K., Klinger, E., Huang, X., Fulop, M., Berkics, M., Panagiotopoulou, P., Sriram, S., Chaudhary, N., Ghosh, A., Vohra, N., Iqbal, D. F., Kurman, J., Thein, R. D., Comunian, A. L., Son, K. A., Austers, I., Harb, C., Odusanya, J. O. T., Ahmed, Z. A., Ismail, R., van de Vijver, F., Ward, C., Mogaji, A., San, D. L., Khan, M. J. Z., Cabanillas, W. E., Sycip, L., Neto, F., Cabecinhas, R., Xavier, P., Dinca, M., Lebedeva, N., Viskochil, A., Ponomareva, O., Burgess, S. M., Oceja, L., Campo, S., Hwang, K.-K., D'Souza, J. B., Ataca, B., Furnham, A., & Lewis, J. R. 2004. Culture-level dimensions of social axioms and their correlates across 41 cultures. Journal of Cross-Cultural Psychology, 35(5): 548–570.Google Scholar
  7. Brislin, R. W. 1986. The wording and translation of research instruments. In W. J. Lonner, & J. W. Berry (Eds.). Field methods in cross-cultural research: 137–164. Beverly Hills: Sage.Google Scholar
  8. Brown, R. D., & Hauenstein, N. M. A. 2005. Interrater agreement reconsidered: An alternative to the rwg indices. Organizational Research Methods, 8(2): 165–184.Google Scholar
  9. Byrne, B. M. 2008. Testing for multigroup equivalence of a measuring instrument: A walk through the process. Psicothema, 20(4): 872–882.Google Scholar
  10. Cameron, K. S., & Quinn, R. E. 2006. Diagnosing and changing organizational culture, Revised ed. San Francisco:Jossey-Bass.Google Scholar
  11. Chan, D. K., Gelfand, M. J., Triandis, H. C., & Tseng, O. 1996. Tightness-looseness revisited: Some preliminary analyses in Japan and the United States. International Journal of Psychology, 31(1): 1–12.Google Scholar
  12. Chelminski, P., & Coulter, R. A. 2007. On market mavens and consumer self-confidence: A cross-cultural study. Psychology & Marketing, 24(1): 69–91.Google Scholar
  13. Cheung, G. W., & Rensvold, R. B. 2002. Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2): 233–255.Google Scholar
  14. Church, A. T., Katigbak, M. S., Locke, K. D., Zhang, H. S., Shen, J. L., Vargas-Flores, J. D., Ibanez-Reyes, J., Tanaka-Matsumi, J., Curtis, G. J., Cabrera, H. F., Mastor, K. A., Alvarez, J. M., Ortiz, F. A., Simon, J. Y. R., & Ching, C. M. 2013. Need satisfaction and well-being: Testing self-determination theory in eight cultures. Journal of Cross-Cultural Psychology, 44(4): 507–534.Google Scholar
  15. CIA. 2018. The world factbook. Accessed Dec. 23, 2018.
  16. Colakoglu, S., & Caligiuri, P. 2008. Cultural distance, expatriate staffing and subsidiary performance: The case of US subsidiaries of multinational corporations. International Journal of Human Resource Management, 19(2): 223–239.Google Scholar
  17. Cortina, J. M., Aguinis, H., & DeShon, R. P. 2017. Twilight of dawn or of evening? A century of research methods in journal of applied psychology. Journal of Applied Psychology, 102(3): 274–290.Google Scholar
  18. Devlin, H. 2018. Attempt to replicate major social scientific findings of past decade fails. The Guardian (Intl. Ed.). . Accessed Dec. 10, 2018.
  19. Diamantopoulos, A., & Papadopoulos, N. 2010. Assessing the cross-national invariance of formative measures: Guidelines for international business researchers. Journal of International Business Studies, 41(2): 360–370.Google Scholar
  20. Fearon, J. D. 2003. Ethnic and cultural diversity by country. Journal of Economic Growth, 8(2): 195–222.Google Scholar
  21. Fischer, R. 2004. Standardization to account for cross-cultural response bias: A classification of score adjustment procedures and review of research in JCCP. Journal of Cross-Cultural Psychology, 35(3): 263–282.Google Scholar
  22. Fischer, R. 2009. Where is culture in cross cultural research? An outline of a multilevel research process for measuring culture as a shared meaning system. International Journal of Cross Cultural Management, 9(1): 25–29.Google Scholar
  23. Fornell, C., & Larcker, D. F. 1981. Structural equation models with unobservable variables and measurement error: Algebra and statistics. Journal of Marketing Research, 18(3): 382–388.Google Scholar
  24. Gelfand, M. J. 2012. Culture’s constraints: International differences in the strength of social norms. Current Directions in Psychological Science, 21(6): 420–424.Google Scholar
  25. Gelfand, M. J., Leslie, L. M., & Fehr, R. 2008. To prosper, organizational psychology should…adopt a global perspective. Journal of Organizational Behavior, 29(4): 493–517.Google Scholar
  26. Gelfand, M. J., Nishii, L. H., & Raver, J. L. 2006. On the nature and importance of cultural tightness-looseness. Journal of Applied Psychology, 91(6): 1225–1244.Google Scholar
  27. Gelfand, M. J., Raver, J. L., Nishii, L., Leslie, L. M., Lun, J., Lim, B. C., Duan, L., Almaliach, A., Ang, S., Arnadottir, J., Aycan, Z., Boehnke, K., Boski, P., Cabecinhas, R., Chan, D., Chhokar, J., D’Amato, A., Ferrer, M., Fischlmayr, I. C., Fischer, R., Fulop, M., Georgas, J., Kashima, E. S., Kashima, Y., Kim, K., Lempereur, A., Marquez, P., Othman, R., Overlaet, B., Panagiotopoulou, P., Peltzer, K., Perez-Florizno, L. R., Ponomarenko, L., Realo, A., Schei, V., Schmitt, M., Smith, P. B., Soomro, N., Szabo, E., Taveesin, N., Toyama, N., Van de Vliert, E., Vohra, N., Ward, C., & Yamaguchi, S. 2011. Differences between tight and loose cultures: A 33-nation study. Science, 332: 1100–1104 Supplemental supporting online material, Scholar
  28. Gong, Y., Chow, I. H. S., & Ahlstrom, D. 2011. Cultural diversity in China: Dialect, job embeddedness, and turnover. Asia Pacific Journal of Management, 28(2): 221–238.Google Scholar
  29. Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. 2006. Multivariate data analysis, 6th ed. Upper Saddle River:Prentice-Hall.Google Scholar
  30. Harrington, J. R., Boski, P., & Gelfand, M. J. 2015. Culture and national well-being: Should societies emphasize freedom or constraint? PLoS ONE, 10(6): e0127173. Scholar
  31. Harrington, J. R., & Gelfand, M. J. 2014. Tightness-looseness across the 50 United States. Proceedings of the National Academy of Sciences of the United States of America (PNAS), 111(22): 7990–7995.Google Scholar
  32. Harzing, A.-W. 2006. Response styles in cross-national survey research: A 26-country study. International Journal of Cross Cultural Management, 6(2): 243–266.Google Scholar
  33. Harzing, A.-W., Baldueza, J., Barner-Rasmussen, W., Barzantny, C., Canabal, A., Davila, A., Espejo, A., Ferreira, R., Giroud, A., Koester, K., Liang, Y.-K., Mockaitis, A., Morleym, M. J., Myloni, B., Odusanya, J. O. T., O’Sullivan, S. L., Palaniappan, A. K., Prochno, P., Choudhury, S. R., Saka-Helmhout, A., Siengthai, S., Viswat, L., Soydas, A. U., & Zander, L. 2009. Rating versus ranking: What is the best way to reduce response and language bias in cross-national research? International Business Review, 18(4): 417–432.Google Scholar
  34. Hinkin, T. R. 1998. A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1): 104–121.Google Scholar
  35. Hoffmann, S., Mai, R., & Cristescu, A. 2013. Do culture-dependent response styles distort substantial relationships? International Business Review, 22(5): 814–827.Google Scholar
  36. Hofstede, G. 2001. Culture’s consequences: International differences in work-related values. Beverly Hills:Sage.Google Scholar
  37. Hofstede, G., Hofstede, G. J., & Minkov, M. 2010. Cultures and organizations: Software of the mind, 3rd ed. New York:McGraw-Hill.Google Scholar
  38. House, R. J., Hanges, P. J., Javidan, M., Dorfman, P., & Gupta, V. 2004. Culture, leadership, and organizations: The GLOBE study of 62 societies. Thousand Oaks:Sage.Google Scholar
  39. Hu, L., & Bentler, P. M. 1999. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6: 1–55.Google Scholar
  40. Hui, C. H., & Triandis, H. C. 1989. Effects of culture and response format on extreme response style. Journal of Cross-Cultural Psychology, 20(3): 296–309.Google Scholar
  41. Inglehart, R., & Welzel, C. 2005. Modernization, cultural change, and democracy: The human developmentsSequence. New York:Cambridge University Press.Google Scholar
  42. International Monetary Fund. 2017. World economic outlook database. Accessed April, 17, 2018.
  43. International Organization for Migration 2018. World migration report 2018. Geneva:International Organization for Migration, UN Migration Agency Accessed April 17, 2018.
  44. Ioannidis, J. 2015. Psychology experiments are failing the replication test – For good reason. Guardian, August 28. Accessed March 15, 2018.
  45. James, L. R., Demaree, R. G., & Wolf, G. 1984. Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69(1): 85–98.Google Scholar
  46. James, L. R., Demaree, R. G., & Wolf, G. 1993. rwg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78(2): 306–309.Google Scholar
  47. Kuhn, T. S. 1962. The structure of scientific revolutions. Chicago:University of Chicago Press.Google Scholar
  48. Lenartowicz, T., & Roth, K. 1999. A framework for culture assessment. Journal of International Business Studies, 30(4): 781–798.Google Scholar
  49. Leung, K., & Bond, M. H. 2004. Social axioms: A model of social beliefs in multi-cultural perspective. Advances in Experimental Social Psychology, 36: 119–197.Google Scholar
  50. Licht, A. N., Goldschmidt, C., & Schwartz, S. H. 2007. Culture rules: The foundations of the rule of law and other norms of governance. Journal of Comparative Economics, 35(4): 659–688.Google Scholar
  51. Luiz, J. M. 2015. The impact of ethno-linguistic fractionalization on cultural measures: Dynamics, endogeneity and modernization. Journal of International Business Studies, 46(9): 1080–1098.Google Scholar
  52. Malecki, E. J., & Ewers, M. C. 2007. Labor migration to world cities: With a research agenda for the Arab gulf. Progress in Human Geography, 31(4): 467–484.Google Scholar
  53. McGorry, S. Y. 2000. Measurement in a cross-cultural environment: Survey translation issues. Qualitative Market Research, 3(2): 74–81.Google Scholar
  54. McSweeney, B. 2002. Hofstede’s model of national cultural differences and their consequences: A triumph of faith – A failure of analysis. Human Relations, 55(1): 89–118.Google Scholar
  55. Meyer, R. D., Mumford, T. V., Burrus, C. J., Campion, M. A., & James, L. R. 2014. Selecting null distributions when calculating rwg: A tutorial and review. Organizational Research Methods, 17(3): 324–345.Google Scholar
  56. Minkov, M., Blagoev, V., & Hofstede, G. 2012. The boundaries of culture: Do questions about societal norms reveal cultural differences? Journal of Cross-Cultural Psychology, 44(7): 1094–1106.Google Scholar
  57. Minkov, M., & Hofstede, G. 2013. Cross-cultural analysis: The science and art of comparing the world's modern societies and their cultures. Los Angeles:Sage.Google Scholar
  58. Moonsinghe, R., Khoury, M. J., & Janssens, A. C. J. W. 2007. Most published research findings are false—But a little replication goes a long way. PLOS Medicine, 4(2): 218–221.Google Scholar
  59. Open Science Collaboration 2015. Estimating the reproducibility of psychological science. Science, 349(6251): aac4716. Scholar
  60. Pelto, P. J. 1968. The differences between “tight” and “loose” societies. Trans-action, 5(5): 37–40.Google Scholar
  61. Ralston, D. A., Russell, C. J., & Egri, C. P. 2018. Business values dimensions: A cross-culturally developed measure of workforce values. International Business Review, 27(6): 1189–1199. Scholar
  62. Ronen, S., & Shenkar, O. 2013. Mapping world cultures: Cluster formation, sources and implications. Journal of International Business Studies, 44(9): 867–897.Google Scholar
  63. Sample, I. 2015, August 27. Study delivers bleak verdict on validity of psychology experiment results. Guardian, Accessed Jan. 20, 2018.
  64. Satorra, A., & Bentler, P. M. 1994. Corrections to test statistics and standard errors in covariance structure analysis. In A. von Eye, & C. C. Clogg (Eds.). Latent variables analysis: Applications for developmental research: 399–419. Thousand Oaks: Sage.Google Scholar
  65. Saucier, G., Kenner, J., Iurino, K., Malham, P. B., Chen, Z., Thalmayer, A. G., Kemmelmeier, M., Tov, W., Boutti, R., Metaferia, H., Çankaya, B., Mastor, K. A., Hsu, K-Y., Wu, R., Maniruzzaman, M., Rugira, J., Tsaousis, I., Sosnyuk, O., Adhikary, J. R., Skrzypińska, K., Poungpet, B., Maltby, J., Salanga, M. G. C., Racca, A., Oshio, A., Italia, E., Kovaleva, A., Nakatsugawa, M., Morales-Vives, F., Ruiz, V. M., Braun Gutierrez, R. A., Sarkar,A., Deo, T., Sambu, L., Huisa Veria, E., Ferreira Dela Coleta, M., Kiama, S. G., Hongladoram, S., Derry, R., Zazueta Beltrán, H., Peng, T. K., Wilde, M., Ananda, A., Banerjee, S., Bayazit, M., Joo, S., Zhang, H., Orel, E., Bizumic, B., Shen-Miller, S., Watts, S., Pereira, M. E., Gore, E., Wilson, D., Pope, D., Gutema, B., Henry, H., Dacanay, J. C., Dixon, J., Köbis, N., Luque, J., Hood, J., Chakravorty, D., Pal, A. M., Ong, L., Leung, A., & Altschul, C. 2015. Cross-cultural differences in a global “Survey of World Views.” Journal of Cross-Cultural Psychology, 46, 53–70.Google Scholar
  66. Schumann, J. H., Wunderlich, N. V., & von Wangenheim, F. 2012. Technology mediation in service delivery: A new typology and an agenda for managers and academics. Technovation, 32(2): 133–143.Google Scholar
  67. Schwartz, S. H. 1992. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In M. P. Zanna (Ed.). Advances in experimental social psychology: 1–65. San Diego: Academic Press.Google Scholar
  68. Schwartz, S. H. 1994. Beyond individualism/collectivism: New cultural dimensions of values. In U. Kim, H. C. Triandis, C. Kagitcibasi, S. Choi, & G. Yoon (Eds.). Individualism and collectivism: Theory, method, and application: 85–119. Thousand Oaks: Sage.Google Scholar
  69. Schwartz, S. H. 2006. A theory of cultural value orientations: Explication and applications. Comparative Sociology, 5(2/3): 137–182.Google Scholar
  70. Shin, D., Hasse, V. C., & Schotter, A. P. J. 2017. Multinational enterprises within cultural space and place: Integrating cultural distance and tightness-looseness. Academy of Management Journal, 60(3): 904–921.Google Scholar
  71. Smith, P. B. 2004. Acquiescent response bias as an aspect of cultural communication style. Journal of Cross-Cultural Psychology, 35(1): 50–61.Google Scholar
  72. Smith, P. B. 2017. Cultural values versus cultural norms as predictors of differences in helping behaviors and in emotion regulation: A preliminary nation-level test related to the Leung-Morris model. Management & Organization Review, 13(4): 739–766.Google Scholar
  73. Smith, P. B., Dugan, S., & Trompenaars, F. 1996. National culture and the values of organizational employees: A dimensional analysis across 43 nations. Journal of Cross-Cultural Psychology, 27(2): 231–264.Google Scholar
  74. Smith, P. B., Peterson, M. F., & Schwartz, S. H. 2002. Cultural values, sources of guidance, and their relevance to managerial behavior: A 47-nation study. Journal of Cross-Cultural Psychology, 33(2): 188–208.Google Scholar
  75. Tabachnick, B. G., & Fidell, L. S. 2007. Using multivariate statistics, 5th ed. Boston:Pearson.Google Scholar
  76. Taneja, H., & Webster, J. G. 2016. How do global audiences take shape? The role of institutions and culture in patterns of web use. Journal of Communication, 66(1): 161–182.Google Scholar
  77. Toh, S. M., & Leonardelli, G. J. 2012. Cultural constraints on the emergence of women as leaders. Journal of World Business, 47(4): 604–611.Google Scholar
  78. Treviño, L. J. 2016. Institution building and institutional theory in emerging economies. In H. Merchant (Ed.). Handbook of contemporary research on emerging markets: 62–87. Cheltenham: Elgar.Google Scholar
  79. Triandis, H. C. 1983. Essentials of studying cultures. New York:Pergamon Press.Google Scholar
  80. Tung, R. L. 2008. The cross-cultural research imperative: The need to balance cross-national and intra-national diversity. Journal of International Business Studies, 39(1): 41–47.Google Scholar
  81. Uz, I. 2015. The index of cultural tightness and looseness among 68 countries. Journal of Cross-Cultural Psychology, 46(3): 319–335.Google Scholar
  82. Uz, I. 2018. Cross-validation of cultural tightness and looseness measures. International Journal of Psychology, 53(4): 287–294.Google Scholar
  83. Vandenberg, R. J., & Lance, C. E. 2000. A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1): 4–70.Google Scholar
  84. Weijters, B., & Baumgartner, H. 2012. Misresponse to reversed and negated items in surveys: A review. Journal of Marketing Research, 49(5): 737–747.Google Scholar
  85. Whiting, B. B. 1976. The problem of the packaged variable. In K. Riegel, & J. Meachem (Eds.). The developing individual in a changing world, vol. 1: 303–309. The Hague: Mouton.Google Scholar
  86. World Bank. 2000. Worldwide governance indicators. Accessed June 24, 2017.

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • Len J. Treviño
    • 1
    Email author
  • Carolyn P. Egri
    • 2
  • David A. Ralston
    • 3
  • Irina Naoumova
    • 4
  • Olivier Furrer
    • 5
  • Yongjuan Li
    • 6
  • Fidel León Darder
    • 7
  • María Teresa de la Garza Carranza
    • 8
  1. 1.Florida Atlantic UniversityBoca RatonUSA
  2. 2.Simon Fraser UniversityBurnabyCanada
  3. 3.University Fellows International Research ConsortiumWashington, D.CUSA
  4. 4.University of HartfordWestUSA
  5. 5.University of FribourgFribourgSwitzerland
  6. 6.Chinese Academy of SciencesBeijingChina
  7. 7.University of ValenciaValenciaSpain
  8. 8.Instituto Tecnológico de CelayaCelayaMexico

Personalised recommendations