Skip to main content

In search of rigor, relevance, and legitimacy: what drives the impact of publications?

Abstract

In this study, we introduce the concept of legitimacy to the rigor-relevance debate and investigate empirically how rigor, relevance, and legitimacy are related to the process of knowledge dissemination within a research field. We argue that this analysis has been a missing piece in the debate on rigor and relevance when institutional logics about what constitutes both elements lead researchers to act according to what they perceive to be appropriate behavior in the research field. We draw on insights from the micro and macro levels of institutional theory to show how researchers aiming to bestow legitimacy on their own work conform to these “rules of the game”. Using meta-analytical techniques, we focus on the field of strategic entrepreneurship and analyze how rigor- and relevance-related characteristics of studies in this field are linked to their legitimacy and therefore to the impact they have in the research community.

This is a preview of subscription content, access via your institution.

References

  1. Adler NJ, Harzing A-W (2009) When knowledge wins: transcending the sense and nonsense of academic rankings. Acad Manag Learn Education 8:72–95

    Google Scholar 

  2. Aguinis H, Suárez-González I, Lannelongue G, Joo H (2012) Scholarly impact revisited. Acad Manag Perspect 26:105–132

    Google Scholar 

  3. Aiken LS, West SG (1991) Multiple regression: Testing and interpreting interactions. Sage, Newbury Park

    Google Scholar 

  4. Anderson N, Herriot P, Hodgkinson GP (2001) The practitioner-researcher divide in industrial, work and organizational (IWO) psychology: where are we now, and where do we go from here? J Occup Organ Psychol 74:391–411

    Google Scholar 

  5. Augier M, March JG (2007) The pursuit of relevance in management education. Calif Manag Rev 49:129–146

    Google Scholar 

  6. Backes-Gellner U, Sadowski D (1990) Organizational implementation of bibliometric indicators. Scientometrics 19:385–395

    Google Scholar 

  7. Baldridge DC, Floyd SW, Markóczy L (2004) Are managers from Mars and academicians from Venus? Toward an understanding of the relationship between academic quality and practical relevance. Strateg Manag J 25:1063–1074

    Google Scholar 

  8. Bennis WG, O’Toole J (2005) How business schools lost their way. Harv Bus Rev 83:96–104

    Google Scholar 

  9. Bettis RA (1991) Strategic management and the straightjacket: an editorial essay. Organ Sci 2:315–319

    Google Scholar 

  10. Beyer JM, Chanove RG, Fox WB (1995) The review process and the fates of manuscripts submitted to AMJ. Acad Manag J 38:1219–1260

    Google Scholar 

  11. Borenstein M, Hedges L, Higgins J, Rothstein H (2005) Comprehensive meta-analysis, 2nd edn. Wiley, Englewood

    Google Scholar 

  12. Bozart JD, Roberts RR (1972) Signifying significant significance. Am Psychol 27:774–775

    Google Scholar 

  13. Bresser RK (1984) Structural dimensions of university departments and their context: the case of West Germany. Organ Stud 5:119–146

    Google Scholar 

  14. Bresser RK, Dunbar RLM (1986) Context, structure, and academic effectiveness: evidence from West Germany. Organ Stud 7:1–24

    Google Scholar 

  15. Bresser RK, Millonig K (2003) Institutional capital: competitive advantage in light of the new institutionalism in organization theory. Schmalenbach Bus Rev 55:220–241

    Google Scholar 

  16. Buckley MR, Ferris GR, Bernardin HJ, Harvey MG (1998) The disconnect between the science and practice of management. Bus Horiz 41:31–38

    Google Scholar 

  17. Chlosta K, Pull K (2010) The incentive effects of appointment tournaments in German higher education. Schmalenbach Bus Rev 62:378–400

    Google Scholar 

  18. Cohen J, Cohen P (1983) Applied multiple regression/correlation analysis for the behavioral sciences, 2nd edn. Erlbaum, Hillsdale

    Google Scholar 

  19. Combs JG (2010) Big samples and small effects: let’s not trade relevance and rigor for power. Acad Manag J 53:9–13

    Google Scholar 

  20. Conger AJ (1974) A revised definition for suppressor variables: a guide to their identification and interpretation. Educ Psychol Measur 34:35–46

    Google Scholar 

  21. Cooper H (1998) Synthesizing research: A guide for literature reviews. Sage, Thousand Oaks

    Google Scholar 

  22. Covin JG, Green KM, Slevin DP (2006) Strategic process effects on the entrepreneurial orientation–sales growth rate relationship. Entrepreneurship: Theory Pract 30:57–81

    Google Scholar 

  23. Daft RL, Lewin AY (2008) Rigor and relevance in organization studies: idea migration and academic journal evolution. Organ Sci 19:177–183

    Google Scholar 

  24. Dilger A (2000) Plädoyer für einen sozialwissenschaftlichen Zitationsindex. Die Betriebswirtschaft 60:473–484

    Google Scholar 

  25. DiMaggio PJ, Powell WW (1983) The iron cage revisited: institutional isomorphism and collective rationality in organizational fields. Am Sociol Rev 48:147–160

    Google Scholar 

  26. Donohue JM, Fox JB (2000) A multi-method evaluation of journals in the decision and management sciences by US academics. Omega 28:17–36

    Google Scholar 

  27. Duncan WJ (1974) Transferring management theory to practice. Acad Manag J 17:724–738

    Google Scholar 

  28. Dyckhoff H, Gilles R (2004) Messung der Effektivität und Effizienz produktiver Einheiten. Zeitschrift für Betriebswirtschaft 74:765–783

    Google Scholar 

  29. Dyckhoff H, Schmitz C (2007) Forschungsleistungsmessung mittels SSCI Oder SCI-X? Internationale Sichtbarkeit und Wahrnehmung der Betriebswirtschaftslehre von 1990-2004. Die Betriebswirtschaft 67:638–662

    Google Scholar 

  30. Dyckhoff H, Rassenhövel S, Gilles R, Schmitz C (2005a) Beurteilung der Forschungsleistung und das CHE-Forschungsranking betriebswirtschaftlicher Fachbereiche. Wirtschaftswissenschaftliches Studium 34:62–69

    Google Scholar 

  31. Dyckhoff H, Thieme A, Schmitz C (2005b) Die Wahrnehmung deutschsprachiger Hochschullehrer für Betriebswirtschaft in der internationalen Forschung: eine Pilotstudie zu Zitationsverhalten und möglichen Einflussfaktoren. Die Betriebswirtschaft 65:350–372

    Google Scholar 

  32. Ellis PD (2006) Market orientation and performance: a meta-analysis and cross-national comparisons. J Manage Stud 43:1089–1107

    Google Scholar 

  33. Fabel O, Heße F (1999) Befragungsstudie vs. Publikationsanalyse: zur Interpretation von Ranglisten der Forschungsaktivitäten deutscher betriebswirtschaftlicher Fachbereiche. Die Betriebswirtschaft 59:196–204

    Google Scholar 

  34. Fiedler M, Welpe IM, Picot A (2006) Terra Incognita—Forschungsleistungen und Qualifizierungswege des deutschsprachigen Hochschullehrernachwuchses für Betriebswirtschaftslehre. Die Betriebswirtschaft 66:464–486

    Google Scholar 

  35. Fiedler M, Welpe IM, Lindlbauer K, Sattler K (2008) Denn wer da hat, dem wird gegeben: Publikationsproduktivität des BWL Hochschullehrernachwuchses und deren wissenschaftlicher Betreuer. Zeitschrift für Betriebswirtschaft 78:477–508

    Google Scholar 

  36. Fleming P, Spicer A (2003) Working at a cynical distance: implications for power, subjectivity and resistance. Organization 10:157–179

    Google Scholar 

  37. Frey BS, Rost K (2010) Do rankings reflect research quality? J Appl Econ 13:1–38

    Google Scholar 

  38. Friedland R, Alford R (1991) Bringing society back in: symbols, practices, and institutional contradictions. In: Powell WW, DiMaggio PJ (eds) The new institutionalism in organizational analysis. University of Chicago Press, Chicago, pp 232–263

    Google Scholar 

  39. Ghoshal S (2005) Bad management theories are destroying good management practices. Acad Manag Learn Education 4:75–91

    Google Scholar 

  40. Gomez-Mejia LR, Balkin DB (1992) Determinants of faculty pay: an agency theory perspective. Acad Manag J 35:921–955

    Google Scholar 

  41. Gopinath C, Hoffman RC (1995) The relevance of strategy research: practitioner and academic viewpoints. J Manage Stud 32:575–594

    Google Scholar 

  42. Gordon RA, Howell JE (1959) Higher education for business. Columbia University Press, New York

    Google Scholar 

  43. Gottfredson SD (1978) Evaluating psychological research reports: dimensions, reliability, and correlates of quality judgements. Am Psychol 33:920–934

    Google Scholar 

  44. Greenwood R, Suddaby R, Hinings CR (2002) Theorizing change: the role of professional associations in the transformation of institutional fields. Acad Manag J 45:58–80

    Google Scholar 

  45. Gulati R (2007) Tent poles, tribalism, and boundary spanning: the rigor-relevance debate in management research. Acad Manag J 50:775–782

    Google Scholar 

  46. Harzing A-W (2012) Journal quality list, 46th edn, http://www.harzing.com

  47. Hedges LV, Olkin I (1985) Statistical methods for meta-analysis. Academic Press, Orlando

    Google Scholar 

  48. Hedges LV, Vevea JL (1998) Fixed- and random-effects models in meta-analysis. Psychol Methods 3:486–504

    Google Scholar 

  49. Hitt MA, Ireland RD, Sirmon DG, Trahms CA (2011) Strategic entrepreneurship: creating value for individuals, organizations, and society. Acad Manag Perspect 25:57–75

    Google Scholar 

  50. Hodgkinson GP, Rousseau DM (2009) Bridging the rigour–relevance gap in management research: it’s already happening! J Manag Stud 46:534–546

    Google Scholar 

  51. Horst P (1941) The role of prediction variables which are independent of the criterion. In: Horst P (ed) The prediction of personal adjustment. Social Science Research Council Bulletin, New York, pp 431–436

    Google Scholar 

  52. Hunter JE, Schmidt FL (1990) Methods of meta-analysis. Sage, Newbury Park

    Google Scholar 

  53. Hunter JE, Schmidt FL (2004) Methods of meta-analysis: correcting error and bias in research findings, 2nd edn. Sage, Thousand Oaks

    Google Scholar 

  54. Hunter JE, Schmidt FL, Jackson GB (1982) Meta-analysis: cumulating research findings across studies. Sage, Beverly Hills

    Google Scholar 

  55. Jermier JM, Knights D, Nord WRE (1994) Resistance & power in organizations. Routledge, London

    Google Scholar 

  56. Johnson JL, Podsakoff PM (1994) Journal influence in the field of management: an analysis using Salancik’s index in a dependency network. Acad Manag J 37:1392–1407

    Google Scholar 

  57. Judge TA, Cable DM, Colbert AE, Rynes SL (2007) What causes a management article to be cited—article, author, or journal? Acad Manag J 50:491–506

    Google Scholar 

  58. Kacmar KM, Whitfield JM (2000) An additional rating method for journal articles in the field of management. Organ Res Methods 3:392–406

    Google Scholar 

  59. Kailer N (2009) Entrepreneurship education: empirical findings and proposals for the design of entrepreneurship education concepts at universities in German-speaking countries. J Enterp Culture 17:201–231

    Google Scholar 

  60. Kieser A (2005) Wie wichtig sind Forschungsrankings? Wie wichtig werden sie in Zukunft sein? Journal für Betriebswirtschaft 55:169–170

    Google Scholar 

  61. Kieser A, Leiner L (2009) Why the rigour–relevance gap in management research is unbridgeable. J Manage Stud 46:516–533

    Google Scholar 

  62. Kieser A, Leiner L (2011) On the social construction of relevance: a rejoinder. J Manage Stud 48:891–898

    Google Scholar 

  63. Kieser A, Nicolai AT (2005) Success factor research: overcoming the trade-off between rigor and relevance? J Manag Inquiry 14:275–279

    Google Scholar 

  64. Köhler R (2004) Publish or perish. Die Betriebswirtschaft 64:127–130

    Google Scholar 

  65. Kraus S, Eggers F, Harms R, Hills G, Hultman C (2011a) Diskussionslinien der Entrepreneurial Marketing-Forschung: ergebnisse einer Zitationsanalyse. Zeitschrift für Betriebswirtschaft 81:27–58

    Google Scholar 

  66. Kraus S, Kauranen I, Reschke CH (2011b) Identification of domains for a new conceptual model of strategic entrepreneurship using the configuration approach. Manag Res Rev 34:58–74

    Google Scholar 

  67. Laband DN (1986) Article popularity. Econ Inquiry 24:173–180

    Google Scholar 

  68. Laband DN, Piette MJ (1994) A citation analysis of the impact of blinded peer review. J Am Med Assoc 272:147–149

    Google Scholar 

  69. Lipsey MW, Wilson DB (2001) Practical meta-analysis. Sage, Thousand Oaks

    Google Scholar 

  70. Locke K, Golden-Biddle K (1997) Constructing opportunities for contribution: structuring intertextual coherence and “problematizing” in organizational studies. Acad Manag J 40:1023–1062

    Google Scholar 

  71. Long RG, Bowers WP, Barnett T, White MC (1998) Research productivity of graduates in management: effects of academic origin and academic affiliation. Acad Manag J 41:704–714

    Google Scholar 

  72. Long CP, Bendersky C, Morrill C (2011) Fairness monitoring: linking managerial controls and fairness judgments in organizations. Acad Manag J 54:1045–1068

    Google Scholar 

  73. Luhmann N (1998) Die Wissenschaft der Gesellschaft. Suhrkamp, Frankfurt am Main

    Google Scholar 

  74. Maassen GH, Bakker AB (2001) Suppressor variables in path models: definitions and interpretations. Sociol Methods Res 30:241–270

    Google Scholar 

  75. Macharzina K, Wolf J, Oesterle M-J (1993) Quantitative evaluation of German research output in business administration. Manag Int Rev 33:65–83

    Google Scholar 

  76. Macharzina K, Wolf J, Rohn A (2006) Zur Forschungsleistung der Betriebswirte im deutschsprachigen Raum—Eine personen- und institutionenbezogene Längsschnittanalyse. In: Wolf J, Rohn A, Macharzina K (eds) Forschungsleistung in der deutschsprachigen Betriebswirtschaftslehre—Konzeption und Befunde einer empirischen Untersuchung. Gabler, Wiesbaden, pp 1–34

    Google Scholar 

  77. March JG (1996) Continuity and change in theories of organizational action. Adm Sci Q 41:278–287

    Google Scholar 

  78. Mazen AM, Graf LA, Kellogg CE, Hemmasi M (1987) Statistical power in contemporary management research. Acad Manag J 30:369–380

    Google Scholar 

  79. McGahan A (2007) Academic research that matters to managers: on zebras, dogs, lemmings, hammers, and turnips. Acad Manag J 50:748–753

    Google Scholar 

  80. Meyer JW, Rowan B (1977) Institutional organizations: formal structures as myth and ceremony. Am J Sociol 83:340–363

    Google Scholar 

  81. Neter J, Kutner MH, Nachtsheim CJ, Wasserman W (1996) Applied linear statistical models, 4th edn. Irwin, Chicago

    Google Scholar 

  82. Nicolai A, Seidl D (2010) That’s relevant! Different forms of practical relevance in Management Science. Organ Stud 31:1257–1285

    Google Scholar 

  83. Nicolai AT, Schulz A-C, Göbel M (2011) Between sweet harmony and a clash of cultures: does a joint academic–practitioner review reconcile rigor and relevance? J Appl Behav Sci 47:53–75

    Google Scholar 

  84. Nkomo SM (2009) The seductive power of academic journal rankings: challenges of searching for the otherwise. Acad Manag Learn Education 8:106–112

    Google Scholar 

  85. Nunnally JC (1978) Psychometric Theory. McGraw-Hill, New York

    Google Scholar 

  86. Ocasio W (1997) Towards an attention-based view of the firm. Strateg Manag J 18:187–206

    Google Scholar 

  87. Oesterle MJ (2006) Wahrnehmung betriebswirtschaftlicher Fachzeitschriften durch Praktiker. Die Betriebswirtschaft 66:307–325

    Google Scholar 

  88. Oswald AJ (2007) An examination of the reliability of prestigious scholarly journals: evidence and implications for decision-makers. Economica 74:21–31

    Google Scholar 

  89. Palmer D, Dick B, Freiburger N (2009) Rigor and relevance in organization studies. J Manag Inquiry 18:265–272

    Google Scholar 

  90. Park SH, Gordon ME (1996) Publication records and tenure decisions in the field of strategic management. Strateg Manag J 17:109–128

    Google Scholar 

  91. Pedhazur EJ (1997) Multiple regression in behavioral research: explanation and prediction. Harcourt, Fort Worth

    Google Scholar 

  92. Pettigrew AM (1997) The double hurdles for management research. In: Clarke T (ed) Advancement in organizational behaviour: essays in honour of D. S. Pugh. Dartmouth, London, pp 277–296

    Google Scholar 

  93. Pettigrew AM (2001) Management research after modernism. Br J Manag 12:S61–S70

    Google Scholar 

  94. Pfeffer J, Fong CT (2002) The end of business schools? Less success than meets the eye. Acad Manag Learn Education 1:78–95

    Google Scholar 

  95. Pierson FC (1959) The education of American businessmen. MacGraw-Hill, New York

    Google Scholar 

  96. Podsakoff PM, MacKenzie SB, Bachrach DG, Posakoff NP (2005) The influence of management journals in the 1980s and 1990s. Strateg Manag J 26:473–488

    Google Scholar 

  97. Porter LW, McKibbin LE (1988) Management education and development: Drift or thrust into the 21st century?. MacGraw-Hill, New York

    Google Scholar 

  98. Rafols I, Leydesdorff L, O’Hare A, Nightingale P, Stirling A (2012) How journal rankings can suppress interdisciplinary research: a comparison between innovation studies and business & management. Res Policy 41:1262–1282

    Google Scholar 

  99. Rauch A, Wiklund J, Lumpkin GT, Frese M (2009) Entrepreneurial orientation and business performance: an assessment of past research and suggestions for the future. Entrepreneurship Theory Pract 33:761–787

    Google Scholar 

  100. Rosenthal R (1979) The “file drawer problem” and tolerance for null results. Psychol Bull 86:638–641

    Google Scholar 

  101. Rynes SL, Giluk TL, Brown KG (2007) The very separate worlds of academic and practitioner periodicals in human resource management: implications for evidence-based management. Acad Manag J 50:987–1008

    Google Scholar 

  102. Schiele H, Krummaker S (2011) Consortium benchmarking: collaborative academic–practitioner case study research. J Bus Res 64:1137–1145

    Google Scholar 

  103. Schrader U, Hennig-Thurau T (2009) VHB-Jourqual2: Method, results, and implications of the German Academic Association for Business Research’s Journal Ranking. BuR—Bus Res 2:180–204

    Google Scholar 

  104. Scott WR (2001) Institutions and organizations, 2nd edn. Sage, Thousand Oaks

    Google Scholar 

  105. Seglen PO (1997) Why the impact factor of journals should not be used for evaluating research. Br Med J 314:498–502

    Google Scholar 

  106. Seidl D (2005) Glossary to Niklas Luhmann’s terminology. In: Seidl D, Becker KH (eds) Niklas Luhmann and organization studies. Copenhagen Business School Press, Kristianstad, pp 403–410

    Google Scholar 

  107. Shadish WR, Tolliver D, Gray M, Sen Gupta SK (1995) Author judgements about works they cite: three studies from psychology journals. Soc Stud Sci 25:477–498

    Google Scholar 

  108. Shapiro DL, Kirkman BL, Courtney HG (2007) Perceived causes and solutions of the translation problem in management research. Acad Manag J 50:249–266

    Google Scholar 

  109. Shrivastava P (1987) Rigor and practical usefulness of research in strategic management. Strateg Manag J 8:77–92

    Google Scholar 

  110. Singh G, Haddad KM, Chow CW (2007) Are articles in “top” management journals necessarily of higher quality? J Manag Inquiry 16:319–331

    Google Scholar 

  111. Starbuck WH (2005) How much better are the most-prestigious journals? The statistics of academic publication. Organ Sci 16:180–200

    Google Scholar 

  112. Starkey K, Madan P (2001) Bridging the relevance gap: aligning stakeholders in the future of management research. Br J Manag 12:S3–S26

    Google Scholar 

  113. Starkey K, Hatchuel A, Tempest S (2009) Management research and the new logics of discovery and engagement. J Manage Stud 46:547–558

    Google Scholar 

  114. Straub DW, Ang S (2008) Readability and the relevance versus rigor debate. MIS Quarterly 32:iii–xiii

    Google Scholar 

  115. Suchman MC (1995) Managing legitimacy: strategic and institutional approaches. Acad Manag Rev 20:571–610

    Google Scholar 

  116. Thornton PH, Ocasio W (1999) Institutional logics and the historical contingency of power in organizations: executive succession in the higher education publishing industry, 1958–1990. Am J Sociol 105:801–843

    Google Scholar 

  117. Tolbert PS, Zucker LG (1983) Institutional sources of change in the formal structure of organizations: the diffusion of civil service reform, 1880–1935. Adm Sci Q 28:22–39

    Google Scholar 

  118. Weiss CH, Bucuvalas MJ (1980) Truth tests and utility tests: decision-makers’ frames of reference for social science research. Am Sociol Rev 45:302–313

    Google Scholar 

  119. Wiklund J (1998) Entrepreneurial orientation as predictor of performance and entrepreneurial behavior in small firms. In: Reynolds PD, Bygrave WD, Carter NM, Manigart S, Mason CM, Meyer GD, Shaver KG (eds) Frontiers of Entrepreneurship Research. Babson College, Wellesley, pp 281–296

    Google Scholar 

  120. Wiklund J, Shepherd DA (2003) Knowledge-based resources, entrepreneurial orientation, and the performance of small and medium-sized businesses. Strateg Manag J 24:1307–1314

    Google Scholar 

  121. Wolf J, Rosenberg T (2012) How individual scholars can reduce the rigor-relevance gap in management research. BuR—Bus Res 5:178–196

    Google Scholar 

  122. Wolf J, Rohn A, Macharzina K (2005) Institution und Forschungsproduktivität—Befunde und Interpretationen aus der deutschsprachigen Betriebswirtschaftslehre. Die Betriebswirtschaft 65:62–77

    Google Scholar 

  123. Zucker LG (1977) The role of institutionalization in cultural persistence. Am Sociol Rev 42:726–743

    Google Scholar 

  124. Zucker LG (1987) Institutional theories of organizations. Annual Rev Sociol 13:443–464

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Miriam Flickinger.

Appendix A

Appendix A

Appendix A Studies included in the meta-analysis

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Flickinger, M., Tuschke, A., Gruber-Muecke, T. et al. In search of rigor, relevance, and legitimacy: what drives the impact of publications?. J Bus Econ 84, 99–128 (2014). https://doi.org/10.1007/s11573-013-0692-2

Download citation

Keywords

  • Rigor
  • Relevance
  • Legitimacy
  • Citation rates

JEL Classification

  • M10
  • Z1