Rankings, research styles, and publication cultures: a study of American sociology departments
- 581 Downloads
Rankings have become a major form of quality assessment in higher education over the past few decades. Most rankings rely, to varying extent, on bibliometric indicators intended to capture the quantity and quality of the scientific output of institutions. The growing popularity of this practice has raised a number of concerns, one of the most important being whether evaluations of this sort treat different work styles and publication habits in an unbiased manner and, consequently, whether the resulting rankings properly respect the particular modes of research characteristic of various disciplines and subdisciplines. The research reported in this paper looked at this issue, using data on more than one hundred US sociology departments. Our results showed that institutions that are more quantitative in character are more likely to favor journals over books as the dominant form of scientific communication and fare, in general, considerably better on the National Research Council’s assessment than their qualitative equivalents. After controlling for differences in publication practices, the impact of research style declined but remained statistically significant. It thus seems that the greater preference of qualitative departments for books over articles as publication outlets puts them at a disadvantage as far as quality assessments are concerned, although their lagging behind their quantitative counterparts cannot fully be explained by this factor alone.
KeywordsAcademic publishing Research methods Scientific paradigm Sociology of science University rankings
Mathematics Subject Classfication62-07 62J05
JEL ClassificationB500 Y800 Z00
The research reported in this paper was supported by the Hungarian Scientific Research Fund, Grant no. T049106. Katalin Bander, Hanna Kónya, Árpád Rab, Boglárka Simon, and Ágnes Sántha provided invaluable help in the data collection, which is gratefully acknowledged. We also thank Akos Rona-Tas, University of California, San Diego, for useful comments and suggestions.
- Allen, M. P. (2003). The ‘Core Influence’ of Journals in Sociology Revisited. ASA Footnotes, 31, Retrieved September 14, 2013 from the World Wide Web: http://ww.www2.asanet.org/footnotes/dec03/fn11.html.
- American Sociological Association. (2007). Guide to graduate departments of sociology. Reference HM9.A512 2007.Google Scholar
- American Sociological Association. (2011). Report to the American Sociological Association Council Regarding the 2010 National Research Council Assessment of Doctorate Programs.Google Scholar
- Berk, R. A. (2005). Survey of 12 strategies to measure teaching effectiveness. International Journal of Teaching and Learning in Higher Education, 17(1), 48–62.Google Scholar
- Blalock, H. M. (1979). Social statistics (2nd ed.). New York: McGraw-Hill.Google Scholar
- Clarke, M. (2007). The impact of higher education rankings on student access, choice, and opportunity. In Institute for Higher Education Policy (Ed.), College and university ranking systems: Global perspectives and American challenges (pp. 35–48). Washington: Institute for Higher Education Policy.Google Scholar
- Corsi, M., D’Ippoliti, C., & Lucidi, F. (2010). Pluralism at risk? Heterodox economic approaches and the evaluation of economic research in Italy. American Journal of Economics and Sociology, 1495–1529.Google Scholar
- El-Khawas, E., DePietro-Jurand, R., & Holm-Nielsen, L. (1998). Quality assurance in higher education: Recent progress; challenges ahead. Working Paper, No. 21199. World Bank.Google Scholar
- Gillies, D. (2012). Economics and research assessment systems. Economic Thought, 1, 23–47. Retrieved September 4, 2013 from the World Wide Web: http://etdiscussion.worldeconomicsassociation.org/wp-content/uploads/Gillies_ET.pdf.
- Griffith, A., & Rask, K. (2005). The influence of the US News and World Report collegiate rankings on the matriculation decision of high-ability students: 1995-2004 (CHERI Working Paper #76). Retrieved August 29, 2011 from the World Wide Web: http://www.ilr.cornell.edu/cheri/workingPapers/upload/cheri_wp76.pdf.
- Halász, G. (2004). A teljesítményértékelés hazai és európai uniós gyakorlata. [The practice of performance evaluation in Hungary and the European Union]. In M. Kónyáné Tóth & C. S. Molnár (Eds.), Közoktatásunk az Európai Unióban. VI. Országos Közoktatási Szakértői Konferencia (pp. 76–86). Debrecen: Suliszerviz Oktatási és Szakértői Iroda.Google Scholar
- Harley, S., & Lee, F. (1997). Research selectivity, managerialism, and the academic labor process—the future of nonmainstream economics. Human Relations, 50, 1427–1460.Google Scholar
- Kuhn, T. (1970). The structure of scientific revolutions (2nd ed.). Chicago: University of Chicago Press.Google Scholar
- Lee, F. S. (2006). The ranking game, class, and scholarship in American mainstream economics. Australasian Journal of Economics Education, 3(1–2), 1–41.Google Scholar
- Lee, F. S., Pham, X., Gu, G. (2012). The UK research assessment exercise and the narrowing of UK economics. MPRA Paper No. 41842. Retrieved September 4, 2013 from the World Wide Web: http://mpra.ub.uni-muenchen.de/41842/.
- Mishra, S. (2006). Quality assurance in higher education—An introduction. Bangalore: National Assessment and Accreditation Council.Google Scholar
- Mochnacki, A., Segaert, A., & Mclaughlin, N. (2009). Public sociology in print: a comparative analysis of book publishing in three social science disciplines. Canadian Journal of Sociology, 34, 729–764.Google Scholar
- Moed, H. F. (2005). Hirsch index is a creative and appealing construct but be cautious when using it to evaluate individual scholars. Retrieved April 17, 2009 from the World Wide Web: http://www.cwts.nl/hm/Comments_on_Hirsch_Index_2005_12_16.pdf.
- Moksony, F. (1999). Small is beautiful. The use and interpretation of R2 in social research. Review of Sociology, Special issue. 130–138.Google Scholar
- Monks, J., & Ehrenberg, R. G. (1999). The impact of U.S. News and World Report college rankings on admissions outcomes and pricing policies at selective private institutions (CHERI Working Paper #1). Retrieved August 10, 2011 from the World Wide Web: http://digitalcommons.ilr.cornell.edu/cheri/1.
- Oromaner, M. (2008). Intellectual integration and articles in core sociology journals, 1960–2000. The American Sociologist 39, 279–289.Google Scholar
- Ostriker, J. P., Kuh, C. V., & Voytuk, J. A. (2011). A data-based assessment of research-doctorate programs in the United States. Washington: National Academies Press.Google Scholar
- Pedhazur, E. J. (1982). Multiple regression in behavioral research: Explanation and prediction, 2nd ed. New York: Holt, Rinehart & Winston.Google Scholar
- Stella, A., & Woodhouse, D. (2006). Ranking of higher education institutions. Retrieved August 10, 2011 from the World Wide Web: http://www.auqa.edu.au/files/publications/ranking_of_higher_education_institutions_final.pdf.
- Stinchcombe, A. L. (1968). Constructing social theories. New York: Harcourt, Brace & World.Google Scholar
- Ter Bogt, H. J., & Scapens, R. W. (2012). Performance management in universities: Effects of the transition to more quantitative measurement systems. European Accounting Review, iFirst Article, 1–47.Google Scholar
- van Raan, A. F. J., van Leeuwen, T. N., & Visser, M.S. (2010). Germany and France are wronged in citation-based rankings. Retrieved August 10, 2011 from the World Wide Web: http://www.cwts.nl/pdf/LanguageRanking22122010.pdf.
- Well, A. D., & Myers, J. L. (2003). Research design and statistical analysis (2nd ed.). Mahwah: Lawrence Erlbaum Associates.Google Scholar
- Winch, R. F., & Campbell, D. T. (1969). Proof? No. Evidence? Yes. The significance of tests of significance. The American Sociologist, 4, 140–143.Google Scholar