Advertisement

The American Sociologist

, Volume 47, Issue 2–3, pp 264–288 | Cite as

Frame Search and Re-search: How Quantitative Sociological Articles Change During Peer Review

  • Misha Teplitskiy
Article

Abstract

Peer review is a central institution in academic publishing, yet its processes and effects on research remain opaque. Empirical studies have (1) been rare because data on the peer review process are generally unavailable, and (2) conceptualized peer review as gate-keepers who either accept or reject a manuscript, overlooking peer review’s role in constructing articles. This study uses a unique data resource to study how sociological manuscripts change during peer review. Authors of published sociological research often present earlier versions of that research at annual meetings of the American Sociological Association (ASA). Many of these annual meetings papers are publicly available online and tend to be uploaded before undergoing formal peer review. A data sample is constructed by linking these papers to the respective versions published between 2006 and 2012 in two peer-reviewed journals, American Sociological Review and Social Forces. Quantitative and qualitative analyses examine changes across article versions, paying special attention to how elements of data analysis and theory in the ASA versions change. Results show that manuscripts tend to change more substantially in their theoretical framing than in the data analyses. The finding suggests that a chief effect of peer review in quantitative sociology is to prompt authors to adjust their theoretical framing, a mode or review I call “data-driven.” The data-driven mode of review problematizes the vision of sociological research as addressing theoretically motivated questions.

Keywords

Peer review Evaluation Rhetoric Publication Journals Editors Revision Sociology of sociology 

References

  1. Abbott, A. (1997). Seven types of ambiguity. Theory and Society, 26(2/3), 357–91.CrossRefGoogle Scholar
  2. Abbott, A. (2004). Methods of discovery: Heuristics for the social sciences. New York: W. W. Norton & Company.Google Scholar
  3. Abbott, A. (2014). Digital paper: A manual for research and writing with library and internet materials. Chicago: University of Chicago Press.CrossRefGoogle Scholar
  4. Abend, G. (2006). Styles of sociological thought: sociologies, epistemologies, and the Mexican and U.S. Quests for truth*. Sociological Theory, 24(1), 1–41.CrossRefGoogle Scholar
  5. Abend, G. (2008). The meaning of ‘theory’*. Sociological Theory, 26(2), 173–99.CrossRefGoogle Scholar
  6. Bakanic, V., McPhail, C., & Simon, R. J. (1987). The manuscript review and decision-making process. American Sociological Review, 52(5), 631–42.CrossRefGoogle Scholar
  7. Bakanic, V., McPhail, C., & Simon, R. J. (1989). Mixed messages: referees’ comments on the manuscripts they review. The Sociological Quarterly, 30(4), 639–54.CrossRefGoogle Scholar
  8. Bazerman, C. (1988). Shaping written knowledge: The genre and activity of the experimental article in science (1st ed.). Madison: Univ of Wisconsin Pr.Google Scholar
  9. Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45(1), 197–245.CrossRefGoogle Scholar
  10. Bornmann, L., & Daniel, H.-D. (2005). Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of board of trustees' decisions. Scientometrics, 63(2), 297–320.CrossRefGoogle Scholar
  11. Camic, C., Gross, N., & Lamont, M. (2011). Social knowledge in the making (1st ed.). Chicago: University of Chicago Press.CrossRefGoogle Scholar
  12. Cole, S. (2001). What’s wrong with sociology? New Brunswick: Transaction Publishers.Google Scholar
  13. Cole, S., Cole, J. R., & Simon, G. A. (1981). Chance and consensus in peer review. Science, 214(4523), 881–86.CrossRefGoogle Scholar
  14. Dear, P. (1985). Totius in verba: rhetoric and authority in the early royal society. Isis, 76(2), 145–61.CrossRefGoogle Scholar
  15. Espeland, W. N., & Sauder, M. (2007). Rankings and reactivity: how public measures recreate social worlds. American Journal of Sociology, 113(1), 1–40.CrossRefGoogle Scholar
  16. Fox, M. F. (1989). Disciplinary fragmentation, peer review, and the publication process. The American Sociologist, 20(2), 188–91.CrossRefGoogle Scholar
  17. Gilbert, G. N. (1976). The transformation of research findings into scientific knowledge. Social Studies of Science, 6(3/4), 281–306.CrossRefGoogle Scholar
  18. Gilbert, G. N. (1977). Referencing as persuasion. Social Studies of Science, 7(1), 113–22.CrossRefGoogle Scholar
  19. Gilbert, G. N., & Mulkay, M. (1984). Opening pandora’s box: A sociological analysis of scientists’ discourse. Cambridge: Cambridge University Press.Google Scholar
  20. Glaser, B., & Strauss, A. (1999). The discovery of grounded theory: Strategies for qualitative research. Chicago: Aldine Transaction.Google Scholar
  21. Goodman, S. N., Berlin, J., Fletcher, S. W., & Fletcher, R. H. (1994). Manuscript quality before and after peer review and editing at annals of internal medicine. Annals of Internal Medicine, 121(1), 11–21.CrossRefGoogle Scholar
  22. Gross, A. G. (1990a). The rhetoric of science. Cambridge: Harvard University Press.Google Scholar
  23. Gross, A. G. (1990b). Persuasion and peer review in science: Habermas’s ideal speech situation applied. History of the Human Sciences, 3(2), 195–209.CrossRefGoogle Scholar
  24. Guetzkow, J., Lamont, M., & Mallard, G. (2004). What is originality in the humanities and the social sciences? American Sociological Review, 69(2), 190–212.CrossRefGoogle Scholar
  25. Gusfield, J. (1976). The literary rhetoric of science: comedy and pathos in drinking driver research. American Sociological Review, 41(1), 16–34.CrossRefGoogle Scholar
  26. Hargens, L. L. (1988). Scholarly consensus and journal rejection rates. American Sociological Review, 53(1), 139–51.CrossRefGoogle Scholar
  27. Harnad, S. (2000). The Invisible Hand of Peer Review. Exploit Interactive. Retrieved September 8, 2014 (http://cogprints.org/1646/).
  28. Hirschauer, S. (2010). Editorial judgments: a praxeology of ‘voting’ in peer review. Social Studies of Science, 40(1), 71–103.CrossRefGoogle Scholar
  29. Huutoniemi, K. (2012). Communicating and compromising on disciplinary expertise in the peer review of research proposals. Social Studies of Science 0306312712458478.Google Scholar
  30. Knorr, K. D. (1977). Producing and reproducing knowledge: descriptive or constructive? Toward a model of research production. Social Science Information, 16(6), 669–96.CrossRefGoogle Scholar
  31. Knorr, K. D. (1979). Tinkering toward success: prelude to a theory of scientific practice. Theory and Society, 8(3), 347–76.CrossRefGoogle Scholar
  32. knorr, K.D. & knorr, D. (1978). On the Relationship between Laboratory Research and Published Paper in Science. Retrieved November 3, 2014 (https://www.ihs.ac.at/publications/ihsfo/fo132.pdf).
  33. Knorr-Cetina, K. (1981). The manufacture of knowledge : An essay on the constructivist and contextual nature of science. Oxford: Pergamon Press.Google Scholar
  34. Lamont, M. (1987). How to become a dominant french philosopher: the case of Jacques Derrida. American Journal of Sociology, 93(3), 584–622.CrossRefGoogle Scholar
  35. Lamont, M. (2009). How professors think : Inside the curious world of academic judgment. Cambridge: Harvard University Press.CrossRefGoogle Scholar
  36. Lamont, M. (2012). Toward a comparative sociology of valuation and evaluation. Annual Review of Sociology, 38(1), 201–221. doi: 10.1146/annurev-soc-070308-120022.
  37. Lamont, M., & G. Mallard. (2005). Peer review in international perspectives: US, UK and France. Report commissioned by the Social Sciences and Humanities Research Council of Canada.Google Scholar
  38. Langfeldt, L. (2001). The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science, 31(6), 820–41.CrossRefGoogle Scholar
  39. Latour, B., & Woolgar, S. (1979). Laboratory life : The social construction of scientific facts. Beverly Hills: Sage Publications.Google Scholar
  40. Leahey, E. (2008). Methodological memes and mores: toward a sociology of social research. Annual Review of Sociology, 34(1), 33–53.CrossRefGoogle Scholar
  41. Lee, C. J., Sugimoto, C. R., Guo, Z., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17.CrossRefGoogle Scholar
  42. Lynch, M., & Bogen, D. (1997). Sociology’s asociological ‘core’: an examination of textbook sociology in light of the sociology of scientific knowledge. American Sociological Review, 62(3), 481–93.CrossRefGoogle Scholar
  43. McCloskey, D. N. (1998). The rhetoric of economics (2nd ed.). Madison: University of Wisconsin Press.Google Scholar
  44. Medawar, P. (1964). Is the scientific paper fraudulent? Yes, it misrepresents scientific thought. Sunday Review, pp. 42–43.Google Scholar
  45. Merton, R. K. (1968). The matthew effect in science: the reward and communication system of science. Science, 199, 55–63.Google Scholar
  46. Merton, R., & Zuckerman, H. (1971). Patterns of evaluation in science: institutionalisation, structure and functions of the referee system. Minerva, 9(1), 66–100.CrossRefGoogle Scholar
  47. Myers, G. (1985). Texts as knowledge claims: the social construction of two biology articles. Social Studies of Science, 15(4), 593–630.CrossRefGoogle Scholar
  48. Nelson, J. S. (1990). Rhetoric of the human sciences: Language and argument in scholarship and public affairs. Madison: University of Wisconsin Press.Google Scholar
  49. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615–31.CrossRefGoogle Scholar
  50. Pfeffer, J., Leong, A., & Strehl, K. (1977). Paradigm development and particularism: journal publication in three scientific disciplines. Social Forces, 55(4), 938–51.CrossRefGoogle Scholar
  51. Rueschemeyer, D. (2009). Usable theory: Analytic tools for social and political research (1st ed.). Princeton: Princeton University Press.Google Scholar
  52. Shapin, S. (1984). Pump and circumstance: Robert Boyle’s literary technology. Social Studies of Science, 14(4), 481–520.CrossRefGoogle Scholar
  53. Simon, R. J., Bakanic, V., & McPhail, C. (1986). Who complains to journal editors and what happens*. Sociological Inquiry, 56(2), 259–71.CrossRefGoogle Scholar
  54. Wennerås, C., & Wold, A. (1997). Nepotism and sexism in peer-review. Nature, 387(6631), 341–43.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Department of SociologyUniversity of ChicagoChicagoUSA

Personalised recommendations