Frame Search and Re-search: How Quantitative Sociological Articles Change During Peer Review
- 412 Downloads
Peer review is a central institution in academic publishing, yet its processes and effects on research remain opaque. Empirical studies have (1) been rare because data on the peer review process are generally unavailable, and (2) conceptualized peer review as gate-keepers who either accept or reject a manuscript, overlooking peer review’s role in constructing articles. This study uses a unique data resource to study how sociological manuscripts change during peer review. Authors of published sociological research often present earlier versions of that research at annual meetings of the American Sociological Association (ASA). Many of these annual meetings papers are publicly available online and tend to be uploaded before undergoing formal peer review. A data sample is constructed by linking these papers to the respective versions published between 2006 and 2012 in two peer-reviewed journals, American Sociological Review and Social Forces. Quantitative and qualitative analyses examine changes across article versions, paying special attention to how elements of data analysis and theory in the ASA versions change. Results show that manuscripts tend to change more substantially in their theoretical framing than in the data analyses. The finding suggests that a chief effect of peer review in quantitative sociology is to prompt authors to adjust their theoretical framing, a mode or review I call “data-driven.” The data-driven mode of review problematizes the vision of sociological research as addressing theoretically motivated questions.
KeywordsPeer review Evaluation Rhetoric Publication Journals Editors Revision Sociology of sociology
- Abbott, A. (2004). Methods of discovery: Heuristics for the social sciences. New York: W. W. Norton & Company.Google Scholar
- Bazerman, C. (1988). Shaping written knowledge: The genre and activity of the experimental article in science (1st ed.). Madison: Univ of Wisconsin Pr.Google Scholar
- Cole, S. (2001). What’s wrong with sociology? New Brunswick: Transaction Publishers.Google Scholar
- Gilbert, G. N., & Mulkay, M. (1984). Opening pandora’s box: A sociological analysis of scientists’ discourse. Cambridge: Cambridge University Press.Google Scholar
- Glaser, B., & Strauss, A. (1999). The discovery of grounded theory: Strategies for qualitative research. Chicago: Aldine Transaction.Google Scholar
- Gross, A. G. (1990a). The rhetoric of science. Cambridge: Harvard University Press.Google Scholar
- Harnad, S. (2000). The Invisible Hand of Peer Review. Exploit Interactive. Retrieved September 8, 2014 (http://cogprints.org/1646/).
- Huutoniemi, K. (2012). Communicating and compromising on disciplinary expertise in the peer review of research proposals. Social Studies of Science 0306312712458478.Google Scholar
- knorr, K.D. & knorr, D. (1978). On the Relationship between Laboratory Research and Published Paper in Science. Retrieved November 3, 2014 (https://www.ihs.ac.at/publications/ihsfo/fo132.pdf).
- Knorr-Cetina, K. (1981). The manufacture of knowledge : An essay on the constructivist and contextual nature of science. Oxford: Pergamon Press.Google Scholar
- Lamont, M. (2012). Toward a comparative sociology of valuation and evaluation. Annual Review of Sociology, 38(1), 201–221. doi: 10.1146/annurev-soc-070308-120022.
- Lamont, M., & G. Mallard. (2005). Peer review in international perspectives: US, UK and France. Report commissioned by the Social Sciences and Humanities Research Council of Canada.Google Scholar
- Latour, B., & Woolgar, S. (1979). Laboratory life : The social construction of scientific facts. Beverly Hills: Sage Publications.Google Scholar
- McCloskey, D. N. (1998). The rhetoric of economics (2nd ed.). Madison: University of Wisconsin Press.Google Scholar
- Medawar, P. (1964). Is the scientific paper fraudulent? Yes, it misrepresents scientific thought. Sunday Review, pp. 42–43.Google Scholar
- Merton, R. K. (1968). The matthew effect in science: the reward and communication system of science. Science, 199, 55–63.Google Scholar
- Nelson, J. S. (1990). Rhetoric of the human sciences: Language and argument in scholarship and public affairs. Madison: University of Wisconsin Press.Google Scholar
- Rueschemeyer, D. (2009). Usable theory: Analytic tools for social and political research (1st ed.). Princeton: Princeton University Press.Google Scholar