Comments are welcome

Scholars present their new research at seminars and conferences, and send drafts to peers, hoping to receive comments and suggestions that will improve the quality of their work. Using a dataset of papers published in economics journals, this article measures how much peers' individual and collective comments improve the quality of research. Controlling for the quality of the research idea and author, I find that a one standard deviation increase in the number of peers' individual and collective comments increases the quality of the journal in which the research is published by 47%.


Introduction
Scientific progress is fueled by new ideas. During the process of transforming new ideas into research outputs scholars rely on their peers to identify weaknesses in their work, and to find alternative models, methodologies and databases that can improve the quality of their research. Considering the time scholars devote to present draft versions of their papers at conferences and research seminars, and discuss ideas with colleagues, it is reasonable to expect that peers' comments and suggestions should improve the quality of research. However, despite its alleged importance, no study has quantified this contribution yet. The goal of this paper is to fill this gap.
I build a dataset of papers published in economics journals. Based on the acknowledgment section of the paper, I record all the scholars that gave comments on the paper, and the seminars and conferences at which the paper was presented. To obtain unbiased estimates on how these individual comments, seminars, and conferences contributed to the quality of the paper, I control for the quality of the author and research idea. First, Minondo (2020) shows that high-quality scholars are more likely to be invited to present their work at a research seminar. It also reasonable to expect that papers written by high-quality scholars are more likely to be accepted at conferences. Furthermore, high-quality scholars may receive more comments on their work because they have more opportunities to interact with other scholars at seminars and conferences, or because their work is more likely to be followed. Second, it seems reasonable to expect that scholars will choose their most promising project when deciding what paper they will present at a research seminar and what draft they will send to a colleague.
To control for the quality of the author, I use the quality of the institution she is affiliated with. In some specifications, I also use author fixed effects. To control for the quality of the research idea, I use a feature of the job placement process of PhD candidates in economics. During their last academic year, future PhD graduates in economics select a project, among their contemporaneous research ideas, as their job market paper. This paper is the tool PhD candidates use to show their research skills to potential employers. Since PhD candidates want to maximize job offers, they select as job market paper their highest quality project. Thus, the fact that a paper was selected as job market paper provides a signal for the initial quality of a research project. I retrieved information from 2067 PhD candidates in economics, from the top US economics departments, that entered the labor market between 2000 and 2018. When the PhD candidate enters the job market, I identify her job market paper and the additional projects she could also have selected as her job market paper. I follow the job market paper and additional projects until they are published. These publications constitute my estimation sample.
Using the job market status of a paper to control for the quality of the research idea, and author's fixed effect, I find that a one standard deviation increase in the number of individual comments in a paper that received the average number of individual comments, increases the impact factor of the journal in which the paper is published by 16%. A one standard deviation increase in the number of research seminars in a paper that was presented at the average number of seminars increases the impact factor of the journal by 31%. Presenting the paper at conferences has no impact on the quality of the journal in which the paper is published, once I control for the number of individual comments and seminars. I find that comments given by high-quality scholars have a larger positive impact on the quality of a paper than comments received from non-top scholars. Likewise, presenting the paper at a top economics department has a larger positive impact on the quality of the journal in which the paper is published than presenting the paper in a non-top economics department. Receiving comments from other scholars and presenting at research seminars have similar effects on theoretical and empirical papers. This paper is related to the literature exploring how knowledge is produced (Stephan, 2010;Fortunato et al., 2018) and, in particular, how peers contribute to that process. Azoulay et al. (2010); Waldinger (2012); Borjas and Doran (2015); Agrawal et al. (2017);Jaravel et al. (2018); Bosquet et al. (2019) analyzed how the premature death, migration, or arrival of scientists affect collaborators' and other peers' productivity. My paper contributes to this literature by analyzing another channel by which peers' can affect the quality of a scholar's output: the individual and collective feedback on ongoing research projects. Our finding that peers' feedback has a large positive effect on the quality of research is in line with Oettl (2012) who found that a scholar's output quality decreases after the death of a co-author if the co-author was helpful to other colleagues.
My analysis is also linked with studies that have analyzed how conferences and meetings contribute to the flow of ideas, to increase the probability of publication, and to enhance the visibility of a paper. Iaria et al. (2018) found that the ban on Central scientists from participating at international conferences during and after World War I was associated with a drop in citations between Allied and Central scientists. Using data from the Joint Mathematics Meetings between 1990and 2009, Head et al. (2019 showed that a mathematician is more likely to cite the work of another mathematician if they coincided in the same conference. This probability increases if the two scholars coincided in the session in which the cited paper was presented. Using data from a major political science conference that was canceled in 2012, Lopez de Leon and McQuillin (2018) concluded that the probability that a paper is cited increases by five percentage points over a period of four years if it was presented at the conference. Gorodnichenko et al. (2019) found that presenting a paper at major conferences in economics increases the probability of publishing it in a high-quality journal and enhances its visibility. I find that presenting a paper at a leading economic conference is associated with publishing it at a high-quality journal. However, this positive association becomes statistically insignificant once I control for the number of individual comments received by a paper, and the number of research seminars at which it was presented. This paper is close to Brown (2005), who analyzed whether presentations at research seminars, conferences, and comments received from colleagues increase the likelihood that a paper receives a revise and resubmit decision at an accounting journal; and whether individual and collective comments increase the number of citations received by papers published in three leading accounting journals. He finds that presenting at research seminars is the only variable that is positively correlated with receiving an invitation to revise and resubmit, and the number of citations received by a paper. I add to this paper analyzing whether the number of individual and collective comments increase the quality of the journal in which a paper is published. Furthermore, I control for the quality of the research idea and author fixed effects, and explore whether some comments and presentations have a higher impact than others.
The remainder of the paper is organized as follows. Section 2 describes the dataset and presents some summary statistics. Section 3 discusses the results of the regression analyses, and Section 4 concludes.

Data
The sample is composed by PhD candidates from the top 41 US economics departments that entered the labor market between 2000 and 2018. To identify the top US economics departments I use the ranking elaborated by Ideas. 1 Every year, during the fall term, economics departments announce their job market candidates. From the department's web page, I recorded each PhD candidate's job market paper and the projects that she could also have selected as job market paper. These were projects whose sole author was the PhD candidate, or were written with other PhD students. I excluded the papers co-authored with scholars that already had a PhD. 2 I followed the job market paper and papers that could also have been selected as a job market paper until they were published.
Based on the acknowledgment section, I retrieved the information on the number of research seminars and conferences in which the paper was presented, and the scholars that provided comments on the paper. 3 Table A.1 in the Appendix reports the economics departments and the PhD candidate cohorts included in the sample. It also reports, for each PhD program, the number of graduates from which I could retrieve information, and the number of potential job market paper projects that became journal articles. There are differences in the number of PhD candidate cohorts included in the sample across US economics departments. Those differences are explained by the possibility of accessing the information of "old" cohorts. Economics departments provide information about the PhD candidates that enter the labor market in the current year. Few departments also provide links to previous years' job market candidates. To retrieve information for older cohorts, I used the Internet Archive Library (https://archive.org/about/). In some cases, the library has a fairly complete record of the different versions of the web site over time. However, in many cases, the information is scant, or there is no copy archived. This explains why I could retrieve information for "very old" PhD candidates (i.e., 2000) for some economics departments (e.g., UC Berkeley or MIT), whereas I could only retrieve information about the most recent cohort for others (e.g., Ohio State). 4 I measure the quality of a paper with the Scimago Journal Ranking (SJR) of the journal in which it was published. 5 Similar to Smeets et al. (2006), I measure the quality of a PhD candidate by the quality of her placement after graduation. 6 To measure the quality of the placement, I use the worldwide economics institutions ranking elaborated by Ideas. 7 If a paper has multiple authors I add up the quality of individual authors. I compute the individual comments received by a paper counting the scholars that are listed in the acknowledgments section of the paper. I also compute the number of comments given by top 10 scholars. 8 I count the seminars and conferences at which the paper was presented.
I also count the seminars given at top 10 institutions and, following Gorodnichenko et al. (2019), the presentations at the three major economic conferences: the American Economic Association (AEA), the European Economic Association (EEA), and the Royal Economic Society (RES). 9 Table A.2 in the Appendix provides information about the construction and characteristics of the estimation sample. I retrieved information from 2067 PhD candidates that entered the job market between 2000 and 2018. These job market candidates were working on 5118 projects that could have been selected as job market papers. Among those projects, 2070 were selected as job market papers. 10 By December 2019, 551 of the job market candidates (27%) had published their job market paper or another paper they could also have selected as job market paper in a journal included in the SJR. This percentage is in line with the results of previous studies that highlighted the low "publication productivity" of PhD graduates (Conley and Onder, 2014). 11 A total of 806 out of 5118 potential papers, 16%, had been published by December 2019. 47% of these publications were job market papers. This percentage is larger than the share of job market papers among potential projects (40%). 18% of publications had more than one author, and 12% were published in a top 5 economics journal. 12 I have information on the number of individual comments for all papers in the sample. However, there are some articles that use formulas such as "we acknowledge numerous seminar participants", "several audiences", or "seminar and conference participants". Since the number of seminars could not be computed for these publications, the main estimation sample drops from 806 to 685 articles. Table 1 reports some summary statistics on the individual and collective comments received by a publication. It provides statistics for all publications, published job market papers, and other publications. The median publication received 9 individual comments. The distribution is not skewed: the average is 10 and the standard deviation is 7. The minimum number of comments received by a publication is zero, whereas the maximum is 49. There are 61 publications, out of 685, with no individual comments. The median publication was presented at 1 seminar only. The maximum number of seminars at which a publication was presented was 23. There are 263 publications, out of 685, that were not presented at any seminar. Note that the distribution of seminars per publication is skewed, since the average number of presentations is much larger than the median. Finally, the median publication was not presented at any conference. The average and the standard deviation is 1.
There is a paper that was presented at 15 different conferences, whereas 411 publications, out of 685, were not presented at any conference. Job market papers received more individual comments, and were presented at more research seminars than non job market papers. Specifically, the median job market paper received 5 more individual comments, and was presented at 3 more seminars than a non job market paper.
Panel A of Figure 1 plots a scatter diagram of the relationship between the number of individual comments received by a paper and the quality of the journal in which it was published. Job market papers are identified by blue dots and non job market papers by red hollow squares. There is a positive correlation between the number of individual comments received by a paper and quality of the journal in which it was published. In Panel B, I add the seminars and conferences in which a paper was presented, and plot a scatter diagram for the relationship between the number of times a paper was presented and quality of the journal in which it was published. There is also a positive correlation between the number of presentations and quality of the journal in which the paper was presented.
These scatter diagrams suggest that peers' individual and collective comments improve the quality of a paper. However, these correlations may be capturing the positive association between the quality of the scholar and number of individual and collective comments from peers; or the quality of the research idea and comments received from peers. In the next Note: The quality of the journal is measured by the Scimago Journal Ranking. Presentations is the sum of research seminars and conferences at which a paper was presented.
section, I explore the contribution of individual and collective comments to the final quality 8 of a paper once I control for the quality of the author and research idea.

Regression results
To estimate the contribution of peer's individual and collective comments to the quality of a paper, I estimate the following regression equation: where Q pajt is the quality of paper p written by author a who did her doctoral studies at university j and entered the job market at year t. I pajt is the number of individual comments received by paper p; S pajt and C pajt are the number of seminars and conferences at which paper p was presented, respectively. Q ajt is the quality of the author and JM P pajt is an indicator variable that turns 1 if the paper was a job market paper. patj is the disturbance term.
Since some economics departments may have more social ties with journal editors than others (Colussi, 2018), I control for the economics department at which the candidate did her PhD (γ j ). Due to the time elapsing between submitting a paper and being accepted for publication at a journal, papers from "younger" PhD candidate cohorts are less likely to be included in the estimation sample. This may create a sample selection problem in the dependent variable. To address this problem, I introduce cohort fixed effects (γ t ). They also control for other cohort-specific factors that may affect the probability of publishing in a high-quality journal, such as the quality of other PhD candidates that entered the job market in the same year, or the number of PhD candidates that decided to pursue an academic career. Table 2 presents the estimates for the impact of the number of comments given by peers' individually and collectively at research seminars and conferences. 13 I cluster standard errors at the author level. First, I estimate Equation (1) with the number of individual comments variable only (column (1) of Table 2). This estimation uses the full sample of publications: 806. As expected, the ln Comment coefficient is positive and very precisely estimated.
13 Since the number of individual comments, seminars, and conferences enter in logs in Equation (1) (1) to (6) include cohort and PhD institution fixed effects (not reported). Standard errors clustered at the author level are in parentheses. a, b, c: statistically significant at 1%, 5%, and 10%, respectively.
This result indicates that receiving more individual comments is positively correlated with publishing in a high-ranked journal. For example, a one standard deviation increase in the number of comments (7 comments), for a paper that received the average number of comments (10 comments), increases the quality of the journal in which the paper is published by 29% [((ln(17)-ln(10))*0.553]. This increase would lift a paper published in a journal located in the 2nd quartile of the SJR Economics and Econometrics category (e.g., CESifo Economics Studies; SJR score: 0.851) to a journal located in the 1st quartile (e.g., Journal of Industrial Economics, SJR score: 1.059).
In column (2) the number of seminars is the only independent variable. Note that the number of observations is lower than in column (1) since, as mentioned above, there are some papers that do not provide a valid list of seminars. As expected, presenting the paper at research seminars is positively correlated with publishing the paper at a high-ranked journal. For example, a one standard deviation increase in the number of presentations (5 seminars) for a paper that was presented at the average number of research seminars (3 seminars), increases the quality of the journal in which the paper is published by 42% [((ln(8)-ln(3))*0.428]. There is a positive correlation between the number of conferences in which a paper was presented and quality of the journal in which the paper was published (column (3)). Specifically, a one standard deviation increase in the number of conferences (1 conference), for a paper that was presented at the average number of conferences (1 conference), increases the quality of the journal by 23% [((ln(2)-ln(1))*0.337].
Column (4) presents the results when the specification includes all peers' contribution variables: individual comments, research seminars, and conferences. The ln Comment and the ln Seminar coefficients remain positive and very precisely estimated. However, both coefficients have a lower point value than in previous estimations. This result indicates that there is a positive correlation between the number of individual comments received by a paper and number of seminars and conferences at which it is presented. Interestingly, the conference coefficient is close to zero. This result indicates that the positive association between the number of conferences in which a paper is presented and quality of the journal in which it is published disappears once I control for the number of individual comments received by a paper and seminars in which it was presented. According to the coefficients reported in column (4), a one standard deviation increase in the number of individual comments and research seminars, for a paper that has an average number of comments and seminars, increases the quality of the journal in which the paper is published by 49% [((ln(17)-ln(10))*0.397 + (ln(8)-ln(3))*0.288].
In column (5), I introduce the quality of the author as an additional regressor. As expected, the quality of the author is positively correlated with quality of the journal in which the paper is published. There is also a reduction in the ln Comment and ln Seminar coefficients' point estimates, suggesting that these coefficients were partially capturing the positive correlation between the quality of the author and journal. 14 Column (6) presents the results when I control for the quality of the research idea. The job market paper coefficient is positive and very precisely estimated. According to the coefficient reported in column (6), the quality of journals in which job market papers were published was, on average, 63% higher than the quality of the journals in which the rest of projects were published (exp .494). The ln Comment and ln Seminar coefficients remain positive and precisely estimated. However, their point values, specially for ln Seminar, are lower than in column (5). This is consistent with the argument that scholars choose to present their most promising projects when they are invited to give a research seminar. Even when I control for the quality of the author and research idea, a one standard deviation increase in the number of comments and seminars, for a paper with average values of these variables, still increases the quality of the journal in which the paper is published by 34% [((ln(17)-ln(10))*0.348 + (ln(8)-ln(3))*0.156].
Finally, column (7) reports the estimations when the regression equation includes author fixed effects. This estimation controls for all variables that are author specific, such as the capacity to transform new ideas into high-quality publications, or the "contribution-threshold" each author establishes to determine whether or not a peer is included in the acknowledgments section, or author's willingness to present at research seminars and conferences. 15 In this specification, I identify peers' contribution to the quality of a paper with the variation in the number of individual and collective comments among papers written by the same author, who were devised and began to be developed during the same period, and whose initial quality was identified by the author. Although I do not have a natural experiment that generates a random variation in the number of individual and collective comments received by a paper, I argue that, conditional on author fixed effects and the initial quality of the paper, the variation is mostly random. This enables me to lean towards a causal interpretation of estimates.
The sample in column (7) only includes scholars that published more than one of the projects that were initiated when they were doing their doctoral studies. This leads to a large reduction in the number of observations. Despite this drop, and the increase in standard errors, the ln Comment and ln Seminar coefficients remain positive and precisely estimated. This result confirms that peers' individual and collective comments improve the quality of a paper. A one standard deviation increase in the number of comments, for a paper that received the average number of comments, increases the quality of the journal in which the paper is published by 16% [((ln(17)-ln(10))*0.305]; and a one standard deviation increase in the number of seminars, for a paper that was presented in the average number of seminars, increases the quality of the journal by 31% [((ln(8)-ln(3))*0.317]. For example, the combined effect of these increases, 47%, would lift a paper published in Review of Economics and Statistics (8.363) to the quality level of The American Review (11.889). Presenting at conferences does not raise the quality of the journal in which the paper is published.
In previous estimations, I assumed that all individual and collective comments contributed equally to increase the quality of a paper. However, it seems reasonable to expect that individual comments from top scholars, or comments received at presentations at top economics departments, or leading conferences, contribute more to improve the quality of a research project. Table 3 presents the results when individual and collective comments are distinguished by quality. Column (1) shows that comments given by top 10 scholars have a much larger positive correlation with quality of the paper than comments offered by other scholars. Note: The dependent variable is the journal's log impact factor. Estimations in Columns (1) to (6) include cohort and PhD institution fixed effects (not reported). Standard errors clustered at the author level are in parentheses. a, b, c: statistically significant at 1%, 5%, and 10%, respectively.
Column (2) reports that presenting the paper at a top 10 economics department has a strong positive association with publishing the paper at a high-ranked journal. However, presenting the paper at a non-top 10 economics department has no correlation with quality of the paper. Presenting the paper at a major economics conference (American Economic Association, European Economic Association, and the Royal Economic Society) has a strong positive correlation with quality of the journal in which the paper is published. Presenting at other conferences also has a positive coefficient, although its point value is lower. The quality of the author (column (5)) and the job market status of the paper (column (6)) increase the quality of the journal in which the paper is published. In these specifications, the comments offered by top scholars, giving a seminar at top departments, or presenting the paper at a leading conference have a stronger positive association with quality of the paper than comments by non-top scholars or presenting the paper at non-top departments or conferences. When I control for author fixed effects (column (7)), comments given by top 10 scholars have a larger positive impact on quality of the paper than comments given by non-top scholars. However, the difference between the coefficients is not large. I find that presenting the paper at a top 10 economics department has a positive effect on the quality of the paper. However, presenting the paper in a non-top economics department has no effect on the quality of the journal in which the paper is published. The coefficient for top conferences is positive, but imprecisely estimated. Surprisingly, presenting at a non-top conference has a negative effect on the quality of the journal in which the paper is published. 16 As explained above, there is an important number of publications (121 out of 806) that acknowledged the comments received by participants at research seminars and conferences, but did not list the institutions at which these seminars were hold, or the name of the conferences. To test the robustness of my results, I re-estimate all specifications with the whole sample (806 observations instead of 685) and removing the number of seminars and conferences variables from the regression equation. The estimates for the ln Comment coefficient should be taken with caution. Since the number of individual comments is correlated with the number of seminars and conferences, the ln Comment coefficient may also capture the effect that seminars and conferences have on the quality of the journal in which a paper is published. Table A.3 in the Appendix confirms that individual comments have a strong positive effect on the quality of the journal in which the paper is published (columns (1) to (4)). Estimates also confirm that the individual comments given by top scholars have a stronger effect on the quality of the journal in which the paper is published than comments provided by non-top scholars (columns (5) to (8)).
Finally, I analyze whether some type of papers benefit more from individual and collective comments than others. Following the methodology used in Card et al. (2020), I classify papers as theoretical, empirical, structural, or experimental based on the counting of some specific words. 17 61% of papers in the sample are experimental, and 36% are theoretical. I select these categories and compare whether peers' comments have a larger impact on empirical than on theoretical papers. I expand Equation (1) with a dummy variable that 16 I also ran regressions using top 5 as the quality threshold for scholars and seminars. Results, not reported, are qualitatively and quantitatively similar to those presented in Table 3. 17 The words used to identify a theoretical paper are proposition, theorem, lemma, proof, model, and theory; for an empirical paper: data, standard error, table, regression, difference-in-differences, and empirical; for a structural paper: structural, BLP, maximum likelihood, mixture, simulation, and calibration; and for an experimental paper: field experiment, RCT, laboratory, subjects, and survey. The category with the largest number of words determines the type the paper belongs to. turns one if the paper is empirical, and interact the comment, seminar, and conference coefficients with the empirical dummy variable.  (3) and (4) presents the results when peer effects are distinguished by quality. Comments from high-quality scholars have similar effects on theoretical and empirical papers, whereas comments from non-top scholars have a larger positive effect on empirical papers. Research seminars at top economics departments have similar effects on theoretical and empirical papers, whereas seminars at non-top departments have a larger positive impact on empirical papers. Results for conferences are not robust.

Conclusions
A scholar's knowledge is limited and, therefore, is unaware of all the elements that may contribute to improve the quality of her research. To discover these elements, she relies on peers, who at research seminars, conferences, or through conversations, identify limitations in the research project and suggest avenues to improve it. In this paper, I measured how much these comments and suggestions improve the quality of research. Since the number of suggestions a paper receives is not independent from the quality of the research idea and author, I use a sample of papers that enables me to control for these variables: the research projects of job market candidates in economics. I find that a one standard deviation increase in the number of individual comments and research seminars increases the quality of the journal in which the paper is published by 47%. I find that comments provided by top scholars have a stronger positive effect on the quality of the paper than comments given by non-top scholars. I also show that while presenting a paper at top economics departments has a strong positive effect on the quality of the paper, presenting at non-top economics departments has no effect. I find that presenting at conferences, even at the top ones, is not associated with publishing in a high-ranked journal, once I control for the number of individual comments and seminars. Peers' comments have similar effect on theoretical and empirical papers.