This section describes the five components of the approach we used to generate our rankings.
Sample of schools
Our procedure begins by establishing a sample of schools to consider. We include the public policy schools that USNWR ranked 41 or better in its “Best Graduate Public Affairs Programs” in 2016. The universe of schools USNWR considers, we understand, is determined in coordination with NASPAA, the Network of Schools of Public Policy, Affairs, and Administration, and APPAM, the Association for Public Policy Analysis and Management. As stated, that ranking features 272 schools, but due to cost considerations (as discussed below the procedure involved downloading and cleaning listings of faculty) we include only about forty schools. Due to ranking ties, it is impossible to select exactly forty, so we consider forty-four. The resulting sample is not representative of U.S. policy schools, and is especially tilted toward higher-ranked schools. Thus, we wish to be clear that our results may not be externally valid to schools beyond those in our sample.
Table 1 (page 6) lists the schools in our sample in alphabetical order. The first column contains the designation used by USNWR. In most cases this includes the name of the school in parentheses, leaving no ambiguity as to the institution in question. For example, “University of Michigan-Ann Arbor (Ford)” refers to the Gerald R. Ford School of Public Policy at the University of Michigan at Ann Arbor. For cases where a name was not stated in parentheses, column 2 lists the school or program which we considered.Footnote 10 Finally, column 3 lists the names we use to designate schools in subsequent tables, in some cases with abbreviations for the sake of space. Even though some institutions listed are not technically schools—for example in a few cases they are programs or institutes within another academic unit—we will henceforth refer to them as schools. In the text below we will use the designation in column 3 the first time we refer to a specific school; subsequently we will use further common abbreviations or designations.
The next task is to construct lists of names of faculty for each school. We downloaded all the faculty lists for each of the 44 schools from their official websites. We obtained these downloads during the month of June, 2016, and had downloaded all relevant listings by June 30 of that year. Therefore the results do not reflect faculty membership changes past that date. This procedure yielded 4927 unique faculty members—an average of about 112 per institution.
This number might seem large, and part of the reason is that we include all faculty members listed (e.g. adjunct professors, lecturers, visitors, professors emeriti, affiliated professors, etc.). One could certainly make reasonable arguments for a more narrow focus. For example, it might be better to consider only individuals that schools describe as their “core” faculty, or to consider only faculty members who have a primary or sole (as opposed to affiliated or joint) appointment in the policy school.Footnote 11 Arriving at such a focus, however, would have required multiple subjective decisions and/or substantial additional data collection.
To elaborate, inspection of the faculty listings (in some cases combined with specific knowledge that authors have of colleagues’ affiliations) reveals that different schools follow different conventions in terms of how they list and classify faculty. For example, some schools list “core” faculty members, while others do not. A few schools list visitors as part of their core faculty; most do not. Some schools list adjunct professors even if they are not actively teaching that specific year; some provide no adjunct listing at all. Some list affiliated faculty, while others either do not have them or do not provide their names. Arriving at a uniform method to classify faculty would thus have required contacting each school, developing a common set of conventions, and working to arrive at a new listing, an exercise beyond the scope of this paper. As a result, we simply include in a school’s roster any individual that it listed as a faculty member, making no distinction between categories.
The fact that we make no adjustments for faculty composition is a major determinant of our choice to focus on results that rank schools by total research output, as opposed to adjusting output by the number of faculty to calculate productivity-type measures. Specifically, while we do present a few results that adjust for schools’ faculty sizes, we note that constructing these requires building ratios for each school, and we cannot be confident that the denominators are strictly comparable across institutions. We discuss this issue further below, and note in closing that the difficulty in arriving at comparable faculty counts may be one reason why the most widely-publicized research-based rankings typically focus only on universities’ aggregate output.
Matching faculty to bibliographic databases
We then obtained information on the research output of each of the 4927 individuals identified, with the final data collection happening in April of 2017. Thus, the results below reflect the cumulative output of individuals as measured in April 2017, based on their June 2016 affiliations.
We attempted to get measures of output using two bibliographic databases: the Web of Science citation index produced by Thompson Reuters,Footnote 12 and the Scopus index produced by Elsevier.Footnote 13 In our extracts from both databases the basic unit of observation is the publication, and these observations are in turn associated with information like the network of citations the publication has received. Each database uses algorithms to attribute publications to individuals, who can be identified by their name and affiliation.
For example, consider “Jane Doe,” who is currently affiliated with School X at University Y (obviously hypothetical names, although the numbers we discuss below refer to a real individual for whom we engaged in additional investigation to check the matching results). The bibliographic databases do not mention specific schools (e.g. the Ford School at the University of Michigan-Ann Arbor), and so we search for the combination of “Jane Doe” and “University Y.” We find Professor Doe on the Scopus database, for instance, and observe that she has publications listed since 2002. In this particular case, all these data points are confirmed by manual inspection of Professor Doe’s C.V., which we obtained online.Footnote 14 Both Web of Science and Scopus use algorithms to attribute publications to authors, and in this particular case these result in all these papers being attributed to Professor Doe at University Y. We note that this is even though Professor Doe only moved to University Y in 2006, having worked at multiple universities before.
While the outlines of this exercise are possible with both bibliographic databases, in the end we only used the Scopus data. This reflects that Web of Science identifies authors only by first initial and last name. While some faculty members have fairly unique surnames, a name like “J. Smith” can produce a large amount of spurious matches even within a university. This is likely an additional reason why other research-focused rankings consider universities rather than schools.
Even though Scopus uses both first and last names, our procedure can still result in spurious matches—for example, when a university has a “John Smith” in both its school of public policy and its chemistry department. The Scopus platform does try to account for these issues by using other contextual information to arrive at unique matches. While there are still errors and our data almost surely include both false positives and false negatives, this noise is greatly reduced relative to that we would see with Web of Science.
The results of the match, by institution, are in Table 2 (page 9). Despite the concerns about “overmatching” (e.g. matching to faculty members in more than one department who share the same name), it at first appears that the greater issue is having no match at all—only 51 percent of all our names produce a match in Scopus. What we conjecture, based on manual inspection and internet searches, is that in many cases the non-matches are due to listed faculty having no publications in these databases.
For example, young assistant professors who have not yet published will not show up. But we venture that the larger share of these non-matches are due to the inclusion of non-research faculty in our listings. Many adjunct faculty members’ main line of work is not at a university, and hence they may not have made publications in the venues covered by databases like Scopus.
We did discover that some non-matches were due to procedure error. For one example, professors with three words in their last name did not always match, an issue which we were able to correct. On the other hand, faculty members who change their name after marriage, or abbreviate parts of their first names inconsistently raise issues that are harder to address. In addition, through some manual exploration we identified what seem like isolated cases of faculty members who have multiple publications and yet are not matched to the university that our lists (and online confirmation) indicate they are affiliated with.
Where we identified this issue, it is possibly due to the ways in which the Scopus algorithms operate. To cite one example, we found isolated cases of economists who are not matched to a university. We were able to determine that where this happened it was because they have an additional affiliation that they cited very frequently in a way that (for reasons we do not fully understand, since we do not have access to the algorithm) results in their match to this other institution. For example there are some economists with many papers listed in the National Bureau of Economic Research (NBER) working paper series. For a small subset of these individuals, Scopus lists the NBER as the primary affiliation, even though in all the cases we saw they work at a university (which presumably they would say is their main affiliation). We did not correct this problem since it had little if any effect on our rankings. We further decided that trying to correct discipline-specific issues could introduce bias, since we have better knowledge of some disciplines than others. The bottom line is that there are many potential sources of noise in information from these sources, and some such noise remains in our data.
It is also worth noting that the match rate across schools is quite variable. Specifically, in Table 2 (page 9) column 1 lists the number of faculty members listed at each school, and column 2 the proportion found in the Scopus database. Column 3 lists the final number of faculty members (those found) used in all the results below. While Florida State University (Askew) has a match rate of one hundred percent, several schools have rates below thirty percent. This results in changes in the number of faculty between column 1 and column 3. For example, the top four schools by faculty size (from largest to smallest) in Column 1 are Columbia University (SIPA), Harvard University (Kennedy), Carnegie Mellon University (Heinz), and New York University (Wagner). Once one considers faculty actually matched (Column 3), the top four are Harvard, Syracuse University (Maxwell), Columbia, and Princeton University (Wilson). This suggests, for instance, that Columbia, Carnegie Mellon, and NYU likely have a higher prevalence of adjunct instructors than Harvard, Syracuse, or Princeton.
Note also that while less so than in the natural sciences, co-authorship is relatively common (and growing) in the social sciences and other areas schools of public policy are active in. For measuring research productivity, the question arises of how to assign credit among multiple authors. One could, for example, give half credit to each author for two-author works. We decided not to adjust for co-authorship, and all matched authors receive full credit. This is the convention for the standard academic h-index and the norm for promotion decisions in some fields. While there could be potential distortion in the rankings if authors at some institutions systematically co-author more than others, we did not notice that this would have a qualitative impact on our results.
Once faculty are matched the Scopus database immediately allows their individual publications to be extracted. The aggregate output of the 2496 individuals matched consists of 65,896 publications. Of these we will focus on 52,369 items that include articles (46,820), books (1140), and chapters in books (4409).Footnote 15 The set of matched publications go back to the 1970s, but over 90 percent of the publications are for 1982 or later, and more than 50 percentare for 2007 or later. This partly reflects the increasing coverage of Scopus in later years.
Table 3 (page 12) lists the 120 journals that most frequently appear in our publications data.Footnote 16 For each journal, the second column indicates the total number of articles matched to faculty in our data. A third column provides the SCImage Journal Rank (SJR), an indicator of journal quality produced by Scopus. This metric “accounts for both the number of citations received by a journal and the importance or prestige of the journals the citations come from,” according to Elsevier.Footnote 17
To construct our school-level rankings, we aggregate the researcher-level results by school. As a basic quantity measure, we sum across all the faculty at a school to get the total number of publications. We do the same for the number of articles, and for the number of books or book chapters.
To incorporate quality, we take two approaches. First, we use total citation counts to the research output of a school, under the assumption that higher quality work will be cited more often. Second, we use the SJR metric to produce counts of quality-thresholded publications; in particular, we count the number of articles from each school in journals with SJRs above the 99th, 90th, and 50th percentiles.Footnote 18
As discussed in Sect. 2.3, faculty counts are not necessarily comparable across schools; for instance, they may include adjunct or visiting faculty members in some cases and not in others. Still, one may worry that variation in these quantity and quality measures could be driven wholly by differences in faculty sizes and have nothing to do with the distribution of quality or productivity of researchers at a school. To address this issue, we complement the aggregate measures with per-faculty performance. As the denominator in this ratio, we use the faculty count from column 3 of Table 2 (page 9). This column contains, for each school, the number of faculty members that were found in Scopus. Arguably, this number provides a rough approximation of the number of “research-oriented” faculty members who work at each school.
It is important to note that our measure of “research” is limited to academic publications and therefore misses major and important components of research activities. The latter include the production of datasets, code repositories, partnerships with nonprofits or government agencies, consulting contracts, unpublished working papers, journalistic publications, and support of other researchers or graduate students. This omission reflects data availability constraints.
Table 3 (page 12) shows that there is significant diversity in the research undertaken by policy-school faculty, at least if one judges by the venues in which it appears. Journals associated with public administration loom large; for example, Public Administration Review is the periodical most frequently seen (573 observations) in our data, and Journal of Policy Analysis and Management is second, with 397 entries. Economics journals are also well-represented, with American Economic Review, Journal of Public Economics, and Quarterly Journal of Economics seeing more than 250 articles each. Political Science publications are prevalent too, with American Political Science Review and Journal of Politics each accounting for more than 150 articles in the sample. Finally, it is interesting that journals with a natural science emphasis also make an appearance, as this is an area of increasing visibility in public policy schools: Proceedings of the National Academy of Sciences of the United States of America accounts for more than 250 articles, while Science and Nature each account for more than 100 papers.
While our main rankings aggregate all journals together, we also provide a few field-specific rankings. This is partly because the within-field SJR journal rankings—notwithstanding the clear advantages of external generation and cross-field applicability—often do not comport with the opinions of researchers in each field. To address this issue we generated four additional sets of rankings that cover journals with a focus on: economics, natural sciences, political science, and public administration.
For each of these fields we consulted with a small number of colleagues active in the area, and produced between two and five groups of journals. We generate a ranking of schools according to the number of articles published in each group of journals. We emphasize that there is no unique way of doing this, or of gaining the consensus of every observer. An advantage is that we draw on expert judgment, but a disadvantage is subjectivity of journal selection.
Based on our aggregation of colleagues’ input, we proceeded somewhat differently in each case. Specifically, the fields, groups, and journals covered are contained in Table 4 (page 15). The procedure is slightly different in each case, and so merits discussion.
In economics, Group A contains the five journals usually considered to contain the work with, on average, the highest quality. Group B includes all the journals in in Group A plus an additional few, considered somewhat less selective. In this case both groups A and B consist of “general interest” journals that appeal to audiences across different subfields, for example, trade or labor economics. Group C includes all the journals in groups A and B, and adds highly rated subfield-specific journals such as the Journal of Econometrics or the Journal of Monetary Economics.
For natural science, we use similar hierarchical groups. That is, every group also contains the journals in the previous groups, with Group A containing the two most prestigious outlets. In this case, however, there are no subfield-specific periodicals. We omit these because their number is much larger and their frequency is relatively low in our data.
In the case of Political science, conversations with researchers in the field suggested that a hierarchical grouping like that constructed for the previous two fields is harder to achieve. This reflects the fact that different sub-fields tend to publish in different groups of periodicals. In this case we therefore proceed by creating five groups of journals. First, there is a group we label General, containing two journals that most observers indicated appeal to researchers in essentially all subfields. Then, there are four groups that pertain to specific subfields: American politics, comparative politics, international relations, and political theory. In each of these we added three to seven further titles of journals that cater more specifically to these subfields (although they do not always fit neatly in only one, and so we allowed repetition).
Finally, for Public Administration, there are only two groups. These are organized in hierarchical order of quality as done for economics and natural science. Again, we note that this set of journals is subjective and based on conversations with colleagues.
In the case of field-specific rankings we only report aggregate rankings—we do not attempt to construct rankings on output per faculty member. We cannot perform an adjustment for faculty size as done for the general rankings because we do not have information on faculty members’ stated areas of focus. For example, total publications in economics would ideally be adjusted by the total number of professors active in economics. This could perhaps be inferred via publication in field-specific journals, but Scopus does not provide authoritative allocations of journals to fields. One might use newly created field journal lists, like those from Table 4 (page 15), but we choose not to for this paper due to large swings in the denominator depending on specification. In addition, at least some faculty members have publication lists spanning a number of fields. For these reasons, we leave the calculation of per-faculty-member field output for future research.