The fruits of collaboration in a multidisciplinary field
- First Online:
- Cite this article as:
- Bartneck, C. & Hu, J. Scientometrics (2010) 85: 41. doi:10.1007/s11192-010-0242-4
- 752 Downloads
Collaboration between researchers and between research organizations is generally considered a desirable course of action, in particular by some funding bodies. However, collaboration within a multidisciplinary community, such as the Computer–Human Interaction (CHI) community, can be challenging. We performed a bibliometric analysis of the CHI conference proceedings to determine if papers that have authors from different organization or countries receive more citations than papers that are authored by members of the same organization. There was no significant difference between these three groups, indicating that there is no advantage for collaboration in terms of citation frequency. Furthermore, we tested if papers written by authors from different organizations or countries receive more best paper awards or at least award nominations. Papers from only one organization received significantly fewer nominations than collaborative papers.
KeywordsBibliometrics Citations Collaboration Organizations
The human endeavor to understand and improve the world is fractured into disciplines, including the natural sciences, social sciences, engineering and design. A great effort is being made to bridge the gaps between the disciplines [10, 21] because, as Charles Percy Snow  argued “The clashing point of two subjects, two disciplines, two cultures—of two galaxies, so far as that goes-ought to produce creative chances. In the history of mental activity that has been where some of the break-throughs came.”
Many scientists, engineers and designers share this positive vision of the benefits of collaboration. Beaver  lists eighteen reasons why scientists collaborate. It may be for the purpose of accessing expertise, equipment, recourses or funding. People might hope for higher prestige, more efficiency & effectiveness, or the ability to tackle bigger problems. Scientists may also want to increase their social network or share the excitement in a certain topic. Many collaborations are based on the principle of tutoring. Senior staff may collaborate with junior staff to help them with their work and vice versa. experienced
These are all good reasons for collaboration, but takes effort to communicate across organizations, cultures and disciplines. International negotiation, for example, has been studied in great detail in an attempt to minimize the failure of communication . Moreover, different disciplines operate under different paradigms . Scientists, for example, have an emphasis on understanding the world, engineers focus on technology and designers tend to operate intuitively. The friction between disciplines becomes apparent when a multidisciplinary community is confronted with the task to judge the quality of its members’ contributions, such as the diverse submissions to a conference .
But not only do some scientists, some engineers and some designers believe that collaboration has a positive effect on the quality of results and the productivity, also some some policy-makers believe it . Whole research institutes and funding streams are motivated under the assumption that synergy leads to more and better results. The Seventh Framework Programme (FP7) of the European Union is an example of a major funding stream that requires international collaboration from its applicants. Such international collaboration in the sciences can be particularly useful for less developed countries  and it may result in publications with higher citation rates than purely domestic papers . Lee and Bozeman , on the other hand, indicated that collaboration does not necessarily lead to productivity improvements. Moreover, He  pointed out that international collaboration does not have greater epistemic authority compared to domestic or even local collaboration. At times, the best experts in the field may sit just down the corridor.
These studies usually focus on the classical disciplines and it is unclear what effect research collaboration has within an inherently multidisciplinary field. The human computer interaction (HCI) field is a prominent example of a multidisciplinary field. It features several organizations (e.g. SIGCHI), journals and conferences. The Conference on Human Factors in Computing Systems (CHI) is arguably one of the biggest and most important conference in the field of HCI. The conference has experienced nearly exponential growth in the last 15 years . Designers, engineers and social scientists contribute to its success and challenges of multidisciplinary collaboration have surfaced in the past . The proceedings of the CHI conference are therefore an appropriate data source to investigate the effects of collaboration.
We focus on the question whether collaboration improves the quality of published articles. We can distinguish three types of collaboration. Authors from the same organization might work together (local collaboration), authors from different organizations within the same country may team up (domestic collaboration) and authors from different organizations in different countries may join forces (international collaboration). We distinguished between sub sections of organizations. For example, the two IBM research laboratories, Almaden and Watson, have been included separately, because the quality of these sub sections might differ considerably. In most cases, members of sub sections deliberately chose to mention their sub sections.
We did not judge if collaboration partners were from different disciplines. Being a member of a computer science department does not automatically mean that you are a computer scientist. It would be extremely difficult to evaluate the biography of the many authors that contributed to the CHI conference. This study therefore focuses on the three collaboration types mentioned above. The conclusions we can draw for true multidisciplinary collaboration is therefore limited, but we can evaluate if the results from collaboration on a more general level is of interest to the multidisciplinary field of HCI.
The quality of the papers is the prime reason to raise interest of a community and CHI papers have already been through the double blind peer review process. Several attempts have been made to evaluate the quality of the review process, but one conceptual problem remains. For a complete analysis it is necessary to also consider rejected papers. This information is usually not available, and hence we are not able to compare the success of accepted papers with the success of rejected papers. Previous studies used deception to test the review process by resubmitting rejected papers , but given the ethical issues associated with the use of deception, we did not use their method. Instead, we focus our attention on the best paper award. It allows us to analyze nominees, winners, and non winners. The program committee members of the conference can nominate papers and the program chairs make a selection amongst the nominations. They attempt to select “excellent” papers and probably take the results of the review process into account. Since 2004, the nominees and award winners are indicated in the conference brochure, and hence we focus on the years 2004–2008. Henry et al.  have already visually explored the CHI conference. Their study has an emphasis on individual researchers and their collaboration networks. They provide a bibliometric study that includes a co-authorship analysis, but it does not take the citations of the publications into account.
The Research Assessment Exercise (RAE) is the official procedure under which the performance of researchers and their organizations in the United Kingdom is being evaluated. The use of bibliometric indicators for the RAE is being debated for some time now . In 2008, the ACM started to include citation counts for the papers in their digital library. Despite the increasing popularity of bibliometric indicators, several disadvantages have of their usage have been brought forward. MacRoberts and MacRoberts , reviewed objections to the usage of these indicators. An obvious objections is, for example, the fact that not every citation is intended to positively acknowledge previous work. At times, researchers also criticize other pieces of work. Lawrence  argued that the usage of bibliometric indicators has a systematic influence on the behavior of individual researchers. It would change their focus from making groundbreaking discoveries to publishing their papers in high impact factor journals.
What is the overall nature of collaboration in a multidisciplinary conference such as CHI?
Do papers with local/domestic/international authors receive different numbers of best papers awards or nominations?
Do papers with local/domestic/international authors receive different numbers of citations?
We performed a bibliometric analysis of the CHI proceedings to be able to investigate the research questions. We will now discuss the data analysis process. The data processing is similar to to the one used in our previous study . We therefore include only a short summary here.
We used Google Scholar (GS) as the source of our analysis, since it offers the widest coverage of scientific literature in comparison to ISIs Web of Science (WOS) or Elsevier’s Scopus . Moreover, GS is the only index that includes all sections of the CHI proceedings for all years. We received the meta data of the CHI proceedings from the ACM in an XML format. We processed the XML and mapped the records to a list of organizations. We manually checked 5.1% of our records on the correctness of the mapping and were satisfied with the 0.25 error rate. We automatically queried GS on August 22–23, 2008 to receive the citations for each paper. We performed an automatic and a manual check on the results of the queries and the results were satisfying.
We conclude that the citations counts from GS are not perfect, but the low error rate leads us to believe that this method is good enough for the given purpose of acquiring valuable insights into the proceedings of the CHI. Please refer to Bartneck and Hu  for a detailed description of the data processing and testing. Before moving on to the results, we need to define how authors and their organizations were credited for their papers. Each author receives one credit for every paper he/she (co)authored. For example, for a paper written by four authors, each author receives one credit.
Mean and standard deviation of number of organizations per year
The number of organizations had a significant but small correlation with citations (Pearson Correlation = −0.045, p < 0.001). The number of countries also had significant but small correlation with the citations (Pearson Correlation = −0.033, p = 0.006). The number of authors had no significant correlation with citations (Pearson Correlation = 0.008, p = 0.275).
Frequency, mean citation and standard deviation per number of organizations, number of countries and number of authors
Number of organizations
Number of countries
Number of authors
Papers across number of organization and number of countries
N, mean and std. dev. and std. error of citations per collaboration type
Count and expected count of award and collaboration
Discussion and conclusions
The study has some limitations. We have been able to distinguish between the number of authors, organizations and countries, but this does not necessarily indicate multidisciplinary collaboration. It might be possible, for example, that authors from two different design departments collaborated. Moreover, membership of a certain faculty does not automatically indicate the expertise of an author. Designers might work for a design department as well as for a computer science department. Some authors might even have multiple backgrounds or have a different nationality than their hosting organization. For practical reasons, we were not able to reveal the competence and nationality of each author. This study cannot make strong claims about true multidisciplinary collaboration. However, we are able to analyze collaboration on a more abstract level, namely, if collaboration within a multidisciplinary field has benefits. The collaboration between authors from different organizations and countries might still help to create outstanding results that are interesting for a multidisciplinary field.
North America and Europe collaborate most within the CHI conference. More specifically, the USA, UK and Canada co-author many papers, which is not too surprising given their shared language and culture. More generally we observe that although the number of organizations and the number of countries did have an overall significant correlation with the citations, their effect size is so small that it does not have any practical importance. Our results therefore support related studies [9, 14], but since our data is based on an inherently multidisciplinary community, we believe that our results expand their initial results. We can also observe that most papers at CHI are from one to three authors that are spread across one to two organizations that are located in one country. Table 3 clearly shows that the majority of papers come from single organizations. There is no substantial difference in citations within this group.
Overall, we cannot observe a general beneficial effect of collaboration of any type (local, domestic or international) on the quality of the papers, measured by their citation counts. Other factors must play an important role, since our analysis shows that the citations are very unevenly distributed across the papers. A small number of papers receive a high citation count while half of the papers do not have more than four citations yet. This results supports Seglen’s observation that science its skewed” . This leads to the enormous standard deviations that are visible in Table 2 and Fig. 5. An obvious factor could be the competence of the authors. An interesting future direction to study could be whether the citation counts of the individual authors influence the citation count for their co-authored papers. More specifically, it would be interesting to see if the citation counts for papers that have been written by authors with low individual citation counts improve through the collaboration. It might also be interesting to study the reverse effect. Does the citation count for collaborative papers of highly cited authors increase or decrease through the collaboration?
Although we could not find a strong positive effect of collaboration on citations, it did increase the number of best paper award nominations. Papers with international or domestic collaborations seem to receive substantially more nominations than papers with only local collaborations. We do not have a clear answer of why this is and can only offer our speculations. The program committees at the CHI conference are highly international and from many different organizations. It might be easier for the committee members to nominate papers that reflect the structure of the committee. But we cannot offer any hard proof for this speculation. The precise structure of the committees over the years is not available to us.
We do not intend to discourage collaboration, since it may have positive effects that go beyond what a bibliometric analysis can measure. It may foster international networks and may help to build trust amongst different cultures. These motivations to collaborate might be stronger in the CHI community in comparison to the natural sciences, in which authors often have to collaborate to be able to afford expensive research infrastructure. Infrastructure necessary for a study in the area in CHI can be afforded by most individual organizations. We speculate that much of the collaboration within CHI might be due to shared interests, which is certainly a good motivation to work together. Still, our analysis shows that the collaboration in itself does not necessarily result in higher citations. Policy makers should realize that collaboration in itself does not yield good results in all research communities at all times. Collaboration comes with a price. Communication and coordination require extra effort that does not automatically result in increased productivity  or citation counts. Collaboration should therefore not be overemphasized, in particular when it comes to the allocation of research funding or on the progression of individual careers.
We thank Bernard Rous and Ronald Boisvert of the ACM, who made the metadata of the CHI proceedings available to us.
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.