Bibliometric reviews—some guidelines

Like other scholarly outlets, each year the Journal of the Academy of Marketing Science (JAMS) receives many submissions that draw heavily on the use of bibliometric research techniques as a basis for reviewing a specific research domain (area of inquiry). Too frequently, these reviews employ sophisticated software tools to map the structure and development of the domain without also providing novel insights that lead the reader to a better understanding of the focal phenomenon. The result is a well-conducted systemization of existing knowledge that unfortunately adds little to the on-going conversation surrounding the field of study; such submissions are routinely desk-rejected.

The aim of this editorial, then, is to offer researchers high-level guidance to help them make better use of bibliometric tools in crafting reviews. This involves three steps. First, researchers must reflect carefully on their aims for undertaking a bibliometric review and communicate these clearly to the reader. Second, the specific bibliometric analyses used need to correspond appropriately with the chosen review objective(s). Finally, for greatest impact the novel and meaningful insights that emerge from these analyses should typically be combined with more traditional systematic review approaches.

Establishing aims for the review

To provide value and be credible, review papers must be conducted in a systematic manner, with an eye to collecting and then synthesizing an existing body of research in a way that reveals significant new insights (Hulland & Houston 2020). Zupic and Cater (2015; p. 429) observe that scholars “have traditionally used two methods to make sense of earlier findings: the qualitative approach of a structured literature review and the quantitative approach of meta-analysis,” additionally noting that “science mapping … based on the quantitative approach of bibliometric research methods … is being increasingly used to map the structure and development of scientific fields and disciplines.” These techniques have the potential to reveal powerful analytical insights (Mukherjee et al., 2022), yet bibliometric researchers often stop short of generating them.

And novel, substantive insights are the crucial, focal outcome of reviews. Hulland and Houston (2020, p. 353) argue that “the usefulness of the insights generated is what separates highly cited review papers from reviews that are largely ignored.” An insight becomes useful when it offers scholars and managers new ways to think about a specific issue that they care about and suggests new ways to address or respond to it.

Many bibliometric reviews submitted to JAMS include elements that are highly descriptive. These include word clouds, lists of most highly cited papers, numbers of published papers (including numbers by journal and/or year), most cited journals, and network structures showing connections between authors, topics, and author countries. Such elements are easy to generate but provide few compelling insights that challenge readers to think or act differently when planning or conducting future research. What this means is that review-based description is only valuable when combined with synthesis to inspire new research ideas or directions.

The reasons for this disconnect between the need for novel insights and the delivery of mundane descriptive content are twofold. First, the data required to conduct bibliometric analyses are readily available and the required software tools are becoming simpler to implement with practice. This makes it easy to generate lists, tables, and maps. Second, researchers writing reviews often ignore the important distinction between two separate types of bibliometric information that have different uses: performance analysis versus science mapping (Donthu et al., 2021; Zupic & Cater 2015). According to Zupic & Cater (p.431), performance analysis “seeks to evaluate the research and publication performance of individuals and institutions” whereas science mapping “aims to reveal the structure and dynamics of scientific fields. This information about structure and development is useful when the researcher’s aim is to review a particular line of research.”

Selecting appropriate bibliometric tools

For researchers primarily interested in describing the sociological foundations of a focal field of study, the use of performance analysis is appropriate. For example, an examination of author co-citation information can reveal both influential authors and the extent to which different authors are interconnected in terms of their publication profiles. Such analyses can have merit (e.g., for reviews intended to provide an historical account of a field’s evolution) but add value primarily in sociological terms, dealing as they do with the people and institutions that represent the intellectual resources driving knowledge creation (i.e., the “inputs”). Even in these cases, though, scholars should take care to place less emphasis on the “who” per se, and more on the implications of the underlying evolutionary network.

However, for journals (including JAMS) focused primarily on novel intellectual contributions that result from the research process (i.e., the “outputs”), such an orientation has limited value. In these cases, the adoption of a science mapping mindset will prove more effective. For example, co-word analysis, which reveals words that frequently co-occur in documents, can be used to reveal important themes (both established and emerging) within the focal domain.¹

Once the specific aims for a review are established, the researcher needs to select specific bibliometric tools best suited to achieving those aims. Typically, this will involve the use of several different techniques that offer complementary strengths while offsetting the weaknesses. Five of the most used bibliometric techniques are listed in the Table.1.²

Table 1 Summary of some common techniques for bibliometric

In addition to using these traditional tools, researchers can augment their bibliometric analyses with an “enrichment toolbox” that includes network metrics, clustering algorithms, and visualization software (Donthu et al.2021) These auxiliary assessments are often helpful both in identifying and communicating novel insights. For example, Borah et al. (2023) use co-word analysis in conjunction with clustering and visualization tools to derive the structure shown in the Fig.1.³

Fig. 1

Example of thematic clustering

Several insights emerge from this illustration that would not be readily apparent without the use of bibliometric techniques. For example, the themes of customer and brand value (cluster 1), customer experience and satisfaction (cluster 2), marketing strategy and innovation (cluster 4), and firm performance and shareholder value (cluster 6) all lie at the center of the map. This may not be surprising, given that these themes have been central to marketing theory and practice for decades. In contrast, themes such as service-dominant logic (cluster 7), salesperson performance (cluster 9), and corporate social responsibility (cluster 3) can be found in the periphery, indicating that they are less integrated with the core themes. Some of this may be due to the fact that they are newer areas of inquiry, but their positions at the edges of the map suggest opportunities for greater connections between these emergent themes and traditional ones.

Relative position can also be informative. Note that the corporate social responsibility (CSR) cluster is located adjacent to the marketing strategy (cluster 4) and shareholder value (cluster 6) clusters. This reflects the early foundations of CSR research, primarily focused on identifying innovative sustainable business approaches that maximized shareholder value. More recent CSR work has challenged these roots, placing a greater emphasis on what it means to be “socially responsible” as a firm while also broadening the lens to include a diversity of stakeholder perspectives.

More careful reflection on the content of this cluster map suggests two distinct axes, one (unit of analysis) defined by a clear firm focus at one end (clusters 3, 6, and 7) versus an individual customer / employee focus at the other end (clusters 1, 8, and 9). The second axis (level of analysis) extends from customer satisfaction (clusters 2 and 5) to market-level responses (clusters 3 and 4).

Combining approaches for impact

The preceding example illustrates the use of multiple bibliometric tools to drive towards novel, meaningful insights. However, bibliometric methods are often most effective when combined with more traditional systematic review approaches. As Zupic & Cater (2015; p. 436) conclude, “bibliometric methods are not a substitute for but a complement to traditional methods of review [through] increased objectivity.” Similarly, Mukherjee et al. (2022; p.103) caution that these “techniques, on their own, are not a panacea for advancing theory and practice.”

When used appropriately, bibliometric techniques can add to systematic domain reviews both by facilitating a more complete understanding of the existing literature and by revealing novel insights. In doing so, they can go beyond simple description to enrich what Hulland and Houston (2020; p. 353) refer to as synthesis, an effective way of organizing existing studies “to guide the reader toward a better understanding of the focal phenomenon.” This synthesis represents the core of any review paper, whereas bibliometric tools are just one potential method for describing the literature. Thus, bibliometric approaches can aid the identification of what Hulland and Houston describe as “significant new insights that emerge from looking across the body of scholarship (i.e., insights that are not apparent from a careful examination of any individual study or small group of studies),” but they should be viewed as means to an end rather than as ends themselves.

Conclusion

The advent of bibliometric techniques has created new opportunities for researchers to study and understand important research domains. However, mindless application of these tools results in findings with no meaningful scholarly value. To organize the existing literature in novel and insightful ways, scholars must invest significant cognitive time and effort in the careful interpretation of preliminary bibliometric findings. Without such an investment, the results will be superficial and unsatisfying. Furthermore, they will be completely useless in helping to create a powerful and prioritized agenda to guide future research in the domain. Scholars interested in meaningfully adding to an existing body of domain-based research by using bibliometric techniques must address these challenges; to not do so risks a high probability of desk rejection both at JAMS and at other leading scholarly journals.