Introduction

Research in addiction studies spans an increasingly diverse range of substances and behaviors, and a correspondingly large array of prevalence, determinants, harms, preventions, and intervention efforts. The research approach taken in these studies is determined by the nature of the research question, with the main approaches being qualitative, quantitative, and literature reviews [1]. Qualitative studies can address questions around subjective experiences such as how social context influences recovery, whereas quantitative research addresses questions on “what and how much”, such as what attitudes people hold towards addiction or the degree to which a particular intervention improves symptom severity. Given the vast and diverse accumulation of research and knowledge, there are also reviews that provide summaries or a synthesis of current knowledge in a particular area. Commonly, researchers and reviewers are well versed in one approach or various methods within one approach but are less familiar with other approaches or methods.

Maintaining sufficient knowledge to report or peer review research findings can be challenging when working with a less familiar method. We have provided editorial support or peer review for dozens of journals, and noticed there are common issues in the submitted articles, as well as key information that reviewers miss. The aim of this work is therefore to guide peer reviewers of such research on key issues related to data collection, analysis, and reporting. To guide this discussion, we searched Pubmed, Google Scholar, and addiction journals author instructions for important guidelines pertaining to qualitative research, quantitative research, and literature reviews. We have provided a brief overview of each of these major approaches (Table 1) and their application in addiction studies, as well as an overview of reporting guidelines that are helpful to consider when conducting manuscript peer review. To aid readability, we have used a selection of examples that are focused on addictive substances and behaviors, but we expect that the content is relevant to any health-related field.

Table 1 Overview of qualitative, quantitative and literature reviews in addiction studies
Full size table

Section 1. Qualitative Research

There are many types of qualitative research designs including ethnography, grounded theory, narrative research, case studies, phenomenological analysis, action research, and community engaged research [2,3,4]. Qualitative research can be used to understand the process of assigning meaning to an experience or phenomenon as it occurs in a specific context or environment, and demonstrate changes in social, cultural, or environmental contexts [5]. Qualitative research is also used to explore new fields of study. For example, qualitative research was one of the first methodologies to demonstrate the role of social context on illicit drug use [6].

Data Collection in Qualitative Research

Key to peer reviewing qualitative research is determining the robustness of the data collection and analysis. The focus of qualitative research is to offer an in-depth look at experiences rather than gathering data to generalise to whole populations. Because qualitative research explores the experiences of participants, sample selection is highly important. Participants should have specific expertise related to the research questions. Depending on the data generation approach, qualitative research may involve small samples when the approach is interviews or in situations in which the experience is new or not frequently experienced in the population (e.g., 5–10 people) but typically the sample size is around 15–25 participants [7]. Decisions on sample size are guided by data saturation, which is defined as the point at which new data collection is no longer adding substantially to insights on experience [8].

Data generation can be through interviews, observations, focus groups, visual or audio media, or analysis of archival or publicly available data or documents. In health care, inclusive of addiction studies, interviews or focus group discussions are the most common data collection techniques [4]. Interviews involve in-depth, semi-structured conversations about participants’ personal experiences and the meanings they assign to them. In contrast, focus groups usually consist of 4–12 people, with the aim of exploring specific issues such as attitudes towards the involvement of significant others in models of addiction recovery [9] through to policy options for vaping [10]. Participants answer facilitator questions and are encouraged to interact with each other, thereby generating data that reflect a shared and contextual perspective through group interaction.

Interviews and focus groups require research questions to be collated into an interview schedule that can be structured (i.e., the same question/s asked of every participant) or semi-structured (i.e., there is flexibility to follow discussions in unexpected directions within specified constraints) [4]. A good interview schedule includes a brief series of open-ended questions (not close-ended questions that prompt yes/no responses) that explore depth rather than breadth [5]. Questions are grounded in the literature so that they advance knowledge rather than being created in isolation for a particular project. Because the interview schedule shapes the range of participant responses and perceptions, it is summarised in the methods and/or included as a table, appendix, or supplementary file. Ideally, the interview schedule and interview process should be piloted or reviewed by a panel with lived experience of the topic to identify the optimal set of questions within the allocated interview time [4].

Data Analysis in Qualitative Research

When conducting peer review of qualitative research, attention is paid to the proposed approach to data analysis and the degree of coherence between the planned approach and eventual outcome. Broadly, qualitative analysis is undertaken by sorting data into meaningful groups, themes, or categories to address the research question, aims or objectives. The way data are grouped can be via deductive or inductive categorisation depending on the research design. An inductive approach means the researcher derives categories or a coding framework from the data, while a deductive approach means the researcher already has a theory or existing knowledge that will be applied to the data [11, 12]. Data analysis can be done by manually using the transcripts or MS Excel, or via specific data analysis software such as NVivo.

Qualitative analyses generally require data familiarisation and initial coding, followed by grouping codes into similar categories and themes. Data analysis is more than grouping responses to questions and instead there should be analysis based on a specific technique. Two commonly used data analytic techniques are Thematic Analysis and Content Analysis. Thematic Analysis is a method for systematically identifying, organizing, and interpreting patterns of meaning across a dataset, involving specific steps for undertaking the analysis [13]. For example, researchers might identify themes such as barriers and facilitators to the uptake of smoking cessation services [14] through to challenges experienced by family impacted by addiction [9]. In recent years, Thematic Analysis has been extended to acknowledge the active role of the researcher in theme development terming it Reflexive Thematic Analysis [15, 16]. When reviewing a paper using Reflexive Thematic Analysis, one would expect evidence of an iterative approach to identifying, refining, and naming themes. Additionally, there should be a statement on the researcher’s role and an acknowledgment of how they might have influenced the data collection and analysis process [4]. In contrast, Qualitative Content analysis [12, 17] systematically organizes larger amounts of text, visual, or other non-numeric data into themes and meanings via a pre-existing or emergent coding scheme or data dictionary [17, 18]. The data are then systematically organized into the coding scheme by two or more coders, and frequencies are reported to help identify patterns, relationships, and trends in the data.

Quality and trustworthiness of qualitative analysis can be improved by addressing potential biases that can influence how data are collected and analysed. Researchers can report potential bias by disclosing their own experiences of the subject matter or check whether their interpretation of experience is consistent with research participants and other members of the research team [3, 18]. For example, data analysis of research in gambling studies could be biased by the researcher receiving funding from the gambling industry. Equally, analysis can be biased when the researcher is involved in anti-gambling or advocacy groups, or receives funding to provide the service that is the focus of the research [19]. Content analysis generally requires two or more coders, and inter-coder agreement is reported. However, other approaches, like Reflexive Thematic Analysis, can involve just one coder who debriefs with peers and reaches an agreement on themes which can be helpful in demonstrating transparency and rigor [15].

The synthesis and interpretation of qualitative results varies by research design, subject discipline, and analytic approach. The main themes, interpretations and findings are presented alongside evidence, usually in the form of quotes from participants. For each broad theme, one or two illustrative quotes are sufficient and long strings of quotes should be avoided [2]. Quotes should be presented so that the person cannot be identified but include any participant information that is relevant to the research question such as age, gender, or symptom severity. Interpretation and implications should move beyond description of a phenomena and connect to wider knowledge and literature to make an original contribution [3]. Depending on the research design, the development of a theory or model or comparison with the wider literature may occur in the results or discussion section of a manuscript.

Reporting Standards for Qualitative Research

There are a range of reporting standards for qualitative research that reviewers can use as a guide. These include the Standards for Reporting Qualitative Research (SRQR) [3], the 32-item Consolidated Criteria for Reporting Qualitative research (COREQ) for interviews and focus groups [4], and the Critical Appraisals Skills Programme [20]. Most addiction journals now accept qualitative studies and require adherence to one of these standards to indicate a commitment to quality and rigour in the research [21, 22]. As such, authors and reviewers should be familiar with these standards and carefully check the requirements of the relevant journal.

The most common issue identified in peer review of qualitative work is the lack of detail on study design and how the analysis was conducted [4, 21,22,23]. Information that is sometimes missing includes how the sample was recruited, the characteristics of the sample and their expertise on the topic (often presented as a participants characteristics table), what questions were asked, who collected the data, and how the data were analysed. Limitations reported in qualitative research can range from trustworthiness, sample selection, sample size, how the data were collected, and potential biases in data analysis [22].

Section 2. Quantitative Methods

Quantitative research can be employed to describe the characteristics of a population or phenomenon but also to examine associations between an exposure and outcome or relationships between variables. Quantitative studies can have experimental or observational research designs that are carried out in controlled or natural settings. In experimental studies, exposures are assigned to the participants by the researchers. Such studies may take the form of a randomised-controlled trial (RCT) where the exposure (i.e., an intervention) is assigned at random to one subsample of the study population while the other subsample receives the established standard of care, no care, or a placebo [24]. Random allocation in this type of study ensures limited confounding by baseline variables and can provide strong evidence of causal relationships between the exposure and outcome [24, 25]. Alternatively, quasi-randomised controlled trials involve non-random allocation to intervention, which may be necessary because of ethical or practical concerns [26]. In contrast, observational studies examine the influence of real-life exposures without any manipulation of the exposure [27]. Examples include cohort studies, in which participants are followed over time and outcomes of interest may or may not become apparent during subsequent timepoints, and case control studies, which follow participants over time but select participants based on the presence (i.e., the cases) or absence (i.e., controls) of an outcome and in which factors (i.e., the exposures) associated with the outcome are assessed retrospectively [28, 29]. Finally, cross-sectional studies are also observational in nature but there is no time component involved. Instead, any exposures and outcomes are measured at the same point in time, commonly making such studies less costly but unable to provide information on causal relationships [30]. Table 2 provides an overview of a selection of quantitative study designs.

Table 2 Selection of quantitative study designs in addiction studies
Full size table

Data Collection in Quantitative Research

A key task when reviewing a quantitative study is to assess the degree to which the study population and methods are appropriate to answer the research aims. Related to this are the external and internal validity of the study findings as a result of the study design. External validity refers to the extent to which the research findings are generalizable to the wider population or other settings [31, 32]. Sample sizes should be large enough to be representative of the population of interest and to detect an effect, while other considerations may be attrition in studies with follow-up evaluations and the number of variables to be included in the specified analysis [33, 34]. Researchers may use formal power calculations to justify the number of participants recruited into the study [32, 35]. Population studies may have a heterogeneous composition to be reflective of the general population to which they aim to generalize their findings. For example, a repeated cross-sectional study examining the rates of alcohol and marijuana abstinence, co-use and disorders in the United States used data from an annually conducted survey of 187,722 young adult students and non-students, resulting in a heterogeneous sample representative of the youth population [36]. However, other study designs may apply a strict set of selection criteria to limit confounding or enable generalization to a specific population. For example, a cross-sectional study which assessed predictors of alcohol misuse among Canadian Aboriginal youth had a smaller (191 youths) and less heterogeneous sample as the target population was more specific and it may not have been feasible or practical to recruit more participants [37].

In contrast, internal validity refers to the degree to which relationships between exposure and outcome are a true effect which is not distorted by the influence of extraneous or confounding variables. High internal validity means there is low systematic error in the design, conduct or analysis of a study such as participant selection, randomization, treatment fidelity or participant attrition [32]. Experimental studies like RCTs may use a narrow set of selection criteria, resulting in a more homogeneous population to limit potential confounding of the association between exposure and outcome. Although internal validity is a prerequisite for external validity, increasing the former may affect the latter. For example, studies investigating interventions for alcohol addiction commonly have exclusion criteria related to comorbidity and factors that predict attrition, but applying such criteria makes a large proportion of the target population, often those with more severe mental health issues, ineligible for participation, thereby reducing generalizability of findings [38].

Quantitative research collects numerical data and relies on the use of systematic and standardised approaches to do so, such as questionnaires, experiments, and observations [39]. To ensure study findings reflect what researchers set out to measure, it is essential to incorporate clear definitions of all relevant concepts and operationalise concepts by stating how they are measured [40]. Where possible, best practice is to use well-established scales that have been assessed on their psychometric properties. Widely used standardized instruments, such as the Addiction Severity Index [41], have been validated in various settings and languages [42]. When a topic has not been extensively studied, standardized and validated instruments may not be available and researchers may opt to adapt an instrument which measures a related concept or construct a bespoke instrument. When an adapted or bespoke instrument is employed, researchers are required to indicate the points of adaptation or provide the instrument in the supplementary materials.

Data Analysis

When conducting peer review of quantitative research, it is helpful to keep in mind some key points. Data analysis should be fully described in the methods including how the data were cleaned, prepared for analysis, and the specific statistical techniques used, with justification in terms of the research question, the sample size, and the data in question. In essence, data analysis will assess the difference between groups on two or more variables or the relationship between variables. Most statistical models underly one or more assumptions regarding the nature of the data, such as distribution of the data and presence of outliers, and models may produce misleading results when these are violated [43]. Missing data can also influence the results, especially when more than 10% of data is missing [44]. Missingness might be at random, but is more commonly missing at non-random, as when those with certain individual characteristics are less likely to provide a response to specific survey items or be retained in a longitudinal study. For example, RCTs and cohort studies have shown higher drop-out from in-person substance use disorder treatment programs among individuals with lower income [45]. Similar to participant recruitment and selection, missing data can result in underrepresentation or overrepresentation of population groups, and systematic differences underlying the missing data might lead to biased results and impact generalizability of the findings [46]. There are various analytical approaches to control for confounding and address missing data [44, 47] and significant statistical expertise may be important to advise on the most appropriate analysis technique. Most journals have statistical experts who can review methodology and therefore when peer reviewing, it is important to alert the editor to your skill set and the need for expert statistical review, if required.

Reporting Standards in Quantitative Research

There are a range of reporting standards that provide guidance on the information that should be included in manuscripts describing quantitative research. Table 2 provides an overview of standards including the American Psychological Association Journal Article Reporting Standards (JARS) for quantitative research in psychology and its extensions [40], the Consolidated Standards of Reporting Trials (CONSORT) [32], and the Strengthening The Reporting of Observational studies in Epidemiology (STROBE) [35]. When conducting peer review, it is important to use the relevant reporting guideline to check that key information is present and logically presented. Quantitative studies are often published in journals with limited word counts, but the evidence-based rationale for the study should still be established clearly and concisely. There should be sufficient justification and description of how and when data were collected, what type of target population was involved, and who the participants were, in terms of their age, gender, race or ethnicity, and any clinical or social characteristics that might affect the generalisability of the study findings. Authors should also clearly justify the analyses undertaken and report the results clearly according to expected standards. Almost always, there should be a clear distinction between results and discussion with interpretation undertaken later in the discussion section. The discussion should discuss the results and their implications in the context of existing research and theory, along with practice, policy, and future research implications, where relevant. Issues often noted by editors include authors simply restating the results in the discussion, without linking to other relevant literature or extrapolating well beyond their results.

Risk of bias tools can also help pinpoint potential sources of bias. Risk of bias refers to the likelihood that a study may report misleading results; hence, risk of bias tools assesses factors that can introduce bias, such as the methods of participant selection, allocation to groups, and outcome assessment and reporting. Risk of bias tools vary according to the study methodology because they assess specific components of studies. For example, there are specific tools to assess randomized [48] and non-randomised trials [49].

Section 3: Literature Reviews

Literature reviews provide a method to synthesise findings from quantitative and qualitative studies to answer specific questions about what is known about a particular topic. The value of literature reviews lies in their ability to provide a summary of current knowledge that is more comprehensive than any single study. Review purposes range from informing clinical decision-making to setting research priorities. Literature reviews can be undertaken to summarize, synthesize, appraise, or scope primary quantitative or qualitative published or unpublished literature. The current paper focuses on reviews that use a systematic search, as these reviews are increasingly prioritised for publication by journals. Table 3 provides an overview of common review types that use systematic searches. To reduce bias and increase transparency of data collection and analysis, reviews are prospectively registered (e.g., PROSPERO, Open Sciences Framework [50, 51]).

Table 3 Overview of a selection of literature review types with systematic search
Full size table

Key to peer review is checking there is a rationale for the selected review type. This selection is informed by the research aim, and each review type differs somewhat in its data collection, analysis, and synthesis. Systematic reviews use a systematic approach to identifying relevant studies and aim to explain or analyze the available evidence [52,53,54,55]. Examples include understanding the effectiveness of interventions like stigma reduction methods for substance use disorders [56] or understanding whether the concentration of THC in cannabis has changed over time [57]. Meta-analyses use a statistical approach to combine results from two or more studies. For example, meta-analyses can be used to determine the average rate of occurrence across settings or populations, such as the rates of help-seeking by people who gamble at different levels of gambling risk [58]. All meta-analyses are based on a systematic review but not all systematic reviews use a meta-analysis. Rapid reviews are pragmatic, time-sensitive, and emerge through stakeholder consultation [8, 59, 60]. Rapid reviews conducted during the recent pandemic have helped summarize the literature on topics such as the impact of COVID-19 public health responses on the risk for illicit drug overdose [61]. Scoping reviews use a systematic approach to data identification but are used to map the type and amount of evidence on a new or emerging area or to clarify definitions or concepts in the literature [62,63,64,65,66]. Unlike other review types, scoping reviews do not answer questions about feasibility, appropriateness, meaningfulness or effectiveness and, as such, should not be synthesising or reinterpreting the evidence [67]. An umbrella review, or review of systematic reviews, can be helpful when multiple systematic reviews have been conducted on the same topic. Umbrella reviews can explain and contrast similarities and differences across studies and synthesise the findings into a single summary [68, 69]. For example, there is a large body of evidence on the impact of advertising on gambling-related harm, and synthesizing this evidence has the potential to affect policy change [70].

Data Collection in Literature Reviews

Peer review of literature reviews includes careful examination of data collection methods. Data in literature reviews are secondary in that they are identified and extracted from a set of previous studies. Data are collected through systematic searches for published or unpublished literature. A systematic approach to identifying data is needed when the review purpose is to provide a comprehensive account of the available literature. The development of a systematic approach starts with finely developed eligibility criteria that are directly linked to the research aim and often relate to participants, interventions, comparisons, and outcomes [52, 54]. The inclusion and exclusion criteria are informed and justified by previous studies, such as restricting the review to prospective studies only or adolescent samples or solely unpublished studies. Eligibility can be narrow, like when the aim is to understand how marketing impacts binge drinking for adolescents [71] or broader, like the risk and protective factors for alcohol and tobacco-related disorders [72].

To locate relevant evidence, a search strategy is devised in conjunction with content or information management specialists [67, 73]. The search strategy details which databases will be searched and with which keywords. The selection of databases depends on the research aims and topic, but at least four databases ensure relevant studies are located [73, 74]. This combination could include Web of Science Core Collection, PubMed, Embase, MEDLINE, and Google Scholar, as well as domain-specific resources like PsycINFO for psychological studies, CINAHL for health-focused reviews, and the Cochrane Registry database for clinical trials. Databases do not list exactly same journals, and there is often a lag between publication and indexing, which is why the strategy starts with multiple databases [74]. The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA-S), a guide specifically developed for systematic searches, suggests a full list of unpublished sources (i.e., grey literature) to be searched, such as registries, and recommends that citation searches through reference lists are reported [75, 76]. Grey literature searches can include databases such as registries or repositories, Google searches, targeted websites, and consultation with experts [77]. Often in grey literature searches, there is no abstract, and instead, the executive summary or table of contents are screened. When conducting a Google or Google Scholar search, usually the number of records screened is limited (e.g., the first 100 records on a specific date), which should be justified and noted in the review [77].

Peer review should check that each step of the data collection process is documented in sufficient detail to enable replication of the study. Steps include extracting identified records into one of the purpose-built review or referencing systems (e.g., Covidence, Rayyan, Endnote) or a spreadsheet like Microsoft Excel and removing duplicates. The title and abstract of each record are assessed against the study eligibility criteria by one or more reviewers. A shortlist of studies that may meet the eligibility criteria are then fully screened by reviewing the full text. Data from studies meeting the eligibility criteria are extracted into a standardized spreadsheet or purpose-built review system (e.g., Covidence) reflective of the research aims, and commonly include study characteristics, such as population size, study focus, and setting, as well as data reflective of the research question, such as number of people with an outcome per group [67, 75]. More rigorous systematic reviews employ double data extraction by two researchers, with conflicts in decisions resolved through researcher discussion [78].

Data Analysis in Literature Reviews

When conducting peer review of literature reviews, it is helpful to examine how data have been synthesised and that the approach is consistent with the review type. Data analysis involves the synthesis or merging of data from included studies on each key variable through a narrative summary, critical appraisal, and, where appropriate, statistical calculation of the size of the effect via a meta-analysis [54, 79]. A narrative summary describes the results for each variable with a combination of frequency counts and thematic grouping, allowing for new insights. Systematic reviews, meta-analyses, rapid reviews, and reviews of reviews include a critical appraisal to assess the quality of the studies (see Table 3 for an overview). Assessing quality helps with the interpretation of findings as it provides an indicator of internal and external validity of included studies [80]. Meta-analyses produce a statistical summary of the included studies and use subgroup and sensitivity analyses to examine the effect with and without certain variables [54, 79, 81]. For example, in a study looking at global rates of self-exclusion from gambling venues using population prevalence data, an alternative rate was estimated by limiting the analysis to data extracted from higher-quality studies [82].

Reporting Standards in Literature Reviews

The role of peer review of literature reviews is to assess the quality of the manuscript against specific reporting standards. The main standard is PRISMA, which offers guidelines for systematic reviews and meta-analyses [83]. There are PRISMA extensions that guide reporting for different types of reviews, from diagnostic tests to health equity and mixed methods reviews that include both qualitative and quantitative data. There are also specific requirements for reporting systematic reviews that contain qualitative data only and that use qualitative methods for data analysis and synthesis [84]. Rapid reviews streamline the search strategy so transparent reporting is important [8, 59, 60, 85]. Presentation of the systematic search uses a flow chart that summarizes the count at each step [83]. The PRISMA flowchart differs according to the type of review and sources used to identify studies like websites, organisation, and reference list searching. Data synthesis almost always involves a table of the characteristics of included studies that is discussed in the text, and where there is a meta-analysis, results are presented as a forest plot, which is a graphical presentation which summarizes individual and pooled study results [54, 79]. Given scoping reviews aim to map the available literature, findings can be presented as a numerical count in a chart, table, or graphic [67].

Reporting standards are important for replication and for being able to assess the quality of the evidence. For example, a review of the methodological quality of reporting in systematic reviews for e-cigarettes and cigarette smoking reported that almost all reviews were of low quality because they missed reporting key content to enable replication, such as how the search was conducted and the lack of pre-registration [86]. Key areas to notice for peer review are that the inclusion and exclusion criteria reflect the research aims and that there was a protocol or plan made prior to starting the review.

A risk of bias tool, such as those mentioned in the section on quantitative research [48, 49] should also be applied in systematic reviews or meta-analyses that seek to conduct a comprehensive analysis of the quality of available literature or that aggregate data. Peer reviewers can also use risk of bias tools specifically designed for systematic reviews, such as AMSTAR 2 [87], to identify potential sources of bias within such reviews.

Conclusions

The best peer reviews are those that are balanced, fair, helpful, and directed towards the research. For this reason, peer review is most useful when it follows established guidelines, avoids personal preferences, and refers to ‘the research’ rather than ‘the author’. Furthermore, peer review is not about deciding if a paper submission fits with the journal scope, but instead involves advising the editor with an expert assessment of submitted research. Best practice peer reviews identifies study strengths but also weaknesses, such as poorly defined selection criteria that are not in line with the research aims.

Researchers are often called on to peer review studies that have topics which are consistent with their background, but that use a research approach or methods with which they are not as well acquainted. The purpose of this review was to provide information to support peer review in these situations. This review aimed to provide useful links and resources to guide the reader across these three commonly used methods. In addition to the presented content, peer reviewers may have to grapple with new methods like the use of AI or studies to develop consensus guidelines, but we believe that the same principles apply irrespective of the methods - consult the relevant reporting guidelines and consider the degree to which the submission has internal logic. That is, have the authors articulated a clear evidence-based rationale for the research and the methods used? The reviewer then considers the degree to which the research has been conducted as intended and that the discussion works to advance knowledge and locate the new findings within what is already known.