Background

Animal experimentation in scientific research is required for the acquisition of new knowledge of mechanisms in biology and medicine, or to answer specific scientific questions [1], and so represents a basic tenet of biomedical research. Animal experiments are also an important aspect of preclinical research. As a bridge between basic research and clinical trials, the quality of animal experiments affects the achievements and conclusions of research studies in many fields [2, 3].

Over recent years, the number of animal experiments published in biomedical journals has increased dramatically. Increasing numbers of studies have shown that [4,5,6,7], even in animal experiments published in top journals, the quality of reporting remains unsatisfactory. A review by The National Centre for the Replacement, Refinement and Reduction of Animals in Research, the NC3Rs [8], demonstrated that many funded and published animal experiments had inadequate reporting of important information such as study design, implementation, and analysis. In 271 animal experiments included in the review, 41% did not state the hypothesis or objective of the study or the number and basic characteristics of the animals used in the experiments, and 30% of them did not describe the statistical methods or did not present the results with correct statistical indicators. In addition, good animal welfare is closely related to the reliability and repeatability of animal experiments. Seeking new methods to better promote animal welfare is also a concern of the 3Rs center [9]. The inadequacy and incompleteness of the quality of articles and poor quality of reports have seriously hindered the utilization of the scientific and practical value of those animal experiments. Therefore, in order to improve the quality of animal experimental reports, based on the CONSORT Statement [10], the NC3Rs developed the ARRIVE (Animal Research: Reporting of In Vivo Experiments) guidelines, which were published in 2010. The guidelines consist of six parts: title, abstract, introduction, methods, results, and discussion, with 20 items and sub-items [11]. Since 2010, researchers outside China have studied the quality of animal experimental reports published in published in international journals based on the ARRIVE guidelines [12,13,14], whereas in China, similar research has only been published by Liu et al [15] Furthermore, their research is limited to specific diseases. In addition, no studies have explored whether publication of the ARRIVE guidelines has improved the quality of animal experimental reports.

Therefore, we collected comprehensively animal experiments published in China, reviewed and analyzed the quality of the experimental reports and existing problems based on the ARRIVE guidelines, in order to provide a reference for the promotion of quality of animal experimental reports in China.

Results

Selection process and results

A total of 21,713 potentially relevant studies were initially selected. After the exclusion of duplicates and those that did not fulfill the inclusion criteria, 4342 studies were ultimately included, in which 4925 animal experiments were conducted (as some included multiple animal experiments). The selection process and results are shown in Fig. 1. Each sub-item in each study was deemed either “yes” (total or partial compliance) or “no” (noncompliance), where the former was defined as a study reporting all or a part of at least one information sub-category items in detail. The compliance rate for the 39 sub-items deemed “yes” was calculated (See “Methods” for details).

Fig. 1
figure 1

Flow chart for the process of article selection and results. *Retrieved databases and selected studies as follows: CNKI (n = 1213), WanFang (n = 5500), VIP (n = 1702), CBM (n = 13,298)

Full size image

Basic characteristics of the studies included in the review

The number of citations for each included study ranged from 0 to 12. More than 70% (73.0%, 3171/4342) had been cited fewer than 5 times, of which nearly 50% (47.6%, 1510/3171) had not been cited at all. Only 29.0% (1261/4342) had been published in journals included in the Chinese Science Citation Database(CSCD), which is a collection of outstanding Chinese and English journals in various fields in China and known as “Chinese SCI”. The greatest proportion of first authors were clinicians (45.1%, 1957/4342) (Table 1).

Table 1 Basic characteristics of studies included in the review
Full size table

Of the 4925 animal experiments included in the review, the most frequently-used species of animal was the mice (81.3%, 4002/4925), then rabbits (15.1%, 742/4925), and dogs (1.91%, 94/4925). Of the type of intervention, the top three were drugs (82.2%, 4050/4925), surgery (7.39%, 364/4925), and acupuncture (3.70%, 182/4925) (Fig. 2). In coverage and classification of related diseases, the top three were circulatory diseases (10.9%, 538/4925), digestive diseases (10.6%, 524/4925), and tumors (9.66%, 476/4925).

Fig. 2
figure 2

Laboratory animal species and interventions of animal test reports published in Chinese journals

Full size image

Reporting quality of studies included in the review (Fig. 3, Table 2)

The ARRIVE guidelines have 6 sections with 20 items and 39 sub-items. Only 28.2% (11/39) of the sub-items were more than 90% compliant. The “yes” compliance rate of approximately half of the sub-items (51.3%, 20/39) was less than 50%, of these “yes” poorly compliant items, 90% (18/20) were less than 30% compliant, while 65% (13/20) were less than 10% compliant.

  • ① Items relating to the title, abstract, and introduction.

Fig. 3
figure 3

Assessment of reporting quality of included studies

Full size image
Table 2 Assessment of reporting quality of included studies
Full size table

For items 1–2, involving the title and abstract, the “yes” compliance rate was higher than 90%. The “yes” compliance rate of items 1 and 2 in studies published after 2010 was higher than that of studies published before 2010. The differences between the two groups for both items 1 (P = 0.012) and 2 (P < 0.001) were statistically significant.

For items 3a-4, which involved the abstract, the “yes” compliance rate was higher than 90%, although the “yes” compliance rate of item 3b was only 37.1%. The “yes” compliance rate of items 3a (P < 0.001), 3b (P < 0.001) and 4 (P = 0.003) in studies published after 2010 was higher than that of studies published before 2010, and the differences between the two years of publication were statistically significant, for item.

  • ② Items relating to methodology.

For 28 items (items 5-13c) involving methodology, the “yes” rate of compliance for items 6a, 6b, 6c, 7a, 8a, 8b, 10a, 11b, 12, 13a, and 13b was greater than 50%. The “yes” compliance rate of items 6b, 6c, 7a, 8a, 11b, and 12 was greater than 90%, but less than 50% for items 5, 6d, 7b, 7c, 7d, 9a, 9b, 9c, 10b, 10c, 11a, and 13c. Except for item 7d, the “yes” rate of compliance in the latter group was less than 30%, and for items 5, 6d, 7b, 9c, 10b, and sub-item 10c it was even less than 10%. The compliance rate for items 5 (P = 0.003), 6a (P < 0.001), 6b (P = 0.002), 6d (P < 0.001), 7d (P = 0.007), 8a (P < 0.001), 9a (P < 0.001), 9b (P < 0.001), 10a (P < 0.001), 11a (P < 0.001), 11b (P < 0.015), 12 (P < 0.006), 13a (P < 0.001), 13b (P < 0.001), and 13c (P < 0.001) in studies published after 2010 was slightly higher than those before 2010, but not for 6c (P = 0.890), 7a (P = 0.242), 8a (P = 0.268) or 10b (P = 0.297). The differences between the two publication year groups were significantly different for items. However, the “yes” compliance rate for items 7b (P < 0.001), 7c (P < 0.001) in studies published after 2010 was even lower than in reports published before 2010 but not for 9c (P = 0.939) and 10c (P = 0.474).

  • ③ Items relating to the results and discussion.

For items 14-17b involving the results, the “yes” compliance rate of items 15a, 15b, and 16 was greater than 50%, while for items 14, 17a, and 17b it was less than 6%. The “yes” compliance rate for items 14 (P < 0.001), and 17a (P = 0.002) for studies published after 2010 was higher than in studies published after 2010, but not for items 15a (P = 0.063), 15b (P = 0.402), or 17b (P = 0.337).

For items 18a-20 involving the discussion, except for the “yes” compliance rate of item 18a, which was greater than 90%, that of the other items was lower than 10%. The “yes” compliance rate of the 5 items in studies published after 2010 was higher than those of studies published prior to 2010 for items 18a (P = 0.001), 18b (P = 0.032), 18c (P = 0.048), and 20 (P = 0.005), but not for item 19 (P = 0.128).

Discussion

The present study is the first comprehensive and systematic review of the reporting quality of animal experiments published in Chinese journals based on the ARRIVE guidelines. Following publication of the guidelines, the compliance rate of the majority of items improved to a certain extent although there remains continued inadequate reporting of specific items, e.g. study design, procedures, sample size, adverse events, and discussion.

Methods section

High-quality research is inseparable from scientific design and rigorous implementation. These guidelines (ARRIVE) aimed to maximize the output of animal research by optimizing the information provided in publications regarding experimental design, implementation, and analysis [11]. In the ARRIVE guidelines, methods include nine aspects: ethical statement, study design, experimental procedure, animals, housing and husbandry, sample size, allocation of animals to experimental groups, experimental outcomes, and statistical methods. The present research showed that:

  • ① Ethical Statement: Of the animal studies published in Chinese journals, only 6.59% reported an ethical statement (item 5). The compliance rate was slightly higher in studies published after the publication of the ARRIVE guidelines, but only to 7.74%. Nikki Osborne et al. conducted a review of the literature that had published original studies on animals, and the results indicate that a considerable number of journals have not made this a mandatory requirement [16]. A number of studies have shown [17] that at least 20 million animals are used in scientific experiments every year in China, of which approximately 25% suffer pain from the procedure, but approximately 10% are not administered painkillers. An analysis of the ethical status of experimental animals in China by Chen Jie et al. [18] demonstrated these serious problems, where animals are used as a tool to obtain experimental data without considering any suffering they may have to tolerate in the process of experimentation. Highly distressed laboratory animals may lead to unreliable conclusions and/or unnecessary changes in scientific output, thereby affecting the reliability and repeatability of experiments, therefore every effort should be made to reduce unnecessary harm to them [9]. Although all countries have successively introduced laws and policies on animal welfare, and have even established animal protection organizations to supervise the regulations [19, 20], Olsson [21] and other researchers have also discussed how better to promote the issue. However, efforts on animal welfare remain unsatisfactory. Therefore, it is necessary to further regulate animal experimentation through legislation. More attention should be paid to animal ethics, adhering to the “3Rs” principle, and respecting the value and rights of animals.

  • ② Study design: After 2010, although more studies used time-line diagrams or flow charts to illustrate the entire process of the design of the study and conduct of the research (item 6d), the actual compliance rate was only 2.31%. A number of studies have shown that more than half of preclinical studies are unrepeatable [22], due to deficiencies in study design, incorrect statistical analysis and inadequate reporting [8, 23, 24]. Therefore, a clear and detailed flow chart can be effective in displaying the entire process of the experiment and improve transparency of the experimental implementation.

  • ③ Experimental procedures: Of animal studies published in Chinese journals, only 6.20 and 11.8% of studies provided time of day when experimental procedures take place (item 7b) or where the experiments take place (item 7c), respectively. One study [25] demonstrated that it was not possible to scientifically evaluate the reliability of the results in 76 articles describing animal experimentation (with more than 500 citations) due to a lack of reporting of key information such as experimental process and procedures in the methodological part of the experiments. Detailed reporting of experimental processes and procedures is a key measure to ensure replication of the results is possible and improve their utilization and conversion rate.

  • ④ Housing and procedures for husbandry: Although research published after 2010 showed that the compliance rate for housing (item 9a) and husbandry (item 9b) improved (P < 0.001), actual compliance rates were only 29.5 and 28.4%, respectively. It is well known that the type of cage or housing, style of settlement, light/dark cycle, temperature, and water quality will impact experimental results. As Hooijmans et al. [2, 26,27,28] pointed out in their research that there can be a four-fold difference in light intensity between cages at the top and bottom of a rack, and small differences such as these have been associated with animal reproduction and behavior.

  • ⑤ Sample size: Of 4342 animal studies published in Chinese journals, only one study reported the algorithm and formula for calculating sample size (item 10b), and only six studies indicated the number of independent replications of each experiment (item 10c). The National Institute of Neurological Disorders and Stroke (NINDS) reached a consensus in 2012 on how to improve reporting in animal research reports for submission of articles and for applying for funding, in which they noted [29] “insufficient sample size may lead to false-negative results, and miss some potentially important discoveries”. Therefore, it is necessary to report the number of animals used in each group and the statistical methods used to calculate sample size.

  • ⑥ Allocation of animals to experimental groups: Although the rate of reporting of specific grouping methods for experimental animals published after 2010 (item 11a) was significantly higher than those published before 2010 (P < 0.001), the actual coincidence rate was only 12.0%. A number of studies have shown [8] that the process of randomization, allocation concealment, and blinding are important aspects that can reduce the risk of bias in an interventional animal experiment.

Results and discussion

In the ARRIVE Guidelines, the results and discussion sections include “baseline data, numbers analyzed, outcomes and estimation, adverse events, interpretation/scientific implications, generalizability/translation, and funding”. The research in the present study showed that:

  • ① Adverse events: The majority of animal experiments published in China (4278/4342, 98.5%) did not report details of important adverse events in experiments (item 17a). Researchers also need to pay attention to adverse events and analyze their nature, an important aspect for judging the pros and cons of an intervention. Therefore, accurate assessment and reporting of adverse events should not be neglected.

  • ② Interpretation/scientific implications: Only 5.25% of the 4342 animal experiments published in Chinese journals reported limitations (item 18b) and only 0.35% described the significance of their research methods or findings in terms of the 3Rs principle (item 18c). The purpose of scientific design is to reduce any bias that may be experienced during experimentation. However, in terms of specific implementation, such as the animal model used, any potential source of bias in the experimental process, and factors affecting the accuracy of the study will reduce the scientific significance of the study [11]. Therefore, it is necessary for the author to objectively evaluate and explain the reasons in the discussion section of the article in order to ensure that the study remains scientifically valid while helping users understand the conclusions. In addition, the implementation of animal experiments should follow the 3Rs (replacement, refinement, and reduction of animals in research), the core of which is to protect, and use fewer, or zero, animals [1, 30]. Scientific experimental design can reduce the number of experimental animals [31, 32]. Experimental technology and optimized research procedures can reduce pain and anxiety in animals in experimental procedures [32]. Flecknell retrospectively studied the use and analgesic effects of perioperative and postoperative anesthesia in animal experimentation, suggesting that appropriate anesthesia and analgesic programs should be adopted in future animal studies and that animal welfare should be sufficiently considered beyond meeting legal requirements [33, 34]. Therefore, it is necessary for authors to interpret the scientific implications in the discussion section of the paper, which can help readers fully evaluate the research information, promote the implementation and the 3Rs principle, thereby improving the quality of the animal experiments.

There are a number of limitations to this study: ① it was based only on the evaluation of animal experimentation published in Chinese journals. The results may not represent the quality of the research of Chinese researchers published in foreign peer-reviewed journals; ② only interventional animal experiments were included. The results do not represent the quality of reports of other types of animal experiments.

Conclusions

In summary, the “yes” compliance rate for Chinese animal experiments based on the 39 sub-items in the ARRIVE guidelines is not at a high level. In particular, insufficient reporting of methods, including an ethical statement, study design, experimental procedures, sample size, and allocation of animals to experimental groups provides evidence that researchers fail to fully understand the whole process of study design and implementation and, to a certain extent, hinder the transformation and utilization of research achievements. In addition, it should be pointed out that even though the criterion for achieving a “yes” classification in the present study was not particularly strict or rigorous, and the results of reporting quality remain rather poor. Additional problems would be exposed if studies were conducted in compliance with strict criteria. However, a randomized controlled trial conducted by Hair et al. [35] indicated that low compliance during the implementation of the ARRIVE guidelines may be due to the fact that journals only require authors to submit lists when they contribute, rather than editors having to review them according to the lists, hindering any improvement in the quality of experimental animal study reporting. Furthermore, we should also point out a number of existing problems in the current ARRIVE guidelines. Therefore, ARRIVE 2019 [36] has been optimized for rigor and transparency in comparison with the 2010 version, being more convenient for journals, institutions or researchers to understand and use by listing animal care and management separately. Therefore, the ARRIVE guidelines should be popularized, and researchers encouraged to follow them in the future, in particular, the ARRIVE guidelines should be promoted in specialized journals. Additionally, we also suggest that in practical research, animal ethics and welfare issues should be truly embraced. Where improvements to the design, implementation, and reporting of animal experiments are introduced, improvements in quality will ultimately follow.

Methods

Patient and public involvement

No patients or members of the public were involved in this study.

Inclusion and exclusion criteria

Inclusion criteria

Interventional animal experiments with no restriction/limitation on animal species or type of intervention.

Exclusion criteria

Republished studies; Non-medical animal experiments; In vitro experiments based on animal tissues, organs or cells; Experiments involving both animals and humans; Relevant journals from Hong Kong, Macao, or Taiwan, which are limited by the authority.

Search strategy

Searches of the Chinese Journal Full Text Database (CNKI), Chinese Academic Journal Database (Wanfang), Chinese Science and Technology Journal Database (VIP), and Chinese Biomedical Literature Database (CBM) from the inception of each database up to July 2018, were conducted. Search terms included: rats, rabbits, dogs, pigs, sheep, apes, frogs, orangutans, monkeys, animal experiments. A detailed search strategy for each database is presented in File S1.

Methodological quality control

Relevant training was conducted for the researchers (B.Z., K.Y.H., Y.B.J., T.Z., Z.Z.S., W.Y.Z., F.C., F.M., Q.Q.G., and L.Z.) who participated in the project, including gaining an understanding and interpretation of the content of the checklist of the ARRIVE guidelines, evaluation principles and methods, and procedures and methods of literature screening and data extraction. After training, the researchers were tested on independent literature screening, data extraction, and quality evaluation prior to the start of the formal study, using 10% of the literature selected for the formal review. Consistency of the evaluation was evaluated. With a Kappa value > 0.8, consistency was strong, and the training considered qualified.

Screened literature and extracted data

Four of the trained reviewers (B.Z., K.Y.H., Y.B.J., and T.Z.) independently screened, extracted and cross-checked studies based strictly on the inclusion and exclusion criteria. A third party (B.M.) judged any final decision in case of disagreement. A full-text data extraction table was created in advance, and completed during the extraction process, recording ① basic characteristics of the included studies: number of authors, the author’s names, dates, journal, animal species, sample size, intervention measures, etc.; ② Information for all items relating to the ARRIVE guidelines.

Quality assessment of the included studies

Three reviewers (W.Y. Z., W.Y.Z., and F.C.) assessed the quality of the studies included in the review based on the 39 sub-items of 20 items included in the ARRIVE guidelines. We responded each item to “yes” (total or partial compliance) and “no” (noncompliance), where “yes” was defined as a study that reported all or part of at least one information item of a sub-item in detail. Conversely, a paper was “no” if it did not. For example, in Item 6 (Study design, Table 2), the item would only be classified as “yes” if at least one of the sub-items (6a-d) was reported, otherwise it would be classified as “no” (none of the 4 sub-items reported). Finally, the compliance rate of “yes” for the 39 sub-items was calculated, i.e. the total number of studies with “yes” relative to the total number of included studies.

Subgroup analysis

Based on the year of publication (2010) of the ARRIVE guidelines, the included papers were analyzed comparatively (animal experiments published before 2010 vs. animal experiments published in 2010 or\ later). The “yes” compliance rate of 39 sub-items of animal experiments published before 2010 and those published in 2010 or later was calculated, respectively, i.e. the number of papers, of which each sub-item was assessed as “yes”, accounted for the percentage of the total number of animal experiments pre-ARRIVE and post-ARRIVE, respectively.

Statistical analysis

Data were analyzed using SPSS software (v21.0, SPSS, Chicago, IL). Categorical data are presented as frequency (n) and percentage (%). A chi-square test was performed to compare percentages between groups. P-values < 0.05 were considered statistically significant.