Abstract
Purpose This study is to examine whether the term “trend toward statistical significance” is used to describe statistically nonsignificant results in biomedical literature. We examined articles published in five high-impact pediatric journals, including The Lancet Child & Adolescent Health, The Journal of Pediatrics, Early Human Development, Frontiers in Pediatrics, and BMC Pediatrics to identify manuscripts where a “trend” was used to describe a statistically nonsignificant result, from January 2020 to December 2021, and, furthermore, for The Journal of Pediatrics, Early Human Development, and BMC Pediatrics from January 2010 to December 2011. We detected that a “trend toward significance” was used to describe a statistically nonsignificant result at least once in 146 articles (2.7%) during the period between 2020 and 2021 and in 97 articles (4.0%) during the period between 2010 and 2011. We found no significant difference in the proportion of published articles with inappropriate use of “trend” across journals belonging to the first quartile of impact compared to the second quartile or across journals publishing under the subscription model or open access policy compared to journals publishing solely under the open access policy, in any period. The overall proportion of the inappropriate use of “trend” declined significantly between 2010 and 2011 to 2020 and 2021 (p = 0.002, RR 0.66 95% CI 0.51–0.86).
Conclusion: “Trend” statements were sporadically used to describe statistically nonsignificant results across pediatric literature. The inappropriate use of “trend” to describe almost significant differences could be misleading, and “trend” should be reserved only when a specific statistical test for trend has been performed, or in relation to appropriate scientific definitions.
What is Known: |
•Previously, researchers have reported inappropriate use of “trend” in articles across anaesthesia or major oncology journals. |
•In many cases, hypothesized results that are close but not lower than the statistical significance threshold are emphasized as “almost” significant. |
What is New: |
•“Trend” statements were sporadically used to describe statistically nonsignificant results across pediatric literature. |
•Inappropriate use of “trend” was similar in journals with a subscription model compared to those having an open access policy and decreased within a 10-year period. |
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
In biomedical literature, reporting results that fail to reach statistical significance might be challenging [1]. When the p value approaches but does not reach the level of significance, authors commonly adopt the term “trend toward statistical significance” to refer to statistically nonsignificant results [1,2,3,4]; however, this inappropriate use of “trend” is problematic. In statistics, there is no definition of a “trend toward statistical significance” and, therefore, describing “almost significant” results as a “trend” might be misleading. In fact, when it is required to estimate the probability that observed changes in an apparent trend represent true rather than chance differences, formal statistical tests should be used including the chi-square test for linear trend, the Cochran-Armitage test, or the Mann–Kendall trend test [5,6,7]. Besides, “trend” can be used in the biomedical literature to describe apparent changes as per dictionary definitions.
Our aim was to examine whether the term “trend” was used inappropriately to describe statistically nonsignificant results, in articles published in high-impact pediatric journals.
Materials and methods
The selection of the journals was made selecting representative journals of the first and the second quartile of impact (Q1 and Q2 in the Scimago Journal & Country Rank database [8]). The two previously published articles examining the inappropriate use of “trends” in anesthesia [3], and oncology journals [1] have used a similar methodology. After the journal selection, all articles published during 2020–2021 and 2010–2011 were examined for inappropriate use of “trend.” The rationale of choosing those specific periods was that 10 years apart would be a sufficient period to examine if the establishment of the best statistical practice has been applied. We examined for inappropriate use of “trend” throughout the entire paper, including those articles that used the term “trend” to describe any statistically nonsignificant results.
We examined articles published in the journals The Lancet Child & Adolescent Health, The Journal of Pediatrics, Early Human Development, Frontiers in Pediatrics, and BMC Pediatrics to identify manuscripts where a “trend” was used to describe a statistically nonsignificant result, from January 2020 to December 2021, and for The Journal of Pediatrics, Early Human Development, and BMC Pediatrics from January 2010 to December 2011. We compared the results between the two periods, between journals belonging to the first (The Lancet Child & Adolescent Health, The Journal of Pediatrics, Frontiers in Pediatrics) and the second quartile of impact (Early Human Development, BMC Pediatrics) and between journals publishing under the subscription model (The Lancet Child & Adolescent Health, The Journal of Pediatrics, Early Human Development) and solely under the open access policy (Frontiers in Pediatrics, BMC Pediatrics).
Statistical analysis
The proportion of the research articles with inappropriate use of “trend” across journals was compared by estimating percentages and 95% CIs. All tests were two-sided, and a p value less than 0.05 was considered statistically significant (alpha 0.05). The data were analyzed using SPSS Statistics (IBM SPSS Statistics for Windows, Version 24.0. Armonk, NY, USA). This project was exempt from approval by the Institutional Review Board because it was not human participants’ research.
Results
We reviewed in total 5393 articles published during 2020–2021 and 2412 articles published during 2010–2011. A “trend toward significance” was used to describe a statistically nonsignificant result at least once in 146 articles (2.7%) during 2020–2021 and in 97 articles (4.0%) during 2010–2011. Although there were no significant differences in the proportion of the articles where a “trend toward significance” was used inappropriately across the different journals, the overall proportion of the articles that used “trend” inappropriately declined significantly between 2010–2011 and 2020–2021 (p = 0.002, RR 0.66 95% CI 0.51–0.86) (Table 1).
There was no significant difference in the proportion of articles with inappropriate use of “trend” across journals belonging to the first compared to the second quartile of impact during 2020–2021 (2.4% vs 4.0%, p = 0.083, RR 0.58 95% CI 0.40–1.08, respectively) or during 2010–2011 (4.0% vs 4.0%, p = 0.938, RR 1.01 95% CI 0.67–1.53, respectively). Finally, we recorded no significant difference across journals publishing under the subscription model compared to journals publishing under the open access policy during 2020–2021 (2.7% vs 2.7%, p = 0.954, RR 1.01 95% CI 0.72–1.40, respectively) or during 2010–2011 (3.9% vs 4.8%, p = 0.537, RR 0.81 95% CI 0.53–1.55, respectively).
Discussion
In the current study, we found that “trend” statements were used to describe statistically nonsignificant results across pediatric literature. In most of the cases, the term “trend” was used inappropriately to describe a p value that was close but not smaller than the level of 0.05 that typically defines statistical significance.
Applying the term “trend” to report an almost significant result is a common pitfall in biomedical literature. Previously, researchers have reported inappropriate use of “trend” in 3.4% of articles across anesthesia journals [3], and 8.7% of articles in major oncology journals [1]. Interestingly, the inappropriate use of “trend” was consistent across journals of different impacts, across different periods, and, as shown in our study, across journals of different publication policies. Of note, our findings suggested that the overall proportion of the inappropriate use of “trend” across pediatric journals declined significantly between 2010 and 2011 to 2020 and 2021, which might indicate that, during the last decade, statistical interpretation has become more robust. Several journals currently include articles and tutorials helping authors to submit manuscripts with more precise statistics, while we believe that many young researchers are better qualified in medical statistics through several courses and seminars [9]. Besides, based on our findings, the inappropriate use of “trend” across pediatric journals was rarer compared to other fields of medicine, especially in oncology, whereas there was similar to the field of anesthesia.
According to the fundamentals of statistics, a p value is either significant or non-significant [10]. In statistics, a p value, or probability value, describes how likely the data would have occurred by random chance, i.e., the p value represents the evidence against a null hypothesis. The outcome of an inferential test is either rejection of the null hypothesis or failure to reject the null hypothesis [11]. There is no other option, and therefore a statement of “trend of significance” would be misleading.
In many cases, hypothesized results that are close but not lower than the statistical significance threshold are emphasized as “almost” significant. The use of “trend” to describe statistically nonsignificant results commonly arises from the authors’ intention to report their findings suggesting that the nonsignificant differences they observed might have been significant if the sample size increased. However, as suggested by Wood et al., a p value is by no means assured to become smaller even with the addition of quite a substantial proportion of extra data, a finding that could not support any claim of a “trend” towards statistical significance [4]. Although it has been demonstrated that underpowered studies could miss true findings, on the other hand, a failure to demonstrate statistical significance is not always due to sample size [4]. Moreover, statistical significance should be interpreted separately from clinical significance. Even statistically significant results might be meaningless in clinical practice, especially if the analysis does not include estimates and confidence intervals for effect sizes [10]. Besides, to address such issues, the American Statistical Association has released recommendations suggesting that scientific conclusions should not be based solely on a p value threshold and that a p value does not measure the importance of a result and does not provide a good measure of evidence regarding a hypothesis [11].
Although we limited our research to five pediatric journals, we included journals of a high impact, published in different regions, with a broad readership. We, therefore, have no reason to believe that our findings do not reflect a general issue in the pediatric literature.
In summary, we found that “trend” statements were sporadically used to describe statistically nonsignificant results across pediatric literature. However, the inappropriate use of “trend” to describe almost significant differences could be misleading as it could be falsely interpreted as suggesting a real trend. Therefore, “trend” should not be used to describe non-significant statistical differences, unless a specific statistical test for trend has been performed, or in relation to appropriate scientific definitions.
Availability of data and material
Not available.
Code availability
Not applicable.
References
Nead KT, Wehner MR, Mitra N (2018) The use of “trend” statements to describe statistically nonsignificant results in the oncology literature. JAMA Oncol 4:1778–1779
Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, Hing C, Kwok CS, Pang C, Harvey I (2010) Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess 14:iii, ix-xi, 1–193
Gibbs NM, Gibbs SV (2015) Misuse of ‘trend’ to describe ‘significant’ differences in anaesthesia research. Br J Anaesth 115:337–339
Wood J, Freemantle N, King M, Nazareth I (2014) Trap of trends to statistical significance: likelihood of near significant P value becoming more significant with extra data. BMJ 348:g2215
Cochran WG (1954) Some methods for strengthening the common χ 2 tests. Biometrics 10
Armitage P (1955) Tests for linear trends in proportions and frequencies. Biometrics 11
Thomas E (2005) An introduction to medical statistics for health care professionals: basic statistical tests. Musculoskeletal Care 3:201–212
SCImago, (n.d.). SJR — SCImago Journal & Country Rank [Portal]. https://www.scimagojr.com/journalrank.php?category=2735. Accessed 01 Mar 2022
Arnold LD, Braganza M, Salih R, Colditz GA (2013) Statistical trends in the Journal of the American Medical Association and implications for training across the continuum of medical education. PLoS ONE 8:e77301
Morgan CJ (2018) Balancing statistical significance and clinical relevance. J Nucl Cardiol 25:707–708
Wasserstein RL, Lazar NA (2016) The ASA statement on p-values: context, process, and purpose. Am Stat 70:129–133
Funding
Open access funding provided by HEAL-Link Greece.
Author information
Authors and Affiliations
Contributions
D.R. and V.G. conceptualized and developed the study design. Data collection was performed by M.B., F.B., N.D., C.K., and S.G. D.R. performed the statistical analysis, and wrote the initial draft. M.B., F.B., N.D., C.K., S.G, and V.G reviewed and revised the manuscript. All authors approved the final manuscript and agreed to be accountable for all aspects of the work.
Corresponding author
Ethics declarations
Ethics approval
Not applicable.
Consent to participate
Not applicable.
Consent for publication
Not applicable.
Conflict of interest
The authors declare no competing interests.
Additional information
Communicated by Peter de Winter
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Rallis, D., Baltogianni, M., Balomenou, F. et al. The trends for the “trend toward significance” in the pediatric literature. Eur J Pediatr 182, 937–940 (2023). https://doi.org/10.1007/s00431-022-04746-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00431-022-04746-8