Introduction

The inclusion of sex and gender in health research demonstrates the complex dynamics of how these categories are social constructs that influence determinants of health. The inclusion of these categories also reveals the interrelationship between biological and social processes that simultaneously produce health inequities as well as socioeconomic and ethnic-racial vulnerabilities in the lives of girls/women and boys/men (Sen et al. 2007; CIHR 2012).

The scientific evidence points to sex and gender differences producing inequalities when it comes to, for instance, adverse reactions to new drugs; gene expression; the relationship between working conditions and health; the prevalence, onset, and severity of diseases; mental health and subjective experiences of conditions such as depression and chronic pain; sexual behavior and access to health services; sexually transmitted diseases; the complications of diabetes; cardiovascular diseases; COVID-19; responses to health interventions; and the use of healthcare systems and services (Shannon et al. 2019; Theobald et al. 2017; Pelletier et al. 2014; Spence and Pilote 2015; Peckman et al. 2020; Pal and Hurria 2010).

The integration of sex and gender in health research becomes essential to supporting an ethical and more responsible science to address significant and innovative knowledge gaps, offer higher-quality and more relevant evidence, and provide a more potent analysis for policy planning, strategizing, and formulation to achieve more equitable and just health outcomes for diverse populations (Shannon et al. 2019; Williams et al. 2021; Mason 2020; Gogovor et al. 2020; Doull et al. 2014).

Evaluation studies of the inclusion of sex and gender in different health issues and types of studies have been progressively developed over the years (Peckman et al. 2020; Johnson et al. 2009; Day et al. 2016; Geller et al. 2011; Heidari et al. 2012; Springer et al. 2012; Doyal 2001). However, studies that analyze the quality of publications that include these categories are more recent (Rasky et al. 2017; Day et al. 2019; Palmer-Ross et al. 2021; Jahn et al. 2017).

In recent decades, research institutions and public and private funding agencies for global research, editorial policies of journals, and associations of scientific editors have been including sex and gender as important categories to outline the differences between sex and gender in manuscripts of health research (Johnson et al. 2009; LERU 2015; Heidari et al. 2016). To assist in the analysis of the quality of sex and gender integration in health research, methodologies, guidelines, and metrics are designed in order to improve the quality of science in providing answers to health systems.

These methodologies promote the use of categories and analytical approaches for sex and gender by researchers in the preparation of research proposals for the inclusion of men, women, and intersex people, and by journal reviewers, funding agencies, and researchers/evaluators, regardless of the purpose of the study, in order to guarantee that information about sex and gender will be disclosed (Jahn et al. 2017; Heidari et al. 2016; Day et al. 2017). However, gender imbalances still persist in the health research system, including strategic considerations in the definition of priorities and funding policies, and the formulation of research questions, design of methodologies, interpretation and analysis of data in different types of studies, and the implications of the results for public policies (Sen et al. 2007; Day et al. 2016; Gahagan et al. 2015; Johnson et al. 2014; Ovseiko et al. 2016)

The Department of Science and Technology of the Ministry of Health (DECIT/MS) in Brazil funds priority-driven research in order to respond to the needs of health and the healthcare system. Therefore, examining whether these studies include the categories of sex and gender and have published different results for women, men, or other gender identities becomes fundamental for monitoring research policy, especially when there is no specific strategy to guide the inclusion of sex and gender in research and the quality of this integration. Brazilian evidence includes studies on the influence of sex and gender on the health of men and women, on the implementation of health policies, and on the mapping of topics that include a gender perspective in scientific output (Bautista and Barquín 2018; Farah et al. 2018; Andrade et al. 2019; Villela et al. 2020). However, no studies were found for the evaluation of the quality of the integration of sex and gender in health research.

This paper seeks to analyze the quality of the inclusion of sex and gender in scientific articles produced by health research, funded by the Brazilian Department of Science and Technology of the Ministry of Health (DECIT/MS) between 2004 and 2016. The study also examines the use of sex and gender as categories in articles categorized by type of research.

Methods

Context of the study

The Unified Health System, Brazil’s publicly funded healthcare system, recognizes the differences between men and women to achieve its egalitarian goals, in order to reduce social and gender inequalities and improve responses according to health needs. As part of this policy, DECIT/MS, the main government agency responsible for funding research and innovation in the country, defines priorities, implements strategies to promote research, and invests important resources through public calls for studies to produce high-quality knowledge and to promote evidence-based policies aimed at improving the performance of the healthcare system and reducing social inequities in the country.

Study design

This is a desk review analysis of the quality of the inclusion of sex and gender in scientific articles produced by health research funded by DECIT/MS, between 2004 and 2016.

In this study, sex is defined as the set of biological attributes in humans and animals, mainly associated with physical and physiological characteristics, including chromosomes, gene expression, and hormone levels and function, as well as reproductive and sexual anatomy. Sex is generally categorized as female, male, or intersex, but considers a range of binarity in the biological attributes that make up sex and how these attributes are expressed (Sen et al. 2007; CIHR 2012).

Gender, on the other hand, is defined as the socially constructed roles, behaviors, expressions, and identities of girls, women, boys, men, and non-binary people. This category influences how people perceive themselves and each other, how they act and interact, and the distribution of power and resources in society. Gender is generally conceptualized as a binary system (girl/woman and boy/male), but it presents considerable diversity in how individuals and groups understand, experience, and express it (Sen et al. 2007; CIHR 2012; Pelletier et al. 2014).

Data sampling and collection

The identification of articles was carried out by mapping studies that included sex and gender as categories and that were funded by DECIT/MS, during the period between 2004 and 2016. Data were extracted from the “Health Research” public repository (http://pesquisasaude.saude.gov.br/), which contains the titles and summaries of the projects funded by DECIT/MS, the name of the research coordinator, the title of the call for research support, the year of funding, and the Brazilian state and region in which the coordinator's institution is located, among other information. The search for studies occurred on August 21, 2019, based on the keywords sex(es), gender(s), gay, transvestite, man/men, woman/women, masculinity(ies), femininity(ies), transsexual, intersex, intersexual, intergender, transgender(s). This repository does not store complete research information or final conclusions.

The collected data generated 3077 titles and abstracts of studies, names of the main research coordinator, and other data mentioned above, which were recorded and organized in a Microsoft Excel spreadsheet. From this total number, 1585 were excluded due to duplication. After reading and analyzing the titles and abstracts of the studies, 1029 other studies were excluded. The exclusion criteria were the use of the term “gender” to refer to the classification of living beings in studies on diseases transmitted by animals or in plants, tissues, cells, or derivatives. In the end, 463 titles and abstracts of research in humans were considered eligible, in which the term “sex” was used to indicate differences in relation to the problem studied or the description of participants (men, women, intersex) in the sample, and “gender” as a category analysis.

Then, the mapping of articles produced by the 463 eligible surveys was carried out by searching the curricula of the coordinators stored on the Lattes Platform (http://lattes.cnpq.br) for the period between April and June 2020. The criterion for attributing each article as a product of the study was the reference to the title of the original research and/or the source of funding described with the name of the funding notice (data obtained in the initial search). No articles were identified or attributed in 319 studies. A total of 144 studies were considered, with the total production of 350 articles on various topics (Fig. 1).

Fig. 1
figure 1

Flowchart of research search results (title and abstracts) and inclusion of full articles

Full size image

Projects were classified by type of study to examine the trend in use and the quality of sex and gender integration by study type. This classification was performed after reading the full text of at least one article produced by each study, according to the definitions of the Canadian Academy of Health Sciences (CAHS): (i) biomedical research (BR), which investigates mechanisms of health and disease and produces knowledge on the development of diagnostic methods, treatments, and methods for preventing injury and disease; (ii) clinical research (CR), which involves human patients with the aim of improving the diagnosis and treatment of diseases or conditions; (iii) health services research (HSR), which evaluates the health system or services in relation to the organization, financing, access, and costs of healthcare; and (iv) population and public health research (PPHR), which investigates the health determinants of a population (CIHR 2022).

Data analysis

The analysis of the use of the categories sex, gender, or both (S/G) was carried out via the peer review system after reading the titles and abstracts of the 350 articles identified, in July 2020. Differences were resolved by consensus.

The evaluation of the quality of the integration between sex and gender in the articles was carried out using the methodology developed by Day et al. (2017), Essential Metrics for Assessing Sex and Gender Integration in Health Research Proposals Involving Human Participants, which provides a practical and comprehensive strategy to analyze the quality of each step of the research process and instructs researchers on the use of these differences (Day et al. 2017).

The analysis was organized according to the three sections of the adopted methodology: (1) literature review and research objectives, (2) research design, methods, and analysis plan (research population, participant recruitment and retention, data collection tools, data analysis plan), and (3) knowledge translation plan. Each section provides questions that guide the analysis, citing examples, and using the rating scale (excellent, good, fair, poor, not applicable) for each criterion with the respective definitions for evaluating the quality of the integration. The analysis considered elements of sex and gender issues, for example, whether the literature includes or indicates evidence of the importance of sex/gender inclusion in health research; whether it indicates that the participants are men, women, both, or another sex or gender identity; whether the data collection tools capture enough information for the variables of the study to be analyzed and described based on sex and gender; and whether, in a study with a single gender or sex, differences within this population are investigated to present relevant findings based on their sex/gender (Ovseiko et al. 2016).

The analysis of the quality of the integration was carried out from the data collected in a form containing the following variables: (a) identification of the research (title); (b) identification of the research coordinator (name, institution); (c) identification of the articles produced (title, year of publication); (d) metrics of the three sections of the methodology. The review was conducted by peers, and differences were resolved by consensus. Data were documented and coded in the Epi Info 7.0 program and in a Microsoft Excel 2010 version software spreadsheet. This study was approved by the Research Ethics Committee of the University of Brasília, Opinion No. 5.081.867–CAAE: 45274021.1.0000.8093.

Results

The results of this research are organized as follows: (1) the use of sex and gender categories in scientific articles by type of research; (2) the quality of the incorporation of sex and gender in articles produced by health research funded by DECIT/MS.

  1. 1.

    The use of sex and gender categories in scientific articles produced by health research funded by DECIT/MS

In the set of 350 articles attributed to research funded by DECIT/MS, the majority were found to use the categories of either sex or gender or both (181; 51.7%). This finding demonstrates that not all articles produced and published by the studies used these categories (Table 1). Among the articles that used them, a significant proportion was produced by research in population and public health (137; 56.1%) and clinical research (28; 52.8%).

Table 1 Use of sex, gender, or both categories among the total studies and the total research articles
Full size table

Table 2 shows that in the total number of articles (181) that used sex, gender, or both categories, the sex category was the most frequently used (45.9%), followed by both sex and gender (33.1%). When analyzing by type of research, the use of the sex category stands out among the articles published in the CR category (60.7%), a higher proportion than among the articles in the HSR (42.9%) and PPHR (42.3%) categories. However, the proportion of gender use was higher among the articles produced in the HSR category (28.6%), and the use of both sex/gender was highest among the articles produced in the PPHR category (38.0%).

  1. 2.

    The quality of the incorporation of sex and gender in articles produced by health research funded by DECIT/MS

Table 2 Use of sex, gender, or both categories in articles published by type of research
Full size table

The analysis of the quality of the incorporation of sex and gender in the 181 articles (Table 3) applying the metrics with an evaluation scale of the three sections of Day's methodology shows that, in section 1, Literature review and research objectives, the mentioned considerations of sex and gender were classified as poor in a significant proportion of articles (43.1%). In section 2, Research project, methods, and analysis plan, in the description of the criterion population, the evaluation was fair in 33.7% of the articles; in the criterion participant recruitment and retention, it was significantly evaluated as poor (83.4%), due to the fact that women, men, or gender identities were not specified in the recruitment strategies of the articles; in the criterion data collection tools, it was mostly classified as good (37.6%) and excellent (30.4%), because in the description, the instruments sought to identify men and women; and in the criterion analysis plan, most were evaluated positively, between excellent (41.4%) and good (35.4%). In section 3, Knowledge translation plan, the articles were disproportionately evaluated as excellent (38.7%) and good (37%).

Table 3 Classification of the quality of the incorporation of sex and gender in the articles according to the type of research and methodology Essential Metrics for Assessing Sex and Gender Integration in Health Research Proposals Involving Human Participants
Full size table

When analyzing the quality of the inclusion of sex and gender in the articles by types of research, it is observed that in section 1 (the literature review and objectives), the articles were classified as poor, in similar proportions as in the different types of research. In other words, differences in sex were not sufficiently explained in the introduction of the article, as they did not demonstrate what is already known and the impacts that the inclusion of these categories would bring. It is worth noting that the fact of having only two biomedical research (BR) articles makes it difficult to compare the analysis criteria.

In section 2, in the evaluation scale for the “research population” item, the PPHR and CR study types stand out when adding the set of excellent and good classifications, obtaining proportions of 50.4% and 35.8%, respectively. This means that gender as well as gender inclusion and exclusion criteria were justified, and the sample size was sufficient to identify relevant findings based on sex and gender.

The participant recruitment and retention item was the worst evaluated criterion in all research types. Data collection tools are instruments that capture relevant information for the analysis of the influence of sex and gender during the participation of individuals in the research. They include questionnaires, guides, and interviews. As a result, most articles had a good evaluation in the use of the tools, managing to approach and capture data on sex (for example, man, woman) and/or gender identity, achieving satisfactory results.

Regarding the data analysis plan item, significant results of the evaluation can be seen when considering the percentages of the two criteria, excellent and good, in the articles of the PPHR (84%), HSR (71.4%), and CR (46.4 %) categories. This shows that the articles described and deepened the analysis of data in the results section, reporting the differences between the individuals and/or population studied from a gender and sex perspective. Compared to the other sections evaluated, it can be seen that the studies are more immersed in detailing the influence and reflections of these categories with respect to the results of the data analysis, presenting more sensitivity in the analysis of sex and gender in this section.

Section 3, Knowledge translation plan, analyzes the consideration of sex and gender in strategies for disseminating research results to adapt interventions to different population groups, systems, health services, and public policies. Important proportions are observed when considering the two criteria in the analysis, excellent and good, among the articles in the PPHR (81.8%), HSR (64.3%), and CR (53.6%) categories.

Discussion

DECIT/MS has invested significant financial resources since 2004, guided by a National Agenda of Priorities in Health Research that recognizes the need to direct the interest of researchers in the study of social determinants and discrimination in health in different population groups. Our study reveals the use of sex, gender, or both categories in research designs and results published in articles, with an emphasis on population and public health research and clinical research. Usage trends vary across disciplines. The differences by sex are more often reported in articles resulting from clinical research, sex/gender in articles mainly from population and public health research, and gender among articles from research on health services. This trend can be seen in other studies that analyze the efforts of institutions and funding agencies in different countries to encourage researchers to incorporate these categories (Geller et al. 2011; Heidari et al. 2012; Doyal 2001; Johnson et al. 2014).

The findings on the quality of sex and gender integration reveal, in general, low levels of excellent and good evaluations in the analyzed sections of the articles examined and inconsistency in the different sections of the articles. In the literature review and research objectives (section 1), the inclusion and exclusion criteria of the sample, the data collection tools and the data analysis plan (section 2), and the knowledge translation plan (section 3), articles that considered the categories of sex (e.g., male and female) and/or gender identity that were assessed as good ranged from 14.4% to 37% and those assessed as excellent ranged from 18.8% to 38.7%. This is observed in the various disciplines of knowledge, including observational studies and randomized controlled trials (Day et al. 2019; Palmer-Ross et al. 2021; Jahn et al. 2017) in biomedical research (Rasky et al. 2017), and in the production of evidence on diseases and health problems. In the case of the latter, for example, despite the growing evidence demonstrating relevant differences in health outcomes for women and men resulting from COVID-19, studies still do not examine these differences from the planning stages of clinical research, nor do they describe the results appropriately (Palmer-Ross et al. 2021).

The integration of these categories can ensure that the results of the articles are equally reliable for both sexes and modify the relationship between discovery and intervention. Despite this, the studies reveal a lack of in-depth analysis of the data in order to understand the impact of sex and gender on the objects of study.

The most deficient criterion was the recruitment and retention of participants, even without any difference when considering the articles in all types of research, in which the researchers did not describe the “n” of men, women, or other gender identities sample. This finding demonstrates that the studies did not consider the relevance and diversity of the sex and gender that make up the group. When defining the sample, consequently, strategies for participant recruitment and retention were not demonstrated in this section. In most of the articles analyzed, no explanation for this exclusion was presented, which makes it difficult to adequately assess the sample’s diversity and determinants of health. In this sense, the assessment by which some populations are inappropriately excluded without justification for the choices is compromised. Differences exist and permeate from symptoms to clinical manifestations of diseases, reliability of tests, and assertive responses.

Our study also shows the absence or low quality of sex and gender considerations in the literature review and research objectives section (section 1, which is the lowest-performing section), as these tend to obscure subgroup differences between men, women, and other gender identities. Incorporating the diversity of subjects with their biological, sociocultural, and behavioral characteristics and singularities can improve intersectional analyses by disaggregating factors and putting knowledge into practice with results and solutions that are more inclusive of the population (Heidari et al. 2012; Springer et al. 2012). It is likely that authors do not include these categories in the introductory section because they are increasingly concise, without providing much theoretical depth on the research questions, and this directly influences how the research will be designed. There is a systematic lack of literature reviews on studies that address sex and gender differences, and this historical neglect results in many studies not asking research questions based on these categories (Rasky et al. 2017). This, consequently, leads to the publication of articles that are neither well-planned nor inclusive of analysis of sex and gender, leading to literature with several questions and possible unanswered evidence, lacking reflection about the mechanisms behind these differences or about the knowledge gaps (Garcia et al. 2016; McGregor et al. 2016; NIH 2022).

One finding that is worth discussing is the fact that a little less than half of the articles produced did not use these categories for analysis and for the presentation of results, despite being the sections with better classifications. Resistance to and difficulties in incorporating these categories may help to explain this finding (Day et al. 2016; Johnson et al. 2014; Peters et al. 2021). Strategies such as an explicit indication of sex and gender as categories in the priorities and in the funding policy can positively influence their incorporation into the research design and the dissemination of results, even if advances happen progressively over time and in a differentiated way to adhere to the incorporation of sex and gender across disciplines and types of studies (LERU 2015; Johnson et al. 2014; Peters et al. 2021; Haverfield and Tannenbaum 2021).

Taken together, the findings of the study demonstrate the absence or low quality of the integration of sex and gender categories in health research. Improving the quality of reporting on sex/gender results in articles not only allows for the improvement of evidence to adapt to health demands and needs, but also helps to identify more precise and relevant interventions that apply in the real world, with the aim of enhancing the quality and effectiveness of health services and care more broadly for the benefit of everyone (CIHR 2012; Shannon et al. 2019; Ovseiko et al. 2016).

The research in this paper has several strengths. First, the use of available metrics and criteria to assess the quality of sex and gender integration in the different areas of health research with humans ensured rigorousness. The questions that guide the assessment and the grading scales are important tools to identify the challenges in the gaps for the elaboration, execution, and analysis of the research results. Second, it makes it possible to build training and capacity-building strategies for researchers on the relevance of incorporating sex and gender in different types of research. Third, the choice to analyze articles produced by research funded by the Brazilian Ministry of Health makes it possible to provide recommendations on two issues: (1) on the relevance of explaining sex and gender in topics that are prioritized in the calls for research support; and (2) on the commitment to support capacity-building and training for researchers.

There are also some limitations of this study. First, the search for articles was carried out only in the résumés of the research coordinators on the Lattes Platform. Despite this platform being widely used academically and broadly accessible in the search for information, the information described is not always complete and it does not always mention the institution that financed the research project. Second, some articles may have been published after the search and data collection for this study were completed, and were, therefore, not included in the analysis. The next steps to improve this research approach would be to analyze other variables such as the gender of the coordinator of each project and to identify proposed recommendations. And third, the low number of biomedical research articles published in the sample makes it difficult to weigh the interest of researchers in the use of these categories. However, the literature reveals the slow incorporation of sex, gender, and both categories, even in contexts with strong incentives for the incorporation of this approach, in biomedical research (Garcia et al. 2016; White et al. 2021).

Implications for science and technology policies

The challenges of including the categories of sex and gender throughout the research process and translating the results remain. This is especially true for biomedical and clinical research. Funding mechanisms need strengthening to better provide methodological advice, targeting, and evidence-based recommendations and to take into account the integration of sex and gender at all stages of the research process. This would allow researchers to incorporate sex and gender-based analyses in ways that demonstrate their impacts on public policies, services, health systems, and the real health needs of the population.

The use of evaluation tools, such as the essential metrics used in this research, would be of great value to help better direct project coordinators in order to qualify the incorporation of these categories with more enriching analyses and results (Smits and Champagne 2020). However, using metrics alone is not enough. It is recommended that strategies be implemented by DECIT/MS, editors, and reviewers in the country for promoting awareness and training in order to ensure a commitment to a more evident science in its results through mandatory requirements, training of evaluators, and outlining public notices and calls for careful evaluation regarding the appropriate integration of sex and gender into different types of health research and monitoring and evaluating study results and recommendations.

Several experiences demonstrate the efforts of institutions, funding agencies, and journals to progressively refine research into more systematic, reproducible, and applicable scientific results, strengthening themselves as agents of change (Rasky et al. 2017; Day et al. 2019; Palmer-Ross et al. 2021). For example, the Canadian Institute of Gender and Health requires all research departments to analyze how public policies and government programs relate to and affect the population and subpopulations of different genders (Rasky et al. 2017) and the National Institutes of Health in the United States has defined a strategic plan for the period 2019–2023 for women's health biomedical research to integrate and improve their outcomes using a gender and sex lens (Palmer-Ross et al. 2021).

Conclusion

Our results confirm findings from international experiences that point out differences and difficulties in presenting results disaggregated by sex and gender analysis in scientific articles produced by health research funded by DECIT/MS. It is up to funding agencies and public institutions to recognize that taking into account the categories of sex and gender in health research is essential as a starting point to enhance the methodological path of the study, to fill knowledge gaps, and to disseminate results that achieve gender equality in science (Heidari et al. 2016). The use of metrics to assess the integration of sex and gender promotes high-quality health research that addresses these categories from the beginning of the study process. This results in an in-depth understanding and reflection of the health situation for decision-making in healthcare systems and services as well as in evidence-based public policy.