Background

Sickle cell disease (SCD) is a genetic disorder that causes a vaso-occlusive phenomenon and hemolysis along with a myriad of other major complications that could be life-threatening. It is one of the commonest inherited diseases, worldwide, and is inherited as an autosomal recessive disease due to the substitution of valine for glutamic acid at the sixth amino acid of the beta-globin chain [1]. This substitution leads to the production of a hemoglobin variant that is poorly soluble when deoxygenated. The clinical features, which include a vaso-occlusive crisis, are the result of the polymerization of the deoxygenated hemoglobin S. SCD is associated with significant maternal morbidity and mortality in pregnant women. The recognized complications include maternal mortality, preeclampsia, eclampsia, venous thromboembolism, cesarean delivery, intrauterine fetal death, and fetal growth restriction [2].

The prevalence of SCD varies among countries. For example, data from the United States showed that the overall prevalence is roughly about 4.83 per 10,000 deliveries; [3] 28.5% of women with SCD develop a crisis at the time of delivery. The maternal mortality rate was reported to be 1.6 per 1000 deliveries in women with SCD compared to 0.1 per 1000 deliveries in those without SCD [3].

Information about the prevalence of SCD in Saudi Arabia is limited and different among the various provinces, with the highest prevalence reported in the Eastern province followed by the Southwest province [4].

SCD in pregnancy tends to cause higher episodes of painful crises and a higher frequency of blood transfusion [2]. Although the complications of SCD are more commonly associated with the HbSS genotype, patients with other types of SCD such as sickle-beta thalassemia (HbSC genotype) and Hemoglobin SC disease (HbSC genotype) should receive the same level of care as those with HbSS. The development of a multidisciplinary care approach and comprehensive sickle cell centers seem to be associated with a decrease in the incidence of perinatal complications [5, 6].

Clinical Practice Guidelines (CPGs) were defined by the Health and Medicine Division of the American National Academies (formerly, the Institute of Medicine [IOM]), as “statements that include recommendations intended to optimize patient care and are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options” [7]. To date, there are no national CPGs for the management of SCD in pregnant women in Saudi Arabia.

The second edition of the Appraisal of Guidelines for Research and Evaluation Instrument (AGREE II) is the gold standard for the quality assessment or critical appraisal of CPGs. It was first published in its original form in 2003 and lastly updated in 2017 by the AGREE enterprise. AGREE II is a validated quantitative tool that has been cited in more than 1013 articles and endorsed by several healthcare organizations [8, 9]. AGREE II identifies constituents that must be addressed by CPGs to improve their quality, and henceforth, ensure their expected trustworthiness, and positive impact on healthcare outcomes [9].

The objective of this study was to conduct a systematic review and critically appraise the quality of recently published CPGs for SCD in pregnancy using the AGREE II instrument as a part of the CPG adaptation program [10].

Methods

The protocol for this study is published in the International Prospective Register of Systematic Reviews (PROSPERO; https://www.crd.york.ac.uk/PROSPERO/display_record.php?RecordID=145443) [11].

Inclusion and exclusion criteria

Three reviewers independently reviewed the retrieved CPGs, based on the following inclusion criteria: evidence-based with clear and detailed documentation of the development methods; available in the English language; obtained from original sources (de novo developed); had a national or international scope; published, amended or updated between January 1, 2014, and December 31, 2018 (the search was repeated before the submission of the final manuscript to identify new relevant CPGs; those published over the last five years only (2014–2018) were included in this study (because the period between two updates was reported to range from two to five years, in several CPG handbooks). CPGs published by an organization or having group authorship in a CPG database or peer-reviewed journal were also included in this study. Only the most current version of each Source CPG was included [11, 12].

The exclusion criteria included CPGs that were published earlier than 2014, written in a non-English language, adapted from other Source CPG (s), presented as consensus or expert-based statements, and written by a single author [11].

Search strategy and selection of SCD in pregnancy CPGs

Literature searches of bibliographic databases (Medline/PubMed and Google Scholar), EBSCO DynaMed Plus (USA) and relevant CPG databases, such as the ECRI Institute Guidelines Trust, National Institute of Health and Care Excellence (NICE; UK), Guidelines International Network (G-I-N) International guideline library, Scottish Intercollegiate Guidelines Network (SIGN; UK), and the Australian National Health and Medical Research Council (NHMRC; Australia) were performed. Moreover, we searched databases of national and international societies specializing in fields related to the topic of SCD in pregnancy, such as the American College of Obstetricians and Gynecologists (ACOG), Royal College of Obstetricians and Gynaecologists (RCOG), Royal Australian and New Zealand College of Obstetricians and Gynaecologists (RANZCOG), Society for Maternal Fetal Medicine (SMFM), Saudi Society for Obstetrics and Gynecology (SSOG), and Arab Association of Obstetrics and Gynaecology Societies (FAGOS). The keywords used were “sickle cell disease” AND “pregnancy” OR “pregnant women” AND “guideline,” “practice guideline,” “clinical practice guideline,” “practice parameter,” “guidance,” OR “recommendations” [11].

The PubMed electronic search strategy included the following: “anemia, sickle cell“[MeSH Terms] OR (“anemia“[Title] AND “sickle“[Title] AND “cell“[Title]) OR “sickle cell anemia“[Title] OR (“sickle“[Title] AND “cell“[Title] AND “disease“[Title]) OR “sickle cell disease“[Title]) AND “pregnan“[Title] AND (“pregnancy“[MeSH Terms] OR “pregnancy“[Title]) OR (“pregnant women“[MeSH Terms] OR (“pregnant“[Title] AND “women“[Title]) OR “pregnant women“[Title]) AND (“guideline“[Publication Type] OR “guidelines as topic“[MeSH Terms] OR “guidelines“[Title]) AND (Practice Guideline [ptyp] AND (“2014/01/01“[PDAT] : “2019/12/31“[PDAT]) AND “humans“[MeSH Terms]) AND (practiceguideline [Filter]). Furthermore, the PIPOH (Patient Population, Interventions, Professionals, Outcomes, and Healthcare Setting) model was used to support the CPG eligibility process [10]. Three reviewers (YA, MA, YS) independently screened titles and abstracts of the retrieved CPGs and articles that met the inclusion criteria. The screening was re-checked by three different reviewers (GE, AA, OK). Disagreements were resolved by focus group discussions after retrieval and review of the full-text articles or full CPG documents, including links to any accessible online supplementary documents or web resources. The search was repeated before the submission of the final manuscript to retrieve any new eligible CPG.

Assessment of CPGs using the AGREE II instrument

The AGREE II instrument (www.agreetrust.org) consists of 23 items organized in 6 domains: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence [13]. Each item is scored on a Likert scale (1–7). The AGREE II evaluation was guided by utilizing its online version, “My AGREE PLUS,” which supports the presence of a CPG “appraisal group” for each CPG that compiles and calculates the ratings of the items into domain ratings, and comments [13]. The four AGREE II appraisers in the study had the relevant clinical expertise in obstetrics and gynecology (YS, AA), internal medicine (GE, MA), and hematology (GE, MA); furthermore, an expert CPG methodologist (YA) was included. At the outset, the CPG methodologist conducted capacity building sessions for the reviewers via hands-on sessions on the concepts, evidence-based CPG standards, and use of the AGREE II instrument. Each reviewer scored his/her assigned CPGs, and the included CPGs was critically appraised by all the five reviewers. The CPG documents, including any updates, plus any relevant supplementary information or links to online webpages related to the CPG methods or CPG implementation tools were reviewed in full by all the appraisers. For each item, the AGREE appraisers were asked to record the justifications for their scores in the “Comments” section. Wide discrepancies between the assessors’ scores for items or questions (a difference of more than 3 between the scores) were resolved by asking those who had provided the outlying scores to re-assess the questions after discussions with the entire group. The standardized AGREE domain scores or ratings (%) were automatically calculated using My AGREE PLUS. A cut-off point of 70% was set for each AGREE standardized domain score or rating. After the appraisal, we focused on the scores of domains 3 and 5 to facilitate the filtration and final evaluation of the reporting quality of the included CPGs. Similar cut-off values have been reported in previous studies [14, 15]. In addition to the classification of the six AGREE II domains, the evidence-bases of the included CPGs and their references sections were screened for systematic reviews or meta-analyses, specifically for Cochrane reviews. We utilized the PRISMA statement flow diagram and checklist for reporting our review [16,17,18]. There was no patient nor public involvement in this study.

Inter-rater analysis

Inter-rater reliability assessment tests (IRR) were conducted to determine the agreement levels between the raters. Percent agreement IRR was used for every question (or item) in each standardized domain in the four CPGs included in this study to assess the level of agreement among the four raters and the percent agreement of the first overall assessment (OA1) of the AGREE II instrument. Furthermore, the Intra-class correlation coefficient (ICC) was used to measure the consistency in the ratings or capacities of the datasets that were gathered as clusters or arranged into clusters, including the second overall assessment (OA2 or “recommend this CPG for use”). ICC is one of the most prevalent IRR approaches used when the number of raters is more than two. A high ICC (or Kappa; K) value (near 1) indicated a high resemblance between standards from the same set. A low Kappa value (near zero) indicated that the standards from the same set are not alike. One-Way Random analysis of variance was used SPSS Statistics, version 21 was used to analyze the data in this study. The ICC was used due to the diversity in the numerical data obtained from the groups or clusters. It helped us in detecting the reproducibility of the results and in determining how closely the peers resembled each other with regard to certain traits or characteristics. The agreement between two ordinal scale classifications was evaluated, and weighted Kappa (Quadratic Weights) was used because the data was obtained from an ordered scale.

Linear weights were used because the difference between the first and second categories had the same importance as that between the second and third categories, and so on. The agreement was quantified using the K statistic, [19, 20] where K = 1 when there is perfect agreement between the classification systems, K = 0 when there is no agreement better than chance, and K is negative when the agreement is worse than chance. The strength of the agreement based on the K value was classified as follows: < 0.20 (poor), 0.21–0.30 (fair), 0.31–0.40 (moderate), 0.41–0.60 (good), 0.61–0.80 (very good), and 0.81–1.00 (excellent) [21].

Results

Identification of SCD in pregnancy CPGs

The results of the search in the PRISMA statement flow diagram are shown in Fig. 1 [17, 18]. The initial list of 33 records was reviewed and filtered by the assessors. Among them, 29 were excluded because they did not meet the inclusion criteria. The four recent SCDs in pregnancy CPGs that complied with our PIPOH and inclusion criteria were shown in Fig. 1. The CPGs were developed by ACOG in January 2007 (reaffirmed in 2018) [22], NICE in June 2012 (with a minor update in August 2016) [23], RCOG in August 2011 (updated in May 2018) [24], and the US Department of Health and Human Services, Public Health Service, National Institutes of Health, National Heart, Lung, and Blood Institute (NHLBI) in 2014 [25]

Fig. 1
figure 1

PRISMA flow diagram. Systematically searching and selecting the clinical practice guidelines for the management of pregnant women with sickle cell disease. From: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097. doi:https://doi.org/10.1371/journal.pmed1000097. For more information, visit www.prisma-statement.org

Full size image

.

Key characteristics of SCD in pregnancy CPGs

Table 1 highlights the characteristics of all the eligible CPGs; two CPGs were developed by US-based (n = 2, 50%), and two CPGs (n = 2, 50%) by UK-based organizations. The four CPGs were developed by two reference specialized professional organizations (ACOG, RCOG), and two national evidence-based healthcare improvement organizations (NICE, NHLBI). All the organizations are from high-income countries [22,23,24,25].

Table 1 Characteristics of the included CPGs
Full size table

Reporting the quality of SCD in pregnancy CPGs

The standardized domain ratings of AGREE II are summarized in Table 2, and the appraisers’ comments are presented in Table 3.

Table 2 AGREE II standardized domain scores for the included CPGs
Full size table
Table 3 Reviewers’ comments on the four CPGs organized according to the standardized domains in AGREE II 22–25*
Full size table

Domain 1: Scope and purpose

The AGREE II standardized score for domain 1 ranged from 76–93%. The scores of three CPGs were greater than 70% in domain 1 (NICE-2012, 93%; RCOG-2018, 89%; and NHLBI-2017, 88%).

Domain 2: Stakeholder involvement

The standardized scores for domain 2 ranged from 33–85%. The scores of two CPGs were greater than 70% (NICE-2016, 85% and RCOG-2018, 76%).

Domain 3: Rigor of development

The standardized scores for domain 3 ranged from 41–90%; the scores of three CPGs were greater than 70% (NICE-2016, 90%; RCOG-2018, 73%; and NHLBI-2017, 71%).

Domain 4: Clarity of presentation

The standardized scores for domain 4 ranged from 63% to 89%, and the cores of three CPGs were greater than 70% (NICE-2016, 89%; RCOG-2018, 83%; and NHLBI-2017, 83%).

Domain 5: Applicability

The standardized scores for domain 5 ranged from 24–90%. Only one CPG scored greater than 70% in this domain (NICE-2016 = 90%).

Domain 6: Editorial independence

The standardized scores for domain 6 ranged from 19–77%. The scores of two CPGs were greater than 70% (RCOG-2018, 76% and NICE-2016, 85%).

Overall assessment

The AGREE II standardized domain scores for the first overall assessment ranged from 46–83%. Three CPGs scored greater than 70% (NHLBI, NICE, RCOG), consistent with their high scores in the six domains. The calculated AGREE II domain scores are shown in Figs. 2 and 3. The radar maps illustrate the final scores, expressed as percentages, for every included CPG in each of the six domains in Fig. 2 and each of the 23 questions in Fig. 3. The higher standardized domain scores are mapped toward the periphery (closer to 100%) and lower domain scores are plotted toward the center. The graphs illustrate a visual display of the relative strengths or weaknesses of each CPG by domain, question, and OA1 when compared to the other plotted CPGs.

Fig. 2
figure 2

Using a Radar chart to map the AGREE II 23-questions, 6-domains, and the first overall assessment for eligible appraised clinical guidelines. Abbreviations: ACOG: American College of Obstetricians and Gynecologists, AGREE: Appraisal of Guidelines for Research and Evaluation, CPG: clinical practice guideline or guidance, D: AGREE II Domain, NICE: National Institute of Health and Care Excellence, NHLBI: National Institutes of Health, National Heart, Lung, and Blood Institute, Q: AGREE II Question (or Item), RCOG: Royal College of Obstetricians and Gynaecologists, and SCD: Sickle cell disease.

Full size image
Fig. 3
figure 3

Radar map of the AGREE II final standardized domain scores for eligible appraised clinical guidelines. Abbreviations: ACOG: American College of Obstetricians and Gynecologists, AGREE: Appraisal of Guidelines for Research and Evaluation, CPG: clinical practice guideline or guidance, NICE: National Institute of Health and Care Excellence, NHLBI: National Institutes of Health, National Heart, Lung, and Blood Institute, OA1: AGREE II overall assessment 1, RCOG: Royal College of Obstetricians and Gynaecologists, and SCD: Sickle cell disease.

Full size image

Recommending the CPGs for SCD in pregnancy for use in practice

The second (overall) assessment with regard to the recommendation for the use of the CPG in practice revealed a consensus between the reviewers. Two of the appraised CPGs were recommended for use without modification (RCOG and NHLBI), while the other two were recommended for use with modifications (NICE and ACOG).

The strengths and limitations of the included CPGs are summarized in Table 3, based on consensus and the comments of the CPG appraisers for each item in AGREE II. All four CPGs cited systematic reviews in their references list. The largest number of systematic reviews were cited in the RCOG CPG (n = 6), and five (83%) of them were Cochrane reviews. Overall, the different options of care in the management of SCD in pregnant women were similar in these CPGs (Table 4).

Table 4 Summary of key recommendations in the four CPGs from ACOG, NHLBI, NICE, and RCOG*
Full size table

Inter-rater analysis

The results of the IRR tests showed a high strength of agreement for every question in every domain in the four CPGs among the four raters. As well as the percent agreement of the first overall assessment (OA1) in Fig. 2. Most of the K values were between 0.50 and 1.00, denoting good to excellent agreement. Only two evaluations (Fig. 3) pertaining to question 6 in domain 2 (D2Q6) and question 8 in domain 3 (D3Q8) revealed poor strengths of agreement (K = 0.0) in the ACOG CPG. As seen in Table 5, out of the 24 questions in the ACOG CPG, one demonstrated excellent agreement (K = 1), 16 showed good agreement (K = 0.5), 5 presented with very good agreement (K = 0.6–0.8), and two with poor agreement (K = 0.00); the results of the overall assessment (1) demonstrated good agreement (K = 0.5). In the RCOG 2011 evaluation, none of the 24 questions demonstrated excellent agreement, 15 presented with good agreement (K = 0.5), and 9 with very good agreement (K = 0.6–0.8); the overall assessment (1) showed good agreement (K = 0.5). In the NICE 2012 evaluation, one out of 24 questions demonstrated excellent agreement (K = 1), 16 presented with good agreement (K = 0.5), and seven with very good agreement (K = 0.6–0.8); the overall assessment (1) showed good agreement (K = 0.5). In the NHLBI evaluation, none of the 24 questions demonstrated excellent agreement, 15 presented with good agreement (K = 0.5), and 9 with very good agreement (K = 0.6–0.8); the overall assessment (1) showed good agreement (K = 0.5; Table 5). The results of the K value among the raters for the four guidelines with regard to the second overall assessment (OA2) revealed the following: number of observed agreements, 6 (37.50% of the observations); number of agreements expected by chance, 4.0 (25.00% of the observations); K = 0.167; standard error of K = 0.138 (95% confidence interval, − 0.103–0.437); and weighted K = 0.077.

Table 5 Classification of the strength of agreement among the four raters for the four clinical practice guidelines
Full size table

Discussion

To the best of our knowledge, this is the first study to systematically evaluate the quality of recently published CPGs for SCD in pregnancy using the AGREE II instrument.

Four CPGs addressing the management of pregnant women with SCD were assessed using the AGREE II instrument. Several areas of improvement in the methodological rigor of the included CPGs were highlighted. One CPG (ACOG) had significant gaps in its rigor of development (domain 3), which is the largest and core domain, and three CPGs demonstrated areas for improvement in their applicability (Domain 5). The weights of these two domains have been emphasized in this study. The NICE CPG received the highest reviewer agreement ratings [23]. All four CPGs included in this review had commonalities and differences in their clinical recommendations (Table 4). The common factors included genetic screening (ACOG, RCOG), genetic diagnosis (ACOG, NHLBI, RCOG), counseling during pregnancy (all four CPGs), transfusion or prophylactic exchange transfusion (ACOG, NICE, RCOG), fetal surveillance (ACOG, NHLBI, RCOG), and contraception (NHLBI, RCOG).

One discrepancy was observed in the form of a lack of clearly articulated recommendations for the vaccination status before pregnancy in three CPGs (ACOG, NHLBI, NICE). Two CPGs (ACOG and RCOG) were more specific about pregnancy when compared with the two other CPGs that contained general recommendations for SCD and smaller sections that focused on pregnancy. Out of the two specific CPGs, RCOG consistently presented with higher scores in all the domains. This systematic and objective assessment of the available CPGs is beneficial in supporting the decision to adopt or adapt the CPGs in clinical practice. After reviewing the four CPGs and given the appropriate rigor, consistently high scores, and clinical relevance of RCOG, we decided to adopt all the recommendations of this CPG in our clinical practice.

The findings of the current study revealed that the CPG assessment was accurate. There was excellent/ very good inter-rater agreement between the four assessors who evaluated the four CPGs using AGREE II. The proposed approach can be considered as a model for similar systematic reviews and quality assessments of CPGs.

Furthermore, the statistical analysis in this study illustrated the practicability of the AGREE II instrument as a valuable tool for the critical appraisal of CPGs, without compromising on the quality. Conceivably, inexperienced staff or non-professional reviewers would not have reached similar agreements with regard to the clinical features or characteristics in the CPGs, which could impact the judgment related to the provision of care to pregnant women with SCD.

In the first half of 2019, more than eight systematic critical appraisals of CPGs in obstetrics and gynecology were published using the AGREE II instrument. These included high priority health topics such as induction of labor[26], planned cesarean Sect. [27], recurrent pregnancy loss[28], packed red cells versus whole blood transfusion for severe pregnancy-related anemia and obstetric bleeding[29], gestational diabetes mellitus[30,31,32], and bladder pain syndrome/interstitial cystitis [33]. These studies identified several gaps, including differences, discrepancies, lack of evidence-base, and inconsistencies in some clinical recommendations among the CPGs; in addition, a few commonalities and similarities in some recommendations with advice to improve these variabilities were observed in CPGs [26,27,28,29,30,31,32,33].

Strengths and limitations

One of the strengths of the current study is that the appraisal conducted in this review was performed by a specialized clinical team of obstetricians and gynecologists, internists, and hematologists guided by an expert CPG methodologist, which adds a layer of strength to the AGREE II assessment. The other advantages of this study include the following: (i) the use of an international, rigorously structured, and validated CPG appraisal tool: the AGREE II instrument; (ii) the appraisal of each CPG by four raters including three clinical topic experts and a CPG methodologist; (iii) a comprehensive search within several databases; and (iv) statistical assessment of the inter-rater differences.

Care providers for pregnant women with SCD must be encouraged to adopt and merge the principles of “evidence-based” and “eminence-based” healthcare in their daily practice through continuous training and education about the standards of high-quality CPGs and their appraisal tools [34,35,36,37,38]. The results of this review can be used as a basis for CPG development or adaptation for pregnant women with SCD. Furthermore, they highlight the importance of the inclusion of the AGREE II criteria during capacity building, which will aid the clinicians in identifying and adopting the CPGs for use in daily practice.

Our study also has several limitations. First, the AGREE II instrument has several updates and different versions. Some of the disadvantages of AGREE II have been addressed in the recently developed “AGREE-REX” (Recommendation EXcellence) tool that reports the clinical credibility of the CPG recommendations. AGREE-REX has been validated and shared publicly on the website [39]. The selection of 70% as a cut-off point for standard domain ratings is another potential limitation because the original AGREE II does not mandate such a cut- off; however, some studies have used this cut-off value [40]. Other limitations include the following: (i) the inclusion of English language CPGs only may have resulted in the exclusion of relevant CPGs intended for use in non-English speaking healthcare settings; (ii) this review mainly focused on CPGs for the management of pregnant women with SCD, due to its known burden and priority for maternity health, and did not evaluate other subcategories because it was out of the scope of this study; and (iii) the included CPGs belonged to two different healthcare systems (i.e. US-based and UK-based).

Implications for practice: guidance for guideline uptake

The adaptation of CPGs has been identified as a valid and feasible alternative to de novo development, which is a resource-extensive process [41]. Evidence-based practice initiatives in some countries, especially those with low- and middle-income economies, have opted to utilize CPG adaptation rather than development [10]. Several formal adaptation methodologies are now available and could be further customized to local contexts [42]. Similar reviews like ours could inform relevant CPG adaptation or development projects for the SCD in pregnant women especially for groups with little experience, in using the AGREE II instrument.

The current critical appraisal highlights the importance of performing quality assessments of CPGs by clinicians to ensure the transparency and strength of the CPG development process according to international CPG standards and to support the provision of the best practice for pregnant women with SCD. We recommend incorporating the AGREE II appraisal of CPGs in the capacity building plans for obstetricians, gynecologists, and hematologists.

Conclusions

The methodological qualities of three evidence-based CPGs were superior to that of the expert consensus. NICE followed by the RCOG and NHLBI SCD CPGs presented with the highest qualities and were recommended for use in practice. We recommend using the AGREE II criteria and the methodologies utilized in the RCOG and NICE CPGs as models.