Background

Exercise Doppler echocardiography (EDE) is standard practice for the evaluation of patients with coronary artery disease. The procedure is now increasingly used for the assessment of the right heart and the pulmonary circulation [1,2,3,4,5]. Echocardiography of the right heart mainly relies on estimates of right chambers dimensions (diameters/areas/volumes) and function (i.e. fractional area change, tricuspid annular plane systolic excursion (TAPSE) and of tissue Doppler–derived tricuspid lateral annular systolic velocity (S’) [2, 3]. Furthermore it estimates the components of the pulmonary vascular resistance equation, that is pulmonary artery pressure (PAP) from the maximum tricuspid regurgitation velocity (TRV), or the right ventricular outflow tract (RVOT) acceleration time (Act) of PA flow, wedged PAP from the ratio of transmitral flow E and mitral annulus e’ waves and cardiac output (CO) from the left ventricular outflow tract (LVOT) aortic flow. The RIGHT Heart International NETwork (RIGHT-NET) study was designed to comprehensively define limits of normal in right heart function and pulmonary circulation hemodynamics during EDE (diagnostic value) and to investigate the impact of abnormal responses on clinical outcome in individuals with overt or at risk of developing pulmonary hypertension (prognostic value) [6, 7]. The present report aims to provide a quality control analysis of left, right heart and pulmonary circulation resting and EDE measurements among 19 echocardiography laboratories with proven experience participating in the RIGHT-NET study [6, 7].

Methods

The echocardiography Core laboratory of the Institute of Clinical Physiology-CNR in Pisa (LG) coordinated the quality control procedure of all investigators at different centres participating in the RIGHT-NET study. Each center designated one operator that performed or reported at least 100 stress echocardiography studies per year. All readers were certified by national and/ or international societies. The quality control process was designed to be simple, reproducible and sustainable. The echocardiography Core laboratory issued a User Manual with a detailed description on how to measure each parameter, according to the most recent American and European Recommendations and Guidelines [8,9,10,11]. The User Manual was sent to all Participating Centers including the reference for transthoracic echocardiography assessment. All participating centers followed the recommended standard operational procedures in terms of data storage (data format, transfer procedure), and data processing (software used and measurement procedures). All operators performing and reading echocardiographic exams adhered to the quality control protocol. The echocardiography Core laboratory sent ten complete echocardiographic examinations in DICOM format through a safe file sharing platform (Fig. 1). All participating investigators were invited by email to join the platform, which was protected by user-specific passwords. The platform includes also detailed instructions on how to start the training and allows downloading and uploading of external files. Each reader was blinded to core laboratory measurements and to clinical history of the patients. All images and videos were completely anonymized to protect patients’ confidentiality’, in compliance with the EU’s General Data Protection Regulation 2018 [6].

Fig. 1
figure 1

Quality control procedure

Full size image

Reading sessions

The echocardiography core laboratory randomly selected 10 cases including healthy subjects and at least one group of patients with overt and/or at risk of pulmonary hypertension (PH), according to clinical classification of PH (Table 1) [12]. Echocardiographic examinations were performed with commercially available equipment on all subjects (Vivid E9, GE Healthcare, Milwaukee, WI, USA). Data were collected on patients undergoing EDE on a semi-recumbent cycle ergometer with an incremental workload of 25 W every 2 min up to the symptom-limited maximal tolerated workload including resting, 50 W, peak stress and recovery acquisition, as previously described [6]. All operators directly measured the requested parameters by uploading the same ten cases from the web platform to their echocardiography machine. The DICOM format enabled to perform assessment of variables in the respective centres. All operators were then asked to enter their measurements in a dedicated excel file, which was then sent to the Coordinating Center for analysis. Table 2 provides the list of the left and the right heart parameters measured by all operators. The gold standard value for each measurement was established by the values measured by the echocardiography Core laboratory, according to the recommendations for echocardiographic assessment of the left and right heart by the American Society of Echocardiography/European Association of Cardiovascular Imaging [8,9,10,11].

Table 1 Demographic and clinical characteristics of 10 subjects included in quality control procedure
Full size table
Table 2 List of parameters measured in the quality control procedure, A Left Chambers Parameters. B Right Chambers Parameters
Full size table

Statistical analysis

Statistical analysis was performed using standard software (MedCalc version 14.8.1, MedCalc Software Ltd., Belgium; SPSS version 20.0, SPSS, Inc., Chicago, IL). Continuous variables were described by mean values ± standard deviation (SD). Normal distribution of the continuous values was assessed by the Kolmogorov-Smirnov test. Accuracy (in %) for each observer was estimated by comparison with the reference standard (core lab reading). Intra-class correlation coefficient (ICC) was calculated along with the 95% confidence interval, in order to quantify the reliability of measurement process. An ICC of > 0.8 indicated good agreement, ICC > 0.9 indicated excellent agreement with the core lab. Inter-observer variability among 19 observers were examined for resting and peak exercise TRV, RVOT Act, TAPSE, S’, right ventricular fractional area change (RV FAC), LVOT velocity time integral (VTI), mitral early inflow pulsed wave Doppler velocity (E), early diastolic mitral annular lateral and septal velocity by TDI (e’), left ventricular ejection fraction (LVEF). Data are presented as mean of the absolute and relative differences (in %) between measurements of all nineteen observers, and ICC for each single parameter was calculated along with the 95% confidence interval.

Intra-observer agreement was tested in 2 observers who volunteered to repeat the measurement session on 2 separate days and ICC was calculated.

Results

Nineteen observers completed all reading sessions. Figure 2 shows a summary of the accuracy (in %) of each center compared with the gold standard core lab for all parameters at rest and at peak of exercise. The average accuracy of 19 readers for all parameters was excellent in about 99.8% (range from 99.7 to 100%) (Table 3). ICC was > 0.9 for all observers. The average agreement of the 19 readers for all parameters was excellent (ICC = 0.98). Therefore there was no need to conduct personal feed-back and a second slot of measurements for anyone. Moreover the average agreement among readers remained excellent at rest and at peak exercise for all measurements (ICC = 0.98 and 0.99, respectively) (Table 4). Inter-observer variabilities among all observers for main exercise TTE measurements were reported in Table 5. Close inter-observer variabilities were found for both resting and peak exercise TRV (3.8 and 2.4%) (ICC = 0.97 and 0.98), RV S’ (2.9% for both) (ICC = 0.95 for both), E (5.7 and 8.3%) (ICC = 0.99 and 0.98) and e’ (6 and 6.5%) (ICC = 0.97 for both). Inter-observer variabilities of the RVOT Act and LVOT VTI were of 9.7% (ICC = 0.95) and 7.4% (ICC = 0.98) at rest, 12% (ICC = 0.92) and 9.6% (ICC = 0.97) at peak exercise, respectively. TAPSE showed less resting (5.3%) (ICC = 0.97) than peak exercise variability (8%) (ICC = 0.95). LVEF revealed lower resting and peak exercise mean relative differences of 7.6 and 9% (ICC = 0.99 and 0.98), respectively. Moderate inter-observer variability was found for resting and peak exercise RV FAC (15 and 16%, respectively) (ICC = 0.82 and 0.80) (Table 5).

Fig. 2
figure 2

Accuracy of each center compared with the gold standard core lab for all, right and left parameters (a), for all parameters at rest and peak (b), for right paramaters at rest and at peak (c) and for left parameter at rest and at peak (d)

Full size image
Table 3 Accuracy, ICC and 95% Confidence Interval of each Center for all parameters
Full size table
Table 4 Accuracy, ICC and 95% Confidence Interval of each Center for all parameters at rest and peak
Full size table
Table 5 Interobserver variability of main exercise Doppler echocardiographic measurements of all participating centers at rest and at peak exercise
Full size table

The intra-observer quality control analysis revealed an excellent ICC of 0.97 (95% Confidence Interval: 0.96 to 0.99). All ICC > 0.95 remained excellent at rest and at peak exercise for all measurements, except for RV FAC (ICC = 0.85 and 0.82, respectively). Each ICC showed p value < 0.0001.

Discussion

Before any acquisition of pooling echocardiographic data for research and clinical applications, a process of quality control and reading harmonization measurements should be undertaken [13,14,15,16]. The present results demonstrate that a rigorously designed protocol with a strong focus on quality assurance and certification can yield very strong ICC and limited variability among the 19 participant experienced centers to a large prospective EDE study of the right heart and the pulmonary circulation.

Previous studies

The inter-observer variability during EDE right heart and pulmonary circulation studies may be not negligible. Few such studies have been previously reported and all were mono-centric [3, 17]. Argiento et al. reported in 124 healthy subjects (62 women and 62 men; age 37 ± 13 yrs) (single center study) an inter-observer variability for pulmonary artery systolic pressure (PASP) and cardiac output (CO) estimates of 1.9 and 4.9% at rest, and 7.9 and 13.9% at maximum exercise, respectively [18]. D’Alto et al. reported in 90 healthy subjects (45 male, mean age 39 ± 13 years) inter-observer variabilities between two readers at rest and peak exercise of 1.9 and 7.9% for PASP, 4.9 and 13.9% for stroke volume, 2.6 and 6.8% for TAPSE, and 5.4 and 8.7% for S′, respectively [19]. Kusunose et al. reported in a subgroup of 15 randomly selected subjects with isolated moderate to severe mitral regurgitation a close inter- and intra-observer variability for resting TAPSE (8.8%) and exercise TAPSE (9.5%) [20]. As these data remain limited, more validation appeared necessary for a multi-centric study like the RIGHT-NET.

Uniqueness of the present study and clinical implications

To the best of our knowledge, this is the largest EDE multicenter study that comprehensively provides a detailed quality control analysis of both the right heart and the pulmonary circulation measurements. One major finding was that the accuracy and agreement were remarkably high among 19 experienced investigators, with no significant differences between resting and exercise measurements. These results provide a valid evidence of reliability of TRV, E/e’ ratio, LVOT VTI and LVEF during exercise. The inter-observer variability of RVOT Act was higher than that of TRV. RVOT Act measurements were collected during exercise, in keeping with a recent report advocating its combination with TRV for the assessment of the pulmonary pressures both at rest and during exercise [21]. The interest of this combination is that the feasibility rate of RVOT Act may be higher than that of TRV [22]. Furthermore our findings suggested that exercise TAPSE and S′ may be used as reproducible measures of the RV longitudinal systolic function. Larger resting and exercise variability of RV FAC may be caused by plane-dependency and reliance on a complex definition of the RV endocardial border [23].

Study limitations

Few study limitations need to be discussed. First, the present study did not validate the echocardiographic measurements against invasive gold standard evaluation of the pulmonary circulation (PAP, wedged PAP and cardiac output), and right ventricular function (indices derived from pressure-volume loops). Second, accuracy and precision were defined by comparison only with the core laboratory measurements. In this regard, it was not logistically possible to repeat the echocardiographic examination of the same patient in each participating center. Third, the study results could have been potentially influenced by the quality images acquired only by the echocardiography core laboratory. For this reason we randomly selected 10 cases with different clinical conditions from a large database to avoid possible bias of selection of best images. Fourth, the number of patients studied was relatively small (n = 10). However, each of the 19 participating centers provided a total of 35 left and right heart echo-Doppler variables at rest, peak exercise and after 5 min of recovery.

Conclusions

When protocols for acquisition and analysis are provided upfront and in experienced echocardiography laboratories EDE represents a reproducible tool to comprehensively assess the right heart and pulmonary circulation. This quality control study represents a solid bedrock for future RIGHT-NET studies, aiming to evaluate the diagnostic and prognostic role of EDE in the clinical settings of patients with cardiorespiratory diseases.