Advertisement

The International Journal of Cardiovascular Imaging

, Volume 34, Issue 11, pp 1725–1730 | Cite as

Crowdsourcing consensus: proposal of a novel method for assessing accuracy in echocardiography interpretation

  • Stephanie Minter
  • Alicia Armour
  • Amanda Tinnemore
  • Karen Strub
  • Anna Lisa Crowley
  • Gerald S. Bloomfield
  • John H. Alexander
  • Pamela S. Douglas
  • Joseph A. Kisslo
  • Eric J. Velazquez
  • Zainab Samad
Original Paper
  • 59 Downloads

Abstract

Quality in stress echocardiography interpretation is often gauged against coronary angiography (CA) data but anatomic obstructive coronary disease on CA is an imperfect gold standard for a stress induced wall motion abnormality. We examined the utility of crowd-sourcing a “majority-vote” consensus as an alternative ‘gold standard’ against which to evaluate the accuracy of an individual echocardiographer’s interpretation of stress echocardiography studies. Participants independently interpreted baseline and post-exercise stress echocardiographic images of cases that had undergone follow up CA within 3 months of the stress echo in two surveys, 2 years apart. We examined the agreement of consensus on survey (survey participant response (> 60%) for one decision) with the stress echocardiography clinical read and with CA results. In the first survey, 29 participants reviewed and independently interpreted 14 stress echo cases. Consensus was reached in all 14 cases. There was good agreement between clinical and consensus (kappa = 0.57), survey participant response and consensus (kappa = 0.68) and consensus and CA results (kappa = 0.40). In the validation survey, the agreement between clinical reads and consensus (kappa = 0.75) and survey participant response and consensus (kappa = 0.81) remained excellent. Independent consensus is achievable and offers a fair comparison for stress echocardiographic interpretation. Future validation work, in other laboratories, and against hard outcomes, is necessary to test the feasibility and effectiveness of this approach.

Keywords

Stress echocardiography Cardiac catheterization Continuous quality improvement Echocardiography Image acquisition 

Notes

Compliance with ethical standards

Conflict of Interest

Author Stephanie Minter, MS, RCS declares she has no conflict of interest. Author Alicia Armour MA, RDCS declares she has no conflict of interest. Author Amanda Tinnemore BS, RDCS declares she has no conflict of interest. Author Karen Strub BS, RDCS declares she has no conflict of interest. Author Anna Lisa Crowley, MD declares she has no conflict of interest. Author Gerald Bloomfield, MD, MPH declares he has no conflict of interest. Author John Alexander, MD, MHS declares he has no conflict of interest. Author Pamela Douglas, MD declares she has no conflict of interest. Author Joseph Kisslo, MD declares he is a speaker and consultant to Philips Ultrasound and GE Ultrasound and has grants from Medtronic and Novaseed. Author Eric Velazquez, MD declares he has no conflict of interest. Author Zainab Samad, MD, MHS declares she has no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

References

  1. 1.
    Mark DB, Anderson JL, Brinker JA, Brophy JA, Casey DE Jr, Cross RR et al (2014) ACC/AHA/ASE/ASNC/HRS/IAC/Mended Hearts/NASCI/RSNA/SAIP/SCAI/SCCT/SCMR/SNMMI 2014 health policy statement on use of noninvasive cardiovascular imaging: a report of the american college of cardiology clinical quality committee. J Am Coll Cardiol 63:698–721CrossRefGoogle Scholar
  2. 2.
    Picard MH, Adams D, Bierig SM, Dent JM, Douglas PS, Gillam LD et al (2011) American Society of Echocardiography recommendations for quality echocardiography laboratory operations. J Am Soc Echocardiogr 24:1–10CrossRefGoogle Scholar
  3. 3.
    Kisslo JBB, Geiser EA, Gresser C, Gillam LD, Watkins-Ivy T, Kerber RE et al (1995) Recommendations for continuous quality improvement in echocardiography. J Am Soc Echocardiogr 8(5):Part 2: S1–S28CrossRefGoogle Scholar
  4. 4.
    Douglas PS, Chen J, Gillam L, Hendel R, Hundley WG, Masoudi F et al (2009) Achieving quality in cardiovascular imaging II: proceedings from the second American College of Cardiology—Duke University Medical Center think tank on quality in cardiovascular imaging. JACC: Cardiovasc Imaging 2:231–240Google Scholar
  5. 5.
    Daubert MA, Yow E, Barnhart HX, Rabineau D, Crowley AL, Douglas PS (2015) Quality improvement implementation: improving reproducibility in the echocardiography laboratory. J Am Soc Echocardiogr 28:959–968CrossRefGoogle Scholar
  6. 6.
    Johnson TV, Symanski JD, Patel SR, Rose GA (2011) Improvement in the assessment of diastolic function in a clinical echocardiography laboratory following implementation of a quality improvement initiative. J Am Soc Echocardiogr 24:1169–1179CrossRefGoogle Scholar
  7. 7.
    Johri AM, Picard MH, Newell J, Marshall JE, King ME, Hung J (2011) Can a teaching intervention reduce interobserver variability in lvef assessment: a quality control exercise in the echocardiography lab. JACC: Cardiovasc Imaging 4:821–829Google Scholar
  8. 8.
    Pellikka PA, Nagueh SF, Elhendy AA, Kuehl CA, Sawada SG, American Society of Echocardiography (2007) American Society of Echocardiography recommendations for performance, interpretation, and application of stress echocardiography. J Am Soc Echocardiogr 20:1021–1041CrossRefGoogle Scholar
  9. 9.
    Monaghan MJ (2006) Role of real time 3D echocardiography in evaluating the left ventricle. Heart 92:131–136CrossRefGoogle Scholar
  10. 10.
    Samad Z, Minter S, Armour A, Tinnemore A, Sivak JA, Sedberry B et al (2016) Implementing a continuous quality improvement program in a high-volume clinical echocardiography laboratory: improving care for patients with aortic stenosis. Circ: Cardiovasc Imaging.  https://doi.org/10.1161/CIRCIMAGING.115.003708 CrossRefGoogle Scholar
  11. 11.
    Feigenbaum H (1994) Echocardiography. Lea & Febiger, PhiladelphiaGoogle Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  • Stephanie Minter
    • 2
  • Alicia Armour
    • 2
  • Amanda Tinnemore
    • 2
  • Karen Strub
    • 2
  • Anna Lisa Crowley
    • 1
    • 2
    • 3
  • Gerald S. Bloomfield
    • 1
    • 2
    • 3
  • John H. Alexander
    • 1
    • 2
  • Pamela S. Douglas
    • 1
    • 2
    • 3
  • Joseph A. Kisslo
    • 1
    • 2
  • Eric J. Velazquez
    • 1
    • 2
    • 3
  • Zainab Samad
    • 1
    • 2
    • 3
  1. 1.Division of Cardiology, Department of MedicineDuke University Medical CenterDurhamUSA
  2. 2.Cardiac Diagnostic UnitDuke University Medical CenterDurhamUSA
  3. 3.Duke Clinical Research InstituteDurhamUSA

Personalised recommendations