Skip to main content
SpringerLink
Log in

Interrater and intrarater agreement superior for three-dimensional digital subtraction angiography (3D-DSA) over 2D-DSA classification for detecting remnants after intracranial aneurysm clipping, a GRRAS Reliability and Agreement Study

  • Original Article - Vascular Neurosurgery - Aneurysm
  • Published:
Acta Neurochirurgica Aims and scope Submit manuscript

Abstract

Background

Growing evidence suggests that three-dimensional digital subtraction angiography (3D-DSA) is superior to 2D-DSA in detection of intracranial aneurysm (IA) remnants after clipping. With a simple, practical quantitative scale proposed to measure maximal remnant dimension on 3D-DSA, this study provides a rigorous interrater and intrarater reliability and agreement study comparing this newly established scale with a commonly used (Sindou) 2D-DSA scale.

Method

Records of 43 patients with clipped IAs harboring various sized remnants who underwent 2D- and 3D-DSA between 2012 and 2018 were evaluated. Using the 2D and 3D scales, six raters scored these remnants and repeated the scoring task 8 weeks later. Interrater and intrarater agreement for both grading schemes were calculated using kappa (κ) statistics.

Results

Interrater agreement was highly significant, yielding κ-values at 95% CI (p = 0.000) of 0.225 for the first [0.185; 0.265] and 0.368 s [0.328; 0.408] time points for 2D-DSA and values of 0.700 for the first [0.654; 0.745] and 0.776 s [0.729; 0.822] time points for 3D-DSA. Intrarater agreement demonstrated κ-values between 0.139 and 0.512 for 2D-DSA and between 0.487 and 0.813 for 3D-DSA scores.

Conclusion

Interrater and intrarater agreement was minimal or weak for 2D-DSA scores, but strong for 3D-DSA scores. We propose that baseline 3D-DSA characterization may prove more reliable when categorizing clipped IA remnants for purposes of risk stratification and lifelong follow-up.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Abbreviations

IA:

Intracranial aneurysm

2D:

Two-dimensional

3D:

Three-dimensional

DSA:

Digital subtraction angiography

GRAAS:

Guidelines for Reporting Reliability and Agreement Studies)

RA:

Rotational angiography

References

  1. Abdulazim A, Rubbert C, Reichelt D, Mathys C, Turowski B, Steiger HJ, Hanggi D, Etminan N (2017) Dual- versus single-energy CT-angiography imaging for patients undergoing intracranial aneurysm repair. Cerebrovasc Dis 43:272–282

    Article  Google Scholar 

  2. Ahn SS, Kim YD (2010) Three-dimensional digital subtraction angiographic evaluation of aneurysm remnants after clip placement. J Korean Neurosurg Soc 47:185–190

    Article  Google Scholar 

  3. Allcock J, Drake C (1963) Postoperative Angiography in Cases of Ruptured Intracranial Aneurysm. J Neurosurg 20:752–759

    Article  CAS  Google Scholar 

  4. Bakeman R, Quera V, McArthur D, Robinson BF (1997) Detecting sequential patterns and determining their reliability with fallible observers. Psychol Methods 2:357–370

    Article  Google Scholar 

  5. Burkhardt JK, Chua MHJ, Weiss M, Do ASS, Winkler EA, Lawton MT (2017) Risk of aneurysm residual regrowth, recurrence, and de novo aneurysm formation after microsurgical clip occlusion based on follow-up with catheter angiography. World Neurosurg 106:74–84

    Article  Google Scholar 

  6. David CA, Vishteh AG, Spetzler RF, Lemole M, Lawton MT, Partovi S (1999) Late angiographic follow-up review of surgically treated aneurysms. J Neurosurg 91:396–401

    Article  CAS  Google Scholar 

  7. Drake CG, Vanderlinden RG (1967) The late consequences of incomplete surgical treatment of cerebral aneurysms. J Neurosurg 27:226–238

    Article  CAS  Google Scholar 

  8. Fandino J, Taussky P, Marbacher S, Muroi C, Diepers M, Fathi AR, Remonda L (2013) The concept of a hybrid operating room: applications in cerebrovascular surgery. Acta Neurochir Suppl 115:113–117

    PubMed  Google Scholar 

  9. Gruber A, Dorfer C, Standhardt H, Bavinzski G, Knosp E (2011) Prospective comparison of intraoperative vascular monitoring technologies during cerebral aneurysm surgery. Neurosurgery 68:657–673; discussion 673

  10. Hochmuth A, Spetzger U, Schumacher M (2002) Comparison of three-dimensional rotational angiography with digital subtraction angiography in the assessment of ruptured cerebral aneurysms. AJNR Am J Neuroradiol 23:1199–1205

    PubMed  PubMed Central  Google Scholar 

  11. Jabbarli R, Pierscianek D, Wrede K, Dammann P, Schlamann M, Forsting M, Muller O, Sure U (2016) Aneurysm remnant after clipping: the risks and consequences. J Neurosurg 125:1249–1255

    Article  Google Scholar 

  12. Johnston SC, Dowd CF, Higashida RT, Lawton MT, Duckwiler GR, Gress DR, Investigators C (2008) Predictors of rehemorrhage after treatment of ruptured intracranial aneurysms: the Cerebral Aneurysm Rerupture After Treatment (CARAT) study. Stroke 39:120–125

    Article  Google Scholar 

  13. Kang HS, Han MH, Kwon BJ, Jung SI, Oh CW, Han DH, Chang KH (2004) Postoperative 3D angiography in intracranial aneurysms. AJNR Am J Neuroradiol 25:1463–1469

    PubMed  PubMed Central  Google Scholar 

  14. Kotowski M, Farzin B, Fahed R, Guilbert F, Chagnon M, Darsaut TE, Daniel RT, Raymond J (2019) Residual cerebral aneurysms after microsurgical clipping: a new scale, an agreement study, and a systematic review of the literature. World Neurosurg 121:e302–e321

    Article  Google Scholar 

  15. Kottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hrobjartsson A, Roberts C, Shoukri M, Streiner DL (2011) Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol 64:96–106

    Article  Google Scholar 

  16. Kumar S, Gaikwad SB, Mishra NK (2014) 3D rotational angiography in follow-up of clipped intracranial aneurysms. ISRN Radiol 2014:935280

  17. Landis J, Koch G (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174

    Article  CAS  Google Scholar 

  18. Marbacher S, Halter M, Vogt DR, Kienzler JC, Magyar CTJ, Wanderer S, Anon J, Diepers M, Remonda L, Fandino J (2021) Value of three-dimensional digital subtraction angiography for detection and classification of intracranial aneurysm remnants after clipping. Oper Neurosurg (Hagerstown)

  19. Marbacher S, Kienzler JC, Mendelowitsch I, D’Alonzo D, Andereggen L, Diepers M, Remonda L, Fandino J (2020) Comparison of intra- and postoperative 3-dimensional digital subtraction angiography in evaluation of the surgical result after intracranial aneurysm treatment. Neurosurgery 87:689–696

    Article  Google Scholar 

  20. Marbacher S, Mendelowitsch I, Gruter BE, Diepers M, Remonda L, Fandino J (2018) Comparison of 3D intraoperative digital subtraction angiography and intraoperative indocyanine green video angiography during intracranial aneurysm surgery. J Neurosurg:1–8

  21. Marbacher S, Niemela M, Hernesniemi J, Frosen J (2017) Recurrence of endovascularly and microsurgically treated intracranial aneurysms-review of the putative role of aneurysm wall biology. Neurosurg Rev

  22. Marbacher S, Spiessberger A, Diepers M, Remonda L, Fandino J (2018) Early intracranial aneurysm recurrence after microsurgical clip ligation: case report and review of the literature. J Neurol Surg Rep 79:e93–e97

    Article  Google Scholar 

  23. McHugh M (2012) Interrater reliability: the kappa statistic. Biochemia Medica 22:276–282

    Article  Google Scholar 

  24. Pedicelli A, Desiderio F, Esposito G, Rollo M, Albanese A, Verdolotti T, D’Argento F, Bonomo L, Maira G, Colosimo C (2013) Three-dimensional rotational angiography for craniotomy planning and postintervention evaluation of intracranial aneurysms. Radiol Med 118:415–430

    Article  CAS  Google Scholar 

  25. Raabe A, Beck J, Gerlach R, Zimmermann M, Seifert V (2003) Near-infrared indocyanine green video angiography: a new method for intraoperative assessment of vascular flow. Neurosurgery 52:132–139; discussion 139

  26. Sindou M, Acevedo J, Turjman F (1998) Aneurysmal remnants after microsurgical clipping classification and results from a prospective angiographic study (in a consecutive series of 305 operated intracranial aneurysms). Acta Neurochir (Wien) 161:1153–1159

    Article  Google Scholar 

  27. Spiessberger A, Vogt DR, Fandino J, Marbacher S (2019) Formation of intracranial de novo aneurysms and recurrence after neck clipping: a systematic review and meta-analysis. J Neurosurg 132:456–464

    Article  Google Scholar 

  28. Spiotta AM, Hui F, Schuette A, Moskowitz SI (2013) Patterns of aneurysm recurrence after microsurgical clip obliteration. Neurosurgery 72:65–69; discussion 69

  29. van Rooij WJ, Sprengers ME, de Gast AN, Peluso JP, Sluzewski M (2008) 3D rotational angiography: the new gold standard in the detection of additional intracranial aneurysms. AJNR Am J Neuroradiol 29:976–979

    Article  Google Scholar 

  30. Wong SC, Nawawi O, Ramli N, Abd Kadir KA (2012) Benefits of 3D rotational DSA compared with 2D DSA in the evaluation of intracranial aneurysm. Acad Radiol 19:701–707

    Article  Google Scholar 

  31. Yang K, Park JC, Ahn JS, Kwon do H, Kwun BD, Kim CJ, (2014) Characteristics and outcomes of varied treatment modalities for partially thrombosed intracranial aneurysms: a review of 35 cases. Acta Neurochir (Wien) 156:1669–1675

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank Mary Kemper (Glia Media) for medical editing of our initial draft.

Author information

Authors and Affiliations

  1. Department of Neurosurgery, Kantonsspital Aarau, Aarau, Switzerland

    Matthias Halter, Stefan Wanderer, Basil Grüter, Lukas Andereggen & Serge Marbacher

  2. Division of Neuroradiology, Department of Radiology, Kantonsspital Aarau, Aarau, Switzerland

    Javier Anon, Michael Diepers, Philipp Gruber & Luca Remonda

Authors
  1. Matthias Halter
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stefan Wanderer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Basil Grüter
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Javier Anon
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michael Diepers
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Philipp Gruber
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Lukas Andereggen
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Luca Remonda
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Serge Marbacher
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conception and design: Halter, Marbacher, Wanderer. Acquisition of data: Halter, Wanderer. Analysis of data: Marbacher, Wanderer, Grüter, Anon, Diepers, Andereggen, Gruber. Statistical analysis: Halter. Drafting the article: Halter, Wanderer, Grüter, Marbacher. Administrative/technical/material support: Remonda. Study supervision: Marbacher. Critical revision and final approval: All authors.

Corresponding author

Correspondence to Serge Marbacher.

Ethics declarations

Ethics approval

All procedures performed in this study involving human participants were in accordance with the ethical standards of the institutional review board of the Cantonal Hospital Aarau and the Swiss Ethics Commission (EKNZ Nr.2017–001671) as well as with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Competing interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Comments

The purpose of this worthwhile manuscript was to evaluate 2D-DSA versus 3D-DSA for the evaluation of aneurysmal remnants after clipping, and to study inter-rater and intra-rater variability (physicians against each other and against themselves) in the interpretation of the two angiographic techniques. For case material, these authors retrospectively selected cases from their surgical (not endovascular) database and identified two groups of anonymized patients with or without aneurysmal remnants visible remnant on 3D-DSA. Of 43 total cases, 32 patients had remnants and 11 patients had completely clipped IAs.

In consideration of the method, image sets were prepared and blinded by the principal investigator, after which six cerebrovascular specialists, three interventional neuroradiologists and three neurosurgeons, independently categorized the remnants on both 2D-DSA and 3D-DSA images. Two months later, the raters repeated the evaluations, again scoring all cases, which had been randomly rearranged. Inter-rater and intra-rater reliability among the six raters for both 2D-DSA and 3D-DSA scores was determined at both time periods.

They propose that 3D-DSA is the superior method. Inter-rater κ-values were higher for 3D-DSA score than 2D-DSA score in both rating rounds. Additionally, each rater showed a higher κ-value for intra-rater agreement using the 3D score.

It is important to distinguish aneurysmal remnants from de novo recurrence. As the authors propose, “future long-term follow-up studies based on our 3D-DSA score may reveal the true rates of local recurrence, growth, and hemorrhage for clipped IAs with or without IA remnants.”

The study has several limitations. It is retrospective in design, the study size is small at 43 patients (because most clipped IAs in the sample set showed no remnant), it harbors possible selection bias since the blinded rating process included 43 cases not randomly selected but selected to represent all categories of their 3D-DSA scale, and finally because it includes only surgical clipping cases, without consideration (or any discussion) of the rapidly advancing endovascular treatment of IAA and of the grading scales for obliteration (like the Raymond-Roy) that are attendant with those therapies.

I do agree that 3D-DSA holds promise as we continue to assess the risk of aneurysmal remnants after clipping or after endovascular treatment, and to help us better understand the implications and risks of remnants of various sizes, and/or of de novo aneurysm recurrence.

Christopher M. Loftus,

Philadelphia, PA,USA

This article is part of the Topical Collection on Vascular Neurosurgery - Aneurysm

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Halter, M., Wanderer, S., Grüter, B. et al. Interrater and intrarater agreement superior for three-dimensional digital subtraction angiography (3D-DSA) over 2D-DSA classification for detecting remnants after intracranial aneurysm clipping, a GRRAS Reliability and Agreement Study. Acta Neurochir 164, 2173–2179 (2022). https://doi.org/10.1007/s00701-022-05156-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00701-022-05156-3

Keywords

Search

Navigation