Skip to main content
Log in

Comparison of MRI response evaluation methods in rectal cancer: a multicentre and multireader validation study

  • Gastrointestinal
  • Published:
European Radiology Aims and scope Submit manuscript



To compare four previously published methods for rectal tumor response evaluation after chemoradiotherapy on MRI.


Twenty-two radiologists (5 rectal MRI experts, 17 general/abdominal radiologists) retrospectively reviewed the post-chemoradiotherapy MRIs of 90 patients, scanned at 10 centers (with non-standardized protocols). They applied four response methods; two based on T2W-MRI only (MRI tumor regression grade (mrTRG); split-scar sign), and two based on T2W-MRI+DWI (modified-mrTRG; DWI-patterns). Image quality was graded using a 0–6-point score (including slice thickness and in-plane resolution; sequence angulation; DWI b-values, signal-to-noise, and artefacts); scores < 4 were classified below average. Mixed model linear regression was used to calculate average sensitivity/specificity/accuracy to predict a complete response (versus residual tumor) and assess the impact of reader experience and image quality. Group interobserver agreement (IOA) was calculated using Krippendorff’s alpha. Readers were asked to indicate their preferred scoring method(s).


Average sensitivity/specificity/accuracy was 57%/64%/62% (mrTRG), 36%/79%/66% (split-scar), 40%/79%/67% (modified-mrTRG), and 37%/82%/68% (DWI-patterns); mrTRG showed higher sensitivity but lower specificity and accuracy (p < 0.001) compared to the other methods. IOA was lower for the split scar method (0.18 vs. 0.39–0.43). Higher reader experience had a significant positive effect on diagnostic performance and IOA (except for the split scar sign); below-average imaging quality had a significant negative effect on diagnostic performance. DWI pattern was selected as the preferred method by 73% of readers.


Methods incorporating DWI showed the most favorable results when combining diagnostic performance, IOA, and reader preference. Reader experience and image quality clearly impacted diagnostic performance emphasizing the need for state-of-the-art imaging and dedicated radiologist training.

Key Points

In a multireader study comparing 4 MRI methods for rectal tumor response evaluation, those incorporating DWI showed the best results when combining diagnostic performance, IOA, and reader preference.

The most preferred method (by 73% of readers) was the “DWI patterns” approach with an accuracy of 68%, high specificity of 82%, and group IOA of 0.43.

• Reader experience level and MRI quality had an evident effect on diagnostic performance and IOA.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others





DIFFUSION-weighted imaging


Interobserver agreement


MRI tumor regression grade


Negative predictive value


Positive predictive value


Watch & Wait


  1. Oronsky B, Reid T, Larson C, Knox SJ (2020) Locally advanced rectal cancer: the past, present, and future. Semin Oncol 47(1):85–92

    Article  PubMed  Google Scholar 

  2. van der Valk MJM, Hilling DE, Bastiaannet E et al (2018) Long-term outcomes of clinical complete responders after neoadjuvant treatment for rectal cancer in the International Watch & Wait Database (IWWD): an international multicentre registry study. Lancet 391(10139):2537–2545

    Article  PubMed  Google Scholar 

  3. Lambregts D, Boellaard TN, Beets-Tan R (2019) Response evaluation after neoadjuvant treatment for rectal cancer using modern MR imaging: a pictorial review. Insights Imaging 10(1):15

    Article  PubMed  PubMed Central  Google Scholar 

  4. Barbaro B, Vitale R, Leccisotti L et al (2010) Restaging locally advanced rectal cancer with MR imaging after chemoradiation therapy. Radiographics 30(3):699–716

    Article  PubMed  Google Scholar 

  5. Mandard AM, Dalibard F, Mandard JC et al (1994) Pathologic assessment of tumor regression after preoperative chemoradiotherapy of esophageal carcinoma. Clinicopathologic correlations. Cancer 73(11):2680–2686.tumor

    Article  CAS  PubMed  Google Scholar 

  6. Siddiqui MR, Gormly KL, Bhoday J et al (2016) Interobserver agreement of radiologists assessing the response of rectal cancers to preoperative chemoradiation using the MRI tumor regression grading (mrTRG). Clin Radiol 71(9):854–862

    Article  CAS  PubMed  Google Scholar 

  7. Jang JK, Choi SH, Park SH et al (2020) MR tumor regression grade for pathological complete response in rectal cancer post neoadjuvant chemoradiotherapy: a systematic review and meta-analysis for accuracy. Eur Radiol 30(4):2312–2323.tumor

    Article  PubMed  Google Scholar 

  8. Bhoday J, Smith F, Siddiqui MR et al (2016) Magnetic resonance tumor regression grade and residual mucosal abnormality as predictors for pathological complete response in rectal cancer postneoadjuvant chemoradiotherapy. Dis Colon Rectum 59(10):925–933.tumor

    Article  PubMed  Google Scholar 

  9. Patel UB, Brown G, Rutten H et al (2012) Comparison of magnetic resonance imaging and histopathological response to chemoradiotherapy in locally advanced rectal cancer. Ann Surg Oncol 19(9):2842–2852

    Article  PubMed  Google Scholar 

  10. Lee MA, Cho SH, Seo AN et al (2017) Modified 3-point MRI-based tumor regression grade incorporating DWI for locally advanced rectal cancer. AJR Am J Roentgenol 209(6):1247–1255. tumor

    Article  PubMed  Google Scholar 

  11. Haak HE, Maas M, Lahaye MJ et al (2020) Selection of patients for organ preservation after chemoradiotherapy: MRI identifies poor responders who can go straight to surgery. Ann Surg Oncol 27(8):2732–2739

    Article  PubMed  Google Scholar 

  12. Schurink NW, Lambregts D, Beets-Tan R (2019) Diffusion-weighted imaging in rectal cancer: current applications and future perspectives. Br J Radiol 92(1096):20180655

    Article  PubMed  PubMed Central  Google Scholar 

  13. Lambregts DMJ, Delli Pizzi A, Lahaye MJ et al (2018) A pattern-based approach combining tumor morphology on MRI with distinct signal patterns on diffusion-weighted imaging to assess response of rectal tumors after chemoradiotherapy. Dis Colon Rectum 61(3):328–337 tumortumor

    Article  PubMed  Google Scholar 

  14. Santiago I, Barata M, Figueiredo N et al (2020) The split scar sign as an indicator of sustained complete response after neoadjuvant therapy in rectal cancer. Eur Radiol 30(1):224–238

    Article  CAS  PubMed  Google Scholar 

  15. Beets-Tan RGH, Lambregts DMJ, Maas M et al (2018) Magnetic resonance imaging for clinical management of rectal cancer: updated recommendations from the 2016 European Society of Gastrointestinal and Abdominal Radiology (ESGAR) consensus meeting. Eur Radiol 28(4):1465–1475

    Article  PubMed  Google Scholar 

  16. Gormly K (2021) Rectal MRI: the importance of high resolution T2 technique. Abdom Radiol (NY) 46(9):4090–4095

    Article  PubMed  Google Scholar 

  17. van Griethuysen JJM, Bus EM, Hauptmann M et al (2018) Gas-induced susceptibility artefacts on diffusion-weighted MRI of the rectum at 1.5 T - effect of applying a micro-enema to improve image quality. Eur J Radiol 99:131–137

    Article  PubMed  Google Scholar 

  18. Van Griethuysen JJM, Schurink NW, Lahaye MJ et al (2020) Deep learning for fully automated segmentation of rectal tumors on MRI in a multicentre setting. Insights into imaging – ESGAR book of abstracts 11(Suppl 3):SS5.4

    Google Scholar 

  19. Ziegler E, Urban T, Brown D et al (2020) Open health imaging foundation viewer: an extensible open-source framework for building web-based imaging applications to support cancer research. JCO Clin Cancer Inform 4:336–345

    Article  PubMed  Google Scholar 

  20. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46

    Article  Google Scholar 

  21. Antoine, J. Y., Villaneau, J., & Lefeuvre, A. (2014). Weighted Krippendorff's alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation. EACL 2014 (pp. 10-p).

  22. Hayes AF, Krippendorff K (2007) Answering the call for a standard reliability measure for coding data. Commun Methods Meas 1(1):77–89

    Article  Google Scholar 

  23. Nasu K, Kuroki Y, Kuroki S, Murakami K, Nawano S, Moriyama N (2004) Diffusion-weighted single shot echo planar imaging of colorectal cancer using a sensitivity-encoding technique. Jpn J Clin Oncol 34(10):620–626

    Article  PubMed  Google Scholar 

  24. Korn N, Kurhanewicz J, Banerjee S, Starobinets O, Saritas E, Noworolski S (2015) Reduced-FOV excitation decreases susceptibility artifact in diffusion-weighted MRI with endorectal coil for prostate cancer detection. Magn Reson Imaging 33(1):56–62

    Article  PubMed  Google Scholar 

  25. Thian YL, Xie W, Porter DA, Weileng AB (2014) Readout-segmented echo-planar imaging for diffusion-weighted imaging in the pelvis at 3T-a feasibility study. Acad Radiol 21(4):531–537

    Article  PubMed  Google Scholar 

  26. Maas M, Lambregts DM, Nelemans PJ et al (2015) Assessment of clinical complete response after chemoradiation for rectal cancer with digital rectal examination, endoscopy, and MRI: selection for organ-saving treatment. Ann Surg Oncol 22(12):3873–3880

    Article  PubMed  PubMed Central  Google Scholar 

Download references


Other authors in the study group:

Frans C. H. Bakers1; Perla Barros2; Ferdinand Bauer3; Shira H. de Bie4; Stuart Ballantyne5; Joanna Brayner Dutra6,7; Laura Buskov8; Nino Bogveradze9,10,11; Gerlof P. T. Bosma12; Vincent C. Cappendijk13; Francesca Castagnoli14; Sotiriadis Charalampos15; Andrea Delli Pizzi26; Michael Digby17; Remy W. F. Geenen18; Joost J. M. van Griethuysen9,19; Julie Lafrance20; Vandana Mahajan21; Sonaz Malekzadeh22; Peter A. Neijenhuis23; Gerald M. Peterson24; Indra Pieters25; Niels W. Schurink9,10; Ruth Smit26; Cornelis J. Veeken27; Roy F. A. Vliegen28; Andrew Wray29; Abdel-Rauf Zeina30

1Department of Radiology and Nuclear Medicine, Maastricht University Medical Centre, Maastricht, The Netherlands

2Department of Radiology, Instituto Oncológico Fundación Arturo López Pérez, Santiago, Chile

3.Radiologie Zentrum, Kaufbeuren , Germany

4Department of Radiology, Deventer Ziekenhuis, The Netherlands

5Department of Radiology, Queen Elizabeth University Hospital, Glasgow, United Kingdom

6Department of Radiology, Real Hospital Portugues (RHP), Pernambuco, Brazil

7Department of Radiology, Instituto de Medicina Integral Professor Fernando Figueira (IMIP), Recife, Brazil.

8.Department of Radiology, Bispebjerg Hospital, Copenhagen, Denmark

9Department of Radiology, The Netherlands Cancer Institute, Amsterdam, The Netherlands

10GROW School for Oncology & Developmental Biology, University of Maastricht, Maastricht, The Netherlands

11Department of Radiology, American Hospital Tbilisi, Tbilisi, Georgia

12Department of Radiologie Elisabeth Tweesteden Hospital, Tilburg, The Netherlands

13Department of Radiology, Jeroen Bosch Hospital, ‘s Hertogenbosch, The Netherlands

14Department of Radiology, University of Brescia, Brescia, Italy

15Department of Radiology, Hôpital Riviera Chablais, Rennaz, Switzerland

16Department of Innovative Technologies in Medicine & Dentistry, Gabriele d’Annunzio University of Chieti, Chieti, Italy

17Department of Radiology, Glasgow Royal Infirmary, Glasgow, United Kingdom.

18Department of Radiology, Northwest Clinics, Alkmaar, The Netherlands

19Department of Radiology, Gelre Hospital, Apeldoorn, The Netherlands

20Department of Radiology, Maisonneuve-Rosemont Hospital, Montreal, Canada

21Department of Radiology, Apollo Cancer Hospital, Chennai, India

22Department of Radiology, Sion Hospital, Sion, Switzerland

23Department of Surgery, Alrijne Hospital, Leiderdorp, The Netherlands

24Department of Radiology, Spaarne Gasthuis, Haarlem, The Netherlands

25Department of Radiology, Telemedicine Clinic, United Kingdom

26Department of Radiology, Amsterdams UMC, Amsterdam, The Netherlands

27Department of Radiology, IJsselland Hospital, Capelle aan den IJssel, The Netherlands

28Department of Radiology, Zuyderland Medical Center, Heerlen, The Netherlands

29Department of Radiology, Ulster Hospital, Belfast, United Kingdom

30.Department of Radiology, Hillel Yaffe Medical Center, Hadera, Israel


The authors state that this work has not received any funding.

Author information

Authors and Affiliations



Corresponding author

Correspondence to Doenja M. J. Lambregts.

Ethics declarations


The scientific guarantor of this publication is Dr Doenja MJ Lambregts.

Conflict of interest

The authors of this manuscript declare no relationships with any companies whose products or services may be related to the subject matter of the article.

Statistics and biometry

One of the authors, Mr Renaud Tissier, has significant statistical expertise.

Informed consent

Written informed consent was waived by the Institutional Review Board.

Ethical approval

Institutional Review Board approval was obtained.

Study subjects or cohorts overlap

Some study subjects included in the current cohort have been previously reported on the following:

n = 90 in a study focused on retrospectively evaluating staging trends in the Netherlands following guidelines updates (Bogveradze et al Abdom Radiol (New York). 2022;47(1):38-47).

n = 11 in a study focused on common interpretation pitfalls in rectal DWI and their use for teaching (Lambregts et al Eur Radiol 2017; 27, 4445–4454)

n = 80 in a technical study focused on assessing the reproducibility of quantitative imaging features in multicentre study cohorts (Schurink et al Eur Radiol. 2022;32(3):1506-1516).

n = 6 in a study focused on assessing the sigmoid take-off as a landmark to distinguish rectal from sigmoid tumors on MRI (Bogveradze et al Eur J Surg Oncol 2022;48:237-244)

n = 16 in a single-center pilot study investigating the DWI pattern method (Lambregts et al Dis Colon Rectum 2018;61(3):328-337).


• retrospective

• observational

• multicentre study

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information


(DOCX 1726 kb)


(DOCX 17.9 kb)


(DOCX 28.2 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

El Khababi, N., Beets-Tan, R.G.H., Tissier, R. et al. Comparison of MRI response evaluation methods in rectal cancer: a multicentre and multireader validation study. Eur Radiol 33, 4367–4377 (2023).

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: