European Radiology

, Volume 28, Issue 9, pp 3611–3620 | Cite as

Inter-operator variability and source of errors in tumour response assessment for hepatocellular carcinoma treated with sorafenib

  • Francesco TovoliEmail author
  • Matteo Renzulli
  • Giulia Negrini
  • Stefano Brocchi
  • Alessia Ferrarini
  • Andrea Andreone
  • Francesca Benevento
  • Rita Golfieri
  • Antonio Maria Morselli-Labate
  • Marianna Mastroroberto
  • Radu Ion Badea
  • Fabio Piscaglia



To assess the inter-operator concordance and the potential sources of discordance in defining response to sorafenib in hepatocellular carcinoma (HCC).


All patients who received sorafenib between September 2008 and February 2015 were scrutinised for this retrospective study. Images were evaluated separately by three radiologists with different expertise in liver imaging (operator 1, >10 years; operator 2, 5 years; operator 3, no specific training in liver imaging), according to: response evaluation radiological criteria in solid tumours (RECIST) 1.1, modified RECIST (mRECIST) and response evaluation criteria in cancer of the liver (RECICL).


The overall response concordance between the more expert operators was good, irrespective of the criteria (RECIST 1.1, ĸ = 0.840; mRECIST, ĸ = 0.871; RECICL, ĸ = 0.819). Concordance between the less expert operator and the other colleagues was lower. The most evident discordance was in target lesion response assessment, with expert operators disagreeing mostly on lesion selection and less expert operators on lesion measurement. As a clinical correlate, overall survival was more tightly related with “progressive disease” as assessed by the expert compared to the same assessment performed by operator 3.


Decision on whether a patient is a responder or progressor under sorafenib may vary among different operators, especially in case of a non-specifically trained radiologist. Regardless of the adopted criteria, patients should be evaluated by experienced radiologists to minimise variability in this critical instance.

Key Points

• Inter-operator variability in the assessment of response to sorafenib is poorly known.

• The concordance between operators with expertise in liver imaging was good.

• Target lesions selection was the main source of discordance between expert operators.

• Concordance with non-specifically trained operator was lower, independently from the response criteria.

• The non-specifically trained operator was mainly discordant in measurements of target lesions.


Carcinoma, hepatocellular Sorafenib Response evaluation criteria in solid tumours Tomography, X-ray computed Magnetic resonance imaging 



Hepatocellular carcinoma


Progressive disease


Non-target lesion


Response evaluation criteria in cancer of the liver


Response evaluation radiological criteria in solid tumours



The authors state that this work has not received any funding.

Compliance with ethical standards


The scientific guarantor of this publication is Fabio Piscaglia.

Conflict of interest

The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.

Statistics and biometry

One of the authors has significant statistical expertise.

Informed consent

Written informed consent was obtained from all subjects (patients) in this study.

Ethical approval

Institutional Review Board approval was obtained.


• retrospective

• observational/experimental

• performed at one institution

Supplementary material

330_2018_5393_MOESM1_ESM.doc (27 kb)
ESM 1 (DOC 27 kb)
330_2018_5393_MOESM2_ESM.docx (16 kb)
ESM 2 (DOCX 16 kb)


  1. 1.
    Nagilla M, Brown RL, Cohen EE (2012) Cabozantinib for the treatment of advanced medullary thyroid cancer. Adv Ther 29:925–934CrossRefPubMedGoogle Scholar
  2. 2.
    Mocellin S, Baretta Z, Roqué I et al (2017) Second-line systemic therapy for metastatic colorectal cancer. Cochrane Database Syst Rev 1:CD006875PubMedGoogle Scholar
  3. 3.
    Randrup Hansen C, Grimm D, Bauer J, Wehland M, Magnusson NE (2017) Effects and side effects of using sorafenib and sunitinib in the treatment of metastatic renal cell carcinoma. Int J Mol Sci 18.
  4. 4.
    Wang Y, Schmid-Bindert G, Zhou C (2012) Erlotinib in the treatment of advanced non-small cell lung cancer: an update for clinicians. Ther Adv Med Oncol 4:19–29CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    Therasse P, Arbuck SG, Eisenhauer EA et al (2000) New guidelines to evaluate the response to treatment in solid tumors. European Organization for Research and Treatment of Cancer, National Cancer Institute of the United States, National Cancer Institute of Canada. J Natl Cancer Inst 92:205–216CrossRefPubMedGoogle Scholar
  6. 6.
    Lencioni R, Llovet JM (2010) Modified RECIST (mRECIST) assessment for hepatocellular carcinoma. Semin Liver Dis 30:52–60CrossRefPubMedGoogle Scholar
  7. 7.
    Kudo M, Ueshima K, Kubo S et al (2015) Response Evaluation Criteria in Cancer of the Liver (RECICL) (2015 Revised version). Hepatol Res 46:3–9CrossRefPubMedGoogle Scholar
  8. 8.
    Choi H, Charnsangavej C, Faria SC et al (2007) Correlation of computed tomography and positron emission tomography in patients with metastatic gastrointestinal stromal tumor treated at a single institution with imatinib mesylate: proposal of new computed tomography response criteria. J Clin Oncol 25:1753–1759CrossRefPubMedGoogle Scholar
  9. 9.
    Llovet JM, Ricci S, Mazzaferro V et al (2008) Sorafenib in advanced hepatocellular carcinoma. N Engl J Med 359:378–390CrossRefPubMedGoogle Scholar
  10. 10.
    Bruix J, Sherman M, Practice Guidelines Committee, American Association for the Study of Liver Diseases (2005) Management of hepatocellular carcinoma. Hepatology 42:1208–1236CrossRefPubMedGoogle Scholar
  11. 11.
    Renzulli M, Brocchi S, Cucchetti A et al (2016) Can current preoperative imaging be used to detect microvascular invasion of hepatocellular carcinoma? Radiology. 279:432–442CrossRefPubMedGoogle Scholar
  12. 12.
    Sneag DB, Krajewski K, Giardino A et al (2011) Extrahepatic spread of hepatocellular carcinoma: spectrum of imaging findings. AJR Am J Roentgenol 197:W658–W664CrossRefPubMedGoogle Scholar
  13. 13.
    Katyal S, Oliver JH 3rd, Peterson MS et al (2000) Extrahepatic metastases of hepatocellular carcinoma. Radiology 216:698-703Google Scholar
  14. 14.
    Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychological Bulletin 76:352–378CrossRefGoogle Scholar
  15. 15.
    Portney LG, Watkins MP (2009) Foundations of clinical research: applications to practice, 3rd edn. Pearson/Prentice Hall, Upper Saddle RiverGoogle Scholar
  16. 16.
    Bruix J, Qin S, Merle P et al (2017) Regorafenib for patients with hepatocellular carcinoma who progressed on sorafenib treatment (RESORCE): a randomised, double-blind, placebo-controlled, phase 3 trial. Lancet 389:56–66CrossRefPubMedGoogle Scholar
  17. 17.
    Bargellini I, Scionti A, Mismas V et al (2014) Identification of responders to sorafenib in hepatocellular carcinoma: is tumor volume measurement the way forward? Oncology. 86:191–198CrossRefPubMedGoogle Scholar
  18. 18.
    Sato Y, Watanabe H, Sone M et al (2013) Tumor response evaluation criteria for HCC (hepatocellular carcinoma) treated using TACE (transcatheter arterial chemoembolization): RECIST (response evaluation criteria in solid tumors) version 1.1 and mRECIST (modified RECIST): JIVROSG-0602. Ups J Med Sci 118:16–22CrossRefPubMedPubMedCentralGoogle Scholar
  19. 19.
    Seyal AR, Gonzalez-Guindalini FD, Arslanoglu A et al (2015) Reproducibility of mRECIST in assessing response to transarterial radioembolization therapy in hepatocellular carcinoma. Hepatology 62:1111–1121CrossRefPubMedGoogle Scholar
  20. 20.
    Lee S, Kim JH, Lee JH et al (2018) Non-invasive monitoring of the therapeutic response in sorafenib-treated hepatocellular carcinoma based on photoacoustic imaging. Eur Radiol 28:372–381CrossRefPubMedGoogle Scholar
  21. 21.
    Tovoli F, Renzulli M, Granito A et al (2017) Radiologic criteria of response to systemic treatments for hepatocellular carcinoma. Hepatic Oncology 4:129–137CrossRefPubMedPubMedCentralGoogle Scholar
  22. 22.
    Lencioni R, Montal R, Torres F et al (2017) Objective response by mRECIST as a predictor and potential surrogate end-point of overall survival in advanced HCC. J Hepatol 66:1166–1172CrossRefPubMedGoogle Scholar
  23. 23.
    Bruix J, Reig M, Sangro B (2017) Assessment of treatment efficacy in hepatocellular carcinoma: Response rate, delay in progression or none of them. J Hepatol 66:1114–1117CrossRefPubMedGoogle Scholar
  24. 24.
    Ronot M, Bouattour M, Wassermann J et al (2014) Alternative Response Criteria (Choi, European association for the study of the liver, and modified Response Evaluation Criteria in Solid Tumors [RECIST]) Versus RECIST 1.1 in patients with advanced hepatocellular carcinoma treated with sorafenib. Oncologist 19:394–402CrossRefPubMedPubMedCentralGoogle Scholar
  25. 25.
    Gavanier M, Ayav A, Sellal C et al (2016) CT imaging findings in patients with advanced hepatocellular carcinoma treated with sorafenib: alternative response criteria (Choi), European Association for the Study of the Liver, and modified Response Evaluation Criteria in Solid Tumor (mRECIST) versus RECIST 1.1. Eur J Radiol 85:103–112CrossRefPubMedGoogle Scholar
  26. 26.
    Arizumi T, Ueshima K, Takeda H et al (2014). Comparison of systems for assessment of post-therapeutic response to sorafenib for hepatocellular carcinoma. J Gastroenterol 49:1578-1587Google Scholar

Copyright information

© European Society of Radiology 2018

Authors and Affiliations

  • Francesco Tovoli
    • 1
    Email author
  • Matteo Renzulli
    • 2
  • Giulia Negrini
    • 1
  • Stefano Brocchi
    • 2
  • Alessia Ferrarini
    • 1
  • Andrea Andreone
    • 2
  • Francesca Benevento
    • 1
  • Rita Golfieri
    • 2
  • Antonio Maria Morselli-Labate
    • 1
  • Marianna Mastroroberto
    • 1
  • Radu Ion Badea
    • 3
  • Fabio Piscaglia
    • 1
  1. 1.Department of Medical and Surgical SciencesUniversity of BolognaBolognaItaly
  2. 2.Radiology UnitS.Orsola-Malpighi Bologna Authority HospitalBolognaItaly
  3. 3.Radiology and Medical Imaging UnitRegional Institute of Gastroenterology and HepatologyCluj NapocaRomania

Personalised recommendations