Skip to main content
Log in

Inter-operator variability and source of errors in tumour response assessment for hepatocellular carcinoma treated with sorafenib

  • Gastrointestinal
  • Published:
European Radiology Aims and scope Submit manuscript



To assess the inter-operator concordance and the potential sources of discordance in defining response to sorafenib in hepatocellular carcinoma (HCC).


All patients who received sorafenib between September 2008 and February 2015 were scrutinised for this retrospective study. Images were evaluated separately by three radiologists with different expertise in liver imaging (operator 1, >10 years; operator 2, 5 years; operator 3, no specific training in liver imaging), according to: response evaluation radiological criteria in solid tumours (RECIST) 1.1, modified RECIST (mRECIST) and response evaluation criteria in cancer of the liver (RECICL).


The overall response concordance between the more expert operators was good, irrespective of the criteria (RECIST 1.1, ĸ = 0.840; mRECIST, ĸ = 0.871; RECICL, ĸ = 0.819). Concordance between the less expert operator and the other colleagues was lower. The most evident discordance was in target lesion response assessment, with expert operators disagreeing mostly on lesion selection and less expert operators on lesion measurement. As a clinical correlate, overall survival was more tightly related with “progressive disease” as assessed by the expert compared to the same assessment performed by operator 3.


Decision on whether a patient is a responder or progressor under sorafenib may vary among different operators, especially in case of a non-specifically trained radiologist. Regardless of the adopted criteria, patients should be evaluated by experienced radiologists to minimise variability in this critical instance.

Key Points

• Inter-operator variability in the assessment of response to sorafenib is poorly known.

• The concordance between operators with expertise in liver imaging was good.

• Target lesions selection was the main source of discordance between expert operators.

• Concordance with non-specifically trained operator was lower, independently from the response criteria.

• The non-specifically trained operator was mainly discordant in measurements of target lesions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others



Hepatocellular carcinoma


Progressive disease


Non-target lesion


Response evaluation criteria in cancer of the liver


Response evaluation radiological criteria in solid tumours


  1. Nagilla M, Brown RL, Cohen EE (2012) Cabozantinib for the treatment of advanced medullary thyroid cancer. Adv Ther 29:925–934

    Article  PubMed  CAS  Google Scholar 

  2. Mocellin S, Baretta Z, Roqué I et al (2017) Second-line systemic therapy for metastatic colorectal cancer. Cochrane Database Syst Rev 1:CD006875

    PubMed  Google Scholar 

  3. Randrup Hansen C, Grimm D, Bauer J, Wehland M, Magnusson NE (2017) Effects and side effects of using sorafenib and sunitinib in the treatment of metastatic renal cell carcinoma. Int J Mol Sci 18.

  4. Wang Y, Schmid-Bindert G, Zhou C (2012) Erlotinib in the treatment of advanced non-small cell lung cancer: an update for clinicians. Ther Adv Med Oncol 4:19–29

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  5. Therasse P, Arbuck SG, Eisenhauer EA et al (2000) New guidelines to evaluate the response to treatment in solid tumors. European Organization for Research and Treatment of Cancer, National Cancer Institute of the United States, National Cancer Institute of Canada. J Natl Cancer Inst 92:205–216

    Article  PubMed  CAS  Google Scholar 

  6. Lencioni R, Llovet JM (2010) Modified RECIST (mRECIST) assessment for hepatocellular carcinoma. Semin Liver Dis 30:52–60

    Article  PubMed  CAS  Google Scholar 

  7. Kudo M, Ueshima K, Kubo S et al (2015) Response Evaluation Criteria in Cancer of the Liver (RECICL) (2015 Revised version). Hepatol Res 46:3–9

    Article  PubMed  Google Scholar 

  8. Choi H, Charnsangavej C, Faria SC et al (2007) Correlation of computed tomography and positron emission tomography in patients with metastatic gastrointestinal stromal tumor treated at a single institution with imatinib mesylate: proposal of new computed tomography response criteria. J Clin Oncol 25:1753–1759

    Article  PubMed  Google Scholar 

  9. Llovet JM, Ricci S, Mazzaferro V et al (2008) Sorafenib in advanced hepatocellular carcinoma. N Engl J Med 359:378–390

    Article  PubMed  CAS  Google Scholar 

  10. Bruix J, Sherman M, Practice Guidelines Committee, American Association for the Study of Liver Diseases (2005) Management of hepatocellular carcinoma. Hepatology 42:1208–1236

    Article  PubMed  Google Scholar 

  11. Renzulli M, Brocchi S, Cucchetti A et al (2016) Can current preoperative imaging be used to detect microvascular invasion of hepatocellular carcinoma? Radiology. 279:432–442

    Article  PubMed  Google Scholar 

  12. Sneag DB, Krajewski K, Giardino A et al (2011) Extrahepatic spread of hepatocellular carcinoma: spectrum of imaging findings. AJR Am J Roentgenol 197:W658–W664

    Article  PubMed  Google Scholar 

  13. Katyal S, Oliver JH 3rd, Peterson MS et al (2000) Extrahepatic metastases of hepatocellular carcinoma. Radiology 216:698-703

  14. Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychological Bulletin 76:352–378

    Article  Google Scholar 

  15. Portney LG, Watkins MP (2009) Foundations of clinical research: applications to practice, 3rd edn. Pearson/Prentice Hall, Upper Saddle River

    Google Scholar 

  16. Bruix J, Qin S, Merle P et al (2017) Regorafenib for patients with hepatocellular carcinoma who progressed on sorafenib treatment (RESORCE): a randomised, double-blind, placebo-controlled, phase 3 trial. Lancet 389:56–66

    Article  PubMed  CAS  Google Scholar 

  17. Bargellini I, Scionti A, Mismas V et al (2014) Identification of responders to sorafenib in hepatocellular carcinoma: is tumor volume measurement the way forward? Oncology. 86:191–198

    Article  PubMed  CAS  Google Scholar 

  18. Sato Y, Watanabe H, Sone M et al (2013) Tumor response evaluation criteria for HCC (hepatocellular carcinoma) treated using TACE (transcatheter arterial chemoembolization): RECIST (response evaluation criteria in solid tumors) version 1.1 and mRECIST (modified RECIST): JIVROSG-0602. Ups J Med Sci 118:16–22

    Article  PubMed  PubMed Central  Google Scholar 

  19. Seyal AR, Gonzalez-Guindalini FD, Arslanoglu A et al (2015) Reproducibility of mRECIST in assessing response to transarterial radioembolization therapy in hepatocellular carcinoma. Hepatology 62:1111–1121

    Article  PubMed  CAS  Google Scholar 

  20. Lee S, Kim JH, Lee JH et al (2018) Non-invasive monitoring of the therapeutic response in sorafenib-treated hepatocellular carcinoma based on photoacoustic imaging. Eur Radiol 28:372–381

    Article  PubMed  Google Scholar 

  21. Tovoli F, Renzulli M, Granito A et al (2017) Radiologic criteria of response to systemic treatments for hepatocellular carcinoma. Hepatic Oncology 4:129–137

    Article  PubMed  PubMed Central  Google Scholar 

  22. Lencioni R, Montal R, Torres F et al (2017) Objective response by mRECIST as a predictor and potential surrogate end-point of overall survival in advanced HCC. J Hepatol 66:1166–1172

    Article  PubMed  Google Scholar 

  23. Bruix J, Reig M, Sangro B (2017) Assessment of treatment efficacy in hepatocellular carcinoma: Response rate, delay in progression or none of them. J Hepatol 66:1114–1117

    Article  PubMed  Google Scholar 

  24. Ronot M, Bouattour M, Wassermann J et al (2014) Alternative Response Criteria (Choi, European association for the study of the liver, and modified Response Evaluation Criteria in Solid Tumors [RECIST]) Versus RECIST 1.1 in patients with advanced hepatocellular carcinoma treated with sorafenib. Oncologist 19:394–402

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  25. Gavanier M, Ayav A, Sellal C et al (2016) CT imaging findings in patients with advanced hepatocellular carcinoma treated with sorafenib: alternative response criteria (Choi), European Association for the Study of the Liver, and modified Response Evaluation Criteria in Solid Tumor (mRECIST) versus RECIST 1.1. Eur J Radiol 85:103–112

    Article  PubMed  CAS  Google Scholar 

  26. Arizumi T, Ueshima K, Takeda H et al (2014). Comparison of systems for assessment of post-therapeutic response to sorafenib for hepatocellular carcinoma. J Gastroenterol 49:1578-1587

Download references


The authors state that this work has not received any funding.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Francesco Tovoli.

Ethics declarations


The scientific guarantor of this publication is Fabio Piscaglia.

Conflict of interest

The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.

Statistics and biometry

One of the authors has significant statistical expertise.

Informed consent

Written informed consent was obtained from all subjects (patients) in this study.

Ethical approval

Institutional Review Board approval was obtained.


• retrospective

• observational/experimental

• performed at one institution

Electronic supplementary material


(DOC 27 kb)


(DOCX 16 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tovoli, F., Renzulli, M., Negrini, G. et al. Inter-operator variability and source of errors in tumour response assessment for hepatocellular carcinoma treated with sorafenib. Eur Radiol 28, 3611–3620 (2018).

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: