Journal of Digital Imaging

, Volume 27, Issue 1, pp 108–119

Development of a Software for Quantitative Evaluation Radiotherapy Target and Organ-at-Risk Segmentation Comparison

  • Jayashree Kalpathy-Cramer
  • Musaddiq Awan
  • Steven Bedrick
  • Coen R. N. Rasch
  • David I. Rosenthal
  • Clifton D. Fuller


Modern radiotherapy requires accurate region of interest (ROI) inputs for plan optimization and delivery. Target delineation, however, remains operator-dependent and potentially serves as a major source of treatment delivery error. In order to optimize this critical, yet observer-driven process, a flexible web-based platform for individual and cooperative target delineation analysis and instruction was developed in order to meet the following unmet needs: (1) an open-source/open-access platform for automated/semiautomated quantitative interobserver and intraobserver ROI analysis and comparison, (2) a real-time interface for radiation oncology trainee online self-education in ROI definition, and (3) a source for pilot data to develop and validate quality metrics for institutional and cooperative group quality assurance efforts. The resultant software, Target Contour Testing/Instructional Computer Software (TaCTICS), developed using Ruby on Rails, has since been implemented and proven flexible, feasible, and useful in several distinct analytical and research applications.


Segmentation Imaging informatics Image segmentation Radiation oncology Algorithms Teaching 


  1. 1.
    Report 83: prescribing, recording, and reporting photon-beam intensity-modulated radiation therapy (IMRT). J ICRU 10:1–106, 2010Google Scholar
  2. 2.
    Rosenthal DI, Chambers MS, Fuller CD, et al: Beam path toxicities to non-target structures during intensity-modulated radiation therapy for head and neck cancer. Int J Radiat Oncol Biol Phys 72:747–755, 2008PubMedCrossRefGoogle Scholar
  3. 3.
    Garden AS, Lewin JS, Chambers MS: How to reduce radiation-related toxicity in patients with cancer of the head and neck. Curr Oncol Rep 8:140–145, 2006PubMedCrossRefGoogle Scholar
  4. 4.
    Rosenthal DI, Blanco AI: Head and neck squamous cell carcinoma: optimizing the therapeutic index. Expert Rev Anticancer Ther 5:501–514, 2005PubMedCrossRefGoogle Scholar
  5. 5.
    Van Kampen M, Levegrun S, Wannenmacher M: Target volume definition in radiation therapy. Br J Radiol 70(Spec No):S25–S31, 1997PubMedGoogle Scholar
  6. 6.
    Fraass BA: The development of conformal radiation therapy. Med Phys 22:1911–1921, 1995PubMedCrossRefGoogle Scholar
  7. 7.
    Lichter AS, Ten Haken RK: Three-dimensional treatment planning and conformal radiation dose delivery. Important Adv Oncol 1995:95–109, 1995Google Scholar
  8. 8.
    Bortfeld T: IMRT: a review and preview. Phys Med Biol 51:R363–R379, 2006PubMedCrossRefGoogle Scholar
  9. 9.
    Webb S: The physical basis of IMRT and inverse planning. Br J Radiol 76:678–689, 2003PubMedCrossRefGoogle Scholar
  10. 10.
    Smith RP, Heron DE, Huq MS, et al: Modern radiation treatment planning and delivery—from Rontgen to real time. Hematol Oncol Clin N Am 20:45–62, 2006CrossRefGoogle Scholar
  11. 11.
    Gregoire V, Jeraj R, Lee JA, et al: Radiotherapy for head and neck tumours in 2012 and beyond: conformal, tailored, and adaptive? Lancet Oncol 13:e292–e300, 2012PubMedCrossRefGoogle Scholar
  12. 12.
    Schwartz DL, Garden AS, Thomas J, et al: Adaptive radiotherapy for head-and-neck cancer: initial clinical outcomes from a prospective trial. Int J Radiat Oncol Biol Phys 83:986–993, 2012PubMedCrossRefGoogle Scholar
  13. 13.
    van Herk M: Errors and margins in radiotherapy. Semin Radiat Oncol 14:52–64, 2004PubMedCrossRefGoogle Scholar
  14. 14.
    Multi-Institutional Target Delineation in Oncology Group: Human–computer interaction in radiotherapy target volume delineation: a prospective, multi-institutional comparison of user input devices. J Digit Imaging 24:794–803, 2011Google Scholar
  15. 15.
    Njeh CF: Tumor delineation: the weakest link in the search for accuracy in radiotherapy. J Med Phys Assoc Med Physicists India 33:136–140, 2008Google Scholar
  16. 16.
    Fuller CD, Nijkamp J, Duppen JC, et al: Prospective randomized double-blind pilot study of site-specific consensus atlas implementation for rectal cancer target volume delineation in the cooperative group setting. Int J Radiat Oncol Biol Phys 79:481–489, 2011PubMedCentralPubMedCrossRefGoogle Scholar
  17. 17.
    Eskander RN, Scanderbeg D, Saenz CC, et al: Comparison of computed tomography and magnetic resonance imaging in cervical cancer brachytherapy target and normal tissue contouring. Int J Gynecol Cancer: Off J Int Gynecol Cancer Soc 20:47–53, 2010CrossRefGoogle Scholar
  18. 18.
    Peters LJ, O’Sullivan B, Giralt J, et al: Critical impact of radiotherapy protocol compliance and quality in the treatment of advanced head and neck cancer: results from TROG 02.02. J Clin Oncol: Off J Am Soc Clin Oncol 28:2996–3001, 2010CrossRefGoogle Scholar
  19. 19.
    Weber DC, Poortmans PM, Hurkmans CW, et al: Quality assurance for prospective EORTC radiation oncology trials: the challenges of advanced technology in a multicenter international setting. Radiother Oncol: J Eur Soc Ther Radiol Oncol 100:150–156, 2011CrossRefGoogle Scholar
  20. 20.
    Okunieff P, Kachnic LA, Constine LS, et al: Report from the Radiation Therapy Committee of the Southwest Oncology Group (SWOG): Research Objectives Workshop 2008. Clin Cancer Res: Off J Am Assoc Cancer Res 15:5663–5670, 2009CrossRefGoogle Scholar
  21. 21.
    Bekelman JE, Wolden S, Lee N: Head-and-neck target delineation among radiation oncology residents after a teaching intervention: a prospective, blinded pilot study. Int J Radiat Oncol Biol Phys 73:416–423, 2009PubMedCrossRefGoogle Scholar
  22. 22.
    Li XA, Tai A, Arthur DW, et al: Variability of target and normal structure delineation for breast cancer radiotherapy: an RTOG Multi-Institutional and Multiobserver Study. Int J Radiat Oncol Biol Phys 73:944–951, 2009PubMedCentralPubMedCrossRefGoogle Scholar
  23. 23.
    Tai P, Van Dyk J, Yu E, et al: Variability of target volume delineation in cervical esophageal cancer. Int J Radiat Oncol Biol Phys 42:277–288, 1998PubMedCrossRefGoogle Scholar
  24. 24.
    Goodman KA, Regine WF, Dawson LA, et al: Radiation Therapy Oncology Group consensus panel guidelines for the delineation of the clinical target volume in the postoperative treatment of pancreatic head cancer. Int J Radiat Oncol Biol Phys 83:901–908, 2012PubMedCrossRefGoogle Scholar
  25. 25.
    Ng M, Leong T, Chander S, et al: Australasian Gastrointestinal Trials Group (AGITG) contouring atlas and planning guidelines for intensity-modulated radiotherapy in anal cancer. Int J Radiat Oncol Biol Phys 83:1455–1462, 2012PubMedCrossRefGoogle Scholar
  26. 26.
    Nijkamp J, de Haas-Kock DF, Beukema JC, et al: Target volume delineation variation in radiotherapy for early stage rectal cancer in the Netherlands. Radiother Oncol: J Eur Soc Ther Radiol Oncol 102:14–21, 2012CrossRefGoogle Scholar
  27. 27.
    Vorwerk H, Hess CF: Guidelines for delineation of lymphatic clinical target volumes for high conformal radiotherapy: head and neck region. Radiat Oncol 6:97, 2011PubMedCentralPubMedCrossRefGoogle Scholar
  28. 28.
    Wang D, Bosch W, Roberge D, et al: RTOG sarcoma radiation oncologists reach consensus on gross tumor volume and clinical target volume on computed tomographic images for preoperative radiotherapy of primary soft tissue sarcoma of extremity in Radiation Therapy Oncology Group studies. Int J Radiat Oncol Biol Phys 81:e525–e528, 2011PubMedCentralPubMedCrossRefGoogle Scholar
  29. 29.
    Kong FM, Ritter T, Quint DJ, et al: Consideration of dose limits for organs at risk of thoracic radiotherapy: atlas for lung, proximal bronchial tree, esophagus, spinal cord, ribs, and brachial plexus. Int J Radiat Oncol Biol Phys 81:1442–1457, 2011PubMedCrossRefGoogle Scholar
  30. 30.
    Vorwerk H, Beckmann G, Bremer M, et al: The delineation of target volumes for radiotherapy of lung cancer patients. Radiother Oncol: J Eur Soc Ther Radiol Oncol 91:455–460, 2009CrossRefGoogle Scholar
  31. 31.
    Weiss E, Richter S, Krauss T, et al: Conformal radiotherapy planning of cervix carcinoma: differences in the delineation of the clinical target volume. A comparison between gynaecologic and radiation oncologists. Radiother Oncol: J Eur Soc Ther Radiol Oncol 67:87–95, 2003CrossRefGoogle Scholar
  32. 32.
    Allozi R, Li XA, White J, et al: Tools for consensus analysis of experts’ contours for radiotherapy structure definitions. Radiother Oncol: J Eur Soc Ther Radiol Oncol 97:572–578, 2010CrossRefGoogle Scholar
  33. 33.
    Brouwer CL, Steenbakkers RJ, van den Heuvel E, et al: 3D variation in delineation of head and neck organs at risk. Radiat Oncol 7:32, 2012PubMedCentralPubMedCrossRefGoogle Scholar
  34. 34.
    Rasch CR, Steenbakkers RJ, Fitton I, et al: Decreased 3D observer variation with matched CT-MRI, for target delineation in nasopharynx cancer. Radiat Oncol 5:21, 2010PubMedCentralPubMedCrossRefGoogle Scholar
  35. 35.
    Steenbakkers RJ, Duppen JC, Fitton I, et al: Observer variation in target volume delineation of lung cancer related to radiation oncologist–computer interaction: a ‘Big Brother’ evaluation. Radiother: J Eur Soc Ther Radiol Oncol 77:182–190, 2005CrossRefGoogle Scholar
  36. 36.
    Steenbakkers RJ, Duppen JC, Fitton I, et al: Reduction of observer variation using matched CT-PET for lung cancer delineation: a three-dimensional analysis. Int J Radiat Oncol Biol Phys 64:435–448, 2006PubMedCrossRefGoogle Scholar
  37. 37.
    Jansen EP, Nijkamp J, Gubanski M, et al: Interobserver variation of clinical target volume delineation in gastric cancer. Int J Radiat Oncol Biol Phys 77:1166–1170, 2010PubMedCrossRefGoogle Scholar
  38. 38.
    Roelofs E, Persoon L, Qamhiyeh S, et al: Design of and technical challenges involved in a framework for multicentric radiotherapy treatment planning studies. Radiother Oncol: J Eur Soc Ther Radiol Oncol 97:567–571, 2010CrossRefGoogle Scholar
  39. 39.
    Insight Toolkit: National Library of Medicine. Accessed 29 August 2013
  40. 40.
    Jenkinson M: NifTI-1 data format. Accessed 29 August 2013
  41. 41.
    Dice LR: Measures of the amount of ecologic association between species. Ecology 26:297–302, 1945CrossRefGoogle Scholar
  42. 42.
    Jaccard P: The distribution of the flora in the alpine zone. New Phytol 11:37–50, 1912CrossRefGoogle Scholar
  43. 43.
    Crum WR, Camara O, Hill DL: Generalized overlap measures for evaluation and validation in medical image analysis. IEEE Trans Med Imaging 25:1451–1461, 2006PubMedCrossRefGoogle Scholar
  44. 44.
    Rogers DJ, Tanimoto TT: A computer program for classifying plants. Science 132:1115–1118, 1960PubMedCrossRefGoogle Scholar
  45. 45.
    Mitra J, Kato Z, Marti R, et al: A spline-based non-linear diffeomorphism for multimodal prostate registration. Med Image Anal 16:1259–1279, 2012Google Scholar
  46. 46.
    Mitra J, Marti R, Oliver A, et al: Prostate multimodality image registration based on B-splines and quadrature local energy. Int J CARS 7:445–454, 2012CrossRefGoogle Scholar
  47. 47.
    Oguro S, Tuncali K, Elhawary H, et al: Image registration of pre-procedural MRI and intra-procedural CT images to aid CT-guided percutaneous cryoablation of renal tumors. Int J CARS 6:111–117, 2011CrossRefGoogle Scholar
  48. 48.
    Warfield SK, Zou KH, Wells WM: Simultaneous Truth and Performance Level Estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Trans Med Imaging 23:903–921, 2004PubMedCentralPubMedCrossRefGoogle Scholar
  49. 49.
    Gomez D, Komaki R, Yu J, et al: Radiation therapy definitions and reporting guidelines for thymic malignancies. J Thorac Oncol: Off Publ Int Assoc Study Lung Cancer 6:S1743–S1748, 2011CrossRefGoogle Scholar
  50. 50.
    Thomas CR, Kalpathy-Cramer J, Senan S, et al: Development of an expert consensus target delineation atlas for thymic cancers: Initial results of an expert survey. International Thymic Malignancy Interest Group, Amsterdam, 2011Google Scholar
  51. 51.
    Bongers EM, Dahele MR, Slotman BJ: Proton stereotactic body radiation therapy for clinically challenging cases of centrally and superiorly located stage I non-small-cell lung cancer: in regards to Register et al. Int J Radiat Oncol Biol Phys 82:492, author reply 492, 2012Google Scholar
  52. 52.
    Kalpathy-Cramer J, Bedrick SD, Boccia K, et al: A pilot prospective feasibility study of organ-at-risk definition using Target Contour Testing/Instructional Computer Software (TaCTICS), a training and evaluation platform for radiotherapy target delineation. AMIA Annu Symp Proc 2011:654–663, 2011Google Scholar
  53. 53.
    Feng M, Moran JM, Koelling T, et al: Development and validation of a heart atlas to study cardiac exposure to radiation following treatment for breast cancer. Int J Radiat Oncol Biol Phys 79:10–18, 2011PubMedCentralPubMedCrossRefGoogle Scholar
  54. 54.
    Malone S, Croke J, Roustan-Delatour N, et al: Postoperative radiotherapy for prostate cancer: a comparison of four consensus guidelines and dosimetric evaluation of 3D-CRT versus tomotherapy IMRT. Int J Radiat Oncol Biol Phys 84:725–732, 2012Google Scholar
  55. 55.
    Lim K, Small Jr, W, Portelance L, et al: Consensus guidelines for delineation of clinical target volume for intensity-modulated pelvic radiotherapy for the definitive treatment of cervix cancer. Int J Radiat Oncol Biol Phys 79:348–355, 2011PubMedCrossRefGoogle Scholar
  56. 56.
    Matzinger O, Poortmans P, Giraud JY, et al: Quality assurance in the 22991 EORTC ROG trial in localized prostate cancer: dummy run and individual case review. Radiother Oncol: J Eur Soc Ther Radiol Oncol 90:285–290, 2009CrossRefGoogle Scholar
  57. 57.
    Yamazaki H, Nishiyama K, Tanaka E, et al: Dummy run for a phase II multi-institute trial of chemoradiotherapy for unresectable pancreatic cancer: inter-observer variance in contour delineation. Anticancer Res 27:2965–2971, 2007PubMedGoogle Scholar
  58. 58.
    Clark CH, Miles EA, Urbano MT, et al: Pre-trial quality assurance processes for an intensity-modulated radiation therapy (IMRT) trial: PARSPORT, a UK multicentre phase III trial comparing conventional radiotherapy and parotid-sparing IMRT for locally advanced head and neck cancer. Br J Radiol 82:585–594, 2009PubMedCrossRefGoogle Scholar
  59. 59.
    Pawlicki T, Yoo S, Court LE, et al: Process control analysis of IMRT QA: implications for clinical trials. Phys Med Biol 53:5193–5205, 2008PubMedCrossRefGoogle Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2013

Authors and Affiliations

  • Jayashree Kalpathy-Cramer
    • 1
  • Musaddiq Awan
    • 3
  • Steven Bedrick
    • 2
  • Coen R. N. Rasch
    • 4
  • David I. Rosenthal
    • 3
  • Clifton D. Fuller
    • 3
  1. 1.Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Harvard Medical School, Department of Radiology and NeuroscienceMassachusetts General HospitalCharlestownUSA
  2. 2.Department of Medical Informatics & Clinical EpidemiologyOregon Health & Science UniversityPortlandUSA
  3. 3.Head and Neck Section, Division of Radiation Oncology, Department of Radiation OncologyThe University of Texas MD Anderson Cancer CenterHoustonUSA
  4. 4.Department of RadiotherapyAcademic Medical CenterAmsterdamNetherlands

Personalised recommendations