Creating a Medical Imaging Workflow Based on FHIR, DICOMweb, and SVG

This paper proposes a web-based workflow scheme for the organization of medical images using FHIR and DICOM servers equipped with standard RESTful APIs. In our integrated workflow, the client systems (including order placer, scheduler, imaging modality, viewer, and report creator) use standard FHIR and DICOMweb APIs. The proposed scheme also facilitates the creation of reports formatted as standard FHIR resources. This paper leverages W3C Scalable Vector Graphics (SVG) to record the image graphic annotations, and encapsulates the SVG image annotation in FHIR observation. FHIR DiagnosticReports and Observations are used to encapsulate reports, findings, and annotations, thereby facilitating the implementation and integration of the scheme within existing structures. The proposed scheme also provides the potential to make it possible to convert results of Computer Aided Detection/Diagnosis from medical images into FHIR DiagnosticReports and Observations to be stored on a FHIR server. The resulting web-based solution uses FHIR XML and/or JSON data to record and exchange information related to imaging workflow. It can also be used to store imaging reports, findings, and annotations linked to the images using the DICOM WADO-RS protocol. As a result, it is possible to integrate all information that is created in medical imaging workflow. Finally, the proposed scheme is easily integrated with other FHIR systems.


Purpose
In Integrating the Healthcare Enterprise (IHE) Scheduled Workflow (SWF) [1], medical imaging workflows must follow HL7 V2 (Health Level 7 Version 2.x) and DICOM (Digital Imaging and Communications in Medicine) protocols for the sharing of data. The SWF profile addresses actors (information systems) that support HL7 V2 and DICOM protocols for workflow and medical image management. The XDS or XDS-I profile (IHE Cross-Enterprise Document Sharing) is also used for the sharing of medical images outside the hospital [1,2]. The previous versions of HL7 and DICOM standards are very large and complex. It's not easy to implement the imaging workflow handling and imaging data sharing. Thankfully, DICOM and HL7 have issued web RESTful (REpresentational State Transfer) specifications and proposals, which are respectively referred to as DICOMweb (DICOM Web Services) and FHIR (Fast Healthcare Interoperability Resources) [3,4]. These efforts have made it friendlier to develop an integrated imaging workflow.
In this study, we developed a demonstration system in which FHIR and DICOMweb specifications are used to facilitate the integration of conventional HIS (Hospital Information System) and PACS (Picture Archiving and Communication System). Our workflow scheme proposes that the HIS implements server-side support for several FHIR resources and the PACS implements server-side support for several DICOMweb services. The storage and handling of imaging workflow and report data is performed on the FHIR server, whereas medical images are stored on the DICOMweb server for distribution. Thus, it is possible to use standard FHIR XML (eXtensible Markup Language) or JSON-formatted (JavaScript Object Notation) data to record information created during imaging workflows. Additionally, we employed FHIR and DICOMweb RESTful APIs (Application Programming Interface) to facilitate integration with existing systems. The proposed web-based solution is easily implemented, and the DICOMweb-based image sharing is easily integrated with other workflows pertaining to clinical diagnosis and treatment.

Conventional Imaging Workflows
Conventional medical imaging workflows require the integration of information systems. As shown in Fig. 1, standardized data is created in the HIS, RIS (Radiological Information System), and PACS, and then shared across systems based on IHE Integration Profiles. Figure 1 lists the six fundamental systems that must be integrated within a typical imaging workflow. The six systems deal with ordering, scheduling, imaging, storing, inspecting, and reporting. In accordance with the IHE Radiology SWF and Reporting Workflow (RWF) profiles [5], system integration requires support for the DICOM protocol (modality work list, image storage, image query/retrieve) as well as the HL7 V2 protocol (ordering and image availability notification). The system handles imaging reports usually via PDF file, the HL7 Clinical Document Architecture (CDA), or DICOM Structure Reporting (SR) for the storage of reports in the report repository or PACS. It is also important to provide support for IHE XDS or XDS-I profiles using ebXML (Electronic Business using eXtensible Markup Language) web services for the sharing of images and reports among hospitals.

FHIR and DICOMweb Imaging Workflows
In this study, we developed a novel web integration framework based on recent medical informatics standards, HL7 FHIR and DICOMweb. As shown in Fig. 2, the proposed framework allows straightforward implementation, management, and integration with existing systems used in the healthcare environment.
The proposed framework includes two web servers (FHIR and DICOMweb servers) and four clients (Order Placer, Order Filler, Modality, and Viewer). All of the clients and servers use the HL7 FHIR and DICOMweb protocols to facilitate integration, and both servers utilize HTTP RESTful APIs for the sharing of data, which is formatted as XML or JSON and stored on the FHIR server. Data sharing by the DICOMweb server can be in XML, JSON format, or binary data format. Our adoption of RESTful web solutions makes it easy for most information technology (IT) developers to understand the system. In accordance with HL7 FHIR, DICOMweb, and IHE, the data creation and exchange processes are as follows: Step 1. The Order Placer (HIS) initiates an order for imaging by creating a corresponding FHIR ServiceRequest on the FHIR Server (HIS).
Step 2. The order is handled by the Order Filler (RIS) as follows: Step 2.1 The Order Filler periodically queries the FHIR Server for new ServiceRequests Step 2.2 RIS schedules an imaging appointment and creates a new EncounterResource on the FHIR server.
Step 2.3 The FHIR server maps the information in the ServiceRequest and Encounter Resources into a client request to the DICOMweb Server to create the UPS-RS worklist item. [6] Step 3. Modality imaging procedure: Step 3.1 Modality obtains a UPS-RS work list from the DICOMweb server.
Step 3.2 Following completion of the imaging session, the images are stored in the DICOMweb Server via STOW (STore Over the Web). [7] Step 3.3 Mapping from a DICOM Study to the FHIR ImagingStudy Resource.
Step 3.4 Finally, modality uses QIDO-RS to ensure that the images are stored on the server. Note that this is similar to the DICOM storage commitment used in PACS [8].
Step 4. Image retrieval and reporting As shown in Fig. 2, radiologists can use the image viewing and reporting system to obtain images and create reports. This process is described in the following: Step 4.1 The radiology department queries the FHIR encounter resources to prepare the prioritized worklist for radiologists.
Step 4.2 Radiologist examines the images, and may get additional FHIR ImagingStudies (current and previous image exam results) from FHIR server, which might be pre-fetched.
Step 4.3 The viewing system retrieves the selected images via WADO-RS basing on QIDO-RS. This allows the radiologist to browse all of the medical images available for a given patient. [9] Step 4.4 While the radiologist is inspecting the images, they may make annotations on medical images, describe the findings based on the annotations, and create a report of the image exam. According to FHIR standard, each image finding can be represented as a FHIR observation resource. And FHIR DiagnosticReport may reference to all the findings of the image exam. This paper suggests that image annotations also be represented as FHIR observation resource that can be referenced by FHIR finding observations. Consequently, image annotations, image findings, and report are correlated to each other. Report is referenced to image findings, and image finding may be referenced to image annotations. The image annotation (FHIR observation), image finding (FHIR observation), and report (FHIR DiagnosticReport) all could be stored in FHIR Server. 1

FHIR Solution for Interoperability
This paper outlines a novel FHIR solution aimed at facilitating the recording and archiving of data related to medical imaging. As shown in Fig. 3, the reports, findings, annotations, and DICOM images could be referenced together based on standards specified in the FHIR resources.
FHIR DiagnosticReport.ImagingStudy refers to ImageStudies associated with the report, DiagnosticReport.Result refers to findings (observations), and Observation.Derived-From (image findings) is derived from other observations containing annotation data pertaining to a specific medical image. The annotation Observation.Focus should reference to a WADO-RS URL (Uniform Resource Locator) that represents the focus image. We will show the SVG benefits in the following examples that package SVG graphics into FHIR observation for medical image annotation. This solution is easier for developers to understand and implement than DICOM GSPS for image annotation.

SVG Annotation
As described in Step 4.4 above, the radiologist can use the viewer to inspect medical images related to a particular patient and a report creator to make annotations pertaining to the images. The annotations are represented in Scalable Vector Graphics (SVG) format in accordance with W3C (World Wide Web Consortium) standards. Note that SVG annotations can be presented directly in a browser, and are easily dealt with by developers. Adding SVG data to observations in FHIR XML or JSON format requires that the SVG data be base64 encoded for storage in FHIR Observation, as shown in Fig. 4. Figure 4 presents an SVG rectangle on an MRI (Magnetic Resonance Imaging) image. Note that such images are stored on a DICOMweb server and accessed in accordance with the DICOM WADO-RS protocol. A web-based viewer can use JavaScript to access DICOM images that are stored in DICOMWeb server. However, the annotation would not be stored into DICOMWeb server. According to DICOM standard the annotation should be DICOM Structure Reporting (SR) or Presentation State (PS) formatted for storing into DICOMWeb server. Unfortunately, DICOM objects are too complex to be directly handled by the general Web browser. And most system developers are unfamiliar with DICOM standards. Thus, we suggest storing image annotations in W3C SVG format, which feature clearly defined geometric graphics, such as lines, ellipses, rectangles, and polygons, and are familiar to most IT developers. This would also make it possible to create a simple SVG webpage with links to WADO-RS images

FHIR Observation Using SVG Annotation
We suggest that SVG annotations are contained within FHIR Observation for storage on an FHIR server. The annotations usually can be stored as three methods: (1) SVG format in a FHIR Observation, e.g., retrieve annotations from a FHIR DiagnosticReport resource and displaying them directly. (2) SVG or DICOM-compatible formats as a reference stored in ImagingStudy.Endpoint in an ImagingStudy resource.
(3) DICOM-compatible formats in the DICOMWeb server by referring to the URL of WADO-RS/WADO-URI. Under the (2) and the (3) methods, the annotations are retrieved via WADO, a DICOM to SVG broker to convert DICOMbased annotation into SVG coded as FHIR SVG observation, which need an additional server for storing the SVG annotation. As shown in Fig. 3, the annotation (FHIR Observation) can be referred by the image finding (another FHIR Observation), and the FHIR Observation.valueString could be used to store annotation information. Note that FHIR Observations should be in XML or JSON format, and W3C SVG graphics should be in XML format. Note also that containing SVG data directly to XML or JSON FHIR Observation.val-ueString would cause an error when uploading annotations to the FHIR server. Thus, we suggest that SVG data to be encoded in base64 to allow the storing of SVG data into FHIR Observations. The results could then be uploaded to the FHIR server, as illustrated in Example script 2 below.
Example script 2 is a simple FHIR image annotation encoded in base64 and stored in FHIR Observation.val-ueString. The valueBase64binary formatted data could be treated as a special type of string. Decoding the base64 string would restore the SVG annotation data to the form </body></html> Example script 1: SVG web page contains a jpg image and rectangular graphic. shown in Fig. 4. This FHIR Observation has been uploaded to an FHIR testing server, and can be accessed at the following URL: https:// hapi. fhir. tw/ fhir/ Obser vation/ 4961 In Example script 2, Observation.Focus refers to a URL pointing to a DICOMweb server. The URL was created in accordance with the specifications for FHIR reference data, allowing it to be stored on an FHIR server. As shown in Fig. 3, the image annotation observation should be referred by finding observation and could be referred by FHIR DiagnosticReport.
When using a viewer to create FHIR annotations, the observations should contain only SVG data (i.e., without references to WADO-RS image), as shown in Fig. 4. The URL of the target image is stored in Observation.Focus. When deploying the image and annotation results outside the original FHIR and DICOMweb environment, we suggest using a standard front-end web solution compatible with standard browsers. Example script 1 presents a very simple solution showing one image and the corresponding annotation. This scheme can be deployed using HTML (Hyper Text Markup Language) and JavaScript to handle DICOM data objects. This would make it possible to create a portable viewer capable of showing DICOM images and annotations without having to deal with DICOMweb or FHIR servers.

Viewer with Standard HTTP APIs
As outlined in Step 4 above, we developed a viewer capable of querying and retrieving medical images as well as storing annotations and reports (see Fig. 5).
The above viewer provides functionalities of panning, zooming, window leveling, and annotating; however, FHIR APIs can also be used to obtain additional information pertaining to the patient (Step 4.1 above). After identify a patient for inspection, we could query all patient image studies use FHIR get image API (Step 4.2). Basing the query results, we can use WADO-RS to retrieve current and previous exam images (Step 4.3). Finally, the viewer can store annotations in the form of FHIR Observations on the FHIR server (Step 4.4). Conventional PACS viewers use DICOM query/retrieve protocols to access images and the DICOM store protocol to store annotations (DICOM PS objects) or reports (DICOM SR object), both of which are unfamiliar to most developers. The viewer presented in this paper uses FHIR specifications for querying and storing image reports on an FHIR server. Most FHIR developers are familiar with RESTful formatted APIs and XML or JSON formatted resources. Note also that the FHIR-based viewing and reporting systems could be easily integrated with other FHIR systems, such as medication, laboratory, and care planning systems.

Converting DICOM SR into FHIR Resources
Computer-Aided Detection (CAD) systems can also be used in the acquisition and manipulation of diagnostic images. CAD systems may follow DICOM standard to convert the detection results into DICOM SR objects for storage on a PACS server. DICOM SR objects formulated under DICOM Part 16 specifications are very complex. Furthermore, CAD results should include data pertaining to the devices, procedures, measurements, subjects, and reference image information within a single DICOM SR object. This inevitably leads to the duplication of data stored in DICOM SR formatted CAD result. In this paper, we proposed a simplifying method that uses the FHIR resources, which would provide the potential to store CAD results on an FHIR server. As shown in Fig. 3,  Presenting CAD results in the form of FHIR Observations would be far simpler than using DICOM SR, as illustrated by the conversion of CDA DICOM SR data into FHIR Observations in Fig. 6. The illustration presents a HTML web page contains two FHIR annotations corresponding to two mammographic images. Note that the FHIR image annotation is easily integrated with FHIR DiagnosticReports as well as other (e.g., laboratory, pathology, and genomic) reports. In other words, the proposed scheme makes it possible to use a browser or client application to view any clinical results stored as standard FHIR DiagnosticReports and Observations. The current implementation simplifies by just rendering the contour. Future work would address migrating more of the omitted DICOM SR CAD complexity into FHIR Observations, e.g. CAD operating point, finding certainty, verified yes/no, algorithm name, presentation required/ optional, size information, type of finding, summary of detections, etc.
As outlined in Fig. 3, CAD DICOM SR objects can also be converted into FHIR annotation Observations, finding Observations, and DiagnosticReports. Following the conversion, the CAD results are stored on an FHIR server. According the workflow in Fig. 2, the web solution described in this paper makes it possible for the viewing the CAD results and mammographic images on browser simultaneously. And as that shown in Fig. 6 and Example script 3, we can use pure web page to present CAD results and mammography on browser. The procedures involved in querying previous imaging results and presenting them in a pure web page are described in the following.

The viewer queries previous imaging Diagnos-
ticReports for the patient of interest. 2. The viewer obtains the corresponding FHIR annotation Observations by the references in finding Observation and DiagnosticReport. 3. The viewer creates a WADO-RS URL using the ImagingStudy.Endpoint and annotation Observation.Focus, as shown in Example script 2. 4. The viewer retrieves the WADO-RS images for display. 5. The viewer draws annotations over the images. 6. The viewer exports a pure web page containing the target image and corresponding annotation.
Note that the above Procedure 6 (exporting a web page with images and SVG annotations) could be altered to allow the export of the original FHIR DiagnosticReports, Observations, and medical images in the form of web page with JSON data and JavaScript accessible using any browser. In other words, the results could be deployed to any website that does not support FHIR or WADO-RS protocols. The simple web page may be deployed across hospitals that might not have FHIR and DICOMWeb server for viewing medical image and annotation.

Conclusions
This paper introduces a novel workflow for the organization of medical images, based on FHIR APIs and associated resources in conjunction with a viewer using DICOMweb APIs to facilitate integration with existing systems. Unlike conventional imaging workflows, which require the complex HL7 V2, DICOM, and HL7 CDA specifications, the proposed scheme uses the common JSON (or XML) data and RESTful APIs of FHIR and DICOMweb. This approach facilitates implementation and management, and can be integrated with web-based HIS, the Laboratory Information System (LIS), the Pathology Information System (PIS), and other existing systems.
The proposed schemes meant to overcome barriers to system integration. With appropriate security protection, the proposed web solution could be used for the sharing of information throughout all departments inside hospital as well as across hospitals. And the secure specifications must further be addressed for cross hospital image workflows. The more detailed specifications and extensions that might be investigated further are listed as follows: • There are three coordinate systems on the viewer, the DICOM pixel, 3D spatial coordinates and canvas coordinates, all of which could be used in image annotation. Note however that using the viewer to rotate, flip, resize, or shift the DICOM image would alter the relationship between the coordinate systems. This would be necessary to perform coordinate transformation when storing annotations on the FHIR server or displaying the annotations on the viewer. For a single image, we suggest that the DICOM pixel coordinate could be used as the reference for image annotation. However, for a series images or different series images, patient coordinate would be useful in image registry and lesion management. We are investigating the three coordinates used in various clinical domains, such as radiotherapy, surgery, etc., and would demonstrate in the future. • It would be also worthy to note that the WADO-RS rendering might be a potential solution, which could apply to all kind of images (mammo, CT, MR, etc.) and allow the DICOMweb server to handle all the issues with 12-bit resolutions, technical factor annotations, sigmoidal lookup tables, and monochrome image interpretation, etc.
[10]. Viewers usually require the ability to parse the DICOM objects. If the format of the image (e.g., row, column, bit allocation defined in the DICOM image pixel module) were known prior to retrieval, then it would be possible for the viewer to display of medical images. This would make it far easier to develop a front end system to handle imaging data by WADO-RS with rendered resources, which would directly return JPEG images to the Viewer. • This study did not focus on the terminology or coding systems used in imaging workflows; however, we acknowledge the importance of these issues in putting together reports. For example, images for the diagnosis of breast cancer would include data pertaining to the shape, location, size, and margin of detected masses. The shape and margin must also be further defined and coded in accordance with each type of imaging modality (e.g., MRI, ultrasound, and mammography). The codes and terminology should be further investigated and carefully defined. • The proposed FHIR imaging workflow could be used in a single facility or among multiple facilities; however, this will require further mechanisms for security and the protection of privacy. FHIR currently leverages many of the security protection mechanisms developed for the web, such as SSL and OAuth for protecting FHIR APIs [11]. However, access control to the FHIR and DICOMweb servers should also be regulated. Additionally mapping the all possible details (referring physician, accession number, visit number, admitting diagnosis, etc.) between ServiceRequest/Encounter and UPS-RS is as well a significant core items in our future work. •  Following the specification, we may use WADO-RS with parameters to access radiology images and WSI images. And we could also access SVG annotations from FHIR servers. The images and annotation could be rendered and presented properly on a pure web viewer. We hope to present more use cases based on standard RESTful specifications in the future.
• RESTful solution with IoT, AI, and precision medicine will bring a lot of new usages and opportunities. Medical information standards should be very important to support the potential solutions for innovative applications. However, this a huge task to define standards for the system integrating solutions in healthcare. This paper proposes an imaging integrating framework using simplified DICOMWeb and FHIR resources. The simplified specification would be easy to be understood for people who are not familiar with medical informatics standards. And developers can follow the framework described in this paper to develop the general imaging exam and report systems which may quickly be adopted into real use. There still is a lot of work for creating standard specifications for integrating AI or precision medicine system with PACS. Especially, the security specifications for the integration. It will bring lot of overhead for the security specifications to be compatible both with conventional HL7 and DICOM and with FHIR and DICOMWeb. This would constitute a very difficult to implement specifications. For the new medical imaging application, such as AI, the integration of FHIR and DICOMWeb would be the promising solution.