This work describes a systematic technographic review of the functionalities and potential impact of AI applications and the characteristics of the vendors. Some vendors have more than one application, each for a specific task, while others have one application that can perform several tasks. Our approach addressed this by using a relational database  in which many functionalities or workflow items can be assigned to one application. This flexibility allowed us to conduct analysis not only at the application level but also at the functionality level.
Companies and applications
The relatively high number of applications and the fact that most companies are young confirm the recent attention in the literature to AI in neuroradiology. Because of the limited information about funding, we cannot draw definite conclusions, but it is interesting to see that the companies leading the funding list are in China.
For the majority of the applications, no information was available regarding the platform on which the applications run; thus, radiology departments will benefit from more detailed information from the vendors before they determine the application that best fits their needs.
For the majority of applications (68%), the type of AI (machine learning or deep learning) was provided. Some companies provide this information in detail, while others do not or provide it in a superficial manner. For radiologists, it is crucial to know the strength and weaknesses of the technology that they use to improve quality, ensure safety and understand artefacts [19, 20]. Additionally, radiologists need to understand technical information about the applications  to recognize the strengths and pitfalls of AI applications . Information about the training data of algorithms and whether external validation was performed helps radiologists assess the credibility and applicability of an AI application in their hospital . This type of data was limited. Close collaboration between radiologists and vendors is needed to ensure the true clinical utility of algorithms .
In addition to the algorithmic details of an AI application, the way the application can be integrated into the work environment has an impact on the job of the radiologist. Usability is essential to ensure that radiologists use the application in their daily work [25, 26]. The seamless PACS integration of many of the investigated AI applications facilitates radiologists’ efforts in using these applications. Even though our data do not show more detailed information about PACS integration, this finding indicates the awareness of vendors that integration in the daily workflow is essential for the adaption of the applications by radiologists.
The fairly high percentage of approved applications demonstrates that AI in neuroradiology is not only in the state of developing and testing but also available for the radiologist in daily practice. Approval can be a starting point for evaluating the benefits of AI application for the health outcomes of patients, which requires higher levels of evidence than what is often needed for regulatory approval .
Modalities and pathology
Neuroradiology heavily relies on MRI and CT, so it is not surprising that most applications are made to be used with MRI or CT data. The types of pathology that can be handled by the applications in our database reflect the frequently encountered diseases in neuroimaging. However, other major categories of disease for which neurologists and other specialists request imaging are missing, for example radiculopathy and epilepsy. In defining clinical challenges such as these, radiologists can contribute to translational research in artificial intelligence .
Functionalities and workflow
The most numerous functionalities are directly related to the core business of a radiologist: finding and interpreting abnormalities and making the correct diagnosis. This fact indicates that AI companies develop products that are genuinely relevant to radiologists.
The items designated to the category ‘quality assurance’ are mainly designed to improve the workflow. No applications were found that perform a more direct quality assurance task, such as assessment of the completeness or quality of reports.
There is a shortage of applications and functionalities related to the early stages of the workflow (e.g. scheduling, acquisition and pre-processing) and the final stages (e.g. reporting and communication). This indicates the opportunities for companies and radiologists to develop applications in areas beyond image interpretation [29, 30].
Impact on the job of a radiologist
Scientific journals dedicate papers and editorials to the emerging development of AI, wondering “Will Artificial Intelligence Replace Radiologists?” . In general, AI will impact parts of many jobs, but other tasks within these same jobs will not change . This is confirmed by our results. Currently, AI applications do not offer functionalities that can replace radiologists. The few applications that have the potential to replace the radiologist only can do that for a limited set of tasks, such as pre-drafting reports and analysing a stroke patient. In fact, the applications available on the market are still narrow-AI applications, meaning that they focus on one small task. This term can be applied to the AI tools that support or replace the radiologist for a single task, while the radiologist is needed to accomplish a sequence of other tasks. These applications do not check for other related or unrelated findings; therefore, the radiologist still has a task.
This fact does not mean that AI has no impact on the radiologist. Many applications are available, which can support radiologists, especially for the ‘detection’ and ‘interpretation’ of the clinical insights, the two primary responsibilities of a radiologist. Many applications also extend the work of radiologists. Quantitative information and biomarkers will enhance the content of the radiology reports of radiologists who choose to use these applications .
Companies are struggling with both scientific and regulatory validations of their products, though we see that the attempts to have sound scientific validation of the AI products are more limited than the regulatory approval. For only a minority of applications, peer-reviewed publications are available. This indicates that regulatory approval is not the same as clinical validation and confirms the remarks that most current applications are not yet ready to accept clinical deployment [34, 35]. The impact on patient outcome has yet to be assessed for almost all applications.
Reviewing new developments and providing an overview of the available applications is a well-established research approach. For example, Landau et al. provided an overview of AI in cytopathology and described both the literature and commercial landscape in a comprehensive review , Chen et al. described the current status of AI in urology  and Murray et al. performed a systematic literature review on AI applications in neuroradiology . We found no other systematic technographic reviews similar to our study.
The applications that are designed for diseases such as stroke or dementia are specific to the neuroradiology subspecialty. These tools are not directly applicable to other subspecialties. However, the underlying concepts of the applications we investigated are generalizable to other types of pathology in neuroradiology or other subspecialties within radiology. These general concepts are as follows:
the prioritization of studies in the PACS worklist, based on the presence of pathology
the optimization of the workflow
the quantification of anatomical structures and comparison with an age-based control group and the derivation of biomarkers
the automated detection and segmentation of pathology
the automated classification for pathology based on guidelines and specific criteria
This list indicates that radiology will not be the same in the near future. Substantial investments in AI will boost research and development in this domain .
There is wide variation in the quality and completeness of the information on the websites of the vendors. Our results represent all available material that we thoroughly assessed. The characteristics of AI applications that are not publicly available were beyond the scope of our study, including applications that are commonly used in research institutions.
We included only applications that mention “neuroradiology” and “artificial intelligence” (or related words). However, some applications offer advanced AI tools for radiology that are also applicable in neuroradiology. Applications that use automated processing but that do not explicitly use AI were not included. Our results, therefore, might underestimate the applications that have an impact on the job of radiologists working in neuroradiology.
We included applications from exhibitors of several large radiology congresses in Europe and North America. Although several companies from Asia were present in our database, our results might be biased towards Europe and North America. Especially because of the high amount of funding acquired by some Asian companies, a significant contribution to future AI developments from this continent can be expected.
Another limitation is that we did not have interrogated the content of the scientific validation material. We only scored the presence or absence of this.
As mentioned, AI is developing at a high pace. Repeating our study over time helps us keep track of these developments and develop a more accurate overview of their potential impacts on radiology work and the radiology profession. This change over time will also provide valuable information about the development of this market. After-implementation feedback is also very important to determine how an application is actually used in terms of support, extension and replacement.