Keywords

1 Introduction

1.1 Mobile Applications in Improving Radiology Education

About 90 % of medical professionals have used mobile devices in medical practice to access their patients’ electronic health records and medical information. Currently, there are four major application stores in the market: iTunes, Google Play, Windows, and Blackberry. iTunes and Google Play stores contain the majority of mobile applications. iTunes’ application store consist of approximately 20,000 medical mobile applications while Google Play’s application store has about 9,000 medical applications [1].

Radiology is a specialty that requires extensive training in image analysis for decision making. Radiology residents are physicians who are being trained in the specialty, and are increasingly using mobile devices. It is estimated that in professional settings, 78 % of physicians use smartphones for work-related purposes [2]. Smartphone usage has spread to healthcare settings with numerous potential and realized benefits. Mobile applications that are installed on smartphones have provided clinicians with readily available evidence-based decisional tools [3]. Medical applications fall under many different categories, including but not limited to, reference applications, such as the Physician’s Desk Reference (PDR®) or WebMD®, medical calculators, and applications designed to access electronic health records (EHR) or personal health information (PHI) [4].

1.2 Poorly Designed Mobile Application Hinders User Acceptance

Recently, the functionality of mobile applications has increased greatly. This increase in functionality has come at the expense of the usability of these applications. The international standard, ISO 9241-11:1998 Guidance on Usability defines usability as:

“The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction, in a specified context of use of the system” [ 5 ].

While technical evaluations of mobile applications receive much attention, few usability evaluation studies have been conducted, especially, for healthcare mobile applications. As such, there have been few studies that investigated the efficacy of using mobile applications that are used in hopes of assisting in training [4, 6].

Consequently, it is estimated that 95 % of downloaded mobile applications are abandoned within a month [7] and 26 % of applications are only used once, possibly because of the lack of attention to usability [8]. Poor healthcare system design may lower effectiveness [9, 10], decrease efficiency [11], and decrease team collaboration [12]. This, in turn, may lead to high cognitive load [13], medical errors [14], and decreased quality of patient care [15]. These issues are correlated within the scope of usability.

The objective of this study is to review and measure the usability of mobile applications currently available in radiology to support training in diagnostic decision-making.

2 Method

2.1 Setting

The University of Missouri Health Care (UMHC) is a tertiary care academic medical center located in Columbia, Missouri, with a total of 564 beds. With 626 medical staff at clinics throughout mid-Missouri, UMHC had an estimated 553,300 visits in 2012 [16]. Department of Radiology includes a full complement of 28 + highly trained clinicians and researchers, and successful training programs of 25 + resident physicians involving cutting-edge technologies and specialized clinical experiences. The department also operates Missouri Radiology Imaging Center, one of the most advanced resonance imaging facilities in the state.

2.2 Systematic Review of the Mobile Application

To determine usability issues of mobile applications in radiology, systematic review process was conducted. Two online stores with major market share (Google Play and iTunes) were searched on July 10, 2014 for mobile radiology applications that assist in training.

A multi-step review process was utilized by three usability investigators and five radiology experts to identify eligible applications and extract usability reviews. Distinct and broad search terms were used to capture a wide range of radiological applications: radiology, X-ray, ultrasound, MRI, CT, radiography, nuclear medicine, mammogram, mammography, and fluoroscopy.

Through screening of the titles and descriptions, applications were selected if they supported education and training of radiological diagnostic decision-making processes. They were excluded if they:

  1. 1.

    only provided access to journals, books, encyclopedias, or other reference material;

  2. 2.

    were designed solely for trivial medical calculations;

  3. 3.

    were designed solely for specific commercial vendor products;

  4. 4.

    were designed for use by a specific hospital/clinic only; or

  5. 5.

    were written in a non-English language.

The investigators extracted the usability reviews to be analyzed and coded them according to validated Nielsen’s heuristic usability evaluation principles (HE) [17, 18]. Two independent investigators (ABR, MAC) cross-examined the usability reviews of the counterpart to reach agreement. An experienced usability investigator (MSK) adjudicated any disagreement. Finally, the entire team collectively reviewed the findings for validity before analysis was carried out.

3 Results

From 381 applications that were initially identified, 52 applications of the total searched applications were eligible applications with user reviews. Using HE was instrumental in understanding areas in need of improvement. For the studied 52 radiology training applications, the usability-related reviews are 79 reviews. The types of usability issues with frequency by principles discovered were (some usability issues were cross-listed):

  • Naturalness (43)

  • Simplicity (43)

  • Efficient Interactions (21)

  • Consistency (16)

  • Effective Information Presentation (11)

  • Preservation of context (11)

  • Minimize cognitive (10)

  • Effective use of language (5)

  • Forgiveness and feedback (1).

The example usability problems were: applications lacking in content (17 apps), requests for scientifically based information (5 apps), downloading and crashing systems problems (5 apps), more images (3 apps), and more labels (2 apps).

The usability issues were classified according to 3 point usability severity scale [19]. Examples of the catastrophic usability issues (level 3 severity) were: crashing/force closing, and inability to install application to the user’s mobile devices. Level 3 usability issues prevent users from completing their task. Level 2 usability issues include: inadequate content, insufficient cases and quizzes, non-intuitive labels to the images, and inefficient interfacing problems. Level 2 usability issues delay users significantly but eventually allows them to complete the task. Minor usability issues (level 1) include: the small font size, lack of zoom option in on the image. Level 1 issues could only delay user briefly to complete the task.

4 Discussion

4.1 Mobile Applications in Radiology Need Further Usability Evaluation

Most existing mobile applications in radiology fall short in adequately engaging stakeholders and ensuring that the system designs are user-centered. This may be attributed to a number of reasons, for example, a system designers’ lack of understanding of clinical workflow, unclear guidelines in how user-engaged technologies should be implemented and actively used. Of most important is an insufficient understanding of users’ information needs, socio-demographic status, preferences, health and literacy, computer literacy, and values. Medical mobile applications that do not take account of these factors can impair the effectiveness of clinical management.

While the majority of mobile applications in radiology have been evaluated using tablet platforms [20–28], there are very few studies available that investigated the potential benefit of mobile applications on smartphone platforms, potentially because of the small display size [20, 29]. However, advancement of display technology and better mobility will allow increasing use of smartphones in the near future [30]. Thus, it is important to explore the feasibility of mobile applications on smaller mobile devices among physicians.

4.2 User-Centered Design Process in Mobile Application Development

Prior studies showed that mobile applications, including radiology, can be utilized to enhance education, with the potential to improve overall patient care [31, 32] but suffered from poor usability. User-Centered Design (UCD) is a process wherein the needs, wants, and limitations of end users of a product are given extensive attention at throughout the design process [33, 34].

The UCD process involves engagement of users throughout the processes:

  1. 1.

    user needs analysis,

  2. 2.

    algorithm development,

  3. 3.

    cyclical prototype design, and

  4. 4.

    development.

In user needs analysis, the design team collects users’ information needs, wants, and motivations for using mobile applications to acquire an understanding of potential factors that may impact the intended goals. Information items include: basic education-related demographics, prior mobile application experiences, perceived mobile device skills, and expectations (contents, features and functionalities) on usage of the mobile application in their clinical management setting. In algorithm development stage, design team sketches several ideas for potential prototype design through an iterative review process. Collected users’ needs are reviewed to ensure if they could be integrated within the application’s limitations. The team should decide on whether or not to include certain educational contents, features, and functional elements. Once the optimal algorithm is established, the team begins to design and develop a few low-fidelity (Lo-Fi) prototypes (i.e., sketches on paper or slides). The Lo-Fi prototypes will then be evaluated utilizing formative evaluations, such as, heuristic evaluation [17, 18] and cognitive walkthrough [35, 36]. When the final prototype has been selected, the team work to develop the high-fidelity (Hi-Fi) prototype with partial to complete functions. The Hi-Fi prototype are evaluated using summative lab-based usability testing [37].

After implementation, continuous data collection on usability should be warranted to measure usability [5, 38, 39] and acceptance [40] of the mobile application. This continuous evaluation will allow the team to adjust the design of the application for improving user acceptance and maintaining maximal educational effectiveness. Following the UCD process, mobile applications in radiology education and training may ultimately increase usability and therefore decrease cognitive overload, and may increase the quality of healthcare services.

4.3 Weakness to the Study

As with all study designs, there are limitations. First, due to this study’s exploratory nature, this study involved mobile applications in two mobile application stores. Selection of iOS (iTunes) and Android (Google Play) platforms for this study was made based on current trend and it may change in the future considering the fast-changing IT market. As such, evaluation of mobile applications in other platforms, such as, Windows or BlackBerry, should be warranted. Second, while Nielsen’s heuristic usability evaluation principles (HE) is an exemplary evaluation method, summative usability testing [37, 41] comparing multiple representative mobile applications could provide evidence-based guidelines to help choose mobile applications that will yield maximal educational and clinical benefits.

5 Conclusion

This study demonstrates the urgent need for usability evaluation in the development of radiology mobile applications. Investigators suggest that the approval process of any medical mobile applications should undergo a more systematic and rigorous process to improve the applications that will satisfy the users’ experience and meet clinical training goals. In addition, the investigators suggest an institution of systematic and standardized guidelines regarding design and test of healthcare mobile applications to achieve maximal adoption.