Background

The development of telepharmacy, with technology incorporation into pharmaceutical care services, was propelled by the COVID-19 pandemic. Telepharmacy provides remote counseling, medication information, online purchases, adverse effects monitoring, and therapy tracking through digital platforms [1, 2]. Moreover, it creates a virtual channel for pharmaceutical care, connecting pharmacists and patients remotely to facilitate easy health evaluation [3]. Other benefits include reducing direct interaction between healthcare professionals and patients, improving pharmaceutical service quality, and minimizing medication errors and adverse effects through various channels such as application, text messaging, video, and voice calls [2]. Telepharmacy improves cost-effectiveness, healthcare access, and after-hours availability, as well as reduces travel time to healthcare facilities [4, 5], breaking geographical barriers and enhancing healthcare accessibility [6, 7]. However, implementing this system in clinical practice may be challenging due to legal considerations, operational costs, and patient trust-building [8, 9].

Recent guidelines in Indonesia mainly address general aspects of delivering telehealth services during the COVID-19 pandemic, particularly focusing on definitions and procedures [10]. Specific regulations for telemedicine are necessary to ensure compliance with ethics and other regulations [11]. Additionally, evaluation of usability and applicability is crucial for optimal implementation and adoption in real-world settings. Gaining insights into usability from the perspectives of the general population and pharmacists during the design and testing stages can enhance application effectiveness, which is essential for successful implementation and scalability [12,13,14]. Despite the rapid growth of telepharmacy in the country [15,16,17,18], only a few available application is tested [19,20,21], leading to varied usability experiences based on features and limitations encountered [19,20,21]. Many types of Indonesian telepharmacy application lack appropriate study designs and methods to address engagement issues faced by users and pharmacists. Considering the interconnection between program usability and the engagement of users, usability studies are essential for understanding and improving engagement [22]. Usability is defined as a user interface characteristic that facilitates application adoption, effectiveness, efficiency, and satisfaction in a targeted environment [23]. In telepharmacy implementation, poor usability can hinder technology acceptance [24]. Therefore, the products, systems, processes, and procedures constituting telepharmacy must be designed and implemented to be usable, useful [25], accessible, and user-friendly for both healthcare providers and patients [26]. Engaging these users in usability testing can help to address specific needs and preferences, promoting successful technology acceptance and adoption [27].

An Indonesian telepharmacy application known as Tanya Obat, translating to “Ask about Medication”, is developed through design-based study [28] by a team of pharmacists and academicians. This provides a comprehensive ecosystem connecting pharmacies, pharmacists, and technicians with the general population. Furthermore, the application comprises features such as locating nearby pharmacies, online medication purchasing, consultation services, medication use information, and educational resources including webinars, e-modules, and coaching clinics for pharmacists and technicians. Tanya Obat differs from other types of application in Indonesia by offering a dedicated ecosystem for pharmacists, including online medication consultation and educational opportunities to enhance competency in delivering health services. Therefore, this study aimed to evaluate Tanya Obat usability and obtain user feedback on the application at early development stage in Bandung City from the perspectives of the general population and pharmacists. The association between the characteristics of participants and perceived usability will be explored.

Method

Study design

A mixed-methods sequential study design was used to assess usability and obtain user feedback on Tanya Obat application at the early development stage before launching. Furthermore, questionnaires for the general population and pharmacists were distributed from March to May and February to June 2023, respectively. Approval with contract number 670/UN6.KEP/EC/2022 was received from the Research Ethics Commission of Universitas Padjadjaran, Indonesia, and all recruited participants provided written informed consent.

Study population and settings

Participants recruited in Bandung City comprised untrained and first-time Tanya Obat users from the general population, irrespective of being health system users, as well as pharmacists registered in the application or not. One week was provided for Tanya Obat usage without supervision to permit full exploration of the application, and recruiting was performed online through a convenience sampling method. Inclusion criteria for the general population consisted of individuals with (i) age above 18 years, (ii) ability to use a smartphone, (iii) lack of prior application experience, and (iv) willingness to participate. Meanwhile, the inclusion criteria for pharmacists were (i) current employment at a Community Health Center (CHC), hospital, or pharmacy, (ii) ability to use a smartphone, (iii) no prior experience with the tested application, and (iv) willingness to participate. All participants received an explanation regarding the study stages, application installation, and features.

Tanya Obat application

Tanya Obat application was developed by a team of pharmacists and pharmacy academicians in collaboration with software engineers using Dart (https://dart.dev/) programming language. As shown in Fig. 1, the application offers five main features including:

Fig. 1
figure 1

Tanya Obat Application Features

Full size image
  1. 1.

    Finding Nearby Pharmacies

This feature allows users to conveniently search for the price sites of nearby pharmacies and view available medication lists, saving time spent on locating the physical buildings.

  1. 2.

    Medication Purchasing

The medication purchasing feature extends the reach of pharmacies to patients through a digital platform. Users can freely browse categories of non-prescription and prescription medications which can be ordered with the delivery service available on the application.

  1. 3.

    Ask a Pharmacist

This flagship feature enables digital consultation with pharmacists regarding self-medication, regular medications currently in use, and other concerns about consumption and side effects. Additionally, it offers pharmacists the opportunity to gain consultation experience, which can subsequently be converted into a credit unit for pursuing professional competency certification. Recruiting for this feature was performed by distributing information through webinars, scientific events, social media and ads, personal and group chat application, as well as direct visits to pharmacies. Users will be charged IDR 20,000 or USD 1.5 for 15-minute consultations based on regulations established by the Indonesian Pharmacists Association in 2019.

  1. 4.

    Health Article

Tanya Obat application allows both the general population and pharmacists to access accurate medication information in a clear and easily understandable manner. Moreover, the authors of this study were accredited pharmacists, pharmacy graduates, and students with competencies and excellent knowledge in writing health articles. Each published article was successfully curated and reviewed by professionals and an editorial team.

  1. 5.

    Sustainable Educational Development for Pharmacists

This feature can only be accessed by pharmacists, serving as a channel for enhancing educational knowledge and professional competency through webinars, e-modules, and coaching clinics.

Each feature of Tanya Obat application is available for both the general population and pharmacists. However, “Ask a Pharmacist” has slightly different user interfaces tailored for the respective group of participants. This specific feature shows a list of online pharmacists for the general population. Meanwhile, the inability of accounts designed for pharmacists to request consultations as patients, is complemented by the provision of access to current and completed consultation history.

Measurements

Perceived usability was evaluated based on a standardized quantitative System Usability Scale (SUS). Meanwhile, additional feedback from users engaged in this study was assessed through open-ended qualitative questions.

Usability measurement

Usability, often defined as user interface characteristic that facilitates application adoption, effectiveness, efficiency, and satisfaction in a targeted environment [23], was assessed with the Indonesian version of the standardized SUS questionnaire [29, 30]. This evaluation was conducted by identifying potential user interface, functionality, and design issues based on the perspective of untrained, first-time users from the general population and pharmacists. Moreover, SUS questionnaire comprised 10 questions rated on a 5-point Likert scale ranging from [1] “Strongly Disagree”, “Disagree”, “Neutral”, “Agree”, to “Strongly Agree” [5]. Participants responded based on the subjective assessment of usability, as presented in Table S1 Supplementary data. Each question contributed a score, with different scoring conversions for odd [1, 3, 5, 7, 9] and even [2, 4, 6, 8, 10] questions, reflecting positive and negative responses, respectively. Summation of all the points yielded a maximum score of 40 which generated a scale ranging from 0 to 100 through multiplication by 2.5. Higher scores suggested favorable user perceptions of the application, while lower scores signified low usability [31]. A previous study showed a significant association between SUS scores reported as continuous or dichotomous data and outcomes [32]. Reporting as dichotomous data may be more practical in real-world scenarios due to the ease of interpretation [33]. Therefore, in this study, a SUS score exceeding 68 points was considered above average in perceived usability [34], calculated based on a published formula [33].

The questionnaire was tested for construct validity and internal consistency using 30 participants who were not included in the main analyses. Construct validity was assessed with the Pearson Product Moment correlation by correlating the score of each question with the total score [35]. The questionnaire was deemed valid when the correlation coefficient (r-value) exceeded the critical value [28]. The values for each question were > 0.361, signifying correlation with the respective dimension and affirming this construct as a measurement instrument. Additionally, Cronbach’s Alpha coefficient yielded a score of 0.721, meeting the criterion established for internal consistency, namely a value > 0.60 [36].

Additional user feedback

After completing SUS questionnaire, participants provided additional feedback based on open-ended questions to obtain impressions following one week of Tanya Obat usage. Three questions were used to assess the perceptions of participants about the application (e.g.,), potential confusion or difficulties encountered (e.g.,), and desired features for enhanced user-friendliness. These included “What was your first impression of this application?”, “What would you say was the most challenging aspect of using this application?”, and “What are your recommendations to improve the application?”. Subsequently, a summary of interpretations made by the study team was sent to all participants for verification.

Demographic characteristics

The survey collected sociodemographic information about the participants, including factors such as gender (male, female), age, highest education level (junior high school, senior high school, bachelor’s degree, registered pharmacists, master’s degree, doctorate), practice settings, and years of practice (for pharmacists). Additionally, participants provided information on subjective experiences with smartphone application (amateur, beginner, skilled, highly skilled), daily duration of smartphone usage in hours (< 1, 1−2, 3−4, and > 4), and internet accessibility at home (available, unavailable).

Sample size calculation

The sample size was calculated using Slovin’s formula [37] due to a lack of prior knowledge about outcome distributions. To achieve a 95% confidence interval (CI) and a margin error of 0.15 with a statistical power of 80%, a minimum sample size of 100 for the general population and 64 pharmacists was required.

Data collection

The link to both the application and survey was disseminated through a digital leaflet across social media, as well as online personal and group chat platforms. This application was available for download on the Google Play store, and data were collected using the Qualtrics platform. To prevent duplicate entries, participants were restricted to filling out the questionnaire only once from a single email address. For pharmacists, a convenient sample was formally or personally recruited through the Indonesian Pharmacists Association network. The estimated time for questionnaire completion was 10 min, while participants were initially reminded that the study aimed to assess user-friendliness of the application and not the ability to use the device correctly. Following one week of exploring the application, participants were immediately instructed to complete the questionnaire to avoid bias introduced by repeated usage and becoming overly familiar with the application. Additionally, screenshots of the features found were requested as evidence of exploring the application.

Data analysis

Basic demographic characteristics were examined using descriptive statistics, where mean SUS scores were analyzed for each subgroup of the general population and pharmacists. Survey quantitative data were tabulated and presented as frequencies, then a Chi-square test was used to determine the association between the characteristics of participants and perceived usability. Open-ended questions were summarized and clustered with thematic analysis, following steps including familiarization with data, initial code generation, theme development, theme review, theme definition, and conduction of independent qualitative analysis by JAS and DQA using NVivo software version 12. Any disagreements among JAS and DQA were resolved through discussion with a third study team member (SDA). To ensure content analysis reliability, continuous discussion and negotiation regarding the content of keywords, broader concepts, and units of meaning were performed among the team.

Result

Quantitative results

Demographic characteristics

A total of 176 participants (100 individuals from the general population and 76 pharmacists) engaged in this study. Those from the general population were recruited from 26 districts in Bandung City, majorly comprising 18−25 years old (59%) females (71%) with high school qualifications (71%). Furthermore, most participants were ‘skilled’ at using smartphone application (56%), with over four hours of daily smartphone usage (65%), and found to use Wi-Fi at home (72%), as shown in Table 1.

Table 1 The characteristics of participants
Full size table

Subsequently, a total of 76 responses were obtained from pharmacists in Bandung City, Indonesia. These majorly included females (73.7%), working in community pharmacies (79.0%), with over six years of job experience (38.2%), considered ‘skilled’ in using mobile application (64.5%), possessing access to the internet (89%), and known to use the smartphone for more than four hours daily (63.2%).

Associations between the characteristics of participants and perceived usability

The characteristics of the general population and pharmacists were presented in Tables 2 and 3, respectively. This study identified no significant correlations between the characteristics and perceived application usability (p-value > 0.05).

Table 2 Associations between the characteristics of the general population and perceived usability (N = 100)
Full size table
Table 3 Associations between the characteristics of pharmacists and perceived usability (N = 76)
Full size table

Usability results

According to the calculation matrix in Table S2 Supplementary data, the average SUS score of the general population was 63.4, representing a below-average perceived usability. A significant variation in scores was observed, ranging from a high of 87.5 to a low of 30. The average SUS score of pharmacists was 64.1 (Table S3, Supplementary data), which was also below average, with a significant variation in scores ranging from 95 to 42.5.

Qualitative results

Two themes, including concerns and recommendations, were identified from the responses provided to the open-ended questions. These were further divided into subthemes comprising feature functionality for concerns as well as visual enhancements, feature improvements, system improvements, and other parameters for recommendations, as presented in Table 4 along with examples of supporting quotes.

Table 4 General population concerns and recommendations for Tanya Obat application (N = 100)
Full size table

User feedback: the general population

Concerns identified for Tanya Obat were features such as consulting a pharmacist, searching for online medication purchases, and locating nearby pharmacies. Users reported that the application was not functioning optimally and required assistance using features including purchasing medication, redeeming prescribed medicines, and consulting a pharmacist. Additionally, inconsistencies in color schemes and delays or errors when accessing features within the application were noted (Table 4).

Potential enhancements for the application included creating more visually appealing shows in terms of color combinations, layout, and font styles. Additionally, it was suggested to ensure that features function as intended, providing relevant descriptions for each feature, and enhancing filters. System-related recommendations comprised addressing bugs or errors such as loading failures, adding a user guide, improving maps, and enhancing search speed. Other suggestions included introducing Tanya Obat on iOS devices and promoting the application among a wider demographic (Table 4).

User feedback: pharmacists

Concerns observed in using Tanya Obat application were related to medication purchase, address configuration, and specific hurdles in the registration of pharmacists. Another concern was identified with the “consult a pharmacist” feature, where the need for more assistance was reported (Table 5).

Table 5 Concerns and recommendations of pharmacists for Tanya Obat application (N = 76)
Full size table

Several suggestions obtained from feedback in open-ended questions were majorly related to features. Regarding the medication purchase feature, the requirement for expanded stock availability and the incorporation of new functionalities was reiterated. Furthermore, pharmacists stated the need for an online Continuing Professional Development (CPD) feature, which would be incorporated by the Indonesian Pharmacists Association as the issuer of competency certificates. This suggestion was provided to support consistent education through seminars, articles, and learning modules, along with the integration of counseling history or medical records of patients that can be accessed by pharmacists in Indonesia and claimed as credit points for CPD. The feature for registration of pharmacists should be enhanced, while modification of the color palette was recommended to ensure increased visibility and maintain satisfactory show and font choices.

Discussion

Principal observations

In this study, usability and user feedback of the innovative Tanya Obat application were evaluated. Perceived usability was categorized as below average, with average scores of 63.4 and 64.1 for the general population and pharmacists, respectively. The identified concerns primarily included the functionality of some features, while chances for improvement were observed in the areas of visual, features, and system enhancements. These concerns and recommendations slightly differed between the general population and pharmacists.

From the general population perspective, the below-average score obtained might be attributed to disparities in digital literacy, defined as the ability to acknowledge and use information from various sources presented through computers [38]. A national survey conducted among 10,000 Indonesians showed a digital literacy index of 3.49 in 2021 and 3.54 on a scale of 5 in 2022 [39,40,41], and despite the improvement, digital literacy in the country remained at a moderate level. Previous results showed that the general population and pharmacy students in Indonesia had a positive perception and were willing to use telepharmacy services [42, 43], providing an opportunity for successful implementation of the application.

Several areas for improvement were identified to enhance and adapt the future application to the understanding and capabilities of users, thereby leading to a more usable and useful system [25 ]. Moreover, primarily existing problems included complications using the features, attributed to unfamiliarity with technology, lack of confidence in using electronic devices, or fear of committing mistakes [39]. A survey reported that 69% of 10,000 Indonesians have not accessed health services through digital platforms [39]. Complications while running the application can be reduced by simplifying the log-in process, reducing required tasks, and showing fewer buttons on the screen [44,45,46,47,48,49,50,51,52,53,54,55]. Selecting the appropriate design, wording, and development language in a user-centered and participatory design process is crucial and may have an important impact on engagement [56]. These improvements tend to increase satisfaction, which plays a vital role in the implementation and continual use of the application [57, 58]. Therefore, developers must prioritize maximizing the application performance to be more user-friendly [44, 59,60,61,62,63,64].

The similar marginally acceptable score obtained from the feedback of pharmacists signified that more work was needed for usability improvement, as these professionals were expected to be more exposed to telepharmacy application than the general population. This unsatisfying score could be partly explained by the varying levels of readiness to use telepharmacy application [65]. Pharmacists were forced to implement telehealth services without adequate readiness assessment, particularly in a setting without well-established telehealth services before the COVID-19 pandemic, such as Indonesia [66] Furthermore, the main concerns that require adjustment to enhance application usability and reduce errors include expanding the network of available pharmacies and pharmacists with specialties comprising traditional medication and cosmetics. Specialization among pharmacists has various important benefits such as higher adherence and persistence, better clinical outcomes, monetary benefits for both patients and the healthcare system, and higher patient satisfaction [67]. Therefore, expanding the network of the pharmacy sector to include traditional medication specialists and pharmacist cosmetologists will significantly improve the application.

Recommendations of pharmacists regarding giving details about all medication were consistent with a previous study that reported an increment of adherence among patients after providing simple and brief written medication information [68]. Additionally, offering live interaction during counseling led to care quality improvement. Specific concerns need to be addressed, such as ensuring properly configured camera and audio settings, appropriate light quality, stable internet connection, and readiness to assist patients unfamiliar with the technology [69]. Furthermore, pharmacists recommended interactive video learning for application enhancement. This was consistent with another study that showed effectiveness in interactive video [70, 71], with an 89.7% reported increase in learning outcomes [70]. Interactive virtual content had significant effectiveness compared to the online class group method [71]. Another recommendation was related to visualization enhancement, playing an important role in attracting user attention [44] with a tendency to influence perceived usability even when no differences existed in functionality offered [72]. Users preferred simplicity, showing more graphics than crowded text, and consistent style using combinations of colors [45,46,47, 73,74,75]. Consequently, telepharmacy application usability would be optimized with the results of this study before proceeding to evaluate effectiveness because the standards of health professionals need to be met [76].

The characteristics of the general population and pharmacists were found to have no significant association with perceived usability. These results showed that the application could be used for all age groups irrespective of educational background, familiarity with smartphone operation, and internet accessibility level [77]. However, certain studies reported inconsistent results on the association between sociodemographic factors and perceived usability among the general population and pharmacists, respectively [77,78,79]. Significant associations were previously shown between demographic-related variables and usability [80] from an investigation conducted among participants with years of mobile application experience [80]. The less experienced participants using mobile health application in Indonesia and their subjectivity might lead to under or overestimating the experience had with smartphones, which could clarify the non-significant associations observed in this study [81]. Moreover, some unmeasured factors tended to be associated with perceived usability, such as readiness [81], experience in using mobile health application [80], digital literacy [38], and health literacy [82, 83].

Implications and future directions

The mixed-methods evaluation conducted in this study and the results represent the first step in optimizing the development and evaluation of Tanya Obat application. A previous investigation assessed satisfaction with telehealth application using SUS method and showed improvement from 71.8 to 82.5 following enhancements based on the first round of testing [84]. Therefore, SUS score tends to increase when adequate improvements are provided to Tanya Obat application. These results support the importance of incorporating usability studies as part of the digital health intervention design process [85].

Continuous explorations to obtain post-refinement data with a larger sample size from a multicenter study in different provinces in Indonesia are essential to provide more accurate representative results. Additional investigations are required to evaluate the impact of Tanya Obat application, particularly among patients with chronic diseases who regularly consume medications, as usability may vary between different clinical groups. This can help identify context-related issues in the future, such as patient adherence, access to healthcare, and satisfaction.

Strengths and limitations

The strengths of this study include the recruitment of two different subjects, namely the general population and pharmacists, representing potential and relevant end users. Furthermore, the quantitative and qualitative data provided by real-world users are crucial in ensuring the addition of obtained views in the application development to improve usability. The inclusion of real-world participants showed important usability problems and unidentified solutions during the development or expert panel review stages. However, one main limitation of convenience sampling is the risk of bias due to the lack of random selection. Certain groups within the population may be overrepresented or underrepresented due to the inability to control the questionnaire link distribution. This skew might lead to the production of results not accurately representing the entire population. For example, the majority of participants were aged below 50 years and had a moderate to high educational level, both of which were factors increasing the tendency of technological proficiency and willingness to adopt new technology [86, 87]. In future investigations, combination of multiple methods, such as random, stratified, or systematic sampling, can improve the quality and representativeness of the convenience sample, which tends to produce more accurate and reliable results, thereby enhancing generalizability. Furthermore, the subjectivity of respondents may be resulting in under or overestimating their experience in using smart-phones which may lead to the non-significant association with the usability score. There was a challenge in ensuring full exploration of all Tanya Obat application features, and the reliance on screenshots from feedback provided by participants without direct verification constituted potential limitations. Future investigations need to conduct a real-time observation or the recording of screen activity as a more objective measurement [88]. The method of this study was limited to user-based usability evaluation, focusing solely on the satisfaction aspect, and not incorporating the think-aloud method [89]. Additionally, expert usability evaluation through heuristic tests [90] were not performed, generating a less comprehensive scope for the results obtained. Despite these limitations, Tanya Obat application usability was improved before launching by making changes based on the concerns and recommendations of the target user groups.

Conclusion

In conclusion, the results showed that the developed Tanya Obat application was below average in perceived usability. Therefore, specific feature optimizations should be performed, particularly in terms of visual appeal, features, and system functionality, to improve potential acceptance and usability, facilitating successful adoption in Indonesia.