Keywords

1 Introduction

Smartphones are the popular gadget among people nowadays, the penetration is increasing with a fast pace [1]. Application development and usage for smart devices become imperative almost in all sectors. Additionally, the numbers of mobile applications such as game, social networking, entertainment, personal and professional are becoming part of our lives. These applications are however constantly competing users’ time and attention [2] for sustain usage and retentions. There are many factors ensures the mobile application sustainability e.g. application usability and user experience [3]. A mobile application that needs to compete users’ time e.g. students is the mobile learning application. Mobile learning application development is a complex activity as firstly, many stakeholders and role-players [4] involved who have direct and indirect impact on the application design, development and usage. Secondly, it deals with learning and learners that required extensive pedagogical considerations [5, 6] which is by nature a complex concept. Thirdly, the application is targeted for smart gadget which has many restrictions such as screen size, input and output medium etc. Fourthly, the application must attach the users emotionally from user experience perspective [2] and compete students time. As a result, we are dealing with a complex systems which required appropriate methodology to cover all the complexities. The User Centered Design (UCD) framework for m-learning application development [7] is proven successful framework to design and develop a usable mobile learning application. This framework puts intended users of the target application at the center of its design and development. It is important to validate the methodology to make sure that the method fulfilled the essential needs in addition to the usability and user experience. Over the time many approaches has been proposed to validate the software development methodology such as [8] or to select the best software engineering method to design a complex system [9].

1.1 User Centered Design Framework for M-earning Application Processes

The case studies which presented in this papers are based on the User Centred Design (UCD) framework for m-learning application development [10]. The following figure reveals the UCD for m-learning application development phases.

Fig. 1.
figure 1figure 1

User-centered design process for m-learning application development

Full size image

  • User Study – This is done by applying methods such as questionnaires and semi-structured interviews. In this phase, the designer aims to learn about the users’ existing means to handle their work related tasks.

  • Data Analysis – Analysis of the data gathered from the user study phase. This analysis consists of transcript coding of the user interviews [11], and description of users’ tasks and environments [12]. The overall requirements for the target application are identified in this phase.

  • Idea Creation – By using affinity diagrams [13], actions and requirements created in the previous phases are categorized. Use cases and scenarios are also applied as design methods to create a description of the application concept to the target users.

  • Product Concept – Scenarios are shared with 3−5 users for their feedback. A scenario reflects the potential application concept. After the collected feedback is reviewed, the designers conduct a short, semi-structured interview to learn about users’ opinions about the application concepts and functionalities. Users’ feedback is analyzed to validate the feasibility of the concept and to ensure that the users and designers share the same understanding of the potential application. After validating the concept through scenarios, developers are asked to design a low-fidelity or a high-fidelity prototype based on the proposed scenarios. Finally, developers are asked to perform usability evaluation of the prototype on users by applying Nielsen’s heuristic evaluation guidelines [14].

The UCD framework for mobile learning application is an iterative design method which, mandates the users’ involvement in each development phases. This minimizes the applications’ failures and error rates and maximizes the application’s acceptance as an educational means by target users.

2 Methodology and Research Data

The parameters of the UCD Framework evaluation comprises of the analysis of the case studies assessments’ result. The analysis of these assessments demonstrates various aspect of the framework functionalities and performances. The descriptions of the utilized methods on each evaluation parameters are as follows:

  1. 1.

    Analysis of the Case Studies’ Usability Evaluation Results. The acceptance of the application by target users group. (Usability assessments). The evaluation of the usability will be based on ISO 9241 part 11 and part 304 which recommends the measures of usability should cover: effectiveness, efficiency and Satisfaction. General metric to measure the usability as quality metric success rate (Whether users can perform the tasks at all) the measurements is based on the time a task requires to perform the tasks, the error rates, and, users’ subjective satisfaction on the application performance and functionality. In this test, we ensured that the application follow the basic usability guidelines. In this phase, we often utilized the Nielsen [15] heuristic evaluation guidelines to make sure that the application is usable for the target users.

  2. 2.

    Metrics for Evaluating Educational Components. The usability and user experience assessment of the target m-learning application based on the education components (in press Journal). In this assessment, the m-learning application is assessed based on Brusilovsky [16] educational components layers i.e. Presentation, Activities, Communications and Administration. The assessments measures for example in each layer the Adjustability, Delightfulness, Reliability and satisfaction based on the predefined criteria. As an example of an assessment in the presentation layer we measure whether the user is able to personalize and customized the application based on his/her needs, the result of this part assessment indicates whether the application supports the adjustability factor in the learning application. In this phase, we evaluate the user experience and usability by asking users to perform predefined tasks on the application. Table 1 presents the usability criteria and guidelines [17] for M-learning application based on educational components. This acceptance criteria defines the framework performance from the educational perspective as it measures how the application performs based on educational activities.

    Table 1. Usability criteria for m-learning application based on educational components
    Full size table
  3. 3.

    Post Interview on Designers’ Experiences Regarding the Applied UCD Methods. Qualitative research semi-structured interview with developers, designers and the owner of the applications. The interview pursuit to get additional information about the framework as development methodology itself. The questions like “how did you find framework to help the design and development the application?” or “how easy it was to create the concept ”, “How many errors identified at the scenario phased”, “Your overall impressions of the application development?” The analysis of the data helped greatly to get inside on developers, researcher view on the methodology.

3 Case Studies and the Measurements

The following case studies were developed and assessed during 2013−2015, except the M-learning application for Java course, which was developed during 2004.

  • Java Application. The aim of this case study was to analysis all phases of the UCD framework performance. With the help of this m-learning application, students were able to have access to Java course resources e.g. Lecture notes, assignments, and feedback at any time any places. In addition, students were able to submit their lab and home assignment through the m-learning application in case they had access to an internet At the elicitation phase, users study was conducted with six students (n = 6) and two staff (n = 2). Additionally, semi-structured interview and questionnaire were applied to learn how users handled the course related activities. To analysis the collected data, we applied interview transcript coding [18] After categorizing the related task and the actions, a scenario [19] was written and shared with users for their feedback. Finally, application high fidelity prototype was designed, developed which was ready to test.

  1. 1.

    Usability and User Experience Assessment. Two different usability evaluation sessions were conducted, one for students (n = 6), and another for staff (n = 2) each of which had distinct sets of tasks to carry out during the given time. The main tasks that the users requested to perform were, to log in to the application, uploaded and download the lecture materials in a different format, to receive, submit and provide feedback for assignments etc. The assessment results indicated that users were able to carry all the given tasks successfully. Despite the fact that this was the first experience with the m-learning application that users ever had. They were very excited to try the application in real life, even though, the cost of the data exchange was relatively high for students. Users’ initial mental models of the m-learning application were that they had to carry out all the educational activities via SMS, this however changed as soon as they tried the applications with the test devices. The users’ had appreciated the simplicity of the user interface of the mobile learning application statements e.g. “I liked the application, easy to find the features, just go the categories”. This had raised by several users especially during post-interview session. Users specially had complained about the input medium as the keypad were not convenient to write a long text as a result they preferred to use the application mainly for reading.

  2. 2.

    Educational Activities Assessment. The users carried out the given tasks according to the educational components of the criteria mentioned in Table 1, such as presentations e.g. users were able to download course materials such as lecture notes, audio, and video etc. despite the fact, that the technology at the time did not support many proposed functions. Similarly, as the presentation component, user were not very delighted to uses the provided functionalities in activity component because of the inconvenient input keypad. Nevertheless, users were delighted to receive instance feedback for the submitted assignment. Users found the communication functionalities very interesting, especially the capability to have direct communication capabilities with their peers and also teachers securely. Through the communication component, users were able to send/receive SMS, email to individual and group and also chat with teacher or classmate. Users find the email functionality in the communication component very interesting as they do not need to search for the course participants’ email separately including the course staff. Sending an email to classmate and teachers become simple and easy with few clicks. Users also were in favor with the functionalities of administration component. User’s had considered this service as usable in their educational activities. Through this service users were able to register for a course, register for the exam and check the credit transcript.

  3. 3.

    Developer Experiments. Two groups of students carried out the design and development tasks, the first group, consists of four students (n = 4) who had previously studied usability related courses, they had the designer role in the project. This group actually conducted the user studies both with students and staff. The second group was consist of two (n = 2) students, who had previous experience in programming, they had the developers’ role. These two groups interacted continuously to achieve the potential goal. The team especially appreciated the systematic approach of the framework to achieve the goal. Additionally, the role divisions were recognized as an asset in this framework by audience. Moreover, the audience believed that having the scenario as a design methodology helped both designers and developers to conceptualize the overall application themselves. They showed their wiliness to utilize the methodology in future projects “I liked it, clear and helpful to come up with the prototype” was heard repeatedly by developers and designers.

  • Adaptive Driving School. With this case study, we aimed to reassess the UCD phases by applying all the steps defined in UCD framework. In this case study, we designed, developed and assessed an adaptive mobile learning application for driving license candidates. The development process and the application evaluation result was presented and published in ICCA2014 conference [20]. This application helps students to study, learn and assess the compulsory driving school theory lessons on their smart devices. Additionally, the application provides mandatory self-evaluation reports to instructors after each practical driving sessions.

  1. 1.

    Usability and User Experience Assessment. The prototype evaluation report indicates that the application was easy to use and provides the essential learning materials for driving school candidates. Users were especially satisfied that the driving theory lessons were accessible at anytime and anyplace even during driving sessions. The usability test was carried out at Haaga-helia UAS, with five potential users (n = 5) and two instructors (n = 2). The test users were mainly novice users meaning they have not had previous experiences on similar application for driving schools. The usability report indicates that the application provides positive user experience for our test users. Test users were able to carry the given tasks completely no savior errors appeared. The test results and the interview data revealed that the test users were founded the application helpful and easy to use.

  2. 2.

    Educational Activities Assessments. The test was carried out based on the educational components criteria presented in Table 1. The presentation component, users were able to carry their educational related tasks properly, despite the fact, that some presentation features were not implemented. The application content and UI was customizable based on students’ performance on theory lessons evaluation and practical driving session reports. Moreover, the application supported multi-formatted content, e.g. audio, video etc. However, due to the technological inadequacy and lack of development time caused the usability factor e.g. adjustability was not able to test. The activity components, test users were satisfied with the clarity of the tasks and knew what to do next. We were not however able to test the reliability factor in this component. The communication component was implemented in a simple communication between student and instructor, which the test revealed, functioning properly. The administration component also followed the defined criteria in Table 1 accordingly.

  3. 3.

    Methodological Assessments. The interview that was conducted with designers and developers indicates that they were satisfied with the overall result. They show their satisfactions based on the fact that the framework reduced the application errors, before the high fidelity prototype implementation, “We managed to find most of the errors already in scenario review phase”. The designers were specifically happy with the continuous consultant of potential users, which helped them to come up with new features that they had not realized at the elicitation phase “we receive nice to have features both in the scenario and in lo-fidelity”. The stakeholders were also asked about their motivation for using the framework: “in the beginning using the framework work was difficult, did not know what to do, but after the transcript coding everything changed”. The main negative feedback, which we received from developers was that this methodology required extensive user consultant, which consume lots of time. The stakeholders e.g. the owners of the application were happy with the overall results, “very surprised everything works with any errors we tested the application with students and instructors”.

  • Customer guide game

    In this case study, we aimed to specifically test the framework proposed design method. In this project, we developed a multi-platform mobile game application for students and staff to learn the new premises and other important locations. We applied scenario design to learn the potential user’s application preferences. As the customer of the application provided us the requirements list, we mainly focused on the design. The case study development process and the usability evaluation result is submitted at CSEDU 2015 [21].

  1. 1.

    Usability and User Experience Assessment. As soon as the functional application prototype was developed, we conducted a usability test with five test users (n = 5). Unlike the paper prototype usability assessment, which users carried out predefined set of tasks, in this testing users had a freedom to play the games. Users were, however, asked to save the points, starts and stop different sessions. Despite receiving visual design, graphical and sound improvements recommendations, the test demonstrates that the application content and UI in general appeal users. Users believed that the game has enough challenges only two test users (n = 5) demanded that the game needs to be more demanding, these two test users were considered as expert users. The most important issues that raised by all test users were the intuitiveness and easy to use UI and the game, in general. Moreover, the final concept test results show that the scenario design and the first paper prototype identified the majority of the errors and improvement recommendations. The product concept phase the most of feedback were on the items that were not directly detectable at the paper prototype phase such as graphics improvement or the sound effect.

  2. 2.

    Educational Activities Assessments. The presentation component was not fully implemented in this prototype, even though, users were able to adjust the game character. Another adjustability feature was not seen as an appropriate for the game. Users were able, however, to create a profile, select a role image etc. The activity component in this case study required users only to play and explore the game features, which all users done properly. Users demonstrated their delightfulness by saying “wow, this is fun”. We did not implement the communication component as a result, we did not have any feature to test. The administrations component satisfied users as they were able to create, delete profile. Additionally, users showed their satisfaction with this component, “I can see who played and what points they achieved”.

  3. 3.

    Methodological Assessments. The author act as a thesis supervisor in this project, as a result, there was constant involvement with all phases of the design and development process. In addition, we had regular Skype meetings with the application owner after each development phases e.g. scenario, lo-fi, and hi-fi evaluations. The designer viewed the framework as a good method to achieve his goal. The designer specially appreciate the scenario design “Scenario saved lots of development time, I had plan to design an application and then check with users, now I see that users needed game and not mobile application” or “Scenario helped me to see the kind of game users preferred and wanted me to implement”.

    The designer was interested to apply the framework, even though, at the starting phase of the project, considered scenario design as a time-consuming step. The designer, developer, and the customer showed their satisfaction with the overall application concept. The main negative feedback that we received from developers was that this methodology required extensive user consultant which consume lots of time.

  • mhealth Application. In this case study, we aimed to test all phases of the proposed UCD framework. There exists a variety of mHealth applications, but none of them combines essential professional tools for nursing. We tackled this problem in the Finnish elderly house. We applied the UCD framework by conducting user study. Together with 12 nurses, we first identified their profiles and their expectations on work-related mHealth application functionalities. The results were utilized in conceptual design of Context-Aware Nurse Assistant (CANA), which combines the identified functionalities and provides context-sensitive services to consolidate nurses’ work activities. The result of this case study is published at the ICUM 2015 conference [22].

  1. 1.

    Usability and User Experience Assessment. The analysis of data indicates that hundred percent of the test users considered the application as unique and excellent. Statement such as “I never ever thought that I can perform my work related tasks with my mobile” was repeatedly heard from the participants. The surprising fact for the researchers was that even the expert participants, i.e. those who were familiar with smart devices, were surprised to see such an application in the field. This indicates that the user data analysis and the concept development using the UCD framework was successful.

  2. 2.

    Educational Activities Assessments. This case study was designed to provide services for nurses on their job related activities. As this application is not considered as an educational application, so, the educational components cannot be fully applied in this case study. But we have identified the following findings, the communication component users were delighted to see that they have access to all those who worked in the same departments at the same time “nice feature I don’t to go to office and search for the other nurse in the department” or “I can see who I can contact if I need help, great”. Similarly, the assessment results demonstrated that users were able to carry their activities properly without any difficulties. We assessed the administration component ourselves and realized that we were able to see, search, print in addition to add, remove users.

  3. 3.

    Methodological Assessments. The discussion meeting was organized with those who were involved in this case study e.g. application designers, developers, and owner. The user study expert considered the application concept development straightforward. The overall assessment of the methodology was positive e.g. “It is an excellent method to design and develop an application without benchmarking”. The application developers considered the methodology very effective specifically as the whole team was involved in development. The designers rather see more design guidelines on the mobile application development in the framework “I wish we had design guidelines to help us to get the best user experience for the design”. The designers most motivation factor was the continuous feedback from users, especially the scenario design helped them very much to design the potential application look and feel. According to the designers, scenario-design helped them to conceptualize the final application. The designers promoted the scenario as a strength of the proposed framework. The discussion notes indicate that all stakeholders of the application are satisfied with the outcome “Good model to design mobile application, I’ll apply the UCD framework on my next project”.

  • Application for Tourism In this case study, a mobile application is developed especially for outdoors’ athletics. With this case study, we aimed to test the UCD framework idea creation and user study phases. The potential customer of the application was the small tourism companies in Finland. Through this application, the tourism companies were able to provide the outdoors activities maps as a web service to their potential users such as kayakers. Additionally, the portal is designed and developed for tourism companies to design content for their customers independently. The details of the application and the development process with usability test result was published at CSCEM2014 [23].

  1. 1.

    Usability and User Experience Assessments. The first field test happened by kayakers on the Baltic sea by Kayakers. The application usability and functional testing with kayakers had ensued that we re-design the initial product concept. The hi-fi prototype was upgraded to a new hi-fi Beta version based on the collected feedback, which was reassessed with users. The overall feedback that we received was good and optimistic, phrases such as, “Very clear with colored arrows pointing in the right direction”. Or “it’s simple and easy to follow”. In the third round of the evaluation, we conducted additional usability test with fifty potential users (n = 50) of four tourism companies (n = 4). The overall test results indicate that users were happy with the application performance despite the fact that several new features have proposed by users.

  2. 2.

    Usability and User Experience Assessments. This was not an educational application, as a result, the second assessment criteria was not directly applied. The main activity component tasks were that the users had to plan routes in advanced which done smoothly without difficulties. The admiration component also was implemented properly, the tourism companies were able to provide proper services to their customers.

  3. 3.

    Methodology Assessments. The overall result indicates that the UCD Framework increased the touch point from a business perspective. “It helped us to find new business needs”. The methodology helped very much to identify draw backs through hi-fi testing “prototyping was good it save a lot of error fixing cost in the productions” before developing the actual product concept. As a result, the developers and the companies shown special attention to this capability of the UCD Framework. The overall feedback on UCD was promising e.g. “regardless of the stakeholders’ divers interest in the mobile application development the methodology demonstrates that it covers all the stakeholders’ interest as the application fulfil the potential users’ needs, as well as the company’s interest”. The developers and designers found the methodology easy to use, and straightforward approach “simple, and easy to follow”. Directly jumping to design the prototype posed additional prototype design iterations in the end they realized that was not a right approach “I wish we use scenario first it could have saved us lots of development time”. The outcome of the application had satisfied all the stakeholders, despite the fact, that this process had taken time and extensive efforts to get the final concept “as a tourism company owner I am very happy with the outcomes”.

4 Discussion

UCD framework for m-learning application development helped to design and developed several case study applications for varies sectors. The nature of these applications mainly was educational, but the presentation styles were various e.g. game. We have conducted three different assessments to validate the framework 1. The application usability assessment 2. The application assessed based on educational component and finally 3. The framework assessment. The application usability assessments have carried in several iterations, such as in concept design, low-fidelity prototype, and product assessments. The framework assessment, however, carried out at the final stage of the product development.

  1. 1.

    Usability and User Experience Assessments. The usability assessments of these case studies revealed that in almost all cases, users were able to carry the predefined tasks. Those case studies, which were developed as a mobile learning application, we had predefined sets of tasks, so it was easy to statistically check the error rates and tasks performance rates. But, in game application we did not have any predefined tasks, but the users were asked to play the game and think aloud, the video analysis of the test sessions indicates that users were able to explore all the features and functionalities of the game. The result of these cases studies, indicates that the UCD framework was efficient enough to produce an application that satisfy the essential needs. Additionally the tasks performance rates indicate that UCD framework has resulted in application that easy to use by all users types e.g. novice and expert. The errors rates also identified in these case studies were almost zero, this demonstrates that the UCD framework in general is efficient enough to produce effective, efficient and error free application.

  2. 2.

    Usability and User Experience Assessments. The actual analysis of the educational components for each of these cases is huge, which is out of the scope of this paper. The measurement and acceptance criteria are well suited to the mobile learning application, but it is also valid criteria to assess other mobile application types e.g. game. The overall assessments of the UCD framework from the educational perspective indicates that the case study applications provided adjustable, delightfulness, reliable with satisfactions to educational related components such as presentation, activities, communications, and administration. In some cases, we were not able to fully assess all components, either because of the technology did not support or the prototype had lack of proper functionalities e.g. adjustability in the presentation component in java application. The UCD framework promotes the m-learning application design based on educational components. This has influenced that our UI design and the development, as a result, we gained positive feedback in our m-learning application assessment.

  3. 3.

    Methodology Assessments. The overall results of the case studies show that the stakeholders e.g. designers and developers were skeptical with the UCD framework at beginning of each project. This trend changed after the project progressed, we received this types of feedback mainly from developers who like to code as soon as they have the project idea e.g. in outdoor application we experienced that they developed many prototypes just because they ignored the scenario design. Having waiting to collect the requirements at the beginning were also considered as a waste of time, which were not however, a valid claim e.g. in java, mhealth and adaptive driving application we followed all the UCD framework phases, which took relatively shorter time to design and develop the application high fidelity, compared with the outdoor activity app. There were also recommendations for adding the design guidelines for m-learning application development based on UCD framework. This is also not feasible to the nature of each m-learning application is different, which the design must follow the context and the content of the potential application.

5 Conclusion and Future Work

We demonstrate that the UCD framework for m-learning application is an efficient methodology for designing and developing the m-learning application. The case studies were developed for various purposes and contexts, this indicates that UCD framework support various application development contexts. The assessments criteria demonstrated that UCD framework result robust application concept, which fulfill the essential needs. These case studies, however, were not assessed from pedagogical perspective, as a result, assessing these cases from the pedagogical perspective is important to see whether the application effects the learning process?