Keywords

1 Introduction

Information and Communication Technologies (ICTs) are becoming very popular among people with cognitive disabilities and Autism Spectrum Disorder (ASD) [1, 2]. Besides, they are excellent tools to help them in their daily life activities. However, the number of research studies related to this topic is still limited [3]. In this sense, Holzinger et al. [4] discussed the tolerance of individuals to introduce new devices in their lives and discovered that the acceptance. The authors concluded that acceptance is related to previous exposure to technology. Therefore, popular technologies such as smartphones and tablets remain as optimal choices for this kind of studies, since the previous or future exposure is guaranteed by the society. In fact, choosing a device of high acceptance level decreases the risk to become an abandoned device through time, as studies pointed out [5].

Touch devices are making computing more accessible for a wide variety of population. The simplicity and benefits of the touch interaction and their portability have lowered the barriers for interacting with ICTs [6]. These devices allow users to interact through natural gestures and manipulate the content directly, allowing them to express themselves in a more physical way, generating better comprehension [7]. This way of interacting help users to focus on the content while they enjoy doing the activities, which helps them acquire a deeper knowledge of the topic presented [8, 9].

Two particular areas of interest related to assistive technologies for cognitive disabilities are job insertion [10] and education [11]. Technologies in general, and mobile technologies in particular, seem to fit perfectly in the formation and training process of both. Traditionally, the necessary skills to promote the job insertion of these people are acquired by repeating the typical tasks several times. During these sessions, educators usually provide oral instructions and help but also paper manuals with the list of instructions [12]. In some cases, these manuals are enriched with pictograms or diagrams. This approach presents some challenges such as difficulties to relocate when they get lost or to look for a certain instruction [13]. Moreover, during the learning process caregivers have to monitor their performance, which is time and human resources consuming. To solve this issue, we chose smartphones as a platform to develop an adapted system to support activities of daily living, AssisT-Task, an application that provides pervasive assistance thanks to the use of smartphones in combination with QR codes.

Regarding the education area, the use of tablets for people with special needs has produced lots of excitement among the education community and their preference for touchscreens has long been documented [14]. However, this excitement has brought developers to build hundreds of apps to help people with cognitive disabilities, making it difficult for teachers to identify the useful ones [15]. Moreover, sometimes teachers do not achieve their educational objectives since they are not able to provide suitable multimedia content [16], mostly due to the lack of flexibility of the available tools. Therefore, we also designed DEDOS-Project, a system to design and play educational activities for multiple platforms, such as personal computers, digital whiteboards, tablets and multitouch tabletops.

This paper is organized as follows: after this introduction, we describe our two approaches in detail. Then, we present our experiences in the classroom and finish with the conclusions extracted from them.

2 Mobile Technologies

2.1 AssisT-Task

AssisT-Task is a mobile system that provides adapted and pervasive guidance to do sequential tasks. Thanks to the use of smartphones in combination with QR Codes, users can receive assistance at any time and place. Moreover, this help is adapted to the user, her needs and context. To do that, caregivers have to design the sequence of steps that composes the task. Then, all the necessary information to identify the task is coded into a visual mark (the QR Code) that should be printed and put near the place where the task should be done. This way, users only have to start the application and point with their phones to the tag to receive the assistance.

The guidance is provided by means of sequential instructions supplemented with photographs. In order to ease the process, the system can be configured to automatically read the instruction aloud. This way, users receive the information by multiple modes and channels. Besides, AssisT-Task offers an alarm mechanism to prevent users from being blocked. In case users do not interact with the smartphone for a period (configurable), the system automatically reads aloud the instruction again to warn them and recover from the block. Finally, it records a detailed registry so users’ performance can be analyzed afterwards.

Caregivers can create, edit and adapt the sequences for their users. To do that, AssisT-Task counts with a PC authoring tool. This software has been developed considering users’ needs (no necessary technical profiles), so its functioning is very similar to standard office software. Figure 1 shows a screenshot of the interface. It is divided in two parts: on the left side, there is a navigation bar including all the available tasks in the system. The main frame contains all the steps that composes a task. In this example, the sequence corresponds to the “make toasts” task. As can be seen, it is composed of 12 steps, each one contains a text (the action) and an image. In this case, some of the images are pictograms while others are actual photographs. Depending on users’ needs and preferences, caregivers may decide to include one or another or, even more, to edit the images (with any external image editor) to highlight a detail or the most important elements of the picture.

Fig. 1.
figure 1

AssisT-Task authoring tool. In the example the “Make toasts” tasks is shown

Full size image

This software also allows caregivers to adapt the sequence to users’ needs by removing the amount of help provided. They can choose any step or steps from the original sequence and remove them for a particular user. This way, the assistance is adapted to users’ progress.

Both the interface and the interaction of the client have been developed in collaboration with therapists and educators. The main objective was to design a simple interface, easy to understand. Therefore, we designed the interfaces shown in Fig. 2(a) Presents the user selection screen. In order to provide a personalized experience, we included it so users could look for themselves using the arrows. Thanks to the names and photographs displayed, the task is easier, according to experts’ opinion. Once the user is selected, a QR capturing screen is displayed. It shows an actual view from the camera. When a QR Code is detected, it is captured and decoded automatically. This way, all the available information of the task is loaded and the sequence of steps starts. In Fig. 2(b) a screenshot of a step is presented. On the top of the screen there is a title bar. The three dots on the right access the configuration options. Right under it, we included the instruction. As can be seen the font is big and clear enough to be easily read. Regarding the text, task designers are in charge of making it simple, direct and easy to understand. The image is under the text. As can be seen, it takes most of the available space of the screen. This way, users may not miss any detail. Finally, at the bottom of the screen there are two buttons to navigate through the sequence. They have different colors, colorblind-proof and with a subtle intention: the next button is green, as a metaphor of positive reinforcement. Pressing it means advancing within the whole task, whereas the previous button is yellow, meaning a neutral connotation – it is not negative to go back and retry if you feel lost but you have not advanced in the process. The arrow symbols printed on them gives the meaning of both buttons. Educators explained that users tend to respond well to arrow indications representing directional messages.

Fig. 2.
figure 2

Screenshots of AssisT-Task mobile client. (a) Presents the user selection screen. (b) Shows a step interface (Color figure online)

Full size image

2.2 DEDOS

The main objective of DEDOS-Project was to provide educators with the suitable tools to design and play educational and collaborative activities for any student. It is composed of two different tools: DEDOS-Editor and DEDOS-Player. On the other hand, DEDOS-Editor is an authoring tool designed to allow teachers to create educational activities in a simple and flexible manner. Our aim was to leverage the design potential of the teaching community by empowering it to create, adapt and modify educational activities. This process can be done in traditional computers and novel touch devices, including collaborative tabletops. Moreover, the application as specifically designed to be used by users with little technological expertise and disregarding the device where the activities will be performed (DEDOS-Player will adapt the content designed by the teacher to the device where the students will solve the activities).

With this idea in mind, we decided to design DEDOS-Editor as a card-based metaphor where each activity is a card-based game. It offers four types of simple activities: single and multiple choice, pair-matching, point connection and mathematics, but they can be combined to design more complex exercises.

The elements used to create the activity are dragged from the toolbar and dropped into the edition area. Designers can use all the elements at hand to create the activity they want and by reusing concepts, elements and operations to create multiple educative activities we favor the users’ learning curve, making them more confident on their mastery and making it easier to learn further concepts. Finally, in order to provide a familiar interface to teachers, we based DEDOS-Editor interface on standard office software, since most teachers know and use it in regular basis. Figure 3 shows the user interface of DEDOS-Editor.

Fig. 3.
figure 3

User interface of DEDOS-Editor

Full size image

Once the educational project is designed, the students can solve it using DEDOS-Player. As can be seen in Fig. 4, the player runs in multiple devices, such as digital blackboards (left) and tablets (right). Before starting the activities, teachers can configure it depending on the educational objectives they want to achieve and the students’ needs. In addition, DEDOS-Player allows, (a) to configure the number of students that will perform the project, (b) how the feedback is going to be given, (c) if students will solve the activities by turns or all at the same time, (d) how many students should answer the activity, (e) if students must answer correctly the activities and, (f) if they have to give the same answer.

Fig. 4.
figure 4

DEDOS-Player example

Full size image

3 Experiences in the Classroom

We strongly believe that including mobile technologies in job training and special education will greatly help them to acquire important competences that will improve their autonomy. In order to test how technology influences students in the both aforementioned main areas, we carried out two trials where mobile technologies were used to help in the learning processes of job related and educational skills.

3.1 AssisT-Task

The AssisT-Task educational experience involved 10 people, aged between 16 and 19 years old, with cognitive disabilities and ASD. They were students of a labor-training program at a special education center. In order to evaluate the impact of mobile technologies as a catalyst in their formation process, we included a smartphone loaded with AssisT-Task as part of the materials of the course.

In order to develop an interesting and useful activity (in regards to the educational process), educators suggested working on the lamination task. As we said before, these tasks are usually trained by means of oral instructions or cards with pictograms. Although many users are able to follow the instructions, they have to be continuously supervised by the educator, which is time and human resources consuming. Therefore, we designed the “lamination” task. It consisted of a set of six instructions supplemented with actual photographs of the process. During the experiment, participants only received the assistance of the smartphone and indirect supervision from their educator. In every session, the educator took notes and registered if the user finished the activity correctly or not.

The study consisted of 90 sessions and took approximately two months and each user did the task 9 times. From the records (both the smartphone and educator’s notes) we found that all of the participants did the activity correctly during the experiment. Therefore, we have a 100 % completion rate. This is a very important factor since the educational process tries to make them capable to do the tasks and, afterwards, work on it to do it in the less possible time. In this sense, we also measured the completion time. In Fig. 5 we have plotted the average time needed to do the task for every session. As can be seen, there is a decreasing tendency, which means an improvement in users’ efficiency. According to educators, this time could not be reduced much more, since lamination machine takes a time to heat up and the lamination process is slow. These issues were also reflected on the registries: the steps corresponding to heat up and laminate took longer than the others in general. Moreover, in many cases the alarm sounded although users were not blocked. In order to provide a view of users’ progress regarding blocking situations, we cleaned up the registries by removing these alarms (false positives) and plotted the average number of alarms per session in Fig. 6. As can be seen, there is also a decreasing tendency, reaching the 0 value at the end of the experiment. Regarding session 4, and its larger standard deviation, educators reported that one of the users was very nervous and did not pay any attention. Therefore, the alarm triggered several times (10 times according to phone’s registries) and provoked this mislead.

Fig. 5.
figure 5

Average time needed per session to do the lamination task

Full size image
Fig. 6.
figure 6

Average number of alarms per session

Full size image

Finally, regarding the use of the smartphone, they did not present any serious trouble and the interaction was fluent. Moreover, it was motivating and made them more willing to participate in the activity.

3.2 DEDOS

The DEDOS experience involved 15 students (10 boys and 5 girls) from 12 to 20 years old with cognitive disabilities. Some of them were also diagnosed with Autism Spectrum Disorder (ASD). In addition, more than a half of the participants presented low reading skills. The study was carried out in three sessions during three weeks and each participant had to perform two educational projects using tablets. Each student completed both projects at least once a week during three weeks total. The duration of each session was not fixed beforehand in order to prevent participants to get nervous. In total, each activity took from 7 to 15 min.

The first project was composed of 24 activities about musical instruments. There were two types of questions: the name of an instrument or its type. The second educational project contained by 17 activities and it focused on questions related to Theory of Mind (ToM) and daily life activities. We can see an example of the activities designed for both projects in Fig. 7.

Fig. 7.
figure 7

Left: activity about musical instruments/Right: example activity about ToM

Full size image

In both projects, all the activities contained visual and textual information. Besides, texts were easily readable, so they would not affect student’s performance. The images provided were real musical instruments and real pictures in order to improve the transfer of knowledge from the educational activities to the real world. Finally, since some users had lower reading skills, we included pictograms in addition to the texts.

Regarding the sessions of this study, although we observed a general increase in the number of correct answers throughout the sessions, some participants’ performance dropped severely in the third session, which led us to analyze the data gathered using direct observation. During this last session, some students tried to solve the activities as quickly as possible since they wanted to perform better than their partners did. This fact added up to their low reading skills, resulted in many unwanted mistakes.

In order to analyze the answers given by the participants, we checked if there were any activities (A) that posed problems to them, so we could identify if they gave wrong answers in the same activity due to the activity itself or if the mistakes were different in each participant. This will help us to understand if there are any topics that were troublesome for the students or if there were any flaws in the design of the activities. Tables 1 and 2 show the number of wrong answers given for each activity and each session (the maximum number of mistakes per activity and session could be 15 since there were 15 participants). We found that the activities which caused more problems to the students where A8, A13 and A18 so we decided to analyze what could be wrong.

Table 1. Wrong answers in musical instruments’ project. Part 1
Full size table
Table 2. Wrong answers in the musical instruments’ project. Part 2
Full size table

In activity A8, we asked the name of an instrument called “bandurria”, which is in form, very similar to guitars. Since guitar was one of the answers, the participants mistook the instrument and most of them chose that answer, failing the activity. In A13 we asked for the type of instrument of the piano and as commonly happen when not having too much knowledge about musical instruments, most of the students chose the percussion option instead of string. The participants made the same mistake in A18, where they were asked again about the type of instrument the piano is. When asked again about this instrument in A24, they the right answer. The rest of errors where distributed among all the participants and they were only due to each participant knowledge of this topic, since as one chose the right answer in the first session, the chose it wrongly in the second session, and the other way around. This happened because they sometimes got distracted or they wanted to finish the project as fast as possible, not paying enough attention to the questions asked.

We also wanted to check if students had any problems with a specific activity of the second project. The data gathered is shown in Table 3. As expected, since the students performed well when working with this project, we could not find any specific activity that was difficult for them. However, we would like to note that the participants could not distinguish well between the pictograms that represent being scared and being surprised, which in some cases made them to finally choose the wrong answer (A2, A12). In addition, most of the students guessed the correct answer by discarding those answers that they knew for sure they were wrong. For instance, and using the example of Fig. 7 they guessed the correct answer because they knew that the face was not expressing neither happiness nor sadness.

Table 3. Wrong answers in emotion and daily life activities’ project
Full size table

4 Conclusions

From both of the studies, we can conclude that participants increased their performance as the sessions progressed. Combining visual and textual information help students with cognitive disabilities and ASD to focus on the contents presented, avoiding usability and accessibility issues, and therefore improving their learning process while they are having fun interacting with new technologies.

From the results of both experiences, we can conclude that the use of technology could have a positive influence in students when performing learning activities. Moreover, the use of technology motivates them and, combined with traditional methods and other learning sources, tutors, teachers and students can achieve better and more easily their objectives. The portability and accessibility provided by smartphones and tablets make them interesting tools to be used in learning environments for students with special needs. By promoting smooth and direct interaction with tactile devices, we facilitate the students to engage with the activities they have to solve, reducing their frustration and increasing their willingness to interact with a certain application or device.