Keywords

1 Introduction

A challenge for parents with children with autism disorder is communication. The children cannot exchange information with their parents because of the delay in the development of language [1].

A first approach to fill the existing communication gap of children with autism was the Picture Exchange communication system. The system aimed to act as a bridge and connect the children with autism with other people. The main objective was to enhance the exchange information, by introducing a communication augmentation system [2].

Despite the efforts and the research in the last decades in augmented and alternative communication (AAC) systems there are still open issues. One of them is the expression of basic daily needs, required for an autonomous life that a person with autism usually cannot express, and therefore became dependent. To overcome this problem, we have developed an application which will help the users to express their feelings, so they can easily communicate using emoji. Although the target audience of the application is the child with autism, it can also be used by the elderly as well as speech impaired people who cannot express themselves for their needs.

In the proposed mobile application, we have used a picture (emoji) with their respective sounds. When the user touches the picture representing their needs or emotions, it will sound like the respective emotions or need so they will be confident about their needs and learn about their emotions and needs listening to the voice that was included in the application.

This paper is organized into seven sections: “Background”, describing the current state of the art, including Express Feelings using mobile application; “Design and Implementation”, explaining how the application works and its main features; “Experimental Design”, describing the methods used to test the application, the type of audience, the evaluation variables (effectiveness, efficiency and user motivation); “Apparatus”, describing the devices used to run the application; “Practical Results”, describing the test users feedback, and “Conclusion”, discussing the evaluation of the results obtained during the field testing.

2 Background

Autism is a Spectrum of disorders which is characterized by problem with social interaction and communication. People with such disorder shows respective and repetitive behaviors which makes them upset [3]. Usually it appears gradually, when the human development began to unfolds [4]. Autism was first recognized by Leo Kanner (1943) as a completely different disorder which was not described before under childhood psychosis. Later this disorder was named as Kanner’s Autism. Autistic children have poor social and language skills with less interest in people and they are rigid up to their daily routines, as described by Kanner in ‘Autistic disturbances in Affective Contact’ [5].

According to the Centers for Diseases Control and Prevention (CDC), about one in every fifty-nine child have autism spectrum disorder [6]. Autism, or Autism Spectrum Disorder, is a combination of condition characterized by challenges to speech and nonverbal communication, skills and repetitive behaviors.

Autistic children have problems regarding understanding and expressing their emotions [7]. So, expressing their emotions through mobile application can be good medium to communicate with their family members.

Due to which autistic children are caught in an impasse of social isolation, studies show that intervention focused on teaching social interaction, interpersonal problem solving, and affective knowledge has resulted in positive social interaction with peers by triggering their ability to share experiences, emotion and cast interest [8]. So, a platform that can simplify expressing of their emotion, and the connection between child and parents will act as a communication bridge and break the communication barrier for autistic children. Currently there are some apps, available in the apps’ stores that aim helping children to cope their behavior. A selection of these applications is presented in the next subsections:

2.1 Awesomely Autistic Test

The Awesomely autistic test application (Fig. 1) was developed by Harry Marcus Ltd, and aims to measure autistic traits. It is based on the Autism Quotient (AQ) created by the psychologist Simon Baron-Cohen and his team at Cambridge’s Autism Research Center.

Fig. 1.
figure 1

Autism Quotient (AQ) result in Awesomely autistic test

Full size image

The test contains 50 choice-based questions (with a Likert: slightly disagree, definitely disagree, definitely agree and slightly agree). Based on the answer it generates AQ and based on AQ it expresses the level of autistic in the test taker. Overall, this app provides a simple screening mechanism that could be used for further autism evaluation.

In Google play it has 5000 downloads and 4.0 stars by 30 persons [9]. It does not have iOS version. It has a file size of 5.24 MB in version 1.05.1 and latest updates was 22 Aug 2016.

2.2 Autism Therapy with MITA

The Autism therapy with MITA application (Fig. 2) provides visual puzzles that facilitates early development for children. It was developed by three Ivy League researchers to help children with autism to learn using pivotal response treatment (PRT) method. MITA stands for Mental Imaginary Therapy for Autism. According to the authors, the app provides puzzles that help the children to improve attention, visual skills, and languages. It includes several games, namely; arithmetic and auditory memory games that train auditory working memory; and combing toys for developing spatial direction choice.

Fig. 2.
figure 2

Game play of Autism therapy with MITA

Full size image

The application is organized by users’ age categories. Three age groups are defined: Preschool (2-5); Children (6-12); and Adolescents (1317). In Apple App Store, it has not enough rating and reviews. Google Play Store reports 5000 download with rating of 4.6 (687 voters) [10]. This app also got the prize of Best Autism app for 2017 on the healthiness’ list. The app was released on 27 Jul 2015 offered by ImagiRation LLC, Boston, MA 02135. The last update was on 25 Jun 2018, version is 4.0.5 with file size: 273 MB.

2.3 Autism Emotion

The Autism Emotion application (Fig. 3) shows a story line to choose an emotion from four given options. Based on the choice (happy, sad, proud and calm), it shows a photo and a sentence describing the photo. The consecutive five photos and in the last slide it has a music button to play the story about it. This app is developed by “modelmekids” that produces teaching tools for children with Autism, Asperger’s Syndrome, PDD-NOS, and nonverbal learning disorder.

Fig. 3.
figure 3

Game play of Autism Emotion

Full size image

This app is only compatible with iOS and has a rating of 3.7 stars with 30 votes [11]. First version of the app came to the store on August 10, 2012. The most recently updated was on Nov 16, 2017 version 4.0 and file size of 23.5 MB that is compatible with IOS 11.

2.4 Upcard

Autism children may not articulate word all the times, but they have a visual vocabulary. Based on this, Royden James created this app for his autistic son.

Upcard application (Fig. 4) uses the picture exchange communication system (PECS) [12], developed for people with cognitive, physical and communication disabilities. This app is made of “picture cards” that users can place in sequences to form simple statements and requests. Sign in and login validation is presented in the app. Through the app parents can make any plan, can keep timer for the child so on specific that message gets popup and on the other hand the autistic child can speak based on the cards or icon he selects from the permitted icons that parents has validated. It also has Flash card game based on the icons used before. So, the UI of the app is slick, but on usability of the app is hard for the children to understand. For the children, it is more time consuming and challenging at prior hand.

Fig. 4.
figure 4

Gameplay of Upcard

Full size image

It has both iOS and android versions. In Apple App Store, it has not got enough rating and reviews. In Google Play Store it reports 1000 download and a rating of 3.9 stars with 8 votes [13]. In Google play the size of the app is 22 M with current version 0.7.3 which requires android 4.1 and up. In IOS, app size is 76.8 MB and requires IOS 9.0 and up.

2.5 Summary

The analysis of the applications allowed to identify their pros and cons, summarized in Table 1. We found the limitation of the existing apps are: difficulty and more time consuming in understanding the app structure are only designed for specific game propose.

Table 1. Overview of the current applications (scale from — to +++)
Full size table

The aforementioned app do not provide platform for children with autism to develop important social skills and express their feeling to their family and friends.

3 Design and Implementation

The study of the state of the art revealed the need for a general app to help autistic child express their feeling and speak through the app. The “Express Feelings” application was designed taking into account the following requirements:

  • Have an intuitive design.

  • Target demographic of people with Autism.

  • Clear feedback of user actions.

The user interface was designed to be used by children with Autism, so well-known icons derived from emoticons (or emojis) were used to give visual representation of the user’s emotion.

3.1 Implementation

The application was designed to work on Android devices with android version from Jelly Bean to Oreo without connecting to internet. To increase the user base, the app was made free and open source at GitHub [14].

3.2 Using the App

When the user starts the application, a short description of the app can be seen along with a set of instructions to use it.

After clicking “Get Started”, a screen to save password is shown as illustrated in Fig. 5. The role of password to keep the settings unchangeable by the autistic user in course of usage.

Fig. 5.
figure 5

Setting password for the for App

Full size image

On selecting a password, the user is allowed to select at most 5 contacts to send the text message by the autistic user as illustrated in Fig. 6 (a).

Fig. 6.
figure 6

(a) Page to select the Contacts (b) Grid of emotion icons (c) Dialog showing prompting the user to send the text.

Full size image

On selecting the contacts, the app is ready to use. The app shows two tabs titled with “Feelings” and “Send to”. The “Feelings” tab shows a grid of icons representing various emotions the user may feel to express with clear distinction between the icons using white spaces. Nine general feelings expressing emoji are represented in the tab for fast and reliable emotion expression.

The app consists of emoticon icons which represents certain emotion of the user which is shown in Fig. 6 (b). When the user taps on an emoticon icon, an audio representing the emotion is played. A dialog box appears which shows the details of the emoticon including a text to be sent to the listed contacts – Fig. 6 (c).

After clicking “Send”, the user can send a predefined message to the selected list of contacts. If “Cancel” is pressed, then the dialog closes and the app resumes to show the list of emotions.

When the user clicks on an icon, respective speech that indicates that respective feelings is generated and a dialog appears along with the which shows the visual representation of the emotion, textual form of the emotion and the list of contacts who will get the message.

After clicking “Send”, the user can send a predefined message to the selected list of contacts. If “Cancel” is pressed, then the dialog closes and the app resumes to show the list of emotions.

The “Send to” tab offers the ability to include or exclude previously selected users from getting the message.

A dotted vertical icon is shown at the top right corner of the application. Clicking this shows a menu to go to “Settings” of the app. Accessing “Settings” requires a password which was set by the user earlier. After setting correct password the user can edit contacts and password.

The State Chart Diagram, presented in Fig. 7, summarizes the application usage, describing all the states and important events.

Fig. 7.
figure 7

State chart diagram

Full size image

4 User Testing

With the help of user testing (UT), we study if the target group can use the proposed application in easy and effective way. The approach was a method of formative evaluation or think aloud as it serves as a “window on the soul”, letting users do all the tasks they will be doing in the future through our application.

During the UT, we looked for critical incidents i.e. moments that strongly affect usability, task performance (efficiency or error rate) and came to following points:

4.1 Methods

User test were performed in two phases: using an application in real time with people with autism; and questionnaire. In the first phase, while the user is using the application, data is collected by a member of the research team.

To test the performance of the user a questionnaire was performed by the caregiver, considering the communication barriers with people with autism.

4.2 Participants

Four participants were chosen from a child care with autism people. The people or caregiver were asked to fill the authorization form prior to the data collection. These kids were chosen by Special education teacher at SERC School (Special Education and Rehabilitation Center for Disabled Children) in Kathmandu, Nepal.

Participants characterization is presented in Table 2.

Table 2. Participants
Full size table

4.3 Experimental Design

The participants will be given mobile phone with them all the time for that day. And we asked the caregiver to take notice of their activities. Questionnaire is made for care giver as follows:

  • Q1: At what time did you get the SMS notification?

  • Q2: What was the response did you get?

  • Q3: Does the response match with the actual emotion?

  • Q4: How much was the kid satisfied after looking at the excitement level of the user?

The successful criteria of the system are defined if the SMS response is matched with the response that actually user wants to deliver after the care giver approach to the user just after getting SMS.

4.4 Apparatus

For the user tests, we used a Mobile phone (Samsung Galaxy Note 9 sm-g960f). We choose this phone because we believe it could be easier to see all the emotion pictures easily for the user to interact with app more easily.

4.5 Results and Discussion

The data collection before testing an application is compared with the data collection after the testing the application. Specially, the usability of an application was tested with autism people, but questioner is asking with the caregiver or their parents and sometime with the doctor and nurses. The performance of the application or the impact of application was evaluated according to the behavior changes by the autism people.

The data collected with this application is described in Table 3.

Table 3. Representing the result of Questionnaire
Full size table

Figure 8 shows the user satisfaction chart for using the application which was done when the care taker notices the emotion of the respective user after getting the SMS and dealing with the user afterward.

Fig. 8.
figure 8

Application satisfaction chart

Full size image

From the chart we can see that, Participant 1 gave 5 responses in which 50% of his response are very satisfying and other 50% was satisfactory. And 25% of the response of Participant P2 was very satisfying and 50% of the responses were satisfactory and remaining 25% was unsatisfactory. For Participant P3, 75% of the responses were satisfactory where 25% of the response was unsatisfactory. Whereas the 25% of the responses of Participant P4 was very satisfying and 50% of the responses were satisfactory and remaining 25% was unsatisfactory.

5 Conclusion

In this paper, we presented an android application for children with autism to express their emotions to their parents and relatives. The application allows speech output by selection an emotion in the mobile application. Emotions are chosen from a preset of emoji as they can represent emotions in a user-friendly way. Moreover, emoji are a common and well accepted icon representation of emotions.

The result indicated that this application can be implemented to all kind of user having autism. Their response showed that they are interested to use the app as it acts as a bridge between the people. Globally, the obtained result indicates all kind of user were successful to use the app. But the user with pervasive developmental disorder was more satisfied than the user having Classic autism and Asperger syndrome.

For future work, we will try to add more emotions that users can give so that their all the needs can be solved and put the Native speech instead of English when they click the emotion.