1 Introduction

Autism Spectrum Disorder (ASD) is a group of disorders characterised by impairments in three core domains: social interaction, social communication and social imagination [25]. Social interaction deficit social interaction, includes lack of understanding others’ intentions, emotions, and mental states. For example, individuals with ASC have difficulties in analysing common social situations, are unable to react to them, or react with delay or in an unusual way. Communication deficits include delay in the development of language skills and sometimes total lack of spoken language. Social imagination is defined as the ability to detach from normal routines and look at them from a different perspective.

Various educational interventions have been designed which help reduce the core difficulties in individuals with ASD [12, 16, 23]. A particular example of an educational intervention which is widely used by practitioners and parents is Social Stories\(^\mathrm{TM}\), defined as short stories written following ten specific criteria, which describe social situations, social skills or concepts [18]. Social Stories\(^\mathrm{TM}\) address the social interaction and communication difficulties encountered by children with ASD. They are implemented to improve social understanding and celebrate the achievements of children with ASD.

One problem with Social Stories\(^\mathrm{TM}\) is that, even if the child reads the story, there is no guarantee that he/she has understood the concept [18]. Therefore it is important to test the child’s understanding through a comprehension check method. However, anecdotal evidence from Constantin revealed that, in spite of being aware of this, practitioners do not check Social Story\(^\mathrm{TM}\) comprehension in children with ASD because they find it difficult and time consuming, partly due to the lack of standard tools [6].

Recently, with the spread of mobile devices, there is an increased interest in exploring the potential of technology to support children with ASD by addressing their challenges, such as joint attention [2], social and communication skills [17] and behavioural problems [20]. Empirical data shows promise for the effectiveness of educational technology for children with autism [13, 15]. Moreover, most of these children have an attraction for technology [7]. This justifies researchers’ efforts in designing new applications to aid children with ASD.

In this project we explore how technology can be designed and developed to best support Social Story\(^\mathrm{TM}\) comprehension checking in children with ASD. In particular, we propose the S\(^{2}\)C\(^{2}\) tablet app for children with ASD, which practitioners can use with them within the school environment to check their Social Story\(^\mathrm{TM}\) comprehension.

The project is in its preliminary stages and we decided not to include children with autism before getting evidence that our solution may work for them. Therefore, in the initial stages that we report here, the design was informed by research literature (related to methods for checking Social Story\(^\mathrm{TM}\) comprehension in children with ASD, designing technology for children in general and for children with autism), and empirical studies with typically developing (TD) children and experts in the fields of ASD and Human Computer Interaction (HCI).

The results of the pilot evaluation were very promising, and suggest, based on the feedback received from TD children and experts in ASD and HCI, that the new technology could be used to aid in Social Story\(^\mathrm{TM}\) comprehension checking in children with ASD. However, caution must be taken in extending the results until design and evaluation studies involving children with ASD are conducted - which is what we are planning for the near future.

2 Designing the App

To design the app, we first reviewed the literature on methods used or recommended for checking Social Story\(^\mathrm{TM}\) comprehension in children with ASD. We identified the following: (1) asking “5W+H” questionsFootnote 1 after the story [1, 8]; (2) partial sentencesFootnote 2 [18], and (3) using computer-based games [3, 5]. Additionally, we reviewed the literature on design guidelines for technology for children (e.g. [4, 10]) and for children with autism (e.g. [9]).

Although including children with ASD in a software design process is possible through a number of methods and techniques, it is also very time and resource consuming. Moreover, TD children (bringing their age-related expertise) and HCI experts can contribute to the design of a technology-based tool with good usability for children, and experts in ASD can provide initial feedback about what is appropriate for children with ASD for checking Social Story\(^\mathrm{TM}\) comprehension. TD children were also successfully included in the past as proxies because of the difficulties to carry out studies with children with ASD [14, 19]. We therefore decided, for the first iteration of the design of the app, to involve TD children and experts in ASD and HCI. Our next step in this project, presented as future work, is to involve children with ASD in further design studies.

We ran a design workshop with TD children between the ages of 7 and 12. As research reveals that rewards embedded into educational technology can be motivational and enhance learning performance [11, 21], this workshop was also an opportunity to investigate what kinds of rewards children between the ages of 7 and 11 enjoy. An iterative process of developing a prototype was carried out and formative evaluation was conducted with four experts in ASD and HCI. The main aims of the formative evaluation were to improve the usability of the S\(^{2}\)C\(^{2}\) app, its suitability for checking Social Story\(^\mathrm{TM}\) comprehension and its appropriateness for children with ASD.

The final prototype included one example story (about sharing) and three methods to check its comprehension which the child could select based on their preferences (Fig. 1).

Fig. 1.
figure 1

Screenshots: [left] a question for checking comprehension and a correct answer; [middle] a partial sentence for checking comprehension; [right] a game for checking comprehension

Full size image

Once children read a story, they can check (with or without support from practitioners) their comprehension using their favourite method. In the question and answer method, three choice answers are provided. If they choose the incorrect answer, they are encouraged to try again. If they choose to try again, they are re-directed to the page with the question they got wrong, but this time the incorrect answer(s) they chose earlier is (are) not available. Similarly, for the partial sentences and for the game, if children choose an incorrect solution, they are encouraged to try again. Then, if they opt for trying again they are re-directed to the screen with the proposed solutions, but the incorrect solution(s) they had chosen is (are) not available. In this way, we avoided the alternative of providing children with negative feedback when they give wrong answers, which can lead to frustration and disappointment. However, our approach may have the disadvantage of leading to the correct answer if all the incorrect ones are eliminated, and this makes it unclear whether the child understood or not the concepts taught in the story. To compensate this, a log file was implemented to record the number of attempts taken to get the correct answer. Future development will provide this information in the practitioner interface (which was not our focus at this stage).

3 Evaluation

An evaluation study was conducted with sixteen TD children aged 7–12 and five experts in ASD and HCI. The main aim of the study was to understand to what extent the new technology supports children with ASD in checking Social Story\(^\mathrm{TM}\) comprehension. Additionally, the study aimed to evaluate whether the S\(^{2}\)C\(^{2}\) app is easy to use by the target population and to collect suggestions for further improvements of the app to make it appropriate for children with ASD.

The children were asked to independently read the story about sharing and then to complete the three comprehension check methods on an Android tablet. At the end, they were asked a series of questions related to their experience in using the app and preference with regards to the checking method. Each session lasted 30–40 min.

The experts used the Think Aloud protocol [22] while exploring the app individually. Then, they were asked a series of questions including their experience with the app, their perception about the suitability of the activities for checking Social Story\(^\mathrm{TM}\) comprehension, and appropriateness of the app for children with ASD. It is worth mentioning that all experts had some knowledge of Social Stories\(^\mathrm{TM}\) and two of them had experience in working with Social Stories\(^\mathrm{TM}\) with children with ASD. Each evaluation session with an expert lasted about 45 min.

All the sessions (both with children and experts) were video recorded (with the participants’ written consent) and the videos were transcribed.

4 Results and Preliminary Findings

To make sense of the data, we employed open coding and axial coding as described by Saldaña [24]. The results showed that both children and experts received the app positively.

4.1 General Experience

All the children answered that they enjoyed using the app and were enthusiastic during its exploration. Child C7 stated:

“I really like the comprehension check methods as they are really easy to understand”.

Out of the sixteen children, five inquired whether the app was available for download on the app stores, and stated that if it was they would want to download it.

All the children said they enjoyed the storyline of the game where they were collecting fruit for their virtual friend (e.g. an elephant), while also helping others on the way.

All the experts commented that they liked the app:

“Overall I like the application, it is well built and I think it has great potential.” (E3)

4.2 Ease of Use

None of the children had any issues with navigating through the app or understanding what the steps to accomplish their goals were.

All the experts found the app easy to use. Expert E5 commented:

“I have young children. They are typically developing, but I believe they would be able to use this application very easily”.

4.3 Appropriateness for Children with ASD

All the experts considered that the app is appropriate for the target users due to the language it uses and its simplicity:

“Using this application, there is nothing which would cause any issues to the target user, as there are no distracting features and the overall language used is simple enough”. (E1)

4.4 Suitability for Checking Social Story\(^\mathrm{TM}\) Comprehension

All the experts who evaluated the app considered that the three methods were very good in extracting the important information from the story and considered that the app fulfilled its purpose for checking comprehension. Expert E3 explained:

“The questions and answers and partial sentences extract the important information from the Social Story\(^{TM}\) and I believe it would be a good way for testing Social Story\(^{TM}\) comprehension”.

Also, expert E4 made the following comment:

“I really like how the questions asked in the application go from a general concept to more specific situations in regard to the story read [...]”.

4.5 Rewards

All the children and experts appreciated the use of rewards within the app. Child C13 stated:

“It was really fun answering the questions and collecting the stars as rewards”.

Another child, C5, also commented:

“I really liked how the fruit I collected in the game were then used to feed my character at the end”.

Similarly, expert E4 added:

“I also like how the game comprehension check method ends with the user sharing their fruit with the character they chose, as this tests the concept of the Social Story\(^{TM}\) I read which was about sharing”.

Child C16 suggested:

“I would like it if there was a big reward when all the comprehension check methods were completed”.

5 Conclusions and Future Work

This research explored how technology could aid in checking Social Story\(^\mathrm{TM}\) comprehension in children with ASD by involving TD children and experts in ASD and HCI in the design and evaluation of an app, S\(^{2}\)C\(^{2}\). Preliminary results showed that both children and experts had a positive experience with the S\(^{2}\)C\(^{2}\) app and perceived it as easy to use. Experts considered that it has potential to support children with ASD in checking Social Story\(^\mathrm{TM}\) comprehension. Our research brings the following contributions to the INTERACT community, particularly to researchers with interest in educational technology for ASD: (1) a design solution for facilitating Social Story\(^\mathrm{TM}\) comprehension checking in children with ASD informed by research literature and empirical data; (2) a tablet app implementation for this solution; (3) empirical evidence from the evaluation with TD children and experts in ASD and HCI that the app may aid in checking Social Story\(^\mathrm{TM}\) comprehension for children with ASD.

The next step is to implement the suggestions collected from TD children and experts (i.e. animations, sound effects and customisation of the game characters). Then, we will recruit children with ASD from schools for children with special needs from across Edinburgh, and by contacting the Lothian Autistic SocietyFootnote 3, for a second round of evaluation.

Alongside further improvements to the child interface to the S\(^{2}\)C\(^{2}\)app that these steps would result in, we would like to further extend this work by adding a practitioner interface for the school environment. Such an interface would allow practitioners to create Social Stories\(^\mathrm{TM}\) and set up comprehension check methods for them which are adapted to each child’s abilities and preferences, but also to the content of the story. For example, practitioners could use editors to prepare their own questions and answers using different fonts and colours, or to create comprehension check games using a child’s favourite characters and environments. Moreover, we could add more types of rewards that practitioners could choose from depending on the child. We would like to evaluate these ideas with practitioners from schools for children with special needs from across Edinburgh.

In the long term, this work could also be extended to include other target groups (e.g. children with learning disabilities or hearing impairments) and other educational interventions.