1 Introduction

Digital technologies can be used as therapeutic, educational, or accessibility tools [1]. This can include using technologies with a leisure function to facilitate participation and enhance quality of life. Members of the autistic community find technology-based supports particularly acceptable [2]. Subsequently, there has been an exponential increase in the number of technology-based supports for the autistic community [3,4,5]. In the past decade, technology-based supports for autism have extended beyond the use of simple desktop computers. They now involve robots [6,7,8], touch screen devices [9], speech-generating devices [10], virtual reality [11,12,13,14], tangibles [15], and wearables [16, 17]. Several systematic reviews highlight the intensity of research on supports for autistic people based on technologies such as virtual or augmented reality [18,19,20,21], mobile or wearable devices [22, 23], robots [24], computerized instruction [25, 26], and serious games [27, 28]. At the same time, parents/caregivers often worry about their autistic children (young people or adults) spending too much time on digital technology (e.g., games and social networks), expressing fears about obsessive behaviors or social isolation [29,30,31,32]. It is, therefore, pertinent for this group that technology-based supports are selected based upon solid evidence of appropriateness and that they are designed with autistic users in mind.

Although the results of studies evaluating technological supports vary in terms of the benefits for autistic people, the overall findings are positive. Technology-based supports often result in benefits such as increased motivation, decreased external signs of anxiety or distress, increased attention, and sometimes increased learning compared to traditional methods [33]. The literature also shows that technology-based supports are beneficial in the areas of social communication [34], emotion recognition [35,36,37,38] as well as academic skills acquisition and improvement [39, 40]. Technology, both in educational and home settings, is being used in a variety of supportive ways to increase independence, reduce anxiety, and create social opportunities for young autistic people [41,42,43,44,45]. With the advent of tablets and smartphones, digital technologies have become easier to use. They are accessible for a broader range of the autism spectrum, including very young children and individuals with lower reading and language abilities [31]. Studies have also reported participants' eagerness to engage with technology [46, 47] and high motivation to complete their work [48, 49].

Although a wide range of technology is available to support autistic people, evidence of its effectiveness is still a matter of debate [4, 50]. Despite almost five decades since the first studies on technology-based supports for autism [51, 52], and the intense research on this topic during the last fifteen years, technology-based supports are still perceived as "emerging" and lacking clinical validity (National Autism Center, www.nationalautismcenter.org). Evidence-based practice (EBP), when applied in healthcare, advocates for clinical decisions to be guided by the integration of best available research, clinical expertise, patient values, circumstances, and healthcare system policies [53,54,55]. Evidence of efficacy is based on relevant scientific methodologies, with meta-analyses and randomized controlled trials (RCT) considered "gold standard" methods [56]. Given the importance of non-clinical supports and educational approaches in autism support practices and a dearth of RCTs in this field, some argue in favor of a more inclusive range of methodologies to be considered, such as single-case designs [57, 58]. Ramdoss et al. [34] systematically reviewed the use of computer-based communication interventions for autistic children for their certainty of evidence. Conclusive evidence was found for only two studies. Two others had strong but not conclusive evidence (categorized by the authors as "preponderance level" of evidence), and six had suggestive evidence. A meta-analysis conducted by Grynszpan et al. [4] identified only 22 pre-post group design studies out of 379. Their findings indicated that digital supports could be effective for autism. Still, only a very small proportion of the evidence from group-design studies (6%) provided sufficient evidence to make an informed decision whether to use digital support.

Although the field of technology for autism largely lacks evidence from RCT, there are signs that this gold-standard methodology is becoming more commonplace [8, 36, 37, 59,60,61,62,63,64,65,66,67,68]. For instance, three RCT studies [59, 60, 62] report computer-assisted learning to teach attention to and recognition of facial cues. An RCT [63] compared a conventional (low-tech) version of the Picture Exchange Communication System [69] with an open-source digital (high-tech) version (called FastTalker). Another RCT [67] found significant improvements in socialization for autistic participants receiving intervention with Superpower Glass, an artificial intelligence-driven wearable behavioral intervention. However, advances in the quality of evidence are somewhat undermined by the observation that technology-based supports having robust evidence of effectiveness are rarely commercially available [70], while commercially available therapeutic technologies are infrequently evaluated in research [34, 71]. For example, from an analysis of 695 commercially available apps for autism, only 5 (< 1%) were found to be supported by direct evidence [50].

EBP focuses on evaluating the effectiveness of a finished product and is less concerned with its design process. Numerous EBP-inspired studies have assessed the effect of using technological supports on outcome measures. However, they have rarely analyzed how the design process was informed by evidence from autistic users [72]. Technology-based support requires not only a robust evidence base but also the involvement of the autistic community in the research and development process [30, 73,74,75] to ensure the needs of the community are met. Several studies have involved autistic end-users with diverse competency levels at different stages of the design process, which has benefitted the development of assistive technologies [76]. These methods are referred to as user-centered design (UCD)' [54]. UCD is an iterative design approach based on an explicit understanding of users, tasks, and environments, driven by user evaluation and multidisciplinary perspectives. Involving potential users in the design process and testing prototypes is a hallmark of UCD [77]. Existing UCD models for health technologies advocate collecting evidence from end-users through various means, such as field observation, interviews, focus groups, literature reviews (including grey literature), surveys, demographic data, web analytics, prototype testing, and participation in the design process [78,79,80]. To the authors' best knowledge, there are currently no scaling instruments to evaluate the level of user input and other sources of evidence contributing to the design process as it relates to technology in autism. The goal of the present study was to create such a scale, primarily intended for autism specialists to apply without needing specific expertise in technological design.

The goal of the present study was to develop and conduct a systematic literature review to test a rating instrument, named the User-based Information in Designing Support (UIDS), that enables systematic evaluation of user input and other sources of information about end-users informing design processes reported in the technological support literature. This instrument was meant to supplement a newly developed framework of evidence-based practice for technological supports for autistic people [32]. This framework was the output of a Delphi study, a methodology used to integrate opinions from diverse experts [81]. The study involved 27 panel members, including autistic people, families, autism-related professionals, and researchers who were asked to list and rate possible sources of evidence on technological supports. Outcomes revealed that the panel valued academic research, expertise of design teams, and inclusion of autistic people in design teams. Their conclusions thus suggested a need for a standardized instrument to evaluate sources of evidence used to guide the design process in studies on technological supports for autism.

2 Methods

2.1 Material: the User-based Information in Designing Support (UIDS) scale

The User-based Information in Designing Support (UIDS) scale was developed to assess the degree to which the design process of a piece of technology meant for support was informed about autistic users. It integrates ISO (International Organization for Standardization, www.iso.org) norms for human–computer interaction (ISO 9241-210:2010) and Druin's [82] taxonomy of user participation. As UCD requires understanding users' incentives, capacities, needs, and goals, we included criteria in the UIDS scale that evaluated the extent to which end-users were involved in iterative cycles of technological development. In addition, in compliance with Evidence-Based Software Engineering (EBSE) [54], the UIDS scale gauges whether the design process involved literature reviews. To translate these requirements in the context of a rating instrument for examining research reports, we created a scoring system that considers whether the developer of the technology conducted a systematic literature review, referred to the literature or other relevant sources, made a replicable design with end-users or one with proxies (e.g., parents, autism professionals), a non-replicable design with end-users or with proxies, and whether they observed end-users in their environment (see Table 1). The items' weights were determined based on a consensus between authors. Most of them followed an ordinal scale. Some were given half points if they were considered of lesser quality. To the best of the authors' knowledge, no other similar instrument could be used as a reference base to assess the quality of information about users in studies reporting on the design process of technology-based support.

Table 1 Point system for the design process and literature review
Full size table

2.1.1 Scoring of the UIDS scale


Three criteria were scored: design process, literature review, and user’s role (Tables 1, 2). They were presented in two separate tables for the sake of clarity.

Table 2 Point system for the users’ role
Full size table

The terms mentioned above were defined as follows:

Replicable design: Enough details are provided for another team to be able to run the same study.

End-users: Eason [83] defined system users as primary (people using the system), secondary (occasional users, or users through an intermediary), and tertiary (people affected by the usage or people making decisions about its purchase). Our study focused on primary users, whom we refer to as “end-users,” meaning the autistic people for whom the system was designed.

Proxies: People affiliated with the target population, such as parents or autism professionals. Typically developing people of the same developmental age as end-users can also be counted as proxies if they are involved in the design process to collect data relevant to the end-users due to similarities in abilities, incentives, goals, or needs. Some UCD methods require verbal reports from end-users testing the technology [84]. This can be an issue when designing technology for minimally verbal individuals, in which case resorting to proxies may be an acceptable workaround [74]. However, a non-replicable design with proxies associates with only a half-point. It is considered less than the average quality expected from a standard design process from a UCD perspective.


Systematic literature review: Overview of primary research on a focused question following a systematic research strategy.


Literature review: Narrative review of a topic.

Reference to literature or official sources: Limited references, which are not sufficient to present a review on a topic, or references to grey literature, e.g., Government technical reports, citations of a few academic sources or material from relevant national organizations such as NAS (National Autistic Society, UK) and CRA (Centres Ressources Autisme, France). This label was associated with only a half-point as it was considered less than the average quality expected from a standard scientific literature review.

For the design procedure criterion, points could be added. For example, if a study involved users' observations and a non-replicable design with proxies, it would be given 1 + 0.5 = 1.5 points. However, no more than 2 points could be awarded in this part. For the literature review criterion, a study could only receive one of these ratings: 0.5, 1, or 2.

These ratings were combined with a rating for the user’s role in the design process. The definition of user’s roles was based on Druin's taxonomy [82, 85] (see Table 2). The UIDS scale assessed the extent to which user needs and goals were prioritized rather than business/institution goals. It also considered the time and number of occasions users were consulted. The user’s role criterion focused on end-users and not proxies, as we considered that the actual end-user needed to be consulted as she/he was to play a role in the technological design. Studies in which end-users were genuine design partners, in other words, "participatory designs,” received the highest scores. Attributing the highest score to participatory designs is consistent with the expert panel consensus yielded by the Delphi study reported in Zervogianni et al. [32]. Points 1–4 were awarded only if end-users (i.e., autistic people) were involved; otherwise, a 0 score was given (Table 2).

Any study could get a minimum of 0.5 and a maximum of 8 points (2 points for Design procedure, 2 points for Literature review, and 4 points for User’s role). UIDS quality was labelled "low" when scores were between "0.5 and 2.5", "medium" for scores between "3 and 5.5," and "high" for scores between "6 and 8". A short scoring sheet is available in the joint online supplementary material (S1 UIDS scoring sheet).

2.2 Literature search and selection procedure

Validity testing of the instrument described above-required material from relevant literature. A systematic literature review was organized following PRISMA's guidelines for the identification, screening, eligibility, and inclusion of papers (Fig. 1). The PRISMA statement is a set of guidelines for the minimum collection of items reported in systematic reviews and meta-analyses on healthcare supports [86].

Fig. 1
figure 1

Review phases adapted from PRISMA guidelines

Full size image

2.2.1 Identification of relevant literature

Our work aimed at screening digital interactive technology for the autistic population. The following keywords pertaining to technology-based supports were used: Computer, Mobile, Smartphone, iPad, Tablet, Virtual, Robot, Haptic, Wearable, and Wii. They were combined with keywords referring to autism conditions, namely autism and Asperger. Although the latter diagnosis is no longer made [87], it has been used extensively during the last decades.

Scientific databases of different foci were searched to accommodate the multidisciplinary nature of the field. More specifically, records were sought in the following scientific databases, covering the domains of education, medicine, psychology, and computer science: ACM (Association for Computing Machinery) Digital Library, IEEE (Institute of Electrical and Electronics Engineers) Xplore Digital Library, ERIC (Education Resources Information Center), PsycINFO, PubMed, and Web of Science.

Relevant papers were identified based on the following fields: Title, abstract, and meta-data (language, publication year, location, number of pages). As technology rapidly becomes outdated, the search focused on recent literature. The date range for the records search was set between 2010 and 2018. The motivation for choosing 2010 as the start year was that it coincided with the launch of the iPad, a platform massively used in healthcare and educational contexts due to its accessibility and portability and the popularity of its manufacturer's products [31, 50]. The database search was conducted in March 2018. The total number of records returned through the process was 5,496. The citation files of each database were exported, and a software script (in Microsoft Visual Studio) was developed to process the files and remove duplicates. The number of records that remained after duplicate removal was 2716.

2.3 Screening

The inclusion and exclusion criteria listed below were applied to the title, abstract, and specific meta-data (language, year of publication, number of pages) in the screening process.

Inclusion criteria:

  • The technology-based support was employed to result in a cognitive and/or behavioral change. Assistive technologies were considered resulting in cognitive or behavioral changes;

  • The technology used was interactive. It involved interactions with the user (e.g., playing computer games) compared to technology where users are passive (e.g., watching television). This criterion was the same as that used in the meta-analysis by Grynszpan et al. [4];

  • Autistic individuals would be end-users of the technology. Technology meant to train or support practitioners or caregivers was discarded. In cases where a study targeted more than one population, including autism, at least 50% of the participants had to be autistic, or the support could not target more than one condition in addition to autism;

  • The paper was written in English.

Exclusion criteria:

  • Technology is only used for clinical evaluation, e.g., screening, assessment, diagnosis, monitoring tools, and fundamental research using technology;

  • Literature reviews;

  • Conference papers less than two pages long.

Two independent reviewers screened the titles, abstracts, and meta-data (language, year of publication, number of pages) of the 2716 records yielded by the literature search. They applied the above-mentioned inclusion and exclusion criteria to filter out irrelevant papers. The lists of papers they each selected were then compared. In case of discrepancy between them, the decision to include or exclude of a paper was made through consensus by consulting a third reviewer. The screening yielded 792 records that met our inclusion criteria. Our main goal was to evaluate the rating scale in a representative sample of individual research reports (i.e., we did not seek to apply it to the entire autism and technology literature). We opted for a sample of sufficient size to assume a normal distribution. From the 792 eligible articles, a random sample of 30% (n = 237) was considered appropriate to evaluate the scale. Selected articles were read in full, and the inclusion and exclusion criteria reapplied to the full text. Twenty-six records were excluded at this stage. The final sample was 211 papers (references can be found in online supplementary material: S2 References of scored papers).

2.4 Reliability and validity testing procedure

Two independent raters scored the sample of 211 papers (see the supplementary material: S3 All UIDS scored papers). The two raters were the persons who had screened the initial records yielded by the literature search to apply inclusion/exclusion criteria. To assess the scale's reliability, a random sub-sample (n = 78), amounting to a third of the included papers (after sampling, before full reading), was scored by both raters and served as a test dataset to determine the level of consensus. We used the intra-class correlation coefficient (ICC), an index used in inter-rater reliability analyses [88, 89].

We also assessed the validity of our UIDS scale. As there are no scales for measuring the quality of information about users guiding the design process of technology-based supports, we could not compare this new scale with an equivalent rating instrument. Instead, we relied on the judgment of an independent researcher trained in user-centered design. This researcher had three years of experience in evaluating human–computer interaction (HCI) and completed a Ph.D., which was awarded soon after the end of this task. Ten papers given a low score on the UIDS scale (0–2.5) and 10 papers given a high score on the UIDS scale (6–8) were selected. The independent researcher was asked to categorize each of these papers as having either high or low user-centered quality. We then evaluated if the researcher's classification matched the grouping derived from UIDS scores.

Finally, we computed descriptive statistics to characterize the sample of 211 papers scored with the UIDS scale for illustrative purposes. Means with confidence intervals were estimated from the sample to evaluate the percentage of studies falling under each scoring label (low, medium, high) of the scale. We also categorized the studies of the sample according to the type of technology they employed.

3 Results

As explained above, a sub-sample of 78 studies was randomly selected to compute ICC between scores given by the two raters. Those scores can be found in the joint online supplementary material (S4 UIDS Ratings for Intra-Class Correlation (ICC)). Inter-rater reliability measured with the ICC for the UIDS scale was 0.93, 95%CI [0.87–0.96]. In addition, regarding the validity assessment of the UIDS scale, the independent researcher in user-centered design agreed 100% of the time with classifications derived from the UIDS scale. As there was no variability between the categorizations yielded by the scale and the independent researcher, it was deemed unnecessary to resort to any further statistical analysis. Indeed, the probability of obtaining such a perfect match by chance is lower than 10–5.

Table 3 displays descriptive statistics for the random sample of 211 studies that were thoroughly reviewed. The mean score on the UIDS scale was 3.57 ± 1.50, 95%CI [3.37–3.77], with a range of possible values between 0.5 and 8.

Table 3 Number of papers with their corresponding rating on the UIDS scale (papers’ references can be found in online supplementary material: S2 References of scored papers)
Full size table

Table 4 presents the type of technologies featured in the sample of 211 scored studies. Mobile devices (tablets and smartphones), robots, desktop computers, and virtual or augmented reality were the most frequent technologies. Less frequent technologies included were smart glasses (e.g., Google Glass [90]), wearable interfaces (e.g., tactile sleeve [91]), tangible user interfaces (e.g., soft haptic toys [92]), and Shared Active Surface (e.g., multi-users digital tabletop [93]).

Table 4 Number of papers categorized according to the type of technology employed (papers’ references can be found in online supplementary material: S2 References of scored papers)
Full size table

4 Discussion

This was the first study to develop an instrument to evaluate information about the users guiding the design process of technology-based supports for autism. The inter-rater agreement for the UIDS scale was in the range of excellent values [88]. Additionally, there was complete agreement between scores obtained on this scale and how an independent researcher trained in user-centered design classified studies as high or low quality. These results are good indications of the relevance and applicability of this scale to evaluate the evidence on which technological design processes of supports for autism are based. Descriptive statistics revealed that the reviewed studies covered a large diversity of technology. As the UIDS scale was not technology-dependent, this diversity was not an issue when scoring papers. Design and development processes are often defined by business requirements, availability of technical means, and generic market research and thus not supported by insights as to what users actually do when using the product. By contrast, user involvement throughout the design and development process can reduce the risk of launching an unsuccessful or unusable product thanks to better-informed design decisions. User-centered design has been linked to more effective, efficient, and safer products being developed, contributing simultaneously to user-acceptance [84, 94]. Additionally, it has been argued that active involvement can improve users' understanding of a technology's potential and help them realize how the tasks they are required to perform may change [95]. A user-centered design approach including autistic users in its different phases gives autistic people opportunities to make their voices heard and control the support they receive.

Most studies received a medium score on the UIDS scale in the sample of papers we reviewed. None of the studies received the maximum possible score. Participatory designs which gain the most points in the implementation phase were rare, potentially due to the cost both in money and time to have participants involved in all stages of analysis and implementation. Lack of awareness of the importance of autistic users shaping the design of a product targeting them may be another factor. The UIDS scale requires evaluating the extent and quality of the literature review performed during a study. The assessed studies often included substantial and well-researched reviews on technology, but did not have sufficient information on autism and approaches used in supporting autistic people. They were usually focused on general technological advancements or on the mechanics of specific algorithms without attempting to scope the potential benefit for the autistic community. This tended to lower their score on the UIDS scale. Such studies merely assessed whether autistic people were willing or able to use a specific piece of technology that was pre-existing, like virtual reality helmets, or developed by the research team, for instance, robotic heads.

When deciding which support is the most appropriate for a person, various factors are taken into consideration, evidence being only one of them [53]. The weight of evidence in the decision process is modulated by other factors like societal values and personal preferences. The level of required evidence depends on the personal goal of using a specific piece of technology. For instance, digital products can be used for the element of fun, in which case stringent evidence criteria may not be critical. In such circumstances, though, ease of use and adequacy with the user's requirements are pivotal to the decision-making process. Assessment of the evidence base of technological support should thus be accompanied by evaluating how users were involved in the technological design process. Additionally, the rapid development of technology-based supports often prevents careful evaluation using long RCTs due to a mismatch between commercial and academic progress [96]. When this is the case, the UIDS scale could be used to evaluate the extent to which the design process was informed by a literature review of autism research and by autistic users' experience.

The rationale endorsed by the UIDS scale favors autistic users' involvement in the design process of technological supports. User participation in health and education digital support design remains limited to providing feedback on designers' ideas [97, 98]. Efforts to involve users as co-designers are often challenging due to the targeted users' abstraction ability [99]. Despite the challenges it poses, with appropriate adjustments, participatory design in the area of technology-based supports can enable digital empowerment and social inclusion of autistic individuals [73]. Additionally, the UIDS scale with minor modifications could be applied to conditions other than autism and extend to research areas such as mental health and education.

The present study has several limitations that should be noted. First, the UIDS scale was not applied to all eligible papers retrieved from the literature search. This was due to the vast number of relevant articles (n = 792) compared to our limited resources. A sample of randomly selected papers (n = 211) was used to test the scale to tackle this problem. It is, therefore, possible that the present study failed to identify specific issues in papers that were not included in the sample. Secondly, the application of user-centered design with autistic participants of younger age or with severe learning difficulties/intellectual disorders may not be so straightforward. For those cases, involving participants as designers can be challenging, and thus decisions often rely on caregivers. These additional difficulties may have lowered the scores on the UIDS scale for studies focusing on such participants.

5 Implications

The present study designed and tested the UIDS scale, which assesses information about users guiding the technological design process of digital supports for autistic users. This scale demonstrated high reliability and validity. As with other scales, the UIDS scale should be used according to its stated procedures, scoring systems, and concepts considered within the framework of the present article. The scaling instrument presented here should help address the critical challenge posed by the current proliferation of studies on technology-based supports for autistic users. The scale could prove useful to national and international organizations responsible for disseminating knowledge about autism, especially those that manage databases of technology-based supports and need a standardized way to display information about the quality of the design of those supports. Moreover, it should be of interest to the entire autism community (including researchers in the field, autistic people, their families, and related professionals) when deciding what digital supports to purchase.