Keywords

1 Introduction

User participatory design (UP) is known as a design method in the area of computer based technology [1]. User participatory design is considered to be one of the best methods for understanding the needs of a target audience, however the participants’ ability to contribute is key to being successful in the process [1]. Therefore, there is a critical need to create high quality, well-designed user participatory methods to meet the needs of persons with all levels of autism. For many persons on the autism spectrum with severe language and cognitive disabilities, traditional forms of user input and participation are either extremely limited or impossible. And without their input, well-targeted designs may be equally limited with regard to their appropriateness. Therefore, there is a critical need to develop a method that allows designers to select appropriate user participatory processes or instruments for including all target audiences regardless of cognitive ability.

Similar to UP methods, direct observation and the recording [2] of behaviors are methods of data collection that are commonly used in autism to understand persons with autism spectrum disorders (ASD) and to model new behaviors. Therefore, these common strategies of data collection can be used for UP for persons with ASD. The following are examples of UP with persons with autism:

  • an anonymous on-line survey of qualitative open ended questions (Putnam and Chong) was given to parents, caregivers, and persons with autism. Open ended questions framed as “in a perfect world describe.” were used to identify trends of software and technology use for persons with autism and preferences in the design of new software or technology products for autism. The findings provided a preliminary understanding of user goals and concerns for the design of software and technologies. The study found no meaningful correlations between gender, verbal ability, diagnosis or age. However 19 percent (22 responders) responded that software be designed with fun as a goal. The study also reported that this disproportionately represented parents of children with autism over those with ADD-NOS and Aspergers [3].

  • In a 2009 study using participatory design with adolescents on the autism spectrum, Madsen et al. identify the issues of cognitive impairments, including memory problems and low reading ability; complications associated with atypical sensory processing; and fine and gross motor difficulties as impacting the ability of persons with autism to participate fully in user participatory design. And for those reasons the researchers chose to use an iterative participatory design process for their research [4].

  • In the blog, UX Matters, Zsombor Varnagy-Toth, identifies verbal difficulties with think-aloud protocols as difficult for persons with autism due to language and communication issues associated with non-verbalness and echolalia. Non-verbal communication was identified as being effective due to the more pronounced use of strong emotion or body language by some persons with autism. Specifically signs of excitement through repetitive behaviors, hand-flapping, rocking and vocalizing were associated with excitement or possible discomfort; signs of focus identified through complete silence and lack of physical movement; and signs of joy in success as interpreted through jumping and laughing out loud. As a conclusion, it is noted that usability testing is critical to the success of products and understanding the atypical responses is important to interpreting the usability test outcomes [5].

1.1 Hypothesis

There is a need for design research that identifies the best practices for collecting meaningful data for persons with autism from one or more protected human subject categories based on effective inclusion criteria strategies in ways that maintain a strict ethical code of conduct. Therefore, the following three areas should be addressed:

  1. 1.

    Research methods vary in their ability to be used by persons with differing cognitive levels in autism and familiar methods from other disciplines may be appropriated.

  2. 2.

    Data collections methods can be combined between quantitative and qualitative to form a richer set of information than either single method can produce for autism.

  3. 3.

    Robust strategies for inclusion criteria need to be part of the research methodology to categorize participants with autism in meaningful ways.

2 Methodology

Four methods of user participatory data collection are discussed: (1) an online survey completed by college students both with ASD and their typical peers; (2) a focus group survey of caregivers and teachers; (3) data collection via an on-site observation of persons with varying levels of ASD; (4) and a user participatory case study of two subjects one with ASD with severe language and cognitive disabilities and a neurologically typical peer. The following four research data collection methods will be discussed with regard to survey instruments, inclusion criteria for participants, implementation and outcomes working with persons with autism from a variety of age and cognitive ability levels.

Online Survey: An online survey was used with university students to compare course content and delivery preferences between neurologically typical students and students with high functioning autism. To be included in the study all participants had to have the ability to independently answer questions using the RAADS-R autism assessment tool as an online survey. The survey was administered in Fall 2014, at Iowa State University. It was sent via e-mail to 33,241 students ages 18-years and older for possible inclusion in this study. The students were from all eight colleges in the university and one academic degree-granting unit. Both undergraduate and graduate students were included in the survey. The responding students provided basic demographic information, completed the Ritvo Autism Aspergers Diagnostic Scale-Revised (RAADS-R) test questions, and answered questions with regard to preferences in course content delivery methods and course evaluation methods. Full approval for this survey was obtained from the Iowa State University Institutional Review Board (IRB). Of the students contacted, 653 responses (226 males, 420 females) were collected. The data was evaluated according to colleges, gender, and RAADS-R scores. A score of 65 and above on the RAADS-R test indicates the presence of ASD. For males who responded to this survey, the mean score on the RAADS-R was 74.8 and for females the mean score was 61.2. All RAADS-R scores across all nine academic units ranged from a low of 4 to a high of 215 with a standard deviation of 37.9 and a mode of 57 [6].

The ethical considerations for this online study were mainly focused on the possible stress of answering symptomatic questions and concerns that might arise in participants with regard to a possible autism condition. Participants were all over 18 years old and were given an informed consent statement with options to discontinue participation at any time. No significant risks or benefits to participants were identified and there were no significant ethical implications.

Focus Group: A Spring 2013 study using qualitative and quantitative data collection methods was given to parents, caregivers and teachers. It included 66 parents and 66 teachers through invitation and snowball effect. It measured their perceptions of the importance of social skills, communication and behavior for children with neurodevelopmental disorders. Each of these three areas as measured by scales were determined to be highly important to these children based on criteria outlined in the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) [7]. The DSM-5 uses three severity specifiers to describe the level of symptomology in the two domains of social communication and restricted/repetitive behaviors. These levels and domains were used to inform the content of this survey. The parent, caregiver and teacher survey that was developed used a series of 5 focus groups to determine the most relevant questions for the development of the mixed method online survey tool. The online survey examined within each of the DSM-5 scale criteria the most important skills for children with autism in higher functioning levels compared to children in low functioning levels as answered by familiar people identified as being part of one of the categories of either parents, caregivers, siblings or teachers. Respondents were asked to rate the importance of social skills (6 questions), communication skills (7 questions), and behavioral skills (7 questions) in performing their roles in taking care of child(ren) with neurodevelopmental disabilities. These issues or measures were formatted using a 1 to 5-likert scale (1 being not very important, 3 as uncertain and 5 as very important). Each of those three skills were subjected to factor analysis. Cronbach’s alpha was also computed to assess the internal consistency of the derived factors [8].

The ethical considerations for the focus group and survey participants were mainly focused on the possible stress of answering questions of a personal nature. Participants were all over 18 years old and were given an informed consent statement with options to discontinue participation at any time. No significant risks or benefits were identified for participants. The most significant factor in this study was to make distinctions between the participants with regard to their relationship to the person with ASD. This gives more accurate interpretations of the responses and allows for analysis of the data to look for patterns based on these respondent categories.

Ethnographic Observation: A data collection tool to observe the natural environment and activities of children with ASD was created and data was collected at a therapy provider site to understand persons in the differing cognitive/developmental levels. Students in the provider programs were divided into groups according to their cognitive and language abilities. For the data collection, an “Activities Observation Sheet” was designed based on the Connectivity Model [9], a research model that uses scales to measure a target audience’s verbal and non-verbal communication in social, emotional, behavioral and physical realms. Through interviews and focus group studies qualitative data sets were collected and combined with quantitative data from survey tools. The qualitative data provides designers with very specific information that allows them to accurately interpret the social and emotional UP data from their target audiences. A series of four on-site ethnographic observations across different weeks were conducted using the Activities Observation Sheet. A team of observers that included designers and autism specialists used the sheets to record social, communication, behavioral and physical verbal and non-verbal data. Demographic data with regard to age, gender, and number of children in the facility was also recorded. The data was used to design educational workshops that were later conducted on the same site with the children from the observations.

The ethical considerations for the ethnographic observations were mainly focused on the possible stress on children in the observation and possible identifying information. The research team was coached on how to create minimal impact on the children by not initiating direct contact while observing and how to give the minimally appropriate verbal responses when directly engaged a child-initiated conversation. Prior to the observations, a letter of support was provided by the collaborating children’s therapy site in support of the research. Because the children were not being directly engaged or interviewed, no specific directions were given to the children being observed. Parents were provided with information and consent/assent forms for their child’s participation.

Case Study: The Play IT Observation method was designed for use in user participatory ethnographic observations in small case study environments. This method was designed to be used with all DSM-5 levels of autism. This specific case study observation used the appropriate DSM-5 level of fascilitated support as indicated by the child’s language and cognitive levels for a child with severe autism. The case being analyzed was the study of an educational software app being used by two children, a 12-year-old boy with severe autism and cognitive disabilities and a neurologically typical 8-year-old female peer. The observations were conducted with a facilitator present during the study. Both verbal and non-verbal data were collected and an autism UP talk aloud protocol with verbal and hand-over-hand prompts was also used because of its familiarity to the child with autism. The data on the use of the product was analyzed with regard to the Play•IT protocols [10]. Play•IT is a UP methodology that incorporates ethnographic research strategies from autism behavioral training and the social, emotional, behavioral and physical scales of the Connectivity Model. This case study collected comparative data on the skills of child with severe language and cognitive disabilities and on the neurologically typical peer. This allowed comparisons to be analyzed between the two children using the device. This data was then used in enhancing the product design for both types of users. The Play•IT method allows social interactions and social rehearsal to take place between target audiences in ways that are natural because it examines the use object being studied in the context of appropriately facilitated interactions between participants. The data is collected by observing and coding interactions based on the Play•IT criteria that involve multiple participants such as a person with autism, their caregiver, and a peer as they interact together with a product in a more natural situation. It also allows for the evaluation of a product within the appropriate level of support and physical assistance as indicated by the DSM-5 for that specific user. This is particularly important in situations where the level of ASD indicates that completely independent functioning is not possible because of language or cognitive limitations.

The ethical considerations for the ethnographic observations were on the possible stress on the children in the observation environment. The research team was coached on how to create minimal impact on the children while observing them interacting with the software app. The researchers were also instructed with regard to how to give appropriate verbal and non-verbal or hand-over-hand prompts that are common in autism behavioral training to facilitate the case study. Because the children were not being directly assessed but rather the software was under review, no specific directions were given to the children. Parental consent for their children’s participation was given.

3 Discussion

In UP for autism, there are three types of participants; (1) persons familar with autism or closely associated to those persons, (2) persons with ASD who have severe limitations in language and cognitive abilities, and (3) persons with ASD who have social issues that may impact their daily lives differently than neurologically typical persons but do not inhibit their ability to function cognitively in a mostly independent way and to use language as their primary form of communication. Research strategies in UP vary with their ability to be used effectively with each of these groups.

When collecting data from respondents who do not have ASD, there is a critical need to accurately assess and categorize them with regard to their relationship to the person with ASD. In studies such as the focus groups conducted with parents, caregivers, siblings and teachers, there was a high degree of correlation between responses. The only points of difference dealt with questions specific to a role such as questions about classroom habits as responded by teachers or daily living questions as responded by persons who are involved in that aspect of the autistic person’s life. Therefore clearly categorizing respondent data with regard to relationships will allow more accurate interpretations to be drawn.

For persons with autism of a severity that significantly limits verbal and cognitive ability, using the DSM-5 levels and domains to inform research categories for inclusion in studies will make the data more accurate and will make the research group more homogenous. Using strategies such as the Connectivity Model and Play•IT aligns the research to the diagnostic criteria and ability levels associated with the target audience. These methods when applied to UP in autism do not imply a diagnosis but rather link evaluation criteria to specific types of persons with ASD. By using DSM-5 criteria to make the UP research participants more homogenous, the data collection strategies can be applied more uniformly and the data analysis outcomes have greater implications for similarly categorized persons with autism.

The Connectivity Model and Play•IT focus on collecting and analyzing important and highly informative non-verbal data for persons with autism with severe cognitive and language impairments. In these research method, data collection through observation of natural environments and third party reporting via focus groups and survey tools are combined to form a more accurate assessment. It is important that observations be done by persons trained in autism. In addition, multiple observations done over a period of time may be more accurate and give a clearer picture of the range of performances and skills typical of the UP group. It is important to acknowledge and respect the DSM-5 levels in each domain for the person or groups being studied. If assessed levels are consistent with a level of required support, researchers should test with this level of support as part of the research design. For example, if a test subject is typically not capable of performing a task due to cognitive or physical constraints, it is not appropriate to expect the test subjects to perform a research task without this required level of support.

For persons with ASD who can function independently to read and answer a survey, using standard autism tools such as the RAADS-R or other tools to categorize levels or sub-groups of social ability is more reliable than a subjec’s self-reported level of autism. In addition, it does not specifically require a participant to divulge medically sensitive information that may be HIPPA protected. The outcomes of the assessment tool are not made known to the participant and therefore any ethical implications of diagnosis are avoided. The RAADS-R tool can be used to give very reliable numerical assessments of participants and clearly indicates both ASD or non-ASD levels.

In addition, with regard to ethical guidelines, researchers need to be trained in how to demonstrate or communicate using an appropriate cognitive and language level the expectations of the person in the study with regard to the tasks that they are to complete. For instance if a test subject is non-verbal, the test situation may need to be exactly communicated visually through several iterations prior to the data collection phase of testing. In addition, researchers must be well informed with regard to signs of non-compliance, agitation, boredom, or frustration from the test subject as indications that they no longer wish to participate in the research activity. Non-verbal communications must be respected as a sign to discontinue the study and disregard the data collected as directed in the IRB compliance agreements.

Ethical considerations involved with each of the four methods vary with regard to informed consent procedures. They also vary with regard to how the study expectations are communicated for user groups that involve children and persons with cognitive impairments. In addition, signs of refusing consent or rescinding consent are based on verbal communication for high functioning populations and based on non-verbal communication indicators for children and persons with cognitive impairments. In each case, the strategies for obtaining consent and plans for ethical discontinuance must be identified and all members of the research team coached in identifying these communications. In cases with children and persons with cognitive disabilities, it is always important to protect the identity and sensitive information collected and to have a plan for destruction of information and identifiers when the study concludes.

4 Conclusions

The findings of this research indicate that methods do vary in their ability to be used by persons with differing cognitive levels in autism. It was also found that familiar UP methods from autism may be appropriated especially those methods for ethnographic observation that are designed for engaging people with autism with a higher severity level as indicated by the DSM-5. For persons with a low level of severity in both DSM-5 domains, the RAADS-R is a highly effective tool for use with both neurologically typical people and high functioning persons with autism who require no support. The RAADS-R effectively identifies two groups of respondents; those with response patterns that are indicative of symptomatic autism and non-symptomatic persons. For persons who lack the ability to read and independently respond to questions, the DSM-5 is an effective tool to provide criteria for designing ethnographic research strategies and observation methods. The DSM-5 can also be used to indicate levels of symptoms in the two domains of social communication and restricted, repetitive behavior. The levels are requiring support, requiring substantial support and requiring very substantial support. It can also indicate that no support is needed and the subject can perform in that domain independently. The leveled support categories can be used to determine if the UX tool should be used or observed independently with the person with autism or if the observation or data collection should be used with the indicated level of support that is indicative of the autism participant.

Inclusion criteria must be clearly defined and the DSM-V is an effective tool for use as a method of discussing autism types with regard to levels of support needed and the ability to function appropriately in the domains of social communication and restricted, repetitive behaviors. Using the DSM-5 will align research outcomes to the diagnosis criteria of the target audience in a meaningful and useful way.

When involving persons with autism in the design process, cognitive ability and communication levels will determine the types of involvement that are possible. Observational data can be very informative and can often be used with persons from any cognitive or communication level. Researcher training is also a critical part of involving persons with autism into the research process. Researchers should also be made familiar with how to appropriately use verbal prompts or physical assists during research collection that involves direct contact or interaction with persons with autism.