1 Introduction

Traditional mental wellbeing assessment methods require people to be aware of their wellbeing status and seek help which can be challenging for those with intellectual disabilities [1]. Individuals with intellectual disabilities can often experience mental wellbeing challenges, but these are frequently overlooked and attributed erroneously to their disability (diagnostic overshadowing) or classified as challenging behaviour [2]. Furthermore, some individuals with intellectual disabilities face additional challenges in expressing their emotions, correctly interpreting social situations and predicting the behavioural consequences of specific actions [3,4,5]. These challenges often result in a significant impact on an individual’s likelihood of engaging with mental healthcare systems.

Mental wellbeing is a state of wellbeing in which an individual realises his or her own abilities to cope with the normal stresses of life and can be impacted by emotions felt [6]. It is typically described as having either a positive or negative valence that, in contrast to emotions, is less specific, less intense and less likely to be provoked or instantiated by a particular stimulus or event [7]. Traditional methods used to assess mental wellbeing often use standardised clinical questionnaires, typically in the form of patient-reported outcome measures (PROMs) or experience sampling [8] to understand longitudinal variability. Self-reporting is used to enable people to record their emotions and stresses which can be assessed and monitored to help establish stressful triggers [9, 10]. However, self-reporting can take considerable time to assess as it must be completed over a long period to gain useful insights [11].

Tangible User Interfaces (TUIs) describe physical objects that are able to translate user actions into input events in the computer interface [12]. As TUIs are physical objects, they rely on users’ knowledge of how the physical world works for interaction [13], making them more intuitive especially for people with less digital knowledge. Matthews and Doherty [14] and Niemantsverdriet and Versteeg [15] have reported that people are more likely to create stronger emotional attachments with physical devices such as TUIs rather than digital interfaces such as apps. Tangible devices provide a technological alternative to traditional self-reporting allowing users to report their current mental wellbeing in real time. Tangible manipulation presents an opportunity to develop novel devices, enabling unique interaction methods and encouraging continued engagement [16]. A technological solution that could actively monitor an individual’s state of mental wellbeing and provide feedback could be extremely beneficial in improving accessibility to mental health tools for all [17].

Co-design, particularly in the context of tangible user interfaces (TUIs) for mental health, plays a crucial role in ensuring the accessibility and effectiveness of such technologies for individuals with intellectual disabilities. Co-design involves actively involving end users in the design process, recognising them as experts in their own experiences and needs. When applied to the development of TUIs for mental health assessment, co-design empowers individuals with intellectual disabilities to contribute their unique perspectives and insights, ultimately resulting in interfaces that are tailored to their specific requirements. For individuals with intellectual disabilities, traditional methods of mental health assessment often fall short in accurately capturing their experiences and challenges. Diagnostic overshadowing, where mental health symptoms are erroneously attributed to the disability itself, can lead to the underestimation or misinterpretation of mental wellbeing issues. Moreover, difficulties in expressing emotions and understanding social situations can further impede effective communication about mental health concerns. Involving individuals with intellectual disabilities in the co-design of TUIs helps address these challenges by allowing them to provide input on the design of interfaces and interaction methods that better align with their capabilities and communication preferences.

TUIs offer unique advantages in the context of mental health assessment for individuals with intellectual disabilities. As physical objects, TUIs leverage users’ knowledge of the physical world, making interaction more intuitive and less reliant on digital literacy. This aspect of TUIs is particularly beneficial for individuals with limited digital knowledge or cognitive impairments. Co-design enables the identification and incorporation of tangible manipulation techniques that resonate with the target user group, fostering engagement and emotional attachment to the devices. By involving individuals with intellectual disabilities in the co-design process, TUIs can be customised to accommodate their specific sensory, cognitive and motor abilities, ensuring a more inclusive and accessible mental health assessment tool. Furthermore, co-design facilitates the exploration and integration of multi-modal sensors within TUIs. This allows for a comprehensive assessment of mental wellbeing beyond self-reporting, which may be challenging for individuals with intellectual disabilities. By gathering data from various sensory modalities, such as heart rate, skin conductance and movement patterns, TUIs can capture a more holistic picture of an individual’s mental state. The involvement of end users in the co-design process ensures that the selection and integration of these sensors align with the needs and preferences of individuals with intellectual disabilities, maximising the accuracy and relevance of the collected data.

Therefore, an iterative co-design process is adopted not to achieve one specific TUI design; instead, a range of different devices was expected to be proposed to suit the needs of all potential users including those who may experience a wide range of limitations. The co-design process aimed to investigate the following research questions:

  1. 1.

    How can the co-design process be adapted to best suit the needs of those with intellectual disabilities?

  2. 2.

    What are the optimal design guidelines for prototyping mental wellbeing tangible interfaces for people with intellectual disabilities?

  3. 3.

    Which technologies can be embedded within TUIs for real-world wellbeing data collection based on people with intellectual disabilities?

To ensure the proposed TUIs are effective and usable by individuals who experience mental wellbeing challenges, including those with intellectual disabilities, it is imperative to design the interfaces with end users. An iterative co-design process is proposed that has been adapted to enable the designing of interfaces and exploration of different sensing and feedback mechanisms with participants from the Nottingham Interactive Community for Education Research (NICER) group at Oak Field School, Nottingham, UK. The group is formed of adults with a range of intellectual disabilities who have a wide range of experience in evaluating assistive technologies, including virtual environments and serious games. In particular, they are interested in research and want to make sure research about intellectual disabilities conducted locally involves people with intellectual disabilities and is informed by their needs and wishes. The developed approach aimed to alleviate communication challenges faced by the target group and gather feedback that resulted in the development of TUIs for monitoring real-world mental wellbeing.

The remainder of this paper is organised as follows: Section 2 provides a literature review on co-design; Section 3 describes the participatory iterative design methodology used to design and develop the interfaces; Section 4 provides a discussion and answers the research questions; and Section 5 concludes and offers recommendations.

2 Background

Co-design is the methodology for actively engaging people directly involved in an issue, place or process in its design, allowing them to make a meaningful contribution to the design process [18,19,20]. Co-design enables the reduction of the gap in knowledge between end users and researchers, allowing non-designers to become equal members of the design team, ensuring designer subjectivity is removed and the technologies developed are suitable for the target population [21, 22]. During the process, design tools are used to empower all of the participants to facilitate a ‘joint inquiry’ where ‘problem and solution co-evolve’ [23]. Co-design brings many benefits to the design of the project by helping the researcher better understand the challenges faced by users and any potential solutions [24, 25].

Co-designing helps solve real-world problems by bringing together people from different backgrounds into the design process, resulting in more inclusive solutions. However, to work most effectively, it is important to select appropriate methods and ways of working which need to match the project being designed and the potential users’ capabilities and limitations. The following co-design approaches have been shown to help empower participants by making things that are normally challenging to observe available as resources for design during the co-design process [26,27,28]:

  • The use of visual, creative methods helps maintain engagement and promotes idea generation [29].

  • Physically making things helps people to explore, verbalise, remember and imagine helping them to generate new solutions to problems they have faced and offer their opinions [22].

  • Creating and telling stories helps put things into context and provides a central way of sharing and communicating. Story sharing can be visual, verbal or include role play [29].

Co-design can be used to promote the inclusion of people with intellectual disabilities when designing new solutions by including their personal experiences, making them more likely to take ownership of the final outcome [30]. People with intellectual disabilities may face barriers such as communication challenges when being involved in the co-design of new assistive technologies, resulting in co-design techniques needing to be modified to fit with participants’ abilities [31]. Previous research has explored methods to help adapt and engage people with intellectual disabilities in the co-design process. Brereton et al. [32] presented early iterations of a design, such as initial prototypes, to engage people with intellectual disabilities in co-design. Alternatively, an iterative design methodology can be used to promote continuous engagement for those with intellectual disabilities [33]. The unfinished design can also be a probe itself to promote discussion and idea generation, such as proposed in the analysis of non-finito products [34]. Finally, focus groups have been used as a methodology for reporting and validating the experience of co-designers with disabilities [35].

Literature reviews [36, 37] explored inclusive research concluding that it adds value when there is a distinctive contribution which only co-researchers with intellectual disabilities can make, when it highlights the contributions people with intellectual disabilities make and when it contributes to better lives for the wider population of people with intellectual disabilities. This shows that co-design is imperative when the research answers questions the authors could not otherwise answer, reaches participants in ways that the authors could not otherwise access, involves reflecting upon the insider cultural knowledge of people with intellectual disabilities or makes impact on the lives of people with intellectual disabilities. This research resulted in the following inclusive research guidelines:

  • Aiming to contribute to social change, that helps to create a society, in which excluded groups belong and which aims to improve the quality of their lives.

  • Being based on issues important to a group, which draws on their experience to inform the research process and outcomes.

  • Exploring which aims to recognise, foster and communicate the contributions people with intellectual disabilities can make.

  • Providing information which can be used by people with intellectual disabilities to campaign for change on behalf of others.

  • Showing that those involved in co-design are ‘standing with’ those whose issues are being explored or investigated.

There has been limited research exploring TUIs for people with ID. Gelsomini et al. explored reflex [38] which provided a number of educational activities where users’ manipulated physical items, demonstrating the benefits of using TUIs for the development of cognitive and social skills. Similarly, other work has demonstrated the benefits of TUIs with children with ID. Children with intellectual disabilities played with a range of tangible interfaces to demonstrate the effectiveness of tangible interfaces for learning [39]. The work concluded that the most efficient paradigm is one with a clear mapping between physical objects and their meanings. Similarly, Polipo a 3D-printed smart toy co-designed with special-education professionals, offers numerous physical affordances and feedback in the form of lights, sounds and music [40]. Polipo was shown to improve fine motor skills in children with intellectual disabilities and encourage them to communicate. Similar benefits were also observed with Poma [41], a TUI designed to improve social and cognitive skills of children with autism spectrum disorder. Overall, while co-designing with participants who have intellectual disabilities has been previously explored, little research has considered the co-design and development of tangible interfaces for mental wellbeing monitoring and real-time feedback.

2.1 Co-designing with people with ID

While co-design has been utilised to develop solutions for people with intellectual disabilities, much research does not involve those users in the co-design process. It is imperative to co-design with people who have intellectual disabilities to ensure solutions take into account the lived experiences of those facing challenges in order to develop the most appropriate solution that can have real-world impact. While there have been limited co-design studies for TUIs, co-design has been used to engage individuals with cognitive disabilities in successfully designing a picture-based remote communication system, helping the participants move from merely being passive onlookers to active participants [42]. Similarly, co-design workshops have engaged people with assisted living needs to develop technologies and services for new care solutions [43], engaged autistic children in co-designing technologies [44], researched accessible apps and games with students with learning disabilities [45] and helped people with complex communication needs to express themselves through art therapy [46]. While involving users with intellectual disabilities in the design process produces additional challenges such as additional ethical considerations [47], it is imperative to ensure the solutions developed meet the needs of the potential users.

A card-based toolkit has been co-designed with people with intellectual disabilities for smart outdoor experiences [48]. This COBO toolkit integrates inspirational cards, interactive smart objects and multimedia contents to guide users during the conception of novel ideas. Participants played a crucial role during the process and introducing the elements; they contributed with very important feedback about the game dynamics and the interactive board prototype current research mainly highlights the benefits of TUIs for children [43]. Similarly, a mobile app has been co-designed to help people with ID use public transport using four specific elements to increase user engagement: a digital prototype, a non-finito feature, an inclusion of a proxy and a co-development opportunity [49]. These four elements confirmed the benefits of prototypes and showed that 30-min co-design sessions were insufficient.

Co-designing with people with ID for mental health has also been conducted to develop and implement creative methods to engage young people experiencing psychosis in an online resource for educational and wellbeing support [50]. Through workshops involving storytelling with emojis, relatable personas, card sorting activities and collaborative design, the research successfully engaged young people, their families and clinicians in both hospital and community mental health settings. These creative co-design sessions, inspired by social media metaphors and tailored to their interests and moods, facilitated active engagement, drew out unique perspectives and ultimately helped develop solutions to effectively meet their needs. A competency-based methodology that was also influenced by social media served as a collaborative effort in designing a web application for imparting life skills [51]. This competency-based approach expands upon ability-based design principles and ensures that the devised designs are accessible to a diverse user base, while also acknowledging the necessity for additional assistance and customization for specific individuals. This approach fosters empowerment by harnessing the competencies of individuals with intellectual disabilities, while also allowing for the advancement and further development of skills.

A game-based learning tool has also been co-designed with young people with ID concluding that during the co-design process, it is necessary to provide clear instruction, be flexible to engage all participants, provide the ability to adapt content to best suit the needs of the participants and provide an emphasis on the consequences of choices [52]. Similarly, a digital skills education solution has been co-designed with people who have ID combining co-design with focus groups and user testing to encourage authentic engagement and inclusion of people with ID and accessibility needs throughout the entire design process [53].

Co-designing with children with ID has been conducted to design a serious game using a flexible approach allowing them to express their voice as user, informant, designer and tester [54]. The study found that when participants were asked to think of a game design, they struggled without existing prompts or prototypes as a concrete starting point. Equity on all aspects was not achieved consistently through the project, but the students’ voices remained empirical for the final solution showing how each stakeholder can bring their own expertise to the task. This study shows an informal, flexible and practical co-design approach is required to promote idea generation and active participation throughout the entire session. Similar research [55] that co-designed with adolescents with autism spectrum disorder also highlighted the benefits of prototyping and conducting an iterative design process. Overall research shows that when participants’ needs, preferences and desires are accounted for, participants with ID can make meaningful contributions to the design process [56].

3 Co-design methodology

We have adopted a participatory approach to co-design solutions for the monitoring of mental wellbeing. This enables the design of products and applications directly with the final users including those with intellectual disabilities in order to develop accessible products. Co-design helps create a sense of empowerment and a feeling of competence, which benefits the participants as they derive satisfaction and fun while feeling useful through their participation [57, 58]. We therefore involved adults with intellectual disabilities throughout the entire co-design process as outlined in Fig. 1 with the assumption that their inclusion will positively impact the quality of the final product in addition to their own experience.

This instantiation of an iterative design cycle took over 1 year in total. Each stage of the co-design process was conducted with the same primary researcher and an experienced facilitator with many years of experience in running co-design workshops. All participants were members of the NICER group with varying disabilities including Williams syndrome (a rare genetic disorder resulting in mild to moderate delays in their cognitive development) [59], Down syndrome [60] and autism [61], but no participants had significant motor skill impairments that would impact their participation. Their role in this process was not as research subjects, they were instead involved in identifying design opportunities relevant to their needs.

A person diagnosed with an intellectual disability [62] shows deficits in intellectual functioning such as reasoning or problem-solving and deficits and impairments in adaptive functioning such as communication or social skills, all during the developmental period before the age of 18 [63]. Intellectual disabilities can be divided into four levels: mild, moderate, severe and profound where for each level the person requires more support [64]. Co-designing with people with disabilities presents additional challenges—such as issues regarding the nature of their cognitive disability, including communicational and memory issues, which may challenge their full participation. However, it is imperative for the voices of end users to be heard, and many of the challenges can be overcome by developing a co-design methodology catering for the requirements of people with intellectual disabilities. The early integration of people with intellectual disabilities into the design processes aimed to prioritise their design decisions and needs. These insights serve as guides to a joint inquiry that seeks to address the challenges of developing TUIs to monitor affective state.

Fig. 1
figure 1

Stages of iterative co-design process

Full size image

The co-design process used in the project is depicted in Fig. 1 adapted from [36, 37] where people with intellectual disabilities engaged in a partnership with researchers. The methodology alternates between focus groups to gather feedback during the preparation, co-analysis and validation phases and interactive co-design workshops completing the ideation and development phases. A novel aspect of this process is that it enabled feedback to be gathered at each stage of the development process including design and prototype development. The co-design process was adapted to include more demonstrations and interactivity to engage participants with less general discussions. Furthermore, the preparation phase introduced participants to the concept of mental wellbeing, and the researchers ensured familiarity during the subsequent workshops and validation phases.

Overall, the co-design process was conducted in conjunction with an advisory panel of adults with intellectual disabilities, focusing on methodological adaptations and special supports developed to facilitate and ensure their participation. The co-design sessions were recorded for future analysis as granted by Nottingham Trent University, ethics application 18/19-43V2.

3.1 Preparation phase focus group

At the beginning of the co-design process, an introductory preparation phase was completed to agree on the scope and aims and objectives of the project. Members of NICER along with teachers of young students with moderate to severe intellectual disabilities acted as an advisory panel with a researcher, experienced facilitator and education specialist leading the session. The focus group was conducted over 1 h, and notes of both verbal and non-verbal communication were recorded for future analysis by the experience facilitator.

An accessible introduction was completed using a presentation to introduce participants to the concept of TUIs and the possibility for them to automatically monitor mental wellbeing. This was completed by showing examples of existing interfaces such as Emoball [65] and Mood TUI [66] to help participants develop a concrete understanding of TUIs and allow them to gain greater knowledge of the devices to be developed. Explanations of hard-to-understand concepts were provided including discussions on emotions and the technologies themselves.

After the project had been introduced, participants were asked whether it was of interest to them and whether they would like to get involved in the subsequent co-design workshops and they agreed and volunteered their involvement. The NICER group is a significant stakeholder in this research, and they developed high expectations of TUIs that could have a positive impact for the intellectually disabled community.

Table 1 Co-design workshop 1 participant characteristics (age, gender and disability severity (moderate or severe)) for total number of participants (N), with Williams syndrome (WS), Down syndrome (DS) and autism
Full size table

3.2 Co-design workshop 1

The first co-design workshop aimed to explore various designs, technologies and requirements for mental wellbeing TUIs. The workshop was conducted over 4 h at Nottingham Trent University to strengthen the participants’ roles as experts and posit them as co-researchers within a university setting and was video and audio recorded for future analysis. The workshop comprised of six participants: four males and two females who have previously been involved with multiple research projects and are experienced co-designers having helped co-design multiple serious games and have been involved in numerous research projects with the same research facilitator in the past. While members of NICER have experience in the co-design process helping them to make valuable contributions and have lived experiences of mental wellbeing challenges, the electronic elements of developing TUIs were new. Therefore, little preparation regarding general co-design practices was required, and the previous preparation phase along with the co-design introduction helped introduce the concept of TUIs and the potential electronics that could be utilised.

Table 1 shows the number (N) of participants and their characteristics including those who have Williams syndrome (WS), Down syndrome (DS) and autism along with information on their gender and level of intellectual disability (moderate or severe) as defined by their condition and previous evaluation by professional carers and teachers. A common challenge faced by all participants is their difficulty expressing themselves verbally.

A common issue with the development of mental wellbeing technological solutions is the lack of ethical considerations [67]. To ensure the co-design workshops were inclusive for participants with intellectual disabilities and caused no harm, all discussions were short and a range of interactive tasks were designed to increase engagement. During the workshop, it was ensured that all participants had a full understanding of the goal of the workshop and their role as a co-designer. The following five co-design methods used within the co-design workshops were designed to support participants’ decisions, help researchers understand the participants requirements and act as conversational instruments based on established methods [68] and previous experience [69, 70].

3.2.1 Introduction, explanation and demonstration

When conducting co-design workshops with participants who have intellectual disabilities, it is imperative to ensure all participants fully understand the goal of the workshop to improve communication. To ensure this, an experienced facilitator introduced the session by clearly explaining the concept of affective tangible interfaces. The concepts of mental wellbeing and emotions were then discussed to ensure all of the participants had an understanding of what wellbeing is and how it impacts them in order to facilitate discussions on solutions to help wellbeing monitoring. Finally, the requirement to develop a device that can monitor wellbeing was discussed to ensure all participants had a full and comprehensive understanding of the purpose of the workshop. Participants had the opportunity to engage in discussions about the mental wellbeing challenges they face as well as what they would require from a technological solution to monitor wellbeing, demonstrating their understanding of both the concept of mental wellbeing and the workshop aim.

Fig. 2
figure 2

Original mental wellbeing interface prototype

Full size image
Table 2 Five participants’ tangible interfaces priorities as ordered during the co-design workshop from highest to lowest priority
Full size table

Challenges can arise from communicating with participants and difficulties interpreting non-verbal interactions [71, 72]. Therefore, in the co-design processes, the adage ‘show me don’t tell me’ [18] is often used, resulting in previously developed prototypes such as the cube shown in Fig. 2 being demonstrated to the participants. Concrete prototypes create opportunities for participants with intellectual disabilities to interact directly with the interfaces and understand the feasibility of developing new interfaces [73]. The prototypes embedded a range of sensors including 9 Degree of Freedom Inertial Measurement Units (9-DOF IMU) to measure motion, force-sensitive resistors (FSR) to measure touch and HR and EDA sensors to measure physiological changes.

3.2.2 Storyboarding and drawing

The design of the interfaces was then explored. Previous work provides many useful strategies for engaging individuals with intellectual disabilities in the design process such as storyboards and pictures and avoiding open-ended questions [74,75,76,77,78]. To better support participants’ way of self-expression, real-time storyboarding was completed where prompts were presented to participants to expand upon, promoting communication. During storyboarding, participants were able to discuss their opinions on the existing interfaces previously demonstrated and share their ideas for new interfaces.

In co-design, methods are used to help participants ‘say, do and make’ [79]. This helps us deepen our engagement with people and strengthens the insights we are able to gather. Using this approach, participants were invited to draw their own interfaces using pen and paper to help promote ideas for new interfaces. This enabled the participants to creatively express their design ideas without the need to verbalise, which some people with intellectual disabilities can find challenging.

3.2.3 Prioritising design requirement cards

The potential requirements of mental wellbeing tangible interfaces were also explored as it is imperative to understand what features users most require to ensure successful devices are developed. A card-based approach was used that enabled participants to prioritise the features they believed were most required. This approach was based on the generative research approach [80] to combine participatory exercises with verbal discussion during the creative idea generation phase. Similar card-based approaches have previously been used due to their accessibility, familiarity and tactile nature which can help promote communication [81]. Six cards were provided to each participant stating a requirement for tangible interfaces including: ease of use, makes me feel better, design, battery life, physical size and understands how I am feeling. Each of the six requirements was explained to the participants by the researcher and experienced facilitator to ensure they fully understood the meaning of the requirements and their role in prioritising the requirement cards.

Each requirement was given a score dependent on the order in which each participant placed the requirement, where the highest priority was given a score of six and the lowest one (Table 2). The most prioritised requirement was makes me feel better (22), closely followed by ease of use (21), then battery life (19), size (16), understands how I feel (14) and finally design (13). This shows that the participants all value the feedback the device could provide to make them feel better as the highest priority. However, this would first require the device to understand how the individual is feeling which was the second least prioritised feature, possibly showing a lack of understanding of the understands how I am feeling requirement. This is potentially due to this requirement being an essential prerequisite to the other options; therefore, participants may have considered this the core functionality rather than a design requirement. Ease of use was the second highest rated priority showing it to be highly valued amongst all participants. Battery life and size followed, although this greatly varies between participants, with some participants rating them as the highest priority and others rating them as the lowest priority. The lowest overall priority was design which was unexpected as participants enjoyed exploring the different prototypes suggesting they see makes me feel better as the overall design goal.

Fig. 3
figure 3

Participants exploring different haptic patterns during a co-design workshop

Full size image

3.2.4 Real-time 3D printing

The process of showing participants with an intellectual disability how their design decisions have a direct real-world consequence in a rapid and concrete way was developed in an earlier study and replicated here [82]. Three-dimensional modelling software was demonstrated by the researcher to demonstrate how the interfaces can be designed and printed to make concrete the relationship between the participants’ decisions and the tangible interfaces produced. A majority vote was conducted to decide on the shape to be printed, and then participants were involved in collaboratively designing the shape using the software. As the workshops were conducted over several hours, there was sufficient time to design and 3D print a small interface, providing opportunity for participants to provide reflection on the design. Creating the interfaces during the workshops resulted in a deeper and more practical understanding about the participants’ experiences [79].

3.2.5 Interactive electronics

When exploring new technological solutions, providing demonstrations is necessary to ensure all participants understand the functionality and how the technology can be used, thereby improving confidence and communication [83]. During the session, a range of non-invasive, easy-to-use sensors were explored through interactive demonstrations. This was designed to increase engagement and ensure participants understood the functionality of the electronics by allowing them to experience the different capabilities offered by each sensor. All electronics were made simple to operate with the electrical circuits pre-built, as used in previous co-design studies [84], to ensure all participants would be able to fully participate [85].

A HR sensor was first explored where participants were able to place their finger on the sensor and lights would flash at the same rate as their pulse. An EDA sensor was also explored as it functions in a similar way to the HR sensor with participants having to place their fingers on the sensor. An FSR was demonstrated next, where as participants pressed harder on the sensor, it caused a haptic motor to vibrate. Finally, a 9-DOF IMU was demonstrated inside a ball; as participants shook the ball, it would vibrate. Overall, this method of exploring the sensors promoted participants’ understanding and enabled them to experience how the sensors will be used in future interfaces.

Varying forms of feedback acting as real-time interventions were also explored giving participants time to reflect and express their feedback between demonstrations (Fig. 3). Visual feedback continued to be explored following the use of multicoloured LEDs to demonstrate the HR sensor. Participants were shown multiple examples including one device where different colours represented different emotions. Auditory feedback was also demonstrated where a speaker was used to play calming sounds from nature. Haptic feedback was the last intervention explored; four different feedback patterns were demonstrated to each of the participants who held the vibration motor to experience the different sensations.

3.2.6 Exploration of tangible self-report methods

Methods to self-report were discussed, as in order to gain a real-world understanding of mental wellbeing, the sensor data along with the user’s state of wellbeing is required. It is necessary to collect self-reports at the point of collection as it is challenging to identify afterwards. On-device self-reporting simplifies the process of by not requiring additional materials, such as questionnaires, and its ease of access promotes frequent reporting. During the session, a range of non-invasive, easy-to-use sensors and on-device self-reporting methods such as buttons and sliding potentiometers that could be used in real-world environments [86] were explored through interactive demonstrations.

A variety of buttons, sliding potentiometers and force-sensitive resistors were demonstrated as on-device methods to label wellbeing. Participants had the opportunity to test each of the labelling methods by simulating using the labelling methods to self-report their current state of mental wellbeing. During the discussion, all participants understood the need for labelling data and expressed their desire for on-device labelling over traditional paper-based labelling stating it is easier to use. Upon considering potential on-device labelling methods, all participants preferred buttons stating I do think they would be better as they were the easiest to use, in particular using different buttons to represent different emotions. When considering the best methods to represent the buttons, Makaton was suggested which is a language programme that uses signs together with symbols to enable communication [87]. Alternatively, different coloured buttons to represent different emotions were explored, such as green and red buttons to represent positive and negative emotions respectively. Using different coloured buttons to label emotional states was preferred by all participants for its simplicity, but the number of labels would be limited to ensure simplicity for those with intellectual disabilities.

3.3 Prototyping

After the co-design workshop concluded, researchers developed initial high-fidelity prototypes using the designs, sensors and feedback mechanisms suggested. Participants were not involved in this process as it required technical skills such as soldering components and took several days, but their feedback during the co-design process guided all of the developed prototypes. This rapid prototype development was conducted to enable participants to physically experience their design suggestions from the workshop, promoting further discussion and the continued refinement of the interfaces (Fig. 4).

Fig. 4
figure 4

Early prototyping with soft toys

Full size image

3.4 Co-analysis focus group

During the co-analysis phase, focus groups were held where the participants who took part in the co-design workshop, along with other members of the NICER group and teachers of young students with intellectual disabilities, were provided with the opportunity to give feedback on the workshop and the resultant ideas. Typically, the focus groups were shorter in nature than the co-design session allowing for reflection on the outcomes of the workshops. During the co-analysis, notes of verbal and non-verbal communication were recorded for future analysis by the experienced facilitator.

Table 3 Co-design workshop 2 participant characteristics including number of participants (N), with Williams syndrome (WS), Down syndrome (DS) and autism
Full size table

The various activities conducted within the workshop were discussed to analyse what activities participants enjoyed and how they though the workshops could be improved, allowing for adjustments to be made to the co-design process where required. The group facilitator asked the NICER group members who attended the co-design workshop to present the main activities and outcomes of the workshop and their implications. Volunteers presented their memories, experiences and design preferences, and this again was all recorded using an accessible storyboard format. Implications for adjustment of co-design techniques and plans for follow-up activities at the next co-design workshop were also discussed.

3.5 Co-design workshop 2

The second co-design workshop advanced upon the findings from the first workshop and co-analysis aiming to refine the developed interfaces. The workshop was conducted over 3 h and was audio recorded for future analysis. The workshop comprised of 8 participants: 5 males and 3 females who again are experienced co-designers. Five participants previously participated in the first co-design workshop; these included 1 with Williams syndrome and 4 with Down syndrome. Table 3 shows the characteristics of the participants including those who have Williams syndrome (WS) and Down syndrome (DS), along with information on their gender and level of intellectual disability.

During the co-design workshop, the five co-design methods previously developed were used. The session was introduced, and the previously developed interfaces were demonstrated to the participants, ensuring all participants were familiar with the interfaces and understood the purpose of the co-design session. After participants understood the aim of the session was to refine the existing prototypes, storyboarding was again utilised to help participants effectively communicate their new ideas for the interfaces. However, following feedback from the co-analysis and the challenges encountered during the first workshop where participants were not able to draw their own interfaces, this was not included within the second workshop. Instead, 3D modelling was performed, and participants experienced 3D printing a new interface during the workshop to help them understand the prototyping process and ensure they understood the impact their design decisions have on the developed interfaces.

The electronics within the interfaces were also explored with participants experiencing the same sensors as previously explored to ensure the suitability of the sensors and help gather additional feedback from new participants who had not previously experienced the electronics. Finally, potential features were not prioritised using cards as conducted in the initial workshop due to this workshop focusing on refining the ideas already produced. Instead, further discussions were held enabling participants to express their opinions on the required functionality and how the prototypes could be improved.

3.6 Product development

Using the feedback from the second co-design workshop, the initial prototypes were refined by the primary researcher. Similar to the prototyping stage, participants were not involved in this process as it required technical skills such as soldering components and took several days, but their feedback during the co-design process guided all of the developed devices. Multiple refined interfaces were developed over several weeks, each considering the feedback gathered throughout the iterative co-design process. These final products allow participants to experience how their design decisions impacted the development of solutions relevant to themselves.

Overall, nine prototypes were developed including both 3D-printed shapes and soft interfaces. The 3D-printed interfaces included 2 cubes as these are easy to hold, a cuboid, a sphere containing sleeves to place fingers within, a spheroid designed to ensure the user’s thumb will rest on the HR sensor and their palm will touch the EDA sensor in addition to the torus shape selected to print during the second co-design workshop. The sensors within each device are shown in Table 4.

Table 4 Description of the 9 developed wellbeing TUIs and the embedded electronics
Full size table

3.7 Validation focus group

The final aspect of the iterative co-design process was to evaluate the developed interfaces through a final focus group. Each of the developed interfaces was demonstrated to members of the NICER group including the participants from the co-design workshops and teachers of young students with intellectual disabilities where notes of verbal and non-verbal communication were taken by the experienced facilitator for future analysis. The participants had the opportunity to experience the different interfaces developed throughout the co-design process and examine how their feedback helped influence the design and functionality of the devices. Participants also had the opportunity to provide their final feedback on all concepts generated, including the design and self-report methods within the interfaces in addition to the co-design process itself and how it was adapted to promote communication and idea generation. This feedback can then be used to refine the developed TUIs as well as the co-design process for future studies.

During the evaluation, participants stated that the interfaces developed thus far are suitable for potential users. During the time, participants experienced the devices they stated that their use made them feel happy; in particular, the fidgeting buttons helped them feel calmer. Teachers and end users agreed, liking the shapes of the interfaces in addition to the ubiquitous nature of the devices with the sensors being embedded within objects. Overall, the developed devices were found to be suitable for their intended purpose of monitoring mental wellbeing and collecting real-world data.

4 Discussion

This co-design study has explored the design, development and evaluation of TUIs to monitor real-world mental wellbeing. A co-design methodology was adopted based on previous research with participants whose mental wellbeing can often be diagnostically overshadowed and who commonly have difficulty in expressing their emotions [2]. The co-design approach addressed the limitations experienced by people with intellectual disabilities (e.g. communication, working memory), enabling them to participate more effectively. This approach takes a practical stance in guiding how co-design methods can be made to work in realistic design settings and adjusted to the needs of participants who may experience intellectual disabilities.

This research has many implications for both affective recognition technologies and the process of co-designing technologies with people who have intellectual disabilities. The five developed co-design methods aimed to increase individuals’ autonomy [88] and promote communication within the workshops. The participants’ investment in the co-design process resulted from their ability to recognise the practical applications of mental wellbeing technologies while also appreciating the impact that they may have on the daily lives of diverse user populations.

When co-designing with people who have intellectual disabilities, it is vital to gradually unfold their creative potential to encourage meaningful participation [89]. Therefore, a number of activities were mediated with education professionals and conducted within each workshop that were designed to elicit design input and opinions, reach consensus and check understanding, such as the prioritisation of requirements.

A number of instructional and conversational instruments were also used during the co-design process to aid inclusivity [68]. These design instruments were used such as demonstrations and hands-on experiences with existing prototypes, sensors and actuators to serve as conversational instruments. During conversations, design instructions were iterated in conjunction with co-design participants, for example it was explained to participants that including all sensors in each design would lead to a TUI that was physically large. Furthermore, all information such as notes including non-verbal communication and video recording was collected in a structured way for subsequent analysis, and major outcomes were recorded on a note board in picture form and simple sentences as a joint record of achievement, which was easy to understand for the entire design team.

A range of interactive activities were also used as the co-design session lasted from morning to early afternoon, allowing them to incorporate physical (drawing, prioritising cards) and digital (3D modelling, exploration of electronics) elements and provide continuity between exploration and prototype testing. This meant that ideas captured from the whole team could be storyboarded and 3D printed all in 1 day, to make concrete the connection between design decisions and embodiment to support cognitive accessibility.

During the ideation phase, the production of prototypes was completed in a stimulating and playful environment, free from pressures that intimidate and block the creativity of the participants, including the use of design kits (drawing materials, cards, electronic components and sensors) as facilitating tools. The participants have previously been involved in the design of assistive technologies, and hence, they bring experience and expectations to be central to design decisions at co-design workshops.

These design instructions and activities along with the five developed co-design methods and the inclusion of analysis and validation phases ensured participants were able to fully participate and provide valuable feedback. The feedback gathered from the co-design process resulted in the development of multiple interfaces for real-world mental wellbeing monitoring.

4.1 Research questions

The following sections use the analysis from the co-design process and feedback from the evaluation session where participants experienced the final developed interfaces to answer the four research questions.

4.1.1 How can the co-design process be adapted to best suit the needs of those with intellectual disabilities?

Co-designing for people with severe, profound and complex intellectual disabilities has shown additional challenges requiring adaptions to the co-design process to increase engagement. Not all tasks were successfully completed by all participants—such as drawing a new interface, as experienced in similar co-design studies [90]. However, the majority of the tasks were completed successfully, particularly the interactive activities within the co-design workshops, involving exploring sensors and feedback mechanisms and the 3D printing of new interfaces to immediately show participants the results of their design decisions. These interactive activities helped maintain engagement, improved understanding of new concepts and improved communication within the workshops. Additionally, storyboarding is an effective method for identifying, understanding and coming to grips with factors that capture and influence people’s experiences [74,75,76,77,78]. A novel approach utilised was combining the interactive sessions with storyboarding to gain feedback, and it has been possible to gain valuable insights, aiding the design and development of future affective interfaces.

During the co-design workshops, prototypes were shown to participants to bridge the gap in knowledge from researchers to participants, helping them gain an understanding of the type of interfaces to be developed. However, to ensure participants’ freedom was not restricted, a number of different prototypes were presented along with different materials and electronics. This helped demonstrate not only example prototypes but also the components used to develop such devices, allowing participants to have greater flexibility when devising their own suggestions. Previous co-design approaches with participants who have intellectual disabilities have similarly used prototypes to elicit feedback [91,92,93] and found it extremely beneficial. We experienced very similar results with participants becoming more engaged when using prototypes, evidenced by an increase in their attention and verbal feedback. Providing participants with low-fidelity prototypes offers much clearer expectation about the upcoming system and offers participants the opportunity to react to features. The use of prototypes and interactive sessions successfully aided idea generation during the co-design workshops while allowing participants to retain freedom in their designs.

Drawing new interfaces was also adapted to meet the needs of participants with ID by helping participants creatively express their ideas that they were not able to verbally communicate such as the sensors and feedback they believed were most important to include. The process of drawing an interface was a novel addition to the co-design process, and it was adapted from similar studies [79, 94, 95] that encourage participants to ‘do’ and ‘make’ although few co-design studies in particular that involved those with ID has provided the creative freedom to participants to envision their own interfaces without requiring verbal communication. Additionally, allowing participants to express themselves through 3D printing during the workshops helped demonstrate how the interfaces are developed and encouraged all participants to consider how differently shaped interfaces could be used. This process helped participants make a concrete connection between their design decisions (their drawings) and how this had a direct impact on the outcome (3D-printed interface) [82]. Linking the drawings from participants to a physical product (3D print) during the workshop is also a novel approach that worked extremely well. Allowing participants to move from the virtual environment to the physical world reinforced how their decisions had a tangible impact. While previous work has developed low-fidelity prototypes during co-design sessions [96,97,98], the use of 3D printing and virtual modelling was a novel approach that goes beyond previous work to produce final products that better resemble what participants envisioned during the workshops.

Ranking features of the interfaces helped participants express what functionality they believed was most important without needing to verbally communicate. While this activity was based on the generative research approach [80, 81], we adapted the card-based approach to meet the needs of participants with ID by enabling them to rank features, thereby allowing time to reflect and make additional suggestions. Physically ranking features was a useful incorporation that has not been used in previous co-design studies with people with ID as it allows participants to consider what they most require while providing prompts to encourage further discussion. This is a novel approach that is often completed verbally during co-design sessions but by providing tangible requirements that participants physically ordered ensured they considered each of the requirements as evidenced through the subsequent discussions.

Finally, exploring the sensors and feedback was enjoyed by all, and the analysis showed it greatly improved attention and engagement in addition to aiding the understanding of each technology. Demonstrating how the electronics work aided participants’ understanding of how the devices are developed and helped them make realistic suggestions regarding the design of future TUIs. While it is common during co-design workshops to explore the technologies being used, having hands-on experience with a range of demonstrations utilising various sensors and feedback actuators simply connected via wires in their raw form rather than within a complete prototype gave the participants a much deeper understanding of the technologies being utilised rather than them being abstracted within the TUI [48, 99] or simulated [100] as other co-design research has previously. While this approach of showing the raw sensors and electronics to participants is more complex to understand, it helped participants gain a greater understanding of what is required when developing a TUI and should not be abstracted away in future studies.

During the workshops, cognitive barriers were a theme that appeared throughout, mainly through lack of communication. Frequently, participants would require prompting when discussing specific topics, for example when explaining which sensors would be most appropriate for use. There was a large variation in the communication skills within the group, with some participants elaborating on their feedback in great detail and others who frequently replied with one-word answers, simply nod or shake their head or always agree with the other participants. These are challenging issues to overcome as some participants may not feel comfortable in expressing their opinions in groups. However, the involvement of an experienced facilitator who understood the commonly occurring communication issues for this group, the use of Makaton [87] where appropriate along with the interactive activities during the co-design workshops, helped improve communication with all participants. Additionally, providing initial low-fidelity prototypes for participants to explore aided their understanding of potential uses for the devices, and drawing an interface helped them to communicate new ideas and features. Furthermore, having the co-design process alternate between co-design workshops and feedback focus groups helped participants gain a clearer understanding of the interfaces and the technologies used within them, helping many to better communicate their ideas and hence provide feedback to help design an effective TUI. This alternating approach ensured participants’ voices were heard during the co-design sessions and allowed them to provide subsequent feedback helping to continue refining the developed TUIs.

Overall, the interactive co-design approach adopted addressed the limitations experienced by people with intellectual disabilities (e.g. communication and working memory), enabling them to participate more effectively. The co-design approach was extremely successful in gathering the requirements for mental wellbeing TUIs mostly by helping concrete participants’ ideas. The interactive activities conducted as part of the co-design workshops were extremely successful and should continue to be utilised in future studies to help participants generate ideas. This approach takes a practical stance in guiding how co-design methods can be made to work in realistic settings and adjusted to the needs of participants who experience intellectual disabilities.

4.1.2 What are the optimal design guidelines for prototyping mental wellbeing tangible interfaces for people with intellectual disabilities?)

1. Inconspicuous design During the workshops, numerous limitations were discussed, but all participants considered it highly important that the interfaces did not appear as a medical device in order to reduce stigma. When a participant previously used medical sensors, they stated I felt awful, I was panicking...the first time I thought I’m not doing this. This makes it vital that any physiological sensors and feedback mechanisms within the interfaces are non-invasive, easy to use and inconspicuous, so as to not induce additional stress. This shows similar results to other studies designing mental health technologies highlighting it as a priority [101]. As the sensors explored during the workshops were all small and unobtrusive, participants believed they were ideal to monitor real-world mental wellbeing. By developing wellbeing interfaces for the general population, as well as for those experiencing mental health challenges, it will reduce the associated stigma by ensuring the devices are suitable for all.

The size of the cube prototype was liked by participants with them stating like that size when referring to the size of new interfaces. Participants discussed numerous usage scenarios for the portable interfaces including using them as work, college and at home. Participants envisioned carrying the device with them and possibly placing it on their desk allowing them to use it whenever they felt necessary to help them understand their state of mental wellbeing or help them relax. This shows the devices must remain small and inconspicuous [102].

The colour and personalisation of devices were repeatably mentioned by participants, and in particular, it was stated the devices should not be black and white and participants would like the ability for the devices to change different colours. When drawing future interfaces, a novel method to personalise the interfaces by attaching extensions that contain additional sensors or feedback was devised to increase engagement. The addition of extensions would enable the devices to adapt to the user enabling the most beneficial sensors and feedback to be included on an individual basis. Additionally, participants suggested the idea of a base interface being developed that can then be customised with different coloured cases to make the device more personal. This highlights the importance of design, while previous work has shown the value of colours, images and layouts in virtual environments [103], this research demonstrates how this translates to physical devices.

2. Tangible methods of physical interaction From the workshops, two main use cases emerged: the first being a tool for recording mental wellbeing state and the second being an interface that could serve as something that helps improve wellbeing. The second use case emerged as participants suggested that they would like toys for fidgeting, stroking, to have it play calming sounds or for recording messages. The most prioritised requirement was makes me feel better. This shows that the participants all value the feedback the device could provide to make them feel better as the highest priority.

When experiencing poor mental wellbeing, people often fidget with objects as fidgeting is a natural response that helps regulate stress [104, 105]. Previous research shows squeezing interactions are preferred by children when angry, but boredom was the most prevalent emotion to trigger fidgeting, and clicking was preferred when bored [106]. One might stroke a furry stuffed toy to calm down or because one is sad; therefore, a number of the co-designed interfaces included mechanisms for fidgeting whether that be stroking a soft toy or pressing buttons on a cube. This demonstrates that the fidgeting buttons enjoyed during the co-design workshops are a beneficial addition and shoulded within affective TUIs resulting in the development of tangible fidgeting interfaces highlighting the benefits of similar previous therapeutic robotics such as Paro [107].

3. Sensors for objective measurement In this work, participants also highlighted the benefits of objective measurement over continuous self-reporting. People with intellectual disabilities can find it challenging to express their emotions; therefore, TUIs to monitor wellbeing may help them better manage their emotions by automatically providing onboard interventions such as haptic feedback or calming sounds. This highlights the importance of going beyond self-reporting wellbeing TUIs as used in many previous studies [66, 108, 109] as the inclusion of physiological sensors can realise the potential for TUIs to provide unobtrusive objective monitoring and automated interventional feedback [110] to improve mental wellbeing.

Unlike many existing TUIs for mental wellbeing [66, 108, 109] which only allow for self-reporting, participants saw the benefits of including sensors to measure physiological and motion data to enable the development of future computational models. While there is a danger of relying on sensors to monitor mental wellbeing as it is inherently personal, advances in artificial intelligence are creating highly precise affective models that could help to greatly benefit populations that are traditionally underrepresented. Collecting physiological sensor data opens up new possibilities for objective data analysis compared with online systems that have previously been co-designed for people with ID [50, 51]. While there remain limitations with deep learning affective modelling such as the models only being as good as the data used to train them, this emphasises the importance of including participants with intellectual disabilities in studies.

4. Privacy preservation Privacy was a key consideration when developing the interfaces especially as they can measure physiological data that links to an individual’s health. Therefore, the interfaces do not communicate with any external devices as all data processing and storage is local. Additionally, microphones were not included within the interfaces to capture audio data as continuous recording of voice would be required to infer wellbeing which may raise privacy concerns due to the highly sensitive nature of the data [111].

A potential alternative to inferring wellbeing from audio data would be to enable participants to verbally record how they are feeling, but this requires participants to continuously understand and report on their wellbeing similar to a self-report diary, rather than automatically and objectively inferring wellbeing which could be of much greater benefit. While the possibility of embedding a microphone for participants to record messages was explored, it was not considered a suitable or valuable method of data collection. Participants never expressed interest in using the microphone for recording and self-reporting, and there are numerous privacy concerns when recording people especially in real-world environments. Furthermore, participants with intellectual disabilities often find it challenging to communicate their wellbeing resulting in little benefit of embedding microphone.

Preserving privacy is a key design consideration that must be considered throughout the design process. TUIs offer many opportunities to collect objective sensor data which could be greatly beneficial, but this must be carefully paired with high levels of privacy to protect users and gain trust.

5. Age appropriateness The results from this study show there is no one-size-fits-all solution to the design of mental wellbeing interfaces. Instead, different devices should be developed for different age ranges with soft devices for young children and 3D-printed interfaces for older children and adults. Participants found that not all sensors would be appropriate for children, such as the HR sensor, as participants believed from past experiences that children would not be able to keep their fingers in continuous contact with the sensor for the device to accurately measure physiological changes. It was suggested that physiological sensors should be reserved for the 3D-printed interfaces and not the soft interfaces, where children in particular may continuously move their hands preventing accurate readings from being recorded.

A number of prototypes were developed and categorised into soft toys potentially for young children aged 5–8 (middle childhood) as this is when children develop relevant social, emotional and cognitive skills [112, 113] and 3D-printed interfaces for older children (8+) and adults. Age is an important factor to consider when developing tangible interfaces to ensure they are accessible and engaging, helping to reduce stigma with mental wellbeing tools.

Even though all of the participants in the workshops were over the age of 18 and suggesting ideas for children’s interfaces, they were all drawing off personal experiences they experienced during childhood such as not being confident to express their emotions, resulting in the devices that can measure touch and motion promoting gentle interactions such as stroking. Furthermore, one participant only recently turned 18 allowing them to base their contributions for children’s interfaces from recent past experiences, and some other participants are involved in working within a school for children with intellectual disabilities allowing them to have personal insight into the features required in wellbeing TUIs for children. The developed interfaces were also shown to teachers of children with intellectual disabilities who agreed with the requirements established during the co-design sessions and believed the resultant devices would be highly beneficial for children. Similarly, the 3D-printed interfaces were well received by all including teachers and highlight the capability to collect real-world physiological data in addition to enabling fidgeting interactions. This shows the need for separate devices to target different populations and demonstrates the success of the co-design workshops.

Overall, participants suggested numerous outputs for the design of future devices. Based upon user feedback, haptic feedback is a key intervention that should continue to be explored as all participants found it relaxing and preferred similar vibration patterns. Visual, haptic and auditory feedback can all be embedded within future devices, although shape-shifting may be more challenging and the addition of a screen would require careful consideration to ensure, it does not induce further stress. Overall, participants believed by including these feedback mechanisms, future interfaces could improve their mental wellbeing in real time.

4.1.3 Which technologies can be embedded within TUIs for real-world wellbeing data collection based on people with intellectual disabilities?

TUIs possess many opportunities when combined with sensing technologies. This co-design study has aimed to identify the most appreciated technologies and sensors that can be embedded within TUIs to help automate mental wellbeing detection including physiological and motion sensors. This is a current issue as advances in AI are helping to automate wellbeing recognition [114,115,116], but there has been a lack of research exploring the real-world devices where this could be used, especially for those with ID.

Previous TUIs for mental wellbeing do not embed physiological sensors and have simply been used to enable the self-reporting of the current state of wellbeing using interaction techniques such as recording emotions by squeezing an electronic ball [65, 117]. While these devices only allow users to report a limited number of emotions, participants believed mental wellbeing and education were the areas where these devices could be of most use. Similarly, Mood Sprite [108] and Subtle Stone [109] are handheld devices that allow users to record their emotions. The devices record the time users create new sprites allowing them to be revisited much like a diary, again showing ways in which TUIs can accompany traditional techniques to make treatment more accessible and user-centric. However, these approaches show that there has been little consideration of sensor-based TUIs for wellbeing. Furthermore, there is no evidence of wellbeing TUIs being co-designed with people with ID even though they could offer great benefit.

In this co-design study, a range of non-invasive sensors were explored using co-design sessions to measure physiological changes and physical interactions. Physiological sensors measuring HR, HRV and EDA present the greatest opportunity to automatically monitor wellbeing due to their correlation with the sympathetic nervous system and their non-invasive nature [118,119,120]. However, the results from the co-design workshops show touch and motion sensors should also be included within all devices due to their simplicity and ease of use, with the more complex physiological sensors reserved for 3D-printed interfaces. Accelerometers could be easily embedded within all interfaces, with participants believing the manner in which the devices will be interacted with will be different depending on the user’s state of wellbeing. Similarly, the ability to measure touch is useful as participants believed stroking the soft interfaces was relaxing as it simulated stroking a pet. Participants enjoyed interacting with touch to activate the feedback, such as pressing hard to enable the visual and haptic feedback as they found this method of interaction intuitive. The way in which the device was touched was also suggested as a mechanism to indicate wellbeing, with users potentially squeezing the device harder when angry. This shows similar results to previous therapy robots [107], but combining these sensors with physiological sensors offers many new opportunities for objective measurements. Overall, the physiological, motion and touch sensors included within the developed interfaces were considered beneficial to objectively measure changes in wellbeing and suitable for the target audience.

A range of feedback actuators to serve as interventions and improve wellbeing were explored including haptic, visual and auditory feedback in addition to fidgeting tools. The co-design workshops helped establish the requirements such as the preference for fidgeting buttons [104, 105] and slow haptic feedback [121] which have been shown to improve wellbeing. Participants believed these feedback mechanisms have great potential to improve wellbeing with some participants finding the fidgeting aspects of the final tangible fidgeting interfaces especially relaxing. This demonstrates the use of the device and the fidgeting buttons themselves can serve as a benefit of using the artefact without the need for additional feedback mechanisms. Visual feedback patterns were also enjoyed with participants finding them engaging and relaxing, but it was found that it is not possible to use light patterns or colours to convey information such as current emotional state to users with intellectual disabilities. Previous work has similarly found that directly informing users of their current state of wellbeing can have a detrimental impact [122]. Therefore, to ensure no additional stress is induced, visual feedback has only been used to display varying patterns to aid relaxation. Finally, while the use of auditory feedback was suggested to play stories and calming sounds, the quality of the sound is of high priority. Visual feedback, slow haptic feedback and fidgeting buttons were found most relaxing and have been embedded within the developed prototypes to act as real-time interventions.

The key design proposals discussed within the co-design workshops including appearance, materials and sensors were all similar to previous research that aimed to co-design TUIs for emotion recognition albeit not for participants with intellectual disabilities [102]. Jingar and Lindgren [102] similarly suggested methods of inferring emotions through touch and motion as suggested for the soft interfaces in this co-design study but did not consider the inclusion of physiological sensors which could have a major impact on the accurate monitoring of real-world wellbeing. There remain limitations in the use of TUIs with physiological sensors as users have to place their fingers on the sensors to record data, unlike alternative devices such as wearables where there is constant connection with the body, but the co-designed TUIs offer novel methods of interaction such as fidgeting which is not capable within wearables, allowing the same device to serve for both monitoring and improving wellbeing.

Overall, we contribute to the body of literature by demonstrating the possibility for a range of sensors and feedback actuators to be embedded within TUIs for mental wellbeing going beyond that of previous research which has mostly focused on tangible methods to self-report without objective data collection [66]. The ability to collect objective data increases the accuracy of wellbeing reporting as previous self-reporting TUIs have shown only a moderate agreement between the expected emotion and the recorded emotion [123].

5 Conclusion

The automatic inference of mental wellbeing would be extremely valuable for individuals with intellectual disabilities as their wellbeing is often diagnostically overshadowed and they can find it challenging to express their emotions. TUIs enable the necessary microprocessor and sensors required to monitor mental wellbeing to be embedded, in addition to providing feedback mechanisms that may help improve mental wellbeing in real time.

Inclusive co-design workshops and focus groups have been conducted to rethink the user design approach of mental wellbeing TUIs. Adjustments to traditional co-design techniques included demonstrations, real-time 3D printing, prioritising cards and interactive electronics to enable successful and practical co-design with people with intellectual disabilities. In particular, the 3D printing of new interfaces improved engagement, ensured participants understood the discussed technologies and demonstrated how their decisions influenced the final designs. Thematic analysis of the qualitative data outlined many recommendations and resulted in a range of new interfaces being developed.

Overall, the participatory process has enabled the successful design and development of mental wellbeing TUIs for people with a range of intellectual disabilities.