Keywords

1 Introduction

Digital systems and connected devices have become an integral part of our living environment: Electronic ‘companions’ make our lives easier and safer and increase comfort in the modern home. Meanwhile, all manufacturers of home- and building technology are relying on ‘smart’ functions for their products. Many of these systems use touchscreens, LC displays or smartphone applications for interaction.

The United Nations Convention for the Rights of People with Disability (UNCRP) [1] is a comprehensive human rights treaty which entered into force on May 3rd, 2008 and has been ratified by 164 countries. In addition to establishing other fundamental rights, the UNCRP “requires countries to identify and eliminate obstacles and barriers and ensure that persons with disabilities can access their environment, transportation, public facilities and services, and information and communications technologies” (Article 9). Looking at every-day-products such as a kitchen stove, washing machine, dishwasher or other household appliances, we can see that accessible user interfaces for such products can be considered a human right covered by the UNCRP.

1.1 Standards and Related Work

Most relevant standards and guidelines for accessible design of domestic appliances include the EN 301 549 (accessibility requirements for ICT products and services) by the European Telecommunications Standards Institute (ETSI) [2], the IEC 63008 (accessibility of household- and similar electrical devices) by the International Electrotechnical Commission [3] and several standards/technical reports of the International Organisation for Standardization, foremost the ISO TR 22411 [4] and the ISO 9241-20 [5].

The problem does not seem to be a lack of standardization or missing expertise in design-for-all principles, but rather a low level of acceptance of these principles in the industry. Although persons with disability could be an attractive customer group – and there is also evidence that accessible user interfaces are preferred by people without disabilities (see also: [9, 10]) – many modern household devices lack even basic accessibility features. In particular, missing haptic feedback and a high complexity of menu navigation impose serious hurdles for many people, especially for people with disabilities and elderly persons [6, 7]. According to the Home designed for all associationFootnote 1, a vast majority of cooktops (hobs) available on today’s market are completely unusable for blind people.

In a comprehensive study involving 52 people with different disabilities and ages, Lee et al. [8] identify a plethora of accessibility issues in modern home appliances and every-day-products, mainly involving problems in discovering functions, perceiving feedback and status messages, and reaching, grasping and controlling functions.

Serafin et al. [9] performed an evaluation of different feedback types for in-vehicle touchscreen systems and found a clear preference for haptic or auditory feedback in addition to the visual feedback. 60–80% of the test subjects would prefer a solution with haptic feedback (the test group consisted of persons without disabilities).

These results could be confirmed by Pitts et al. [10], who conclude in their research that haptic feedback significantly reduces visual workload as well as task completion times when working with touchscreens. They show that haptic feedback compensates information loss which occurs when visual feedback is degraded, and that multimodal (visual and haptic) feedback improves the overall user experience and confidence in the touchscreen interface.

Jiang et al. [11] point out a present trend in home appliance design towards excessive functions and complex operation (e.g. a typical modern microwave oven offers 80–160 functions) and that elderly people are not strongly present in current user experience design studies. They introduce four principles for user experience design, focused on elderly people with decreased learning abilities: Easy-to-Use, Easy-to-Operate, humanization (adequate sensory-, interaction- and emotional experience) and fault tolerance.

Although interesting concepts for multimodal interaction elements have been documented (e.g. the SmartKnobFootnote 2 design by Scott Bezek), to our best knowledge a flexible and accessible user interface for ‘connected’ household appliances does not exist.

In this work, we present the Universal Access Panel (UAP) for enabling barrier-free access to home appliances and IoT devices. This system offers a novel, radically simple, multimodal interaction concept, consisting of just a few interaction elements for accessing dedicated functions of the connected home. Using established application programming interfaces (APIs) and communication protocols such as HomeConnectFootnote 3 or openHab,Footnote 4 various household appliances, consumer electronics products or smart home devices can be integrated and controlled from a single, multimodal interaction console.

2 Method

The system was planned and implemented in a participatory design process, involving two blind accessibility experts who work at the Austrian Federation of the Blind and Partially SightedFootnote 5 (BSVÖ) and two experts from the Home designed for all association (one of them visually impaired). The functional prototype was created over a period of 18 months and evaluated with 5 persons from the target group. The development process consisted of three phases:

  1. 1.

    Initial concept definition: in the first meeting, ideas for the novel interaction concept were discussed and general requirements for the system were defined. For example, functional requirements guarantee haptic, visual and acoustic presentation of important information. Furthermore, various mechanical switches and motorized slider potentiometers were selected for purchase.

  2. 2.

    Implementation with iterative feedback: in several feedback rounds, the user experts contributed important design decisions to the evolving prototype. The selection of mechanical switches was presented so that suitable components could be identified; form and function of the interaction elements and challenges/requirements for software features were discussed. Figure 1 shows design decisions in the process of the participatory development of the UAP.

  3. 3.

    User study: In February 2022, a functional prototype was presented to 5 blind or visually impaired participants and a qualitative user study was conducted. All users had to accomplish predefined tasks using the UAP; subsequently, user interviews were performed and the system was evaluated based on the System Usability Scale (SUS).

Fig. 1.
figure 1

Knob selection for step switch (left), motorized potentiometers (middle), modular interaction components (right)

Full size image

3 Hardware Architecture

Figure 2 depicts an overview of the system architecture. For the main processing module, the M5Stack ESP32 Basic Core IoT development kitFootnote 6 was chosen. This module contains an Espressif ESP32 System-on-Chip (SoC) with dual-core Xtensa 32-bit LX6 microprocessor (up to 240 MHz system clock), built-in 16MB FLASH memory, WiFi- and Bluetooth capabilities. The M5Stack module offers a 2.0-inch color IPS display, an SD-card slot and an audio amplifier/speaker, which are important features for the multimodal characteristics of the access panel. The ESP32 SoC is fully supported with an open source toolchain (Espressif IDFFootnote 7) and drivers/software libraries for all peripherals, as well as a full integration with a real-time operating system (FreeRTOSFootnote 8).

Fig. 2.
figure 2

System overview, Universal Access Panel component interconnection (green box). (Color figure online)

Full size image

3.1 Modular Components

A modular hardware architecture was chosen, which allows the connection of interaction elements to the main module via I2C bus. One or multiple sensor/actuator elements are attached to an 8-bit AVR microcontroller which acts as I2C slave device. The AVR provides analogue and digital input/output and generates a 150 kHz PWM for controlling the motorized fader potentiometer. A custom PCB was made using the Open Source KiCadFootnote 9 design suite, in order to ease connection and reusability for multiple I/O modules. This architecture provides the flexibility to build UAP-variants for particular use cases with minor effort (e.g. a panel with 4 faders for controlling 4 different zones of a cooktop appliance). However, at a particular stage it was decided to focus on a specific panel configuration which is applicable for many different use cases (see Fig. 5). This configuration was utilized in the user study and consists of a) the main module (M5-Stack/ESP32 SoC), b) one interaction element for selection of device/function, c) one motorized fader potentiometer for parameter feedback and manipulation and d) one additional button for committing parameter selections or controlling selected functions. The 3d-printed enclosures were created using the Open Source FreeCadFootnote 10 3d parametric modelling tool.

The main module establishes a wireless (WiFi-) connection to the LAN and accesses local services via an openHab instance which relays communication to other devices, e.g. a KNX/IPFootnote 11 gateway. Furthermore, functions of supported home appliances can be accessed via the HomeConnect-APIFootnote 12 for connected home appliances which offers a cloud-based access to 15 appliance types of 9 major manufacturers of domestic appliances.

4 Software Architecture

The C++ software application for the main processing module was designed for maximum flexibility, in order to support different input modalities and device control standards. The class diagram (see Fig. 3) shows the data structures abstracting real devices and functions in the system, which are described in the following.

4.1 Classes and Data Structures

A function represents a single functional feature of a real device, like ‘hot air’ of an oven or ‘cooktop field 1’ of a stove. Additionally, functions can also refer to single standalone actions like switching a light, or closing shutters. FunctionComposed aggregates several single functions. Different kinds of functions are derived from a base function, depending on which API should be called (e.g. openHAB, HomeConnect, a generic REST API or any other kind of API).

Fig. 3.
figure 3

Data structures/class diagram, representation of devices and functions in the UAP

Full size image

A flexible number of parameters can be associated to a function. Parameters can be changed using a specific interaction component, e.g. the parameter ‘temperature’ of the function ‘hot air’ of an oven could be set using a slider. If a function offers more than one parameter (e.g. temperature and duration), different interaction components could be utilized in order to adjust these parameters individually, or default values for parameters could be used.

A device is the abstraction of either a single real device (like an oven or dishwasher with all its functions) or a collection of different functions of distinct real appliances which are thematically or spatially related (like the user’s favorite functions, or functions located in the living room). Figure 4 shows the program flow of the software application running on the ESP32 SoC, illustrating the user interaction with the UAP.

Fig. 4.
figure 4

Visualization of the program flow of the main software running on the ESP32 SoC

Full size image

4.2 Program Flow and States

The ESP32 SoC queries the connected input devices and receives user interaction parameters via the I2C protocol (see Fig. 2). The IOManager software component processes this information and controls the following states:

  • Select device: user input is used in order to select one of the devices.

  • Select function: once a device is selected, user input is used to select a function.

  • Adjust parameters: if one or more parameters are associated with the selected function, user input is used to adjust theses parameter(s).

  • Execute function: after selecting and configuring a function, an additional user input is used in order to trigger the corresponding API call (execute the action).

5 Results

Figure 5 shows the final configuration of interaction elements and the functional integrated prototype of the UAP which was tested with persons of the target group.

Fig. 5.
figure 5

Configuration of interaction elements (left), functional integrated prototype (right)

Full size image

Every user interaction with the UAP is multimodally perceptible in the following ways: (a) haptically by the usage of mechanical input elements with sensible state – in particular the motorized slider potentiometer, representing the current parameter value and providing force feedback for parameter levels, (b) visually by showing the current state on the LC display of the M5Stack and (c) auditory by announcing the current state as Text-To-Speech (TTS) voice. In order to provide voice feedback for arbitrary messages, the online service ResponsiveVoiceFootnote 13 is used. The audio interface is provided by the ESP8266 Audio library.Footnote 14 To avoid long latencies when downloading voice audio files, sound files are cached locally on the SD card of the M5Stack.

5.1 User Study

A user study was performed with 5 persons of the target group at the ‘LivingLab’ of UAS Technikum Wien in Vienna, Austria. Two of the participants were females and three were males in the age of 52 ± 20.16 years, four of them were blind and one visually impaired. The UAP was preconfigured for controlling different appliances in the ‘LivingLab’, including lights, shutters, oven and dishwasher. After signing the informed consent document, the participants were introduced to the functionality of the prototype. After getting acquainted with the setting, the users had to accomplish 8 tasks using the UAP, including turning on specific lights, starting and stopping the oven, and closing the shutters. The user study started with easy tasks, such as turning the lights on/off, followed by more complex tasks like changing parameters of running devices or stopping functions which are currently running. While the participants performed the tasks, occurring problems were observed. In a subsequent interview, all users were invited to share their thoughts and ideas for improvements. The tasks and observed problems are listed in Table 1.

A technical issue occurred during execution of task 4: the HomeConnect API was blocking remote access to a connected device because this device was accessed locally beforehand. During task 4, the study participants also experienced some difficulties when trying to start the washing machine with the required settings (Wool 60° – 1000 rpm). The reason for these difficulties was that in the current UAP implementation, only one parameter can be changed for a given device function (thus it is required to switch the device function if either temperature or rpm shall be changed). Furthermore, mentionable problems were observed during the final task, where the participants were asked to stop the running programs of the oven and the washing machine. In order to fulfil this task, it is required to navigate to the ‘active functions’ menu and press start button again, which caused some confusion for the participants.

Table 1. List of user study tasks including a description of occurring problems
Full size table

Subsequently, all users completed 10 questions of the System Usability Scale (SUS) [12]. In total, the prototype scored 84.5/100 points in the SUS, indicating good overall usability. The most negative ratings were given for question 10 “I needed to learn a lot of things before I could get going with this system” (7/20 points agreeing) and question 6 “I thought there was too much inconsistency in this system.” (5/20 points agreeing). The high inconsistency rating possibly resulted from an unexpected error that happened in multiple test sessions, causing some parameter values to change without user interaction. The highest positive score (20/20 points) was given for question 1 “I think that I would like to use this device frequently”, which shows an overall positive user experience of the participants. In general, it was observed that younger participants had fewer problems understanding the interaction concept than older ones, but even an 81-year-old person with visual disabilities could perform basic device control using the UAP, which would otherwise have been impossible due to the nature and complexity of the native touch-screen interfaces.

6 Discussion

A novel Universal Access Panel for improving the accessibility of connected devices and home appliances was developed in a participatory design process and evaluated by blind and visually impaired persons. The system provides multimodal feedback and enables access to household appliances and smart home functions which would otherwise be completely inaccessible to different user groups. While the overall feedback of the user evaluation was very good, some weaknesses of the current prototype became obvious: additional user input elements for repeating the last speech output, stopping a program or configuring more than one parameter would be useful. However, adding more interaction elements also makes the device more complex – therefore a proper “trade-off” between functionality and complexity must be found and further evaluated with users. Currently, the configuration of the panel is ‘hardcoded’ and it’s not possible to change it dynamically. It is planned to provide an accessible web-based configuration manager which connects to the UAP and allows adjusting the system to the actual environment of the user.

We want to emphasize that one important purpose of the UAP is to present innovative concepts to the industry, to encourage manufactures to comply with existing accessibility guidelines and to pave the way for a better accessibility of household appliances in general. The goal remains – and it is even more relevant in today’s digitally connected world: Products and appliances shall be useable for everyone!