1 Introduction

Information technology is today transforming many different areas, including health and education. Smart, interactive products are able to transmit learning styles in a way that manages to capture children’s attention and motivate them in their activities.

Children learn in many different ways. Furthermore, they may be affected by some type of disability, or in different skills that each is able to develop at their own pace. A hearing impaired child acquires cognitive skills at a slower pace than a hearing child. Hearing impaired children therefore require a special education to receive an appropriate educational development.

Adaptive learning is a method that has been incorporated in the area of education. It requires a computer system in order to create a customized learning experience [5, 8]. It has also been used in the area of health, where psychological aspects are taken into account to adapt the level of difficulty of a game to the skills of the user using neural networks [20].

The term first appeared in 1970 in the field of Artificial Intelligence (AI), in order to adapt an educational process to the strengths and weaknesses of each user. It can thus be said that an adaptive system has the ability to adjust its operation to the goals, tasks and interests, and adapt other characteristics to the profile of the user [1]. This leads to the consideration of a user model that makes it possible to capture information from users while they interact with the system, to thereby learn about the user and thus classify learning styles according to the needs of the child.

Meanwhile, in the Human-Computer Interaction (HCI) line, specifically the philosophy of User-Centered Design (UCD) is a methodology that follows a process in order to identify the needs of the user. This means that with the help of UCD it is possible to identify the characteristics of the user and with AI analyze the data and intelligently adapt user interfaces according to the skills, behaviors and interests of the user.

The result is that AI techniques are increasingly being used by HCI researchers and the use of machine learning applications in HCI research is visibly increasing. Previous research [23, 24] has mentioned the importance of these two lines in creating smart user interfaces, as user tasks become more and more complicated. AI can help to reduce this complexity for users to provide intelligent adaptive techniques.

In this article, Sect. 2 describes how these two lines, HCI and AI, can contribute in creating a smart model for children with hearing disabilities; in Sect. 3, the learning needs that may be involved in children with hearing impairment are discussed; the conceptual model is then proposed in Sect. 4; and finally, Sect. 5 presents several conclusions and outlines future work.

2 Human-Computer Interaction and Artificial Intelligence

Previous research [1, 2, 4] shows the interest in how the dual lines of Human-Computer Interaction and Artificial Intelligence can contribute to creating a smart, interactive model for children with hearing disabilities. The work proposed by Gonzalez et al. [1] consists in a student model, wherein based on an analysis of the different aspects of the user, the most relevant are selected, such as: personal information, learning styles, personality, context, and psychological aspects. This model is useful for adapting a virtual education system to certain content, and adapting activities to a specific type of user. Furthermore, UCD methodology has been used in order to design interfaces tailored to the user profile.

Adaptive models applied in learning offer an intelligent alternative for adapting content according to user preferences. Adaptive learning models have been used as the basis for building adaptive systems applied to a specific context. In 2008 [8] took the AHAM [7] model as their base, upon which they built a system for learning environments online. The model architecture consists of the following modules: student model, domain model, instruction model, adaptive model, and user interfaces. In 2010, Mascio et al. [5] proposed the design of an intelligent adaptive learning system for people with a low level of text comprehension. The learning system is based on an AHA (Adaptive Hypermedia Architecture) model, consisting of a domain model, user model, environment model and a model of adaptation for the learning process. A set of rules are also included that are correlated with the domain and user models.

User-centered design has meanwhile been the subject of much research, such as [6], which proposes a model of analysis of the user in order to structure the information concerning end users with children between 7-11 years old. In this work, an experimental design was applied, consisting of a set of tasks that each child or the teacher carries out in order to analyze and identify the different characteristics of the users. Moreover, in 2012 [2] with a UCD focus, an analysis was carried out of different types of user model where the most important characteristics, focused on psychological, physical and cognitive aspects, are extracted. These were proposals with the aim of improving levels of usability in the systems and a way of integrating software operation according to the characteristics of the user.

The research works discussed show the importance of integrating these two lines, HCI and AI, in an intelligent adaptation model that helps to provide an interaction and feedback specific to children with hearing disabilities. To design a model of intelligent learning, it is important to focus on the user in order to identify the characteristics that may affect the child’s learning. However, the major challenge facing AI is to interpret the activities of the user and predict the objectives correctly [25]. In addition, children with hearing impairment do not behave the same or learn at the same rate as hearing children [21, 22] - so that different levels of difficulty are needed; or some of them have a different method of communication that may affect the context of use and the learning strategies. This leads to the importance of having a model that can adapt to the needs of the child, whether in communication preferences, context of use, or difficulty level, and so on.

3 Learning Needs

Hearing impairment is an obstacle that renders it impossible to process information linguistically through the ear. This is generally known as deafness. Deaf children who do not have hearing aids have only sign language as their communication channel, so they have difficulty developing concepts in a number of areas. However, some deaf children have benefited from hearing aids such as cochlear implants. These children can go on to communicate verbally, and must learn to receive information by means of sounds, so that they need to learn to recognize the sounds via the cochlear implantFootnote 1.

Hearing impaired children face different challenges - cognitive, educational and socio-cultural. Hearing children develop language skills through sounds (sound-letter-word meaning), which corresponds from letter to sound. As such, the same learning styles cannot be applied to hearing impaired children as to hearing children. In addition, children with hearing impairments fail to develop their skills at the same pace as hearing children, which makes it difficult to identify problems in the development of their basic cognitive skills and this can affect their progress in the acquisition of learning. In addition, each child learns at a different pace, indicating that different children may have different learning styles and in turn the different learning styles will correspond to different teaching styles.

A learning style is defined as the strategies that each individual uses when faced with learning new knowledge, in other words the different ways in which an individual may learn, involving cognitive, emotional and physiological traits that can be used as stable indicators as to how users perceive, interact with, and respond to learning environments [13]. It can thus be said that a learning style is made up of one or several teaching models and strategies. A hearing impaired child can be faced with different learning styles, as he has different ways to communicate, such as sign language (visual and gestural communication), lip-reading (visual communication) and oral communication for children with a hearing aid such as a cochlear implant.

A learning model called VARK [19] can be used to classify individuals according to their preference in order to capture and process the information. The VARK model establishes learning strategies according to the sensory preferences of the child, for example: Visual, Auditory, Reading/Writing, and Kinesthetic. A hearing impaired child has greater development in the visual channel [6]; the manner in which these children reach an understanding of the information submitted to them should thus be supported by pictograms. A child with a cochlear implant, however, requires to make use of the auditory channel to learn how to listen. Therefore, a child with hearing disabilities may have different preferences for receiving information. This could be related to the Felder-Silverman learning model [18], which proposes a test to classify the child in the following categories: entry (visual-verbal), perception (sensory-auditory), organization (inductively deductive) processing (active-reflective), and understanding (sequential-global). A problem can occur in the test due to the fact that when the interaction is with children, and particularly those with a hearing impairment, they do not have adequate language skills to respond to the test on their own.

Elsewhere, there are the cognitive skills for each child. These help to classify the level of difficulty and what form the teaching strategies would take. These cognitive skills can be elicited using psychometric tools [27] that assist in determining the interests of the user. Gagne’s theory [14] proposes a domain of cognitive, emotional and motor learning, corresponding with intellectual skills, cognitive strategies, verbal information, attitudes, and motor skills. In turn, it proposes a set of tasks to be considered in learning: gaining the attention of the student, informing the child about the objectives, stimulating them, and providing feedback relating to prior learning, offering stimulating material, providing guidance to the child, checking on their performance, providing information, assessing their performance and enhancing retention transfer.

Thus, it can be stated that the more information obtained about the user, learning strategies better suited to their cognitive, motor skills and attitudes can be applied. Furthermore, the learning theories are subject to the learning styles that can be involved in the application that will interact in a smart way with the child.

4 Conceptual Model

The proposed model contains the following blocks: user profile, evaluation techniques, intelligent environments, and classification. As shown in Fig. 1, each block performs certain functions that are borne in mind throughout the adaptive model.

Fig. 1.
figure 1

Model based on the needs of children with a hearing impairment

Full size image

In User profile, the characteristics of the child with a hearing impairment are defined according to their needs. To obtain the information, researchers worked with two organizations: the USAER program (Regular Education Support Service Units) in Aguascalientes, Mexico, a program involving several state schools that accept children with special needs in regular education; and the Institute for Blind and Deaf Children of the Cauca Valley, in Cali, Colombia, who have children with cochlear implants. Information was gathered both for deaf children whose communication channel is sign language or lip-reading and for children with cochlear implants whose channel of communication is oral. The children ranged from 7-11 years.

Following UCD philosophy, a number of different Evaluation techniques were carried out on user experience (UX) with the children to identify the user profile. This identification is the continuation of work done in [3], where an analysis model is proposed for deaf children. Through different evaluation techniques adapted to UX, aspects of the children in the teaching of literacy are elicited. A search was also conducted for information from different approaches proposed by authors in the identification of aspects of the user [9–12]. The information obtained was analyzed and the most important aspects for children with hearing impairment were selected, as shown in Table 1.

The identified issues are of use in gathering information about the user, so that the system acquires knowledge about users by means of the various actions carried out, in order to determine which elements to adjust according to their needs. The type of information obtained (Fig. 2) may be of two types: explicit, when the user must register information such as personal data and the type of disability (physical/cognitive); and explicit, when information is captured dynamically as the user interacts with the system, such as cognitive, behavioral/academic, emotion, and motivation.

Table 1. Aspects of user profile
Full size table
Fig. 2.
figure 2

Taxonomy of user profile.

Full size image

Emotions are included in the user profile as they are an important part of the driving force of the learning, as they can influence the learning and motivation of the child on trying out the tool. With the aim of fostering motivation, strategies are used that influence such factors, such as challenges, rewards, etc.

To elicit information about cognitive skills in children, psychometric tools are used that can be incorporated as a pre-test within the tool, where the aim is to adjust the level of difficulty of the activities. As a result, the information that is gathered in these different ways is stored in a database, where multiple correlations are created between the data and bring about the possibility of creating profiles and trends. As such, the database is a key element because it makes recording and storage of data possible, as well as tracking the performance of the child.

The intelligent environments block is related to artificial intelligence approaches. In research to date, the techniques most frequently used are Bayesian networks, decision trees, and neural networks [16, 17]. These techniques are within the branch of machine learning, which aims to develop algorithms that are able to generalize behaviors using information that is unstructured and acquired. Machine learning has automated learning algorithms, including supervised and unsupervised. The task of prediction based on learning styles is related to concept learning, a particular case of supervised learning [15]. This is defined as a correspondence between the inputs (attributes) and desired outputs (classes) of the system. This means that the observations of user behavior that are the inputs of the system help the training of the system in being able to predict future actions of the user.

Bayesian networks determine the probability that an element J belongs to a class Ci, given a set of values Vij……Vnj ​​of attributes {A1…….An} of element J.

$$ {\text{P(C}}_{\text{i}} | {\text{A}}_{1} = {\text{V}}_{{1{\text{j}}}} \ldots \ldots \ldots {\text{A}}_{\text{n}} = {\text{V}}_{\text{nj}} ) $$

If the values of the attributes are independent

$$ {\text{P}}\left( {{\text{C}}_{\text{i}} } \right) = \mathop \prod \limits_{{{\text{k}} = 1..{\text{N}}}} {\text{P}}({\text{A}}_{\text{k}} = {\text{V}}_{\text{kj}} | {\text{C}}_{\text{i}} ) $$

It may be noted that the element J corresponds to each child, classes Ci depend on that which it is desired to classify, values (Vij) correspond to each value captured for each child, and each user profile characteristic corresponds to an attribute (An).

Decision trees, meanwhile, are supervised methods of classification that offer very high readability, since the training result is a set of very easy to interpret if-else type statements. The main idea of this type of machine learning is to interpret the training set with a set of rules that must be learned. The different adaptation techniques differ in cost and may influence the accuracy of the adaptation [26]. It is therefore important to select suitable algorithms, as they in turn can influence user satisfaction in displaying the items tailored to the needs of the users.

Fig. 3.
figure 3

User interface for data capture

Full size image

Finally, the classification block is related to the types of variable that will be trained within the system, i.e. if it is desired to adjust the level of difficulty or according to cognitive skills (attention, perception, memory) of the child so that the system can adjust elements of the user interface. However, it is very important to decide what it is sought to classify, because the number of adaptive models that must be used depends on it.

Using this model, decision trees were able to be applied considering the VARK model, which is subject to a condition in the type of disability (physical/cognitive) of the child. In other words, if children have a hearing impairment at a profound level of deafness and do not have a cochlear implant, it means their preference for sensory feedback is visual, so that making use of sounds in a learning tool would not be worthwhile. What is more, if they communicate through sign language, then that ought to be taken into account in the tool. Meanwhile, if it is sought to classify the level of difficulty the child has in relation to the activity, information is gathered from each of the activities carried out using the tool and a record log is kept, where the information is analyzed and the level of difficulty classified (basic, intermediate, advanced) according to that information. The information it is considered important to store is: time that it takes to carry out the activity, the comparison between the frequency of correct and incorrect responses, and the cognitive skills level. The skills level was weighted in a rating between low, medium and high.

To capture information from the user’s profile, a user interface was implemented where explicit data is gathered, such as personal information (name, gender, age, course), disability (physical/cognitive) and photo of the child, as shown in Fig. 3.

Once the teacher has registered the explicit information of the child, the tool can begin to be used. The first interface that appears is a pre-test that the child has to take, a psychometric tool that consists of a set of questions designed to assess the basic cognitive skills of the child. The skills assessed are visual attention, perceptual discrimination, and visual memory. The psychometric tool used was based on a study conducted by the department of psychology at the University of San Buenaventura in Cali, where a psychometric model called SONAR has been designed for hearing impaired children.

The information captured from the pre-test is vital for adjusting the level of difficulty of the activities of the tool. Each of the skills tested with the child is taken as a model to classify into three classes Ci = {low(L), medium(M), high(H)} and each of the observations made by a child corresponds to X(i) to be used as training data and an \( {\text{X}_{{\rm j}}^{{\rm (i)}}} \), which is a characteristic (j) corresponding to an observation of a child (i). Bayesian networks are applied to find the probabilities of each cognitive level, {P(L), P(M), P(H)}. For the values ​​assigned in low, medium and high, it was considered that if the children manage to score between 0-30 in the assessment, they are classified as low; likewise 30–60 medium; and 70–100 high. The pre-test to be applied in the tool will have a total of 15 questions, distributed as follows: 5 visual attention, 5 perceptual discrimination, and 5 visual memory.

Using this pre-test, it is desired to gather information from the training data and thus apply the most appropriate approach of adaptive learning techniques.

5 Conclusions and Future Work

The relationship between the dual research lines of HCI and AI shows the importance of designing interfaces that can be adapted intelligently to the needs of the user. The adaptation model proposed assists in identifying characteristics of children with hearing disabilities and in turn using the techniques or suitable adaptation algorithms to make a classification according to the outputs desired. HCI is a line of research that can contribute to the requirements of the user model that will be used in AI to employ adaptive algorithms. As future work, it is intended to implement and evaluate the model in a case study on the design of a serious game for teaching literacy to children with hearing impairments that adjusts the level of difficulty of the activity and in turn modifies elements in the presentation.