An assistive technology based on Peirce’s semiotics for the inclusive education of deaf and hearing children

Abstract

In the contemporary educational scenario, heterogeneous classrooms have students with different communication needs. Whether we consider the inclusive education area, one of the most relevant drawbacks is that deaf children have needs that are normally ignored for not fitting with the pattern of other students. Furthermore, a few works have performed scientific researches that deeply investigate the application of models and technologies to tackle differences between deaf and hearing children. Therefore, this work proposes a bilingual literacy model based on Peirce’s semiotics for inclusive education that utilizes software technology to improve the communication process between deaf and hearing children in classrooms. An experiment was conducted to observe the level of communication during games played in class and to investigate the quality of interaction by considering the presence of the Portuguese and the Brazilian Signs Language. We also identified three communication categories between deaf and hearing children: (i) Sign Communication; (ii) Common Vocabulary; and (iii) Formal Signaled Communication. We developed and implemented a semiotic model that supports the learning of Portuguese and Libras alphabets, as well as the process of communication between deaf and hearing children.

Appendix

In this section, the analysis of the results obtained with the controlled experiment will be presented. For this purpose, a proper statistical tool was used to treat and to understand the data. The first method being was the Anderson–Darling Test (ad) [19], which tests whether the data comes from a normal population. Therefore: Anderson–Darling test (ad): $F(x) = P(X \le x)$ being the cumulative distribution function of a normal population. G(x) being the empirical data distribution function (EDF), which can be defined as the cumulative distribution function of the relative frequencies. The test evaluates if $G(x) \approx F(x)$. So, our hypotheses are:

• H0: data comes from a normal distribution

• H1: data does not come from a normal distribution

It is noticeable from Table 3 that more than half of the p-values are smaller than 5%, indicating that we rejected the normality hypothesis for these variables at a 5% level. In this sense, we can use non-parametric tests to compare if the variables of the sample without technology and sample with technology and teacher come from the same probability distribution. For this purpose, we will use the Wilcoxon test [19] for two populations. The test hypotheses are described below, where $F_x$ represents the probability distribution of the random variable X and $F_y$, represents the probability distribution of the random variable Y. The test is performed considering a level of significance of $\alpha = 0.05 = 5\%$.

Table 3 Anderson darling normality test

Our main goal is to test the equivalence of the populations regarding the samples without technology and with technology and teacher, therefore we will have the hypothesis of the type:

• H0: $F_\text {UseOfSign-WithoutTechnology} = F_\text {UseOfSign-WithTechnologyTeacher}$

• H1: $F_\text {UseOfSign-WithoutTechnology} \ne F_\text {UseOfSign-WithTechnologyTeacher}$

Taking as basis Table 4, we noticed that only the Use of Signs samples—Without and With Technology (use of signs by the students) and the Use of Libras—Without and With Technology samples (use of Libras among students) are considered to come from different populations. The p-values are equal to 0.002364 and 0.001366, respectively, and smaller than 0.05.

We can conclude from these samples that the use of Libras signs and signs created among children before the use of the technology and after the use of technology and assistance from the teacher are statistically different at a 5% level. The same conclusion is true when using only Libras signs. We will perform some graphical analysis to validate the conclusions above.

Table 4 Wilcoxon test—group without technology and group with technology and teacher

In the current study, the involved variables are:

• Common Communication between the children, using standard language or any type of language. Ex: notes; group games; gestures; etc;

• Port Use of written or spoken Portuguese;

• Libras Use of standard Libras signs;

• PortLibras Use of Portuguese languages and Libras in the same time frame;

• WithoutCod If there was communication without the use of a standard language or code;

• Deaf If there was interaction with deaf students;

• Hearing If there was interaction with hearing students;

• DeafHearing If there was interaction with deaf and hearing students in the same time frame.

One of the objectives of the study is to check if the model and technology used to stimulate the use of Libras signs and the Signs inherent to the children. Since both are variables that present characteristics of the Poisson distribution, it is possible to think of a Generalized Linear Model, based on the Poisson distribution and with logarithmic connection function.

The methodology adopted in the data analysis considers generalized linear models (GLM) proposed by P. McCullagh and J. A. Nelder, Generalized Linear Models, Chapman and Hall, London, 1989. Furthermore, every regression model in fact explains the mean of the response variable, instead of the response variable itself, as presented as follows.

The classical regression model considers that

$$\begin{aligned} Y = X\beta + \epsilon \end{aligned}$$

where $\epsilon$ is a random error that have to satisfy $\mathrm{{E}}(\epsilon ) = 0$, that is, the mean of error must be zero. Here, $\mathrm {E}(\cdot )$ represents the mathematical expectation of a random variable, or expected value, or population mean of the random variable. The consequence of this assumption is that

$$\begin{aligned} \mathrm {Y} = X\beta + \mathrm {E}(\epsilon ) \Rightarrow \mu = X\beta \end{aligned}$$

in which $\mu$ is the mean of response variable. In time series the same occurs, however the practitioners do not know about this mathematical particularity. Our regression model considers that the response has gamma distribution instead of the normal distribution, and for this reason the model becomes

$$\begin{aligned} g(\mu ) = X\beta \end{aligned}$$

since $\mu \in (0,\infty )$ and the function $g(\mu ) = \log (\mu )$ is appropriated to become the regression similar to normal models in which $\mu \in (-\infty ,\infty )$.

The first model to be tested will try to explain the behavior of the random variable Libras by considering the other study variables as covariates (therefore, they are not considered random variables) and, additionally, the two associated groups, namely: Without technology and With technology and teacher. With this goal, we will create a new covariate, which will have values equal to zero for the group Without technology and values equal to one for the group With Technology and Teacher. This variable is known as the Dummy Variable. Here we will represent it by “ID”. After testing some models, we found out that it was more appropriate to explain the average of Libras with:

$$\begin{aligned} \mathrm{log}\mu _{i} = \beta _{1} + \beta _{2}ID_{i} + \beta _{3}\mathrm{Sign}_{i} + \beta _{4}\mathrm{Deaf}_{i}, i = 1, ..., 30 \end{aligned}$$

Note that the data we have are the covariate values: ID, Sign and Deaf and the values of the response variable, for each one of the thirty observations in the sample. The estimation process of the coefficients of the model considers exactly the values of the covariates and the values of y to obtain the estimates of the $\beta$’s and consequently the estimates of $\mu _{1}$, $\mu _{2}$, $\mu _{3}$, $\mu _{4}$, $\mu _{5}$, $\mu _{6}$ ... $\mu _{30}$, where each $\mu$ represents a different moment of observation.

The method used to obtain an estimate of $\beta$’s is the model of maximum likelihood and to test the significance of the $\beta$’s the quasi-t tests are used, being asymptotic tests (approximately).

In Table 5 we present the estimates of the $\beta$’s if the p-Values associated with the quasi-t tests.

Table 5 Estimates of the $\beta$’s if the p Values associated with the Quasi-t tests

Using Table 5 as basis we notice the rejection of all the hypotheses

$$\begin{aligned} H0:\beta _{1} = 0;\, H0:\beta _{2} = 0;\, H0:\beta _{3} = 0;\, H0:\beta _{4} = 0 \end{aligned}$$

Therefore, all the covariates are considered important to build a group model that does not use technology in the same way as the group that used the proposed technology associated with the help of the teacher. This final model is defined as:

$$\begin{aligned} \hat{\mu }= \mathrm{exp}\{-2.99 + 1.08 ID_{i} + 0.68 \mathrm{Sinal}_{i} + 0.10 \mathrm{Deaf}_{i}\} \end{aligned}$$

Another statistical hypothesis that we still must test is: Is the estimated model adjusted to the data of good quality? We can do this based on residue analysis to check if a given model is an adequate representation of the data. The construction of a regression model involves the definition of the distribution of the response variable, choice of the binding function, and choice of covariates. Typically, residues are based on the differences between observed responses (y) and the estimated average $\hat{\mu }$. For instance, $r_{i} = y_{i} - \hat{\mu }_{i}$ with the residual being a measure of discrepancy between the actual data and the adjusted model. Here we will use the residual component of the deviation, the one used with the highest frequency for generalized linear models.

An important residual graphic is the one of normal probability with simulated envelope, which can be used even when the empirical distributions of the residuals are not normal ([13]). If the model is suitable for the data, we expect that most of the residues are randomly distributed within the envelope bands. Based on Fig. 17a, we see that this occurs with the proposed use of the Libras model, confirming the quality of the adjusted model.

Given that another important variable in the study is the quantity of signs, it seems plausible to fit a regression model with the purpose of explaining the number of signs performed by the students while providing a contrast between the use of signs without the intervention of technology and the use of signs with the intervention of technology. The estimated model for quantity of signs was $\hat{\mu }= \mathrm{exp}\{-2.54 + 2.15ID_{i} - 0.10\mathrm{Common}_{i}\}$. From Fig. 17b, we find that the model is well fitted to the data since the residues are mostly inside the envelope and randomly distributed around zero.

Fig. 17

Graphic of residuals

We should emphasize the effect of the covariates on the average number of communication made among the students using Libras, and this effect is measured based on the estimates of $\beta$’s. If the estimate of a $\beta$ associated with the given covariate is negative, the estimated value of $\beta$ will be as larger as the reduction in the average amount of Libras. On the other hand, if the estimated $\beta$ is positive, it is important to increase the student’s performance the higher this estimate became to be. The opposite happens when the estimate of $\beta$ is positive. For instance, $\hat{\beta }_{2} = 1.08$, with this indicating that the increase of one unit in the indicator covariate implies on increase of $(\mathrm{exp}(1.08) -1) \times 100\% = 195\%$ on the average amount of Libras usage. But this is a covariant indicator, which only assumes the value equal to zero (group without technology) and value equal to one (group without technology and teacher). Thus, adding a unit in this covariate implies going from zero to one, or going from the moment without technology to the moment with technology and teacher. If the child is part of the group using technology and teacher, the use of Libras increases by approximately 195%. If the child uses several signs other than Libras for each additional sign used, the average use of Libras increases by 101% and the communication between deaf and hearing students increases in 12% the use of Libras.

The model $\hat{\mu }_{i} = \mathrm{exp}(-2.99 + 1.08ID_{i} + 0.68\mathrm{Sign}_{i}$$+ 0.10\mathrm{Deaf}_{i})$ also allows to estimate the quantity of Libras signs used for the i-student. For example, let’s consider the student $i = 23$. The covariates values for this student are $x232 = 1.0$, since this is a student who has experienced method interference. $x233 = 4.0$ and $x234 = 4.0$, we have $\hat{\mu }_{23} = 3.3 \approx 3.0$, with the true value of the response being $y23 = 3.0$ This is a good example of the model being set correctly.

The model interpretation for the quantity of signs, $\hat{\mu }_{i} = \mathrm{exp}(-2.54 + 2.15ID_{i} + 0.10\mathrm{Port}_{i} + 0.10\mathrm{Common}_{i})$ is interesting. Notice that $\hat{\beta }_{2} = 2.15$, with this implying that the student experimenting interference of the associated technology with the teacher increases in $(\mathrm{exp}(2.15)-1) \times 100\% = 758\%$ the average quantity of signs used by these students. On the other hand, in a logical way, the communication made through the Portuguese language decreases in $(1.0 - \mathrm{exp}(-0.10)) \times 100\% = 9.5\%$ the use of signs.

We conclude that both through the applied tests and through the regression model generated to verify possible inconsistencies, the process of communication between hearing and deaf children improved quantitatively, with increased use of signs, and qualitative, because with the use of these signs, the children had a larger common vocabulary, making the communicative process more semantic. We also conclude that the generated regression model is adjusted to reality, so that the experiment can be scalable, or easily reproduced in other schools that have similar conditions, obtaining similar results.

1.1 Qualitative analysis

This section describes the qualitative analysis of the performed experiment. As previously mentioned, our experiment lasted for five days, with sections of thirty minutes per day recorded, and aimed to observe how communication between deaf children and hearings would occur before and after the technological intervention.

Based on these observations, we tried to understand whether there was some kind of communication or interaction pattern between deaf and hearing children, and how these patterns would have occurred. In the first two days, we have wanted the children to get used to the environment since they were isolated from the other children in the school. From the third day, we have observed the behavior of the children to identify whether there was any pattern of communication between them. We summarize the most significant moments in our experiment by observing the possible patterns of communication among the target children. We group these moments into 3 categories of communication to show to the reader the most significant points according to the opinion of the researcher: (i) gesture communication; (ii) common vocabulary; and (iii) Signed Formal Communication.

We have observed that moments of communication between the hearing children and their deaf colleagues had a significant semantic gain, because the communication that happened before through notes and gestures created by the children, now happens with the addition of signs of Libras thus making the possible moments of conversation extending even further. We also noticed that there was a greater engagement of the deaf student in class time, especially in relation to her colleagues. A priori, when we idealized this experiment, we imagined that the deaf student would be isolated at the beginning, and with the gradual learning of the signs language made by her and her colleagues, the number of interactions would increase. However, we were wrong about this hypothesis because from the very beginning the deaf student communicated with her colleagues without any difficulties. However, we realized that the moments of communication were limited due to the lack of vocabulary. We realized then that the improvement provided by the technology and help of the teacher was related to the quality of the moments of communication. With the increase in the repertoire of the Libras signs, the children had a greater understanding of the situation in which they were involved. With this, we feel that the objectives proposed in this research were achieved.