Keywords

1 Introduction

Augmented reality (AR) is one of the recent popular technologies which can be used for simplifying people’s lives by bringing virtual information to their environments. It can be defined as a real-time direct or indirect view of a physical real-world environment that has been augmented by adding virtual computer-generated information to it [1].

A number of recent studies have examined the applicability of AR within various areas of life and showed that it can help people with disabilities [1–6]. These studies mostly investigated potentials for using AR from different points of view in regard to educational purposes for the d/Deaf and hard of hearing (D/HH).

For instance, some studies emphasized the use of AR for vocabulary acquisition [2, 3] since learning how to read and write can pose several challenges for deaf persons, especially when they perceive sign language as a first language and written language as the second language. Moreover, Carmigniani et al. [1] pointed out the importance of using AR as a substitute for users’ missing senses, i.e. when they need a sensory substitute like augmenting hearing by the use of visual cues. Similarly, the University of Applied Sciences in Düsseldorf developed a “Deaf Magazine” [5] where mobile AR application helps sign-language users understand written content by providing additional audio-video material.

The above-mentioned studies mainly focused on examining the purposes AR can be used for. However, to the best of our knowledge, there has been a lack of investigative efficiency in D/HH people’s usages of AR mobile applications where paper-based written information is augmented with a virtual sign-language interpreter (SLI) video. Moreover, there is a lack of D/HH users’ perceptions of the concepts of using AR for the purposes mentioned above. Such investigation may be crucial when addressing e-inclusion in terms of implementing the use of AR in education.

The main purpose of the current study was to meet this deficiency by conducting an experiment where we evaluated the efficiency of using AR when learning how to sign particular words in sign-language and participants’ perceptions of the augmented content. In line with that, a mobile AR application was developed and used to augment paper-based pictures, symbolizing signing words, and with SLI videos displayed on a mobile phone. Thus, the following hypotheses were examined:

RQ1: Are there statistically significant differences in the success rates of signing between different ways of acquiring signing words (a picture symbolizing signing words, the AR mobile application and a SLI) for D/HH and hearing people?

We would expect to find significantly higher success rates in the accuracy of signing words when using the AR mobile application compared to using a picture symbolizing signing words in both D/HH and hearing people (=signers and non-signers). In contrast, success rates in accuracy regarding performed signing words when acquiring signing from the AR mobile application would not be expected to be significantly higher than when watching a SLI personally.

RQ2: Are there statistically significant differences in success rates between signing different words within the same level of difficulty and when using the same ways of acquiring signing words (a picture symbolizing signing words, the AR mobile application or a SLI)?

It would not be expected to find that success rates would vary significantly when signing different words within the same level of difficulty.

The paper is organized as follows. We start by presenting the AR application for learning sign language. Next, the research method is presented by describing the design and procedure of the experiment. Then participants’ characteristics, data analyses and results are shown. Finally, discussion and conclusion with study limitations follow.

2 AR Application for Learning Sign-Language

Figure 1 shows the concept of our designed mobile AR application. A photo of SLI was used as a reference image for augmentation. When the user of a mobile AR application positions his/her phone above the photo, a camera captures it and a SLI video appears within the photo frame. Some general requirements for recording SLI videos were followed when preparing the SLI video [8, 9].

Fig. 1.
figure 1

The concept of AR application for augmenting an object in a physical real-world environment with a SLI video.

Full size image

3 Methods

3.1 Design and Procedure of the Experiment

Table 1 shows the design of the experiment which sought to identify whether the use of AR significantly contributed to the learners’ success rates when signing specific words. The experiment was divided into four sessions: (a) introduction to the test, (b) training session, (c) experimental and evaluation session, and (d) final evaluation session.

Table 1. Design of the experiment
Full size table

3.1.1 Introduction to the Test

Prior to starting the training session, the participants were informed about the experiment in written form and in sign-language. Next, they were asked to sign a written consent form. Filling out an initial questionnaire followed, where participants provided information about their genders, ages, types of hearing loss if any, skills at using smartphone, as well as estimations about sign and written language competencies. The competencies were measured using questions developed on the basis of the adjusted Deaf Acculturation Scale [7]. The questionnaire was put online and all questions were entirely presented in both Slovenian written and sign-language.

3.1.2 Training Session

One picture symbolizing a signing word served for the training session. After watching a single picture for six seconds, participants were asked to sign the word by themselves and two SLIs evaluated its accuracy. Next, a smartphone with AR mobile application was given to them and they were asked to position the phone above a new picture, so that the SLI video appeared within the photo frame on the phone. After six seconds, they were asked to sign the word, which was again evaluated by two SLIs. Finally, a SLI was given six seconds to sign the word and participants were asked to sign it. Once again, two SLIs evaluated its accuracy. The results from the training sessions only served for acknowledgement of the procedure and were excluded from the data analysis.

3.1.3 Experimental and Evaluation Session

This session was conducted in three main steps. During each step, participants were asked to sign two different words, which were six words in total in this session.

In the first step, the participants were given six seconds to view a picture symbolizing a sign for one word. Then, the picture was taken away and participants were asked to sign the word. Two SLIs evaluated them. The same procedure was repeated for the second word.

In the second step, the participants were given the smartphone with the AR application installed on it. Once again, they were asked to capture the picture and were given six seconds before being asked to sign one word. Afterwards, two SLIs evaluated them. The same procedure was repeated for one additional word.

In the third step, the participants were given six seconds to look at the SLI standing in front of them when signing another word. Afterwards, they were asked to sign the word by themselves and two SLIs evaluated them. The same procedure was repeated for the second word.

The above-mentioned procedure was performed during three different tests: A, B and C. Table 1 shows the distributions of words within these tests for training, experimental and evaluation sessions. SLIs evaluated the participants’ signings on a scale from 1 (fail) to 10 (excellent).

3.1.4 Final Evaluation Session

Participants were asked to fill out the final questionnaire, where they reported whether they had known the signs for all the included words before participating in the experiment. In addition, they were asked how difficult it was for them to sign the words. Again, the questionnaire was put online and was entirely presented in both Slovenian written and sign-languages.

3.1.5 Further Conditions

Besides the above-mentioned points, some additional criteria were followed regarding the design and procedure of the experiment. Firstly, we selected the words from a nationally recognized set of three books for learning Slovenian sign-language – the Visual dictionary of Slovenian sign-language. This set of books provides explanations of written words in sign-language both in pictures, symbolizing signing words, and in SLI videos.

While these books provided sign language interpretations for a large number of words, we asked one deaf person, one hard of hearing person and one SLI to consensually choose the words for the experiment. The main criteria for selection of words were frequency of using signing gestures among signers and complexity of gestures. Additionally, a precondition told to these persons was that one word and its signing version would be used in the training session and the additional six words would be used in the main experimental session.

These six words were of two different levels of difficulty. Accordingly, three pairs of words were created, where the first word was of lower and the second word was of higher levels of difficulty. Each pair of words was then used once during each of three steps in the experimental session.

Secondly, two SLIs participated in the experiment as evaluators. The main criterion for their selection was that they were certified SLIs on the national level. We involved two of them in order to increase the reliability of evaluating signed words.

Thirdly, the accuracies of the signed words were evaluated by two SLIs based on their mutually agreed criteria, where consistency of their evaluation was to be assured. In addition, both SLIs were chosen strictly from the surroundings of the town in the north-eastern part of Slovenia where the experiment took place and where participants who were sign-language users came from. Consequently, we tried to avoid possible errors in evaluating the signing due to unpredicted variations in sign-language which might occur when SLIs were used to sign some words in different ways than the participants.

By following the above-mentioned criteria, we tried to avoid the inconsistency in evaluating signing which was also the case in a previous study [10] where the authors emphasized the problem of assessing signing on SLIs’ personal knowledge leading to inconsistency in evaluating the signing.

Fourthly, the times of viewing pictures, SLI videos and SLI were defined according to the longest time needed by a SLI for signing any word among predefined sets of words. Accordingly, the durations of all SLI videos, times of viewing pictures and SLI were the same. In this way, our aim was to meet the same conditions for all three steps of the experimental session.

3.2 Participants

The sample for this study was recruited from the D/HH and hearing populations in Slovenia. The participants were personally invited to participate. The majority of hearing participants were students at the University of Maribor and most D/HH participants who responded were members of the Association of the Deaf and Hard of Hearing Podravje, Maribor.

In total, 34 people participated. Out of these, 12 were d/Deaf or hard of hearing and 20 were hearing; two persons reported that they do not know what level of hearing loss if any they had. Twenty-six participants were male and 8 participants were female. On average, they were 30.38 years old (age range: 21–61, SD = 12.03). Nineteen participants were signers and 15 participants were non-signers.

For the purpose of the current study we classified participants in two groups, while other participants were not subjects of this study:

  • Group 1: D/HH signers (n = 11),

  • Group 2: hearing non-signers (n = 14).

3.3 Data Analyses

A two-way mixed-design between-within subjects analyses of variance (ANOVA) [11] was conducted in order to assess the effect of the mode of presenting signing words on the accuracies of the participants’ signing of these words across groups having different hearing status and status of using sign-language. Statistically significant differences between the three independent groups were checked with a one-way ANOVA [12]. All analyses were performed with SPSS version 21.0.

4 Results

4.1 Results on Research Questions

The first research question sought to identify whether success rates regarding accuracy when performing signing words would be significantly better when using the AR mobile application compared to acquiring signing words from viewing a picture and a SLI personally.

We conducted an analysis for hearing non-signers and D/HH signers. In both groups, evaluations of signing gestures for two words in each category, i.e. picture, AR and a SLI, from two different SLIs were summed. Accordingly, a maximum score in evaluation of a single category was 40. Table 2 shows rounded values for the mean values, standard deviations and standard errors of mean values.

Table 2. Level of success rate in accuracy of signing the words when using a picture only, AR application and a SLI.
Full size table

We conducted a two-way mixed-design ANOVA to compare success rates in accuracy of performing signing words between hearing non-signers and D/HH signers when using three different ways when articulating particular signs: a picture, an AR mobile application and a SLI.

The results showed a significant effect on success rates when using different ways of displaying signing words: Wilks’ Lambda = .50, F(1, 23) = 11.15, p < .001, ηp 2 = .50, with both groups together (hearing non-signers and D/HH signers) showing an increase of success rate when signing words were shown with the second and third ways compared to the first way (see Table 2 and Fig. 2). This indicates that when the signing words were acquired by using the AR mobile application and by watching a SLI’s signing, the success rates increased in both groups. We calculated the increment by using the following equation:

Fig. 2.
figure 2

Success rate in accuracy of signing when acquiring signing words from pictures, the AR mobile application and a SLI.

Full size image
$$ increment = \frac{{\overline{{x_{2} }} }}{{\overline{{x_{1} }} }}*100 - 100 $$
(1)

An increment of 35 % was achieved in success rate when using the AR mobile application instead of a picture. The increment in success rate was 9 % when acquiring a signing word from a SLI personally instead of using the AR application. The most obvious increment is evident in the success rate when watching a SLI compared to using a picture −48 %.

However, no statistically significant interaction was found between the success rate in different ways of acquiring signing words and hearing loss groups: Wilks’ Lambda = .992, F (1, 23) = .09, p > .05, ηp 2 = .008.

What is more, the main effect for the two groups was not statistically significant, F (1, 23) = 3.10, p > .05, ηp 2 = .12. It indicated that there was no significant difference in success rate regarding accuracies of signing certain words for hearing non-signers and D/HH signers.

In order to find an answer on the second research question, we conducted a one-way ANOVA in order to inspect statistically significant differences between different tests (A, B and C) in success rates for each way of acquiring signing words (a picture, the AR application and a SLI). For each word we summed the evaluation marks from both SLIs that evaluated participants’ signings. Thus, each word could be evaluated in total up to a score 20.

The results showed statistically significant differences between the three tests in the following ways of acquiring signing words:

  • Picture 1, F(2, 22) = 9.16, p < .01, ηp 2 = .45,

  • Picture 2, F(2, 22) = 9.05, p < .01, ηp 2 = .45,

  • The first video in the AR mobile application (AR 1), F(2, 22) = 5.60, p < .05, ηp 2 = .34,

  • The first word signed by a SLI (SLI 1), F(2, 22) = 4.54, p < .05, ηp 2 = .29.

In order to discover whether the differences occurred within or between the groups, we conducted post hoc comparisons with a Tukey HSD test. For picture 1, significant differences in success rates were found between the first and third words, as well as between the second and third words. Table 3 shows mean values, standard deviations and standard errors of means for the success rates in the accuracy of signing for the words across ways of acquiring signing words and types of tests.

Table 3. Descriptive statistics on different ways of acquiring signing words and specific words
Full size table

Our result thus indicated that the success rates when signing the word “now” were significantly better than those for the word “break”. Likewise, the success rates for the word “listen” were significantly better than those for the word “break”.

Post hoc comparisons of success rates for picture 2 showed that the differences among all three words were significant, except the differences between the words “to exploit” and “claw”.

In addition, in AR 1 the post hoc comparisons showed that signing the word “break” was evaluated with significantly lower scores than signing the words “now” and “listen”. Similarly, when acquiring the signing word from a SLI (see SLI 1 from Table 3), participants’ signing the word “now” were evaluated as significantly better than signing the word “break”.

4.2 Further Analysis

Further analysis showed no significant differences in the success rates when signing for participants’ prior knowledge of signs. Table 4 shows the results for participants’ skills of using smartphone, as well as skills of using sign and written language. The results are presented for both groups together (hearing and D/HH) and for each group separately.

Table 4. Descriptive statistics on participants’ skills
Full size table

5 Discussion

In this study we found significant effects regarding the different ways of acquiring signing words on the success rates in the accuracies of signing these words. When using the AR mobile application, the success rate was significantly better compared to using a picture symbolizing the same word. When participants were watching a SLI, signing specific words, they achieved significantly higher scores in accuracy of signing compared to using a picture or the AR mobile application.

This finding suggests that AR can be understood as a supportive technology not only to the hearing but also to D/HH users. Our study showed that both groups significantly benefited from using it, which is complementary to the finding by Carmigniani et al. [1] who pointed out the benefit of AR in a way that D/HH users can receive visual cues instead of missed audio information.

Our findings, however, also imply that the role of a SLI is still important, since participants achieved the highest scores when a SLI interpreted signs personally when standing in front of them. It may indicate that AR cannot replace a SLI but could be understood as a complementary tool, especially when the physical presence of a SLI is impossible. Naturally, an additional precondition is also that the content which needs to be interpreted into sign-language is previously known and the SLI video could thus be pre-recorded.

The next finding of our study was that the success rates did not significantly differ between hearing non-signers and D/HH signers when signing by taking different ways of acquiring signs (a picture symbolizing signing words, the AR mobile application or a SLI). This finding may indicate that the AR technology has a potential in learning sign-language not only for beginners but also for those who already use sign-language and have a need to expand their vocabulary by learning new signs.

Our next findings were the significant differences between signing specific words within each way of acquiring signs. Accordingly, we analyzed the differences in success rates for signing the words with lower levels of difficulty and when acquiring a signing word by using a picture symbolizing a sign. The same analysis was performed for using the AR mobile application and watching a SLI personally. Generally, we found higher scores for the success rates when signing the word “now” compared to signing the words “listen” and “break”.

For higher level of difficulty we found a significant difference in performance when signing different words only for a situation when using a picture. Signing the word “butterfly” had significantly better scores in success rates than signing the words “to exploit” and “claw”. This finding can be understood through the frequencies of using certain signs in everyday communication and the complexity of gesture to be accurately shown.

6 Conclusion

The findings suggest that AR has the potential to be used for learning sign language not only among the hearing but also among D/HH people, regardless of their proficiencies in using sign-language. Our study showed that the success rate in the accuracy of performing signs increased by 35 % when acquiring signs from the AR mobile application instead of a picture symbolizing signs. When a SLI was physically present and showed the sign directly to participants, the success rate increased by an additional 9 %, compared to acquiring signs from the AR mobile application. These findings can thus contribute to increasing the awareness of how beneficial AR technology for acquiring signs can be, when learning sign-language.

This study had two main limitations. Firstly, there were small numbers of D/HH and hearing participants included within the study. However, it is important to consider that only a relatively small number of D/HH people are available in Slovenia who could be included in the study [13]. In order to address this limitation, we suggest repeating the study on a larger sample in Slovenia, as well as in other countries.

Secondly, participants’ written and sign language skills were not objectively measured but were rather self-reported. Consequently, we could not know how skilled the participants really were and thus we had to rely on participants’ self-estimations of their competencies.