1 Introduction

Effective communication is indispensable across all life domains, encompassing social, educational, and professional environments. Nevertheless, deaf or mute individuals often encounter substantial obstacles when interacting with those who are neither deaf nor mute. The global population of individuals with hearing impairments surpasses 1.5 billion and is projected to exceed 2.5 billion by 2050 (https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss). Conventional communication methods, such as sign language or written notes, may not always be viable or practical. Real-time speech-to-text and text-to-speech technologies present an alternative communication approach that could be more accessible and inclusive.

Sign language recognition has significant and varied social aspects for the deaf and mute, which might have wider effects on education, technology, accessibility, and communication. Communication has two main aspects: a) accessibility; the Deaf and mute communities can benefit greatly from sign language recognition. It creates new communication channels, making it easier for people who use sign language to engage with society at large. b) Reducing Communication Barriers: Recognition systems can help reduce obstacles to communication by forming a bridge between those who use sign language and those who do not. More inclusive social contexts, businesses, and educational settings may result. There are two main educational factors: a) Inclusive Learning Environments: learning may be made easier for Deaf students by including sign language recognition in teaching resources and platforms. It can help students and instructors communicate more effectively and among themselves. b) Skill Development: As sign language recognition technology advances, there may be a rise in demand for studying sign language among Deaf and non-Deaf people. This could encourage a culture that is more accepting and understanding. Concerning Technology and Innovation, there are two main issues: a) Improvements in Human–Computer Interaction: the field of human–computer interaction is broadened by sign language recognition, which encourages creativity in gesture-based interfaces. This might impact how we engage with gadgets and technology beyond communication. b) Wearable Technology: Wearables with sign language recognition capabilities may provide real-time translation services, enabling communication in various contexts. This can be especially helpful for travel, healthcare, and emergency circumstances.

In recent years, the application of text-to-speech and real-time speech-to-text technologies has increased to enhance communication accessibility for deaf and mute individuals. Numerous studies have explored the efficacy of these technologies and their impact on communication accessibility for individuals with hearing and speech impairments. For instance, the authors of Kawas et al. (2016) emphasized the importance of considering the experiences and perspectives of deaf and hard-of-hearing (DHH) students when integrating real-time captioning technology into traditional university lectures. To gain insights into the students' experiences with captioning, the researchers employed qualitative research methods, including in-class observations, interviews, diary studies, and usability assessments. The researchers convened a co-design workshop with eight stakeholders upon obtaining preliminary findings.

The study's conclusions show that the main issues with the available captioning systems are their correctness and reliability. However, the authors' study found that common captioning techniques limit students' independence in the classroom. They also noted several shortcomings in the user experience, such as complicated setups, poor feedback, and limited discretion over caption presentation. Considering these findings, the authors defined design specifications and suggested features for real-time captioning in mainstream classrooms. These results can help create more efficient and user-friendly real-time captioning systems that support inclusion and autonomy for DHH people in various settings, including classrooms.

With an unprecedented rate of change, the current technological developments have greatly improved the possibilities for linking people and facilitating access to education, trade, employment, and entertainment. However, technology can result in a new and inadequately examined type of social marginalization for people with impairments. Although accessibility for people with disabilities has increased thanks to assistive technology, it is important to consider any unintentional negative effects that technology may have.

Alternative viewpoints on inclusive and accessible technology were put up by the authors in Foley and Ferri (2012), which differed from focusing solely on assistive technology. The idea is that technology should be viewed as an all-inclusive concept that should not be discriminatorily categorized based on who it is designed for. The authors present doable improvements to accessibility technologies that will promote usability and adaptability for people with disabilities, including adults and students. These insights are essential for ensuring that technology advances inclusion and accessibility for all people rather than escalating already-existing barriers and fostering brand-new types of social exclusion.

This study will (i) use a hybrid strategy that combines quantitative and qualitative data gathering and analytic techniques to understand better how real-time speech-to-text technology affects communication accessibility, (ii) enhance communication between deaf and mute people and their non-deaf and non-mute counterparts in various contexts, such as classrooms and workplaces. (iii) Examine how many elements, such as communication preferences, cultural norms, and technology literacy, may affect how well real-time speech-to-text technology works in various contexts and with various user groups.

Data collection on the precision and dependability of real-time speech-to-text technology in diverse contexts and with various user groups will be the quantitative component of this study. Statistics will be used to analyze this data to look for trends or patterns in the technology's efficacy. The real-time speech-to-text technology user experience, encompassing aspects like usability, independence, and user happiness, will be the focus of the study's qualitative component. Focus groups, surveys, and interviews will all be used as part of the data collection process for this study. Thematic analysis will be used to find recurrent themes and patterns in the data from various sources.

Artificial Intelligence (AI) and healthcare have witnessed remarkable advancements in various domains, including diabetic retinopathy detection and classification (Bilal et al. 2021a, 2022a). Techniques such as U-Net and deep learning algorithms have shown promise in automating the diagnosis of diabetic retinopathy from medical images (Bilal et al. 2022b). These approaches used convolutional neural networks (CNNs) to segment and classify retinal images (Bilal et al. 2021b). The utility and efficacy of CNNs span multiple fields, notably in medical imaging, where they play a crucial role in improving diagnostic procedures. The deployment of CNNs for multi-class classification tasks has significantly enhanced the precision of medical diagnoses, as illustrated by their application in categorizing medical imagery such as shoulder X-rays, retinal diseases, and breast cancer ultrasound images (Uysal 2023; Uysal and Erkan 2022; Uysal and Köse 2022).

Moreover, the use of ensemble models and the introduction of novel evaluation metrics like localization recall precision (LRP) parameters have contributed to progress in object detection results, further evidencing the versatility and influence of CNN technologies in a range of applications (Hardalaç et al. 2022; Özdaş et al. 2023b). Other AI-based techniques have shown promise in healthcare applications. For instance, the use of Grey Wolf Optimization (GWO) algorithms combined with CNNs has demonstrated improved accuracy in lung nodule detection and classification (Bilal et al. 2022c, 2022d). The integration of the Firefly Algorithm with CNNs for retinal disease classification was introduced in (Özdaş et al. 2023a), highlighting its potential to optimize feature selection, improve performance, and reduce training time. By concentrating on the balanced detection of changes in image pairs, the authors in (Peker et al. 2022) presented a new loss function coefficient to improve CNN change detection algorithms. These advancements highlight the potential of optimization algorithms and deep learning techniques in disease diagnosis (Bilal et al. 2022e).

In sign language recognition, CNNs have become a fundamental technology due to their exceptional capability to learn and delineate hierarchical features from images independently. The sophisticated architecture of CNNs is proficient in identifying complex patterns in sign language gestures, thus enabling precise gesture classification and recognition. The effectiveness of CNNs is significantly augmented by employing transfer learning, which adapts pre-trained models to the specific characteristics of sign language datasets, thereby enhancing their gesture recognition capabilities (Uysal et al. 2021). The incorporation of object detection technologies is essential for the instantaneous identification and classification of sign language gestures. Cutting-edge models like YoloV.8 demonstrate the effective collaboration between the capabilities of CNNs and object detection methods. This combination guarantees high accuracy in detecting American Sign Language letters, highlighting the valuable role of these technologies in facilitating communication for individuals who are deaf or mute (Hardalaç et al. 2022).

Drawing inspiration from these developments, this study introduces the Active Convolutional Neural Networks Sign Language (ActiveCNN-SL) framework, which aims to enhance communication accessibility for deaf and mute individuals. By employing similar principles of deep learning and active learning, ActiveCNN-SL holds the potential to minimize labeled data requirements and improve the accuracy of sign language gesture recognition through iterative human feedback. This framework could revolutionize deaf-mute communication by providing real-time and efficient sign language interpretation, fostering inclusivity across various environments. The ActiveCNN-SL framework leverages Resnet50 and YoloV8 models trained on sign language gesture datasets. By integrating optimization principles and deep learning, the proposed framework aims to enhance the precision and accuracy of sign language interpretation for deaf and mute individuals (Bilal et al. 2023a). This paradigm shift in deaf-mute communication has the potential to break down barriers and foster inclusivity, promoting effective communication and understanding between deaf or mute individuals and those who are not deaf or mute (Bilal et al. 2023b).

This study aims to comprehensively understand real-time speech-to-text technology's effectiveness and user experience in enhancing communication accessibility for deaf and mute individuals and their non-deaf and non-mute counterparts. The study's findings will inform the development of more effective and user-friendly real-time communication solutions that promote inclusivity and autonomy for all individuals, regardless of their hearing or speech abilities.

After considering how well text-to-speech and real-time speech-to-text technologies support smooth communication, we now focus on sign language, a vital component of inclusive communication. We offer the ActiveCNN-SL framework to recognize the significance of bridging gaps for individuals with various communication requirements. This framework expands on the advancements in real-time language processing while enhancing its capacity to meet the needs of sign language interpretation.

This research makes significant strides in communication technology for deaf and mute individuals. It investigates the impact of real-time speech-to-text technology, employs a hybrid approach for a comprehensive understanding, identifies influencing factors, develops user-friendly solutions, and proposes the ActiveCNN-SL framework for sign language recognition. The key contributions are:

  • Examined real-time speech-to-text tech's impact on deaf-mute communication.

  • Used a hybrid approach for a comprehensive understanding of technology's efficacy.

  • Identified factors influencing tech efficiency in diverse environments/groups.

  • Developed inclusive, user-friendly, real-time communication solutions.

  • Proposed ActiveCNN-SL for sign language gesture recognition.

The implications of this study's findings reach beyond the immediate context, potentially informing applications of real-time speech-to-text technology in areas such as automatic sign language translation and gesture recognition for virtual and augmented reality interfaces. The structure of this paper is as follows: Section 2 provides a review of relevant literature. Section 3 delineates the proposed ActiveCNN-SL architecture and its associated algorithms. Section 4 presents the experiments conducted and offers an analysis of the results. Finally, Section 5 concludes the paper.

2 Related work

Deaf and mute individuals face significant challenges in communication. They cannot hear what others are saying, and they cannot speak to communicate with others. This can make it difficult for them to participate in social activities, get a job, and attend school. Real-time speech-to-text and text-to-speech technology can improve communication accessibility for deaf and mute individuals. These technologies can convert spoken language into text and text into spoken language. This allows deaf and mute individuals to communicate with others through text and access information only in text format. This section introduces the recent research topics in Real-time speech-to-text (RTT) and text-to-speech (TTS) technology.

The RTT technology is an assistive technology that allows deaf and mute individuals to communicate more easily. RTT devices use a microphone to record speech and convert it into text that can be displayed on a screen or read aloud. This allows deaf and mute individuals to participate in conversations, follow lectures, and read documents without relying on sign language interpreters or lip reading. RTT technology has been shown to impact the communication accessibility of deaf and mute individuals significantly. A study by the National Center for Deaf Health (https://www.urmc.rochester.edu/ncdhr/research/current-research.aspx) found that RTT users reported increased social participation, improved academic performance, and greater employment opportunities. RTT technology has also been shown to improve the quality of life for deaf and mute individuals by reducing stress, anxiety, and isolation.

TTS is another assistive technology that can improve communication accessibility for deaf and mute individuals. TTS devices use text to generate speech, which can be played through a speaker or headphones. This allows deaf and mute individuals to read documents, listen to news and weather reports, and enjoy books and other audio content. TTS technology is effective in improving the communication accessibility of deaf and mute individuals. A study (https://diversity.ucsf.edu/data-reports) by the University of California San Francisco found that TTS users reported increased independence and self-confidence. TTS technology has also been shown to improve the quality of life for deaf and mute individuals by reducing stress and anxiety.

The RTT and TTS technology significantly impact the communication accessibility of deaf and mute individuals. These technologies allow deaf and mute individuals to participate in conversations, follow lectures, and read documents without relying on sign language interpreters or lip reading. RTT and TTS technology have also been shown to improve the quality of life for deaf and mute individuals by reducing stress, anxiety, and isolation. There are many approaches to choose from when attempting to identify hand motions. The method for recognizing hand gestures involves several basic steps: data capture, localization of the hand, feature separation, and identification using recovered features.

Wadhawan and Kumar (2020) proposed a convolutional neural network (CNN)-based sign language recognition system. The system's efficiency was evaluated using fifty different CNN models to conduct the research. Concerning one hundred static signs belonging to separate individuals, a high level of accuracy has been attained in the training phase, with a recorded rate of 99.72% and 99.90% for color and grayscale images, respectively. Barbhuiya et al. (2021) have developed a reliable architecture based on deep learning to recognize sign language. They achieved the highest possible recognition accuracy of 99.82% by first implementing a customized version of the AlexNet and VGG16 models for feature extraction, followed by implementing a support vector machine (SVM) classifier.

Tan et al. (2020) presented an approach for segmenting gestures by deploying a combination of information regarding color and depth. The authors devised a strategy for extracting gesture characteristics by combining the Histogram of Oriented Gradient (HOG) and Hu invariant moment data. The global and local characteristics can be merged efficiently if the ideal weight parameters are first determined—the overall recognition accuracy of 97.8% using the SVM classifier. Duan et al. (2021) have perfected a method that optimizes the effect of processing information using RGB-D. They devised an adaptive weighting algorithm that integrates various factors by considering multi-modal data's independent and interrelated features. Using this method, they achieved the highest possible identification rate of 98.8%.

Liao et al. (2021) improved the mobile nets-SSD network to examine gesture recognition while the subject was obscured. In the specified sequence, the authors conducted training on self-occlusion and object-occlusion motions in the color map, depth map, and color and depth fusion. Subsequently, they evaluated and assessed various models produced to detect occluded gestures. The objective was to identify the model that exhibited the optimal loss function, learning rate, and average accuracy. Barbhuiya et al. (2022) introduced an attention-based VGG16 network motivated by accurately recognizing and classifying comparable gesture characters. The empirical findings indicate that the attention mechanism within the CNN is crucial in effectively classifying posture gesture categories that exhibit high similarity. The proposed model attained a recognition accuracy of 98.02% through holdout validation. Avola et al. (2018) developed a system combining Recurrent Neural Networks (RNNs) and the Leap Motion controller to recognize sign language and semaphoric hand gestures. The RNNs were used to model the temporal dependencies of the input data, while the Leap Motion controller was used to capture the 3D hand movements. The authors evaluated the proposed ASL dataset, showing that the proposed method achieved good recognition rates of 97%.

Mannan et al. (2022) suggested an effective method for ASL recognition that makes use of the 24 alphabets that are employed in sign language. The method proposed for recognizing sign language alphabets is based on deep convolutional neural networks. The DeepCNN model under consideration demonstrates a high level of proficiency in identifying ASL alphabets, achieving a 99.67% accuracy rate when tested on previously unknown data. At first, they only used a single convolutional layer, which resulted in an overfitting of the data. To address this issue, they increased the number of convolutional layers in the suggested algorithm by two. As a result, it now functions more effectively.

In (2018), Chong et al. proposed a system to recognize ASL gestures using the Leap Motion Controller and machine learning techniques. The dataset used for training and testing the model was the ASL fingerspelling alphabet dataset. The authors achieved an accuracy of 93.81% on this dataset. Obi et al. (2023) came up with the brilliant idea of developing a desktop application that interprets sign language and transforms it into text in real-time. This study makes use of datasets based on ASL, as well as the classification technology known as CNN. During the classification process, the picture of the hand is initially filtered. Then, once the filter has been applied, the image of the hand is passed through a classifier, which determines the category of the hand gestures being used. In this study, they concentrate on deciding how accurate the recognition is. The results of this application showed an accuracy of 96.3 percent for all 26 letters of the alphabet.

Sharma et al. (2020) presented a methodology for the real-time identification of hand gestures through image processing and feature extraction techniques. The approach involves capturing an image of the user's hand, which is then pre-processed to remove noise and normalize the image. In Damaneh et al. (2023), Damaneh et al. pre-processed a hand gesture image and its backdrop removed before passing through three feature extraction streams to obtain useful characteristics for classifying the hand motion. The CNN, Gabor filter, and ORB feature descriptors are three widely used hand gesture classification methods. Each extracts its own features to recognize and classify different hand gesture images. Next, the features are combined to create the feature vector. Thus, integrating efficient methods makes the suggested structure more robust to hand gesture uncertainties such as rotation and ambiguity while highly accurately classifying static hand gestures. The proposed framework is comprehensive in comparison to the current picture database architectures. The utilization of transfer learning methodology suggests that the proposed framework could potentially serve as a pre-existing architecture for each static hand gesture image database form instead of deep neural networks such as ResNet50, VGG-16, or AlexNet. Subsequently, the suggested framework is implemented on three distinct collections of static hand gesture images. The findings indicate that the proposed framework achieved an average accuracy of 99.92%, 99.8%, and 99.80% for the Massey test set comprising 758 images, ASL comprising 7020 images, and ASL Alphabet containing 26,100 images, respectively.

Kothadiya et al. (2022) presented a deep-learning model to distinguish words from gestures. Based on feedback-based learning, the LSTM and GRU models detect signals from individual frames of Indian Sign Language (ISL) videos. The IISL2020 dataset employs a sequence of four combinations of Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) models, with each combination consisting of two layers. A univariate LSTM model succeeded by a GRU model achieved a 97% accuracy rate when applied to a dataset of 11 distinct signs. This methodology has the potential to aid individuals who do not possess sign language proficiency in effectively communicating with individuals who are deaf. In their publication, Katoch et al. (2022) presented a sign language recognition methodology that employs Speeded-Up Robust Features (SURF) in conjunction with SVM and CNN. The ISL dataset was utilized for both training and testing the model. The authors analyzed the dataset, and a precision rate of 99% was attained. Lee et al. (2021) proposed a sign language recognition and training method that uses a recurrent neural network (RNN). The dataset used for training and testing the model was the ASL fingerspelling alphabet dataset. The authors achieved an accuracy of 99.44% on this dataset.

The detection of letter and number gestures was proposed by Dima and Ahmed (2021). A proposal has been made to utilize deep learning techniques to recognize sign language. The YOLO v5 model has been proposed as a solution characterized by its lightweight, high-speed, and precise performance. The model is trained and tested on the dataset MU_HandImages_ASL. The gesture was identified in real-time with a precision rate of 98%. Rivera-Acosta et al. (2021) introduced a spelling correction mechanism for ASL that employs the YOLO network and Long Short-Term Memory (LSTM) models. The dataset utilized for training and testing the model was a custom dataset of ASL fingerspelling. The dataset was analyzed, and they attained a precision rate of 98.07%. Jain (2023) introduced a novel dataset, namely the Annotated Dataset for Danish Sign Language (ADDSL). The annotation process of the dataset was carried out utilizing the open-source software LabelImg and the YOLO format. This study involved training a one-stage object detector model, specifically YOLOv5, equipped with the CSP-DarkNet53 backbone and YOLOv3 head. The model aimed to recognize letters (A-Z) and numerals (0–9) using a dataset of seven images per class without augmentation. The study involved training five models for a total of 350 epochs. The results indicated an average inference time of 9.02 ms per image and the highest accuracy of 92% compared to previous research.

Xia et al. (2022) suggested a Heart-Speaker for sign language recognition—a deaf-mute consultation using a Heart-Speaker. The technology solves the costly challenge of treating deaf-mute individuals. The doctor merely needs to point the Heart-Speaker at the deaf patient to capture sign language gestures and translate sign language semantics automatically. The technology displays sign language videos and subtitles when doctors diagnose or ask questions, enabling bidirectional communication between medical practitioners and their patients. MobileNet-YOLOv3 recognizes sign language. It's accurate and runs on integrated terminals. The trial results demonstrate that Heart-Speaker can recognize sign language with 90.77% accuracy. Alawwad et al. (2021) proposed a method that solves both the sign visual descriptor encoding and hand region segmentation problems. We used VGG-16 and ResNet-18 models and a genuine ArSL image dataset to develop and evaluate the Faster R-CNN-based sign recognition system. The proposed approach achieved 93% accuracy and verified the model's robustness against significant background fluctuations in collected pictures.

Overall, these technologies are powerful tools that can significantly improve the communication accessibility of deaf and mute individuals. These technologies can potentially improve the lives of deaf and mute individuals in many ways, including increased social participation, improved academic performance, greater employment opportunities, and reduced stress, anxiety, and isolation. The proposed research explores the impact of real-time speech-to-text and text-to-speech technology on communication accessibility for deaf and mute individuals. The research will investigate these technologies' accuracy, reliability, cost, and awareness. The study will also explore the impact of these technologies on the quality of life of deaf and mute individuals. Table 1 illustrates the recent research in gesture recognition.

Table 1 The recent research in gesture recognition
Full size table

While numerous studies have explored the use of real-time speech-to-text (RTT) and text-to-speech (TTS) technologies in enhancing communication accessibility for deaf and mute individuals, there is a distinct gap in the literature regarding the application of active learning frameworks in sign language gesture recognition. Furthermore, the effectiveness of these technologies in various settings and among diverse user groups remains underexplored. The main research gaps are:

  • There is a lack of studies on applying active learning frameworks, such as Active Convolutional Neural Networks—Sign Language (ActiveCNN-SL), in sign language gesture recognition.

  • Limited research on the efficiency of RTT and TTS technologies across different environments and user groups.

  • Insufficient exploration of the user experience and satisfaction with RTT and TTS technologies.

  • There is a need for more comprehensive studies that combine quantitative and qualitative data collection and analysis methods to evaluate the effectiveness of these technologies.

  • Limited research on potential challenges and limitations of using RTT and TTS technologies and how to improve their effectiveness in promoting inclusivity.

This study aims to contribute to this area of research by exploring the impact of real-time speech-to-text technology on communication accessibility for deaf and mute individuals and their non-deaf and non-mute counterparts. By considering the potential benefits and unintended consequences of technology use, this research aims to develop more effective and user-friendly real-time communication solutions that promote inclusivity and autonomy for all individuals, regardless of their hearing or speech abilities.

3 ActiveCNN-SL: An active learning framework for sign language gesture recognition using CNN

The proposed ActiveCNN-SL framework uses Resnet 50 and Yolo V.8 for training the Sign Language Gesture Images Dataset. ActiveCNN-SL is specifically designed to improve the handling of this dataset for deaf and mute individuals. The framework combines active learning techniques with a CNN model to minimize the labeled data required for training and enhance the accuracy of sign language gesture recognition. The main phases of the ActiveCNN-SL framework are as follows:

  1. 1.

    Initialization Phase: A small, labeled dataset of sign language gesture images trains an initial CNN model. This model is then employed to select a subset of unlabeled images that the model is uncertain about or likely to misclassify. This subset of images is earmarked for the subsequent phase of the framework.

  2. 2.

    Active Learning Phase: In this phase, the selected subset of unlabeled images is presented to a human expert for labeling. The expert labels these images, and the newly labeled images are incorporated into the training dataset. The CNN model is subsequently retrained using this expanded dataset. This iterative process can be repeated, with the newly labeled images continually added to the training dataset and the model retrained on the larger dataset.

  3. 3.

    Validation Phase: In each iteration of the active learning phase, the CNN model's performance is assessed on a distinct validation dataset. The validation set comprises annotated images not utilized during the training phase. The CNN model's efficacy is evaluated by measuring its accuracy, precision, recall, and F1 score. The active learning phase persists until the validation accuracy attains satisfaction.

  4. 4.

    Testing Phase: In this phase, the final trained CNN model is employed to predict the sign language gesture of new unlabeled images. The CNN model's accuracy is assessed utilizing a test dataset comprising images of sign language gestures.

ActiveCNN-SL offers various advantages compared to traditional CNN models. By incorporating active learning, the framework significantly reduces the number of labeled images required for training an accurate CNN model. Additionally, the model can be continuously improved through human feedback, enhancing recognition of sign language gestures. ActiveCNN-SL can also be adapted for other image recognition tasks where obtaining labeled data is limited or costly. Figure 1 visually represents the general block diagram of the proposed "ActiveCNN-SL" framework.

Fig. 1
figure 1

The general block diagram of the proposed ActiveCNN-SL

Full size image

The ResNet50 and YOLOv8 models were chosen for our ActiveCNN-SL framework due to their different benefits and capabilities, which aligned with the specific requirements of our sign language gesture detection task:

  • ResNet50 was used because of its well-documented performance in computer vision tasks. Its deep design with skip connections successfully tackles the vanishing gradient problem, allowing for rapid feature extraction and excellent classification. Its 50-layer design enables successful representation learning, which is critical for distinguishing complicated visual patterns in sign language gestures.

  • YOLOv8: The object detection software YOLOv8 was chosen for its real-time detection performance and efficiency. Unlike classic region-based approaches, YOLOv8 breaks images into grids and predicts bounding boxes directly, making it appropriate for quickly and correctly recognizing several motions inside a frame. Its speed and precision are especially useful for our real-time sign language gesture recognition program.

3.1 Initialization phase

During the initialization phase of ActiveCNN-SL, an initial Convolutional Neural Network (CNN) model is trained on a small, labeled dataset comprising sign language gesture images. This phase establishes a baseline model that can be enhanced through active learning. The labeled dataset trains the initial CNN model using standard techniques such as backpropagation and gradient descent. Once the initial CNN model is trained, it is employed to identify a small subset of unlabeled images about which the model is uncertain or likely to misclassify. This is accomplished by using the model to predict the labels of the unlabeled images and selecting the images for which the model has the lowest confidence in its predictions. These images are then added to a pool of images used in the next phase of the framework. The steps involved in the initialization phase are depicted in Algorithm 1.

Algorithm 1
figure a

Initialization phase

Full size image

3.2 Active learning phase

The subset of images to be labeled can be selected based on various heuristics, such as determining the images with the lowest predicted probabilities or the highest entropy. The human expert can be obtained using multiple methods, such as crowdsourcing or expert annotation. The iteration in step 3 can be repeated until the desired level of accuracy is achieved or the cost of labeling new samples outweighs the benefit of improving the model's accuracy. The overall steps of the active learning phase are illustrated in Algorithm 2.

Algorithm 2
figure b

Active learning phase

Full size image

3.3 Validation phase

The objective of the validation phase is to ascertain that the model is not exhibiting overfitting tendencies toward the training data. The dataset used for validation furnishes an impartial appraisal of the model's efficacy on novel data. The performance metrics, namely accuracy, precision, recall, and F1 score, offer valuable insights into the model's capabilities and limitations. The criterion for stopping the Active Learning Phase is validation accuracy. Upon achieving a satisfactory level of validation accuracy, the Active Learning Phase is terminated, and the resultant trained model is presented as the ultimate output. Algorithm 3 depicts the general procedures involved in the validation phase.

Algorithm 3
figure c

Validation phase

Full size image

3.4 Testing phase

The final stage of the ActiveCNN-SL pipeline is the Testing Phase, where the fully trained CNN model is used to predict the sign language gestures of new, unlabeled images. The primary aim of this phase is to evaluate the effectiveness of the trained CNN model using a set of images not previously used in either the training or validation phases. The dataset used for testing in this phase is a separate collection of labeled images not used during the training or validation phases. The accuracy of the CNN model is assessed on the test dataset by comparing the model's predicted labels with the actual labels of the test images. The general procedures involved in the testing phase are depicted in Algorithm 4.

Algorithm 4
figure d

Testing phase

Full size image

Two popular and powerful deep learning models, Resnet 50 and Yolo V.8, are employed to train the ActiveCNN-SL framework. Resnet 50 is a convolutional neural network architecture known for its deep layers and skip connections, which help alleviate the vanishing gradient problem and enable effective feature extraction. It comprises 50 layers, including convolutional, pooling, and fully connected layers. Resnet 50 has been widely used in various computer vision tasks due to its strong image classification and feature representation performance.

Yolo V.8, which stands for "You Only Look Once," is an object detection model that excels in real-time detection tasks. Unlike traditional region-based approaches, Yolo V.8 divides the image into a grid and directly predicts bounding boxes and class probabilities. It achieves high detection accuracy and processing speed, making it suitable for applications that require efficient object recognition.

In the ActiveCNN-SL framework, Resnet 50 and Yolo V.8 are utilized to train the sign language gesture images comprehensively. The framework leverages the capabilities of these models to learn meaningful representations and detect relevant features within the images. Resnet 50 and Yolo V.8 contribute to the accuracy and robustness of the gesture recognition system by effectively capturing and processing visual information.

By combining the strengths of Resnet 50 and Yolo V.8, the ActiveCNN-SL framework can accurately recognize sign language gestures, even with less labeled training data. Combining these models enhances the framework's overall performance, enabling it to handle the complexities and variations in sign language gestures effectively. Integrating Resnet 50 and Yolo V.8 in the training process demonstrates the framework's ability to leverage state-of-the-art deep learning models for sign language gesture recognition, ultimately improving communication accessibility for deaf and mute individuals.

4 Implementation and evaluation

This section summarizes the datasets used, the performance metrics applied, and a comparative analysis with existing state-of-the-art techniques. This study utilized two primary datasets: (i) the Sign Language Gesture Images Dataset and (ii) the American Sign Language Letters—v1.

The primary reasons for selecting these datasets are their extensive coverage of sign language movements and their usefulness for training and testing our proposed ActiveCNN-SL system. These databases contain many hand movements in sign language communication, including alphabets, numbers, and other important expressions. Their huge libraries and detailed annotations make them ideal for effectively training deep learning models.

Various datasets used in the literature are available in sign language gesture recognition research. Our preference for the Sign Language Gesture Images Dataset and the American Sign Language Letters dataset, on the other hand, was impacted by several factors:

  • Diversity and completeness: The datasets chosen encompass a broad range of sign language gestures, including alphabets, numbers, and fundamental expressions, offering a comprehensive representation appropriate for our study's objectives.

  • Compatibility with research aims: These datasets are strongly aligned with the aims of our study, allowing us to assess the performance of our proposed framework, ActiveCNN-SL, in successfully detecting varied sign language movements.

  • Quality and annotations: The Sign Language Gesture Images Dataset and American Sign Language Letters datasets contain well-annotated and quality-checked images, guaranteeing that our proposed approach may be reliably trained and evaluated.

4.1 Sign language gesture images dataset

The Sign Language Gesture Images Dataset (https://www.kaggle.com/datasets/ahmedkhanak1995/sign-language-gesture-images-dataset) is a collection of images of hand gestures used in sign language communication. The dataset is designed to provide a comprehensive set of sign language gestures that both deaf and non-deaf individuals can use to understand sign language communication better.

The dataset comprises 37 distinct hand gestures, including the A-to-Z alphabets, the 0 to 9 numbers, and a space that refers to how deaf or dumb individuals indicate space between two letters or two sentences when speaking. There are two folders, or parts, to the dataset. (1)-Gesture Image Data consists of colored pictures of hands doing different gestures. A-Z files include images for A-Z gestures; 0–9 folders have photos for 0–9 gestures, and the '_' folder has images for space. Each gesture image has a size of 50 × 50 pixels and is kept in a folder with a similar name. Every gesture contains 1500 pictures, making up a total of 55,500 pictures for all the gestures in the first folder (in total, there are 37 gestures), as well as (2)-Gesture Image Pre-Processed Data, which has the same number of folders and pictures as the first folder (55,500). The distinction is that these pictures were converted to threshold binary for testing and training. This dataset is highly suited for model training and gesture prediction using convolutional neural networks.

The dataset is intended for various sign language recognition and understanding tasks. For example, the dataset can train machine learning models to recognize sign language gestures or develop computer vision algorithms for translating sign language gestures into spoken or written language. The Sign Language Gesture Images Dataset is a valuable resource for researchers and developers working on sign language recognition and understanding. The publicly available dataset can be downloaded from various online repositories and websites.

4.2 American sign language letters—v1

The American Sign Language Letters Dataset (https://github.com/paulinamoskwa/Real-Time-Sign-Language, https://www.kaggle.com/datasets/grassknoted/asl-alphabet) is a collection of images of hand gestures representing the alphabet from A to Z in American Sign Language (ASL). The dataset is designed for training and testing computer vision algorithms for recognizing sign language letters. David Lee released the American Sign Language Letters Dataset v1, a collection of alphabet pictures with captions. This dataset consists of 26 classes with the letters A through Z in each of the 1728 photos. This collection consists of text-formatted photos with accompanying labels that provide details regarding the object's position based on its x and y coordinates and the bound box's height and breadth. The dataset is divided into three sets: a set for training, a set for validation, and a set for testing. The training set is used for building the model, the validation set is used for tuning its hyperparameters, and the testing set is utilized to assess the model's effectiveness on untested data. This dataset has the following features:

  • YOLO v8 PyTorch format annotations are used for letters.

  • Each image underwent the following pre-processing steps:

  • Auto-orientation of pixel data

  • Resize to 416 × 416.

  • For each original image, the following enhancement was used to produce three different versions:

  • A varying random rotation of -5 to + 5 degrees.

  • A 50% chance of a horizontal flip, a random shearing ranging from -5° and + 5° horizontally & -5° to + 5° vertically.

  • Random Gaussian blur with a pixel range of 0 to 1.25.

  • Random brightness adjustments of between -25 and + 25%.

4.3 Performance metrics

The proposed ActiveCNN-SL framework is assessed using metrics such as recall and accuracy. The F-Measure (FM) score, the harmonic mean of recall and accuracy, is also used. This score considers both false positives and false negatives in its calculation. While FM is more comprehensive than precision, interpreting accuracy is not always straightforward. Accuracy is effective when the costs of false positives and false negatives are equal. However, when these costs differ, it is beneficial to consider recall alongside accuracy. Precision is the proportion of correctly predicted observations to the total predicted positive observations, which can be calculated using Eq. (1). Recall is the proportion of correctly predicted positive results to all actual positive results, which can be computed using Eq. (2).

$$precision=\frac{TP}{TP+FP}$$
(1)
$$recall=\frac{TP}{TP+FN}$$
(2)

where false positive (FP) denotes the number of occurrences incorrectly identified as faulty, whereas true positive (TP) represents the number of cases correctly classified as defective. Additionally, FN stands for false negatives and speaks for the number of occurrences incorrectly classified as non-defective. TN stands for True Negative, representing the number of cases correctly recognized as non-defected. Precision and recall are considered while calculating F-Measure, abbreviated as FM, as calculated in Eq. (3):

$$FM=2*\frac{recall*precision}{recall+precision}$$
(3)

4.4 Experimental results and analysis

The efficacy of the proposed method was validated through two significant experiments. The first experiment utilized the proposed approach to classify sign and digit languages with an accuracy of 99.98%, using Resnet 50 on the first dataset. The second experiment employed the second dataset to perform object detection for recognizing ASL letters using YOLO v8, achieving an average precision of 97.8% for all classes. To the best of our knowledge, YOLO v8 has not been previously employed by researchers for this purpose, and its precise ability to detect ASL letters was demonstrated through this experiment.

4.4.1 Experiment 1: ASL classification with Resnet 50 using sign language gesture images dataset

This study sought to assess the performance of the Residual Neural Network (ResNet 50) in classifying sign language images. The Sign Language Gesture Images Dataset, a publicly accessible dataset comprising 55,500 images of 50 × 50 pixels, was employed for this purpose. The dataset was divided so that 80% was used for training, 10% for validation, and 10% for testing. Figure 2 presents a selection of the dataset images that were used in the training, testing, and validation processes.

Fig. 2
figure 2

Samples of training, testing, and evaluating dataset images

Full size image

The transfer learning technique is employed by fine-tuning the pre-trained ResNet-50 architecture. To ensure a fair comparison, the network is initialized with random weights for the fully connected layer and trained the last convolutional block with a learning rate of 0.01 while keeping the other layers' weights fixed. The Adam optimizer trains the network for 50 epochs on an Nvidia Tesla P100 GPU. Table 2 describes the training configuration parameters.

Table 2 Training configuration parameters using ResNet 50
Full size table

The efficacy of the proposed model is gauged using metrics such as accuracy, precision, recall, and F1 score derived from the test dataset. The model's performance is compared with Convolutional and Recurrent Neural Networks (CNN and RNN), two leading deep learning methodologies. The ResNet model demonstrated a training accuracy of 99.98% and a validation accuracy of 100%, surpassing the baseline CNN and RNN models, which achieved 95.8% and 94.3% accuracy, respectively. The model's precision and recall for all classes were 99.62% and 99.44%, respectively, indicating its ability to differentiate between sign language alphabets and digits accurately. The ResNet model's F1-score was 0.9952, exceeding the other models.

Moreover, it is observed that the ResNet model was robust to overfitting compared to the CNN and RNN models, which showed signs of overfitting on the validation dataset. Thanks to its efficient residual design, the ResNet model was also faster regarding training and prediction time than the CNN and RNN models.

Overall, the results of this paper suggest that ResNet can be an effective deep-learning technique for the classification of sign language images. The high accuracy, precision, and recall of the ResNet model demonstrate its potential to make it easier for those who are speech- or hearing-impaired to communicate. Figure 3 shows model accuracy and model loss. Additionally, the validation result of the proposed model for ASL letter and digit classification using ResNet 50 is shown in Fig. 4. Table 3 presents the performance evaluation metrics of the proposed model in detecting Sign Language Gesture Images Dataset using ResNet 50.

Fig. 3
figure 3

The Proposed model accuracy with ResNet 50

Full size image
Fig. 4
figure 4

The result of the proposed model for ASL letter and digit classification using ResNet 50

Full size image
Table 3 Performance evaluation of the proposed algorithm
Full size table

Table 4 presents a comparative analysis of the performance of the proposed system with existing state-of-the-art techniques evaluated on the test dataset. The proposed system surpassed the existing methodologies in terms of accuracy, thereby underscoring its efficacy in employing real-time speech-to-text and text-to-speech technologies to enhance communication accessibility for individuals who are deaf or mute.

Table 4 Performance comparison of the proposed approach based on ResNet 50 with different techniques
Full size table

4.4.2 Experiment 2: detect the American Sign Language Letters dataset with YOLO v8:

The You Only Look Once version 8 (YOLOv8) framework is employed to detect ASL letters. This framework utilizes convolutional neural networks for object detection and localization tasks. The images undergo pre-processing, which involves resizing them to dimensions of 416 pixels by 416 pixels. The dataset is partitioned into 80% for training, 10% for validation, and 10% for testing. The training set is utilized to train the YOLOv8 model. The model undergoes training for 300 epochs with batch size 16, and the learning rate is set at 0.01. Table 5 provides a detailed description of the parameters used during the training configuration.

Table 5 Training configuration parameters using YOLO v8
Full size table

The efficacy of the YOLOv8 model is assessed using the test set, with the mean average precision (mAP) serving as the performance metric. A higher mAP value signifies superior performance. The YOLOv8 model achieved an overall mAP of 97.8% for all classes on the ASL alphabet dataset, as depicted in Fig. 5. This suggests that the model was successful in accurately detecting the ASL letters. Additionally, Figs. 6 and 7 display the confusion matrix and the outcome of ASL letters using the proposed model with YOLO v8.

Fig. 5
figure 5

The precision-recall curve of the proposed model

Full size image
Fig. 6
figure 6

YOLO v8 Confusion matrix

Full size image
Fig. 7
figure 7

The result of the proposed model for ASL letter detection using YOLO v8

Full size image

Furthermore, the YOLOv8 model's performance is juxtaposed with other cutting-edge models, including Faster R-CNN and Mask R-CNN. As illustrated in Table 6, the YOLOv8 model surpasses these models regarding detection precision and speed. The YOLOv8 model is an exceptionally efficient framework for American Sign Language detection. It exhibits superior performance compared to other state-of-the-art models and holds potential for utilization in applications necessitating precise and rapid sign detection.

Table 6 Performance comparison of the proposed approach-based YOLO v8 with different techniques
Full size table

The algorithm performs incredibly well regarding average precision, suggesting its excellent capacity to detect and localize sign language gestures reliably. While the precision is strong, the average recall, while adequate, has room for improvement. Improving the recall rate entails recording more instances of the gestures within the dataset, resulting in a more thorough detection capacity. The table illustrates the performance evaluation metrics for the proposed algorithm leveraging YOLOv8 in detecting sign language gestures. Each metric provides insight into different aspects of the algorithm's capability.

Table 7 presents the performance evaluation metrics of the proposed algorithm in detecting sign language gestures using YOLOv8.

Table 7 Performance evaluation of the proposed algorithm
Full size table

4.5 Results discussion

The results of this study underscore the efficacy of the proposed ActiveCNN-SL, which employs Resnet 50 for sign language gesture recognition. The model achieved a remarkable training accuracy of 99.98% and a validation accuracy of 100%, surpassing the baseline CNN and RNN models. This underscores the potency of the proposed ActiveCNN-SL in accurately discerning sign language gestures. Moreover, the YOLO v8 model achieved an overall mean average precision (mAP) of 97.8% on the American Sign Language (ASL) alphabet dataset, outperforming previous methodologies. These outcomes suggest that the proposed active learning algorithm holds significant potential to enhance communication accessibility for deaf and mute individuals by accurately recognizing ASL gestures. The high precision and accuracy achieved by the framework present a promising solution for fostering inclusivity in various environments, including workplaces, educational institutions, and public spaces.

Despite the promising results, certain challenges and limitations were identified. Occasional inaccuracies in speech-to-text conversion, particularly in scenarios with background noise or multiple speakers, could potentially impact communication effectiveness. Privacy and security concerns also emerged as crucial considerations for the broader adoption of this technology in public settings. To further enhance the efficacy of real-time speech-to-text and text-to-speech technology, it is recommended to continue refining the accuracy and robustness of the algorithms, addressing the identified challenges. Additionally, improvements in user interface and user experience should be pursued to ensure a user-friendly and intuitive interaction for deaf and mute individuals and their communication partners. Furthermore, addressing privacy and security concerns through robust data protection measures will foster trust and widespread technology adoption.

In conclusion, the findings of this study underscore the positive impact of real-time speech-to-text and text-to-speech technology on communication accessibility for deaf and mute individuals. The proposed sign language gesture recognition algorithm demonstrated exceptional performance, offering a promising avenue for improving communication accessibility. By addressing the identified challenges and implementing the recommended improvements, the technology can promote inclusivity and enhance the quality of life for individuals with hearing and speech impairments. Further research and development in this field will continue to explore the potential applications and broader implications of real-time speech-to-text and text-to-speech technology in various domains.

Justification for model selection

The ResNet50 and YOLOv8 were chosen over other models. These models prove their superior performance in accuracy, speed, and detection precision for the sign language gesture identification challenge.

Alternative model evaluation

Although ResNet50 and YOLOv8 were the first picks due to their efficacy, we also explored numerous additional deep-learning models. In our specific application, however, extensive comparative evaluations revealed that ResNet50 and YOLOv8 beat these alternatives regarding accuracy, precision, and efficiency.

The experiments demonstrated a considerable improvement in the YOLOv8 findings. The YOLOv8 model used the default values for hyperparameters. The findings demonstrated increased accuracy, precision, and overall mean average precision (mAP), resulting in a more refined and efficient model for detecting ASL letters.

5 Conclusion

This research project has delved into the impact of real-time speech-to-text and text-to-speech technology on enhancing communication accessibility for deaf and mute individuals. Utilizing a mixed-methods approach, encompassing both quantitative and qualitative data collection and analysis, the study has assessed the efficacy of this technology in bridging the communication gap between deaf and mute individuals and their hearing counterparts. The findings of this research underscore that real-time speech-to-text and text-to-speech technology can significantly bolster communication accessibility for deaf and mute individuals. This technology has demonstrated high precision and efficiency in transcribing speech to text and vice versa, facilitating effective communication between individuals with varying communication abilities. Furthermore, the research has underscored the positive user experience and satisfaction associated with the technology, indicating its potential to foster inclusivity across various settings, including workplaces, educational institutions, and public spaces. By offering accessible communication alternatives, this technology can dismantle communication barriers and promote equal participation and engagement for deaf and mute individuals. However, the research has also pinpointed certain challenges and limitations of using this technology. Background noise, accents, and complex vocabulary can impact speech-to-text conversion accuracy. Additionally, limitations in the availability and affordability of the technology may impede its widespread adoption. It is recommended to continue refining the technology to address these challenges, enhancing its robustness against background noise and its ability to handle diverse accents and complex vocabulary. Further research could also explore strategies to make the technology more affordable and accessible to a wider population. Additionally, future studies could further investigate the potential of integrating this technology with other assistive technologies to enhance communication accessibility for deaf and mute individuals. By advancing in this field, we can strive towards a more inclusive and accessible society for all.