Real–Time Sign Language Recognition and Multilingual Speech Output Based on Machine Learning

Lavanya N L; H R Sujay; Akash M; Darshan S; Akash G A

doi:10.5281/zenodo.17598489

Authors

Lavanya N L Department of Computer Science and Engineering, East West College of Engineering, Visvesvaraya Technological University, Bengaluru, Karnataka 560064.
H R Sujay Department of Computer Science and Engineering, East West College of Engineering, Visvesvaraya Technological University, Bengaluru, Karnataka 560064.
Akash M Department of Computer Science and Engineering, East West College of Engineering, Visvesvaraya Technological University, Bengaluru, Karnataka 560064.
Darshan S Department of Computer Science and Engineering, East West College of Engineering, Visvesvaraya Technological University, Bengaluru, Karnataka 560064.
Akash G A Department of Computer Science and Engineering, East West College of Engineering, Visvesvaraya Technological University, Bengaluru, Karnataka 560064.

DOI:

https://doi.org/10.5281/zenodo.17598489

Keywords:

Sign Language Recognition, Mediapipe, OpenCV, Machine Learning, Random Forest, Google Translate API, Human–Computer Interaction

Abstract

For people with hearing and speech disabilities, sign language is an essential means of communication. Yet, a communication gap between signers and non-signers remains large because of limited public knowledge. To overcome this limitation, this work proposes a machine learning–based real-time sign language recognition and translation system. The system captures hand movements using a standard webcam and uses the Mediapipe framework to recognize accurate hand landmarks. These landmarks are subsequently categorized using independently trained Random Forest Classifier models for Indian Sign Language (ISL) and American Sign Language (ASL). The identified gestures are translated into text and then audible speech utilizing the pyttsx3 library, and the Google Translate API provides multilingual translation for cross-linguistic communication. Experimental results show that the system proposed performs accurate real-time recognition performance through regular computing hardware alone.

References

Gnanapriya, S., & Rahimunnisa, K. (2023). A Hybrid Deep Learning Model for Real Time Hand Gestures Recognition. Intelligent Automation & Soft Computing, 36(1).

Abdallah, M. S., Samaan, G. H., Wadie, A. R., Makhmudov, F., & Cho, Y. I. (2022). Light-weight deep learning techniques with advanced processing for real-time hand gesture recognition. Sensors, 23(1), 2.

Yaseen, Kwon, O. J., Kim, J., Jamil, S., Lee, J., & Ullah, F. (2024). Next-gen dynamic hand gesture recognition: Mediapipe, inception-v3 and lstm-based enhanced deep learning model. Electronics, 13(16), 3233.

Meng, Y., Jiang, H., Duan, N., & Wen, H. (2024). Real-Time Hand Gesture Monitoring Model Based on MediaPipe’s Registerable System. Sensors, 24(19), 6262.

Zhang, Y., Yuan, B., Yang, Z., Li, Z., & Liu, X. (2023). Wi-nn: Human gesture recognition system based on weighted knn. Applied Sciences, 13(6), 3743.

Lavanya, N. L., Bhat, A., Bhanuranjan, S. B., & Narayan, K. L. (2023). Enhancing the Capabilities of Remotely Piloted Aerial Systems Through Object Detection, Face Tracking, Digital Mapping and Gesture Control. International Journal of Human Computations & Intelligence, 2(3), 147-158.

Garg, M., Ghosh, D., & Pradhan, P. M. (2023). Multiscaled multi-head attention-based video transformer network for hand gesture recognition. IEEE Signal Processing Letters, 30, 80-84.

Lavanya, N. L., Savanvur, A. K. V., Shrivatsa, R. S., & Shetty, U. K. (2024). LANE MORPH: Machine Learning Powered Divider For Traffic Volume Adaptation. International Journal of Human Computations & Intelligence, 3(6), 378-385.

Slama, R., Rabah, W., & Wannous, H. (2025). Online hand gesture recognition using Continual Graph Transformers. arXiv preprint arXiv:2502.14939.

Kale, H., Aswar, K., Yadav, D. Y. M. K., & Mali, D. Y. (2024). Attendance marking using face detection. International Journal of Advanced Research in Science, Communication and Technology, 417424.

Ahmed, S. T., Basha, S. M., Arumugam, S. R., & Kodabagi, M. M. (2021). Pattern Recognition: An Introduction. MileStone Research Publications.

Kumar, S. S., Ahmed, S. T., Sandeep, S., Madheswaran, M., & Basha, S. M. (2022). Unstructured Oncological Image Cluster Identification Using Improved Unsupervised Clustering Techniques. Computers, Materials & Continua, 72(1).