Arseam

International Journal of Advances in Engineering & Scientific Research

EMOTRAX: A MULTI-MODAL AI-POWERED EMOTION RECOGNITION SYSTEM FOR REAL-TIME MENTAL HEALTH SUPPORT USING TEXT AND VOICE INPUTS

Devika D Nair, Harsh Sonker, Dayanitha K, Anushree V, Ayush Kacchap, Dr. Manju Bargavi S K,

Devika D Nair, MCA, Department of CS&IT, Jain Deemed-to-be University

Harsh Sonker, MCA, Department of CS&IT, Jain Deemed-to-be University

Dayanitha K, MCA, Department of CS&IT, Jain Deemed-to-be University

Anushree V, MCOM, School of Commerce, Jain Deemed-to-be University

Ayush Kacchap, MCA, Department of CS&IT, Jain Deemed-to-be University

Dr. Manju Bargavi S K, Professor, School of Computer Science & IT, Jain Deemed-to-be University

DOI : https://doi.org/10.5281/zenodo.15428831 Page No : 35-46

Published Online : 2025-05-16

Download Full Article : PDF Check for Updates

ABSTRACT

With rising mental health concerns, there is an urgent need for intelligent digital tools that provide empathetic and timely support. This paper introduces EmoTrax, a multi-modal emotion recognition system that leverages AI and NLP to detect emotional states in real time. It processes both text and voice inputs to generate personalized mental health recommendations using deep learning models. The system incorporates speech-to-text conversion and sentiment classification to ensure accuracy and contextual relevance. EmoTrax is designed with a user-friendly interface and a scalable backend for seamless interaction. Ethical design is a core focus, with safeguards for privacy, data sensitivity, and user autonomy. Continuous learning is supported through feedback loops and user interaction. The paper also highlights system limitations, such as potential bias in emotion detection, usability challenges, and reliance on third-party APIs. Evaluations of performance and usability show promising results, demonstrating high emotion detection accuracy and positive user engagement. Future developments include integrating facial recognition and physiological signals to enhance emotional insight. These enhancements aim to promote emotional awareness and early intervention. Overall, EmoTrax offers a scalable, responsive solution that bridges the gap between AI technology and mental health support.

Keywords: AI, deep learning, digital wellness, emotion recognition, ethical AI, mental health, multimodal input, NLP, sentiment analysis, user interface

REFERENCES

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805.
Brown, T. et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33, 1877–1901.
Google Cloud. (2023). Speech-to-Text API Documentation. Retrieved from https://cloud.google.com/speech-to-text/docs
MongoDB Inc. (2023). MongoDB: The Developer Data Platform. Retrieved from https://www.mongodb.com/
Gratch, J., & Marsella, S. (2005). Evaluating a computational model of emotion. Autonomous Agents and Multi-Agent Systems, 11(1), 23–43.
Flask Documentation. (2023). Flask Web Framework. Retrieved from https://flask.palletsprojects.com/
Calvo, R. A., & D'Mello, S. (2010). Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications. IEEE Transactions on Affective Computing, 1(1), 18–37.
Picard, R. W. (1997). Affective Computing. MIT Press.
Islam, M. R., Kabir, M. A., & Ahmed, M. U. (2020). Emotion recognition using deep learning approaches: A review. Sensors, 20(18), 5130.
Yin, Z., Zhao, M., Wang, Y., Yang, J., & Zhang, J. (2017). Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Computer Methods and Programs in Biomedicine, 140, 93–110.
WHO. (2022). Mental health and COVID-19: Early evidence of the pandemic’s impact. World Health Organization.
Denecke, K. (2015). Ethical issues in using ambient intelligence in health care. Yearbook of Medical Informatics, 10(1), 17–22.
Yang, Y., & Narayanan, S. (2016). Analysis of emotional speech using voice quality features. IEEE Transactions on Audio, Speech, and Language Processing, 14(4), 1197–1208.
Kocaballi, A. B., et al. (2020). Envisioning ethically-aware AI systems for mental health support. AI & Society, 35(1), 39–51.
Rathi, D., & Soni, M. (2021). Multimodal Emotion Recognition using Machine Learning. International Journal of Computer Applications, 183(22), 8–13.
A. Dhall, R. Goecke, S. Lucey, and T. Gedeon, “Collecting large, richly annotated facial-expression databases from movies,” IEEE MultiMedia, vol. 19, no. 3, pp. 34–41, Jul.–Sept. 2012.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT, 2019, pp. 4171–4186.
T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423–443, Feb. 2019.
K. Schuller et al., “Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge,” Speech Communication, vol. 53, no. 9–10, pp. 1062–1087, Nov.–Dec. 2011.
S. Calvo and D. Peters, “Positive computing: Technology for well-being and human potential,” MIT Press, 2014.
Zhang, Y., Wang, Y., Zhao, M., Yang, J., & Yin, Z. (2021). “Multi-modal emotion recognition using physiological signals and deep learning techniques”.Computer Methods and Programs in Biomedicine, 198, 105787.
Linqin, L., Zhao, X., Wang, J., & Xu, H. (2019). “Emotion recognition based on fusion of speech and text using deep learning.”IEEE Access, 7, 144–155.
Yan, H., Wang, F., Liu, J., & Chen, T. (2022). “Robust multi-modal emotion recognition using physiological signals.”IEEE Journal of Biomedical and Health Informatics”, 26(2), 455–466.

Cite this Article as: Devika D N, Harsh S, Dayanitha K, Anushree V, Ayush K, & Manju B S K. (2025), EmoTrax: A Multi-Modal AI-Powered Emotion Recognition System for Real-Time Mental Health Support Using Text and Voice Inputs”, International Journal of Advances in Engineering & Scientific Research, Volume 12, Issue 2, 2025, pp 35-146. DOI : https://doi.org/10.5281/zenodo.15428831

Article View: 3
PDF Download: 0

REFERENCES

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805.
Brown, T. et al. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33, 1877–1901.
Google Cloud. (2023). Speech-to-Text API Documentation. Retrieved from https://cloud.google.com/speech-to-text/docs
MongoDB Inc. (2023). MongoDB: The Developer Data Platform. Retrieved from https://www.mongodb.com/
Gratch, J., & Marsella, S. (2005). Evaluating a computational model of emotion. Autonomous Agents and Multi-Agent Systems, 11(1), 23–43.
Flask Documentation. (2023). Flask Web Framework. Retrieved from https://flask.palletsprojects.com/
Calvo, R. A., & D'Mello, S. (2010). Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications. IEEE Transactions on Affective Computing, 1(1), 18–37.
Picard, R. W. (1997). Affective Computing. MIT Press.
Islam, M. R., Kabir, M. A., & Ahmed, M. U. (2020). Emotion recognition using deep learning approaches: A review. Sensors, 20(18), 5130.
Yin, Z., Zhao, M., Wang, Y., Yang, J., & Zhang, J. (2017). Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Computer Methods and Programs in Biomedicine, 140, 93–110.
WHO. (2022). Mental health and COVID-19: Early evidence of the pandemic’s impact. World Health Organization.
Denecke, K. (2015). Ethical issues in using ambient intelligence in health care. Yearbook of Medical Informatics, 10(1), 17–22.
Yang, Y., & Narayanan, S. (2016). Analysis of emotional speech using voice quality features. IEEE Transactions on Audio, Speech, and Language Processing, 14(4), 1197–1208.
Kocaballi, A. B., et al. (2020). Envisioning ethically-aware AI systems for mental health support. AI & Society, 35(1), 39–51.
Rathi, D., & Soni, M. (2021). Multimodal Emotion Recognition using Machine Learning. International Journal of Computer Applications, 183(22), 8–13.
A. Dhall, R. Goecke, S. Lucey, and T. Gedeon, “Collecting large, richly annotated facial-expression databases from movies,” IEEE MultiMedia, vol. 19, no. 3, pp. 34–41, Jul.–Sept. 2012.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT, 2019, pp. 4171–4186.
T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423–443, Feb. 2019.
K. Schuller et al., “Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge,” Speech Communication, vol. 53, no. 9–10, pp. 1062–1087, Nov.–Dec. 2011.
S. Calvo and D. Peters, “Positive computing: Technology for well-being and human potential,” MIT Press, 2014.
Zhang, Y., Wang, Y., Zhao, M., Yang, J., & Yin, Z. (2021). “Multi-modal emotion recognition using physiological signals and deep learning techniques”.Computer Methods and Programs in Biomedicine, 198, 105787.
Linqin, L., Zhao, X., Wang, J., & Xu, H. (2019). “Emotion recognition based on fusion of speech and text using deep learning.”IEEE Access, 7, 144–155.
Yan, H., Wang, F., Liu, J., & Chen, T. (2022). “Robust multi-modal emotion recognition using physiological signals.”IEEE Journal of Biomedical and Health Informatics”, 26(2), 455–466.

Submission Instruction

Call for Paper

Submit Paper Online

Join Editorial Board

Author Guidelines

Impact Factor: 6.013

Indexed in

EMOTRAX: A MULTI-MODAL AI-POWERED EMOTION RECOGNITION SYSTEM FOR REAL-TIME MENTAL HEALTH SUPPORT USING TEXT AND VOICE INPUTS

Aims and Objectives

Ethics Policy

Peer Review Policy

Call for Paper

Conference

Instruction To Review

Guide To Authors