Learner Emotions and AI Scoring: Integrating Affective Signals in Language Evaluation

Abstract

The rapid evolution of artificial intelligence (AI) and the Internet of Things (IoT) is reshaping language assessment practices, yet most AI-driven approaches remain limited in their ability to recognize the affective dimensions that shape learner performance. This study explores how real-time emotional and physiological signals-such as heart rate variability and facial expressions-can be integrated into AI-mediated oral assessment to enhance fairness, validity, and learner support. Using a combination of generative AI (GPT-4), OpenAI Whisper, affective computing, and IoT-enabled biofeedback, an experimental platform was developed to capture and respond to learners' emotional states during speaking tasks. Grounded in Weir's sociocognitive validity framework and Shneiderman's human-centered AI principles, the study involved 24 intermediate English learners and examined the correlation between affective data and oral performance, the alignment between AI and human scoring, and learner perceptions of fairness and emotional support. Results showed that emotional states significantly impacted language output and that integrating affective input improved the alignment between AI and human ratings, particularly in fluency and coherence. Participants generally perceived the system as transparent and empathetic but raised concerns about emotional surveillance and data privacy. The findings contribute to emerging discussions on human-centered assessment design by highlighting how affective responsiveness in AI-supported assessment can foster interpretive fairness, reduce performance anxiety, and support learner agency.