Ziyan Che

Central China Normal University, China // zche@mails.ccnu.edu.cn

Chee Kit Looi

The Education University of Hong Kong, China // cklooi@eduhk.hk

Longkai Wu

Central China Normal University, China // longkaiwu@ccnu.edu.cn

Jiyuan Zeng

University of Macau, China // yc58280@um.edu.mo

ABSTRACT: 

While GenAI chatbots are increasingly used in education, a process-oriented understanding of how students learn with chatbots is still limited. To support sound pedagogical use of chatbots, this study adopts an exploratory profiling approach to characterize learner engagement across two course-embedded chatbot activities in a graduate course (N = 29). University students interacted with (1) a multi-persona chatbot that supports open-ended discussion with multiple education theorists and (2) a scaffolding tutor chatbot that guides learners through question–answer–feedback sequences. Multi-modal evidence, including turn-level Higher-Order Thinking codes, behavioral interaction logs, and metacognitive reflections, was used to characterize interpretable engagement signatures (profiles). Rather than proposing generalized learner types for predictive modeling, the profiling approach aims to describe variation in student–chatbot learning processes and to organize complex interaction data into illustrative profiles for pedagogical reflection. Multiple profiles emerged within and across the two chatbot activities, ranging from Critical Evaluators, who demonstrated critique and reflective judgment in open-ended dialogue, to Passive Responders, who primarily relied on structured scaffolding to complete the learning task. These profiles suggest that, under different interaction scripts and task contexts, learner strategies produced diverse observable cognitive, behavioral, and metacognitive engagement patterns. This study further discusses how such profiles can offer a vocabulary for describing interactional variation and generate hypotheses for future adaptive support.

Keywords: 

Generative AI, Higher-order thinking, Process analysis, Instructional design

Ruiqing Du

National Engineering Research Center of Educational Big Data, Central China Normal University, Wuhan, China // duruiqing@mails.ccnu.edu.cn

Zengzhao Chen

National Engineering Research Center of Educational Big Data, Central China Normal University, Wuhan, China // zzchen@ccnu.edu.cn

Debo Ren

National Engineering Research Center of Educational Big Data, Central China Normal University, Wuhan, China // deboren@mails.ccnu.edu.cn

Zhifeng Wang

Faculty of Artificial Intelligence in Education, Central China Normal University, Wuhan, China // zfwang@ccnu.edu.cn

Yawen Shi

National Engineering Research Center of Educational Big Data, Central China Normal University, Wuhan, China // shiyawen@mails.ccnu.edu.cn

ABSTRACT: 

The effective translation of instructional designs into classroom practice remains a central challenge in teacher professional development, yet the field has long lacked both systematic tools for assessing Fidelity of Implementation (FoI) and sufficient empirical evidence on how FoI relates to different dimensions of Teaching Performance (TP). To address these linked gaps, this study first developed and validated an LLM-driven agent integrated with Retrieval-Augmented Generation (RAG) to provide automated formative feedback, and then used this tool to examine the educational effects of FoI-oriented feedback. Its assessments demonstrated robust concurrent validity with human expert ratings. Building on this, a semester-long pre-test/post-test randomized controlled trial was conducted with 30 secondary school teachers, analyzing 240 classroom videos. Analysis of Covariance (ANCOVA) revealed that the experimental group achieved significantly greater gains in both FoI and overall TP compared to the control group. Exploratory correlational analyses further suggested a differentiated pattern, in which gains in FoI were more clearly associated with Classroom Management and Supportive Climate than with Cognitive Activation. These findings suggest a context-dependent pattern: the Agent appeared more effective in supporting procedural aspects of instructional practice than in relation to deep-level pedagogical reasoning. Consequently, this research not only contributes a validated and scalable AI tool but also suggests that automated feedback may be most effective when strategically integrated with human guidance.

Keywords: 

Education assessment, Agent architecture, Instructional content, Instructional design, Large Language Models

Teerawat Kulsuwan

Arts et Métiers Institute of Technology, LAMPA, F-53810 Changé, France // teerawat.kulsuwan@ensam.eu

Sylvain Fleury

Arts et Métiers Institute of Technology, LAMPA, F-53810 Changé, France // sylvain.fleury@ensam.eu

Simon Richir

Arts et Métiers Institute of Technology, LAMPA, F-53810 Changé, France // simon.richir@ensam.eu

ABSTRACT: 

This study investigates the influence of stereotyped pedagogical agents on educational outcomes in immersive virtual environments, with particular emphasis on cross-cultural variation. A total of 107 French and Thai engineering students completed a virtual reality-based instructional module on industrial forging, interacting with either a teacher-style or a peer-style anthropomorphic agent. Drawing on cross-cultural stereotypes in educational traditions and Hofstede’s cultural dimensions framework, the study examined whether preferences for agent instructional style and educational outcomes varied according to national background. Based on empirical evidences between Eastern and Western pedagogical traditions, it was hypothesized that Thai students would prefer teacher-directed guidance, whereas French students would respond more favorably to peer-based interaction. Contrary to expectations, Thai students reported more positive affective responses when interacting with peer-style agents, whereas French students showed little significant improvement in outcomes when interacting with either teacher- or peer style agents. These counterintuitive findings challenge the prevailing assumption that culturally congruent, educationally stereotyped agents systematically enhance learning. Specifically, pedagogical agent stereotypes primarily influenced learners’ affective experiences but had limited effects on cognitive processing or learning outcomes. However, cultural values appeared to shape learners’ emotional responses, although they did not explain differences in knowledge acquisition. Overall, the interpretation of these findings underscores the need for future research to explore alternative theoretical perspectives that may better explain how learners perceive and respond to pedagogical agent stereotypes across diverse cultural and educational contexts in virtual reality-based learning environments.

Keywords: 

Pedagogical agent, Stereotype, Cross-cultural context, Learning, Virtual reality

Nam Ju Kim

Yonsei University, South Korea // namjukim@yonsei.ac.kr

Min Kyu Kim

Georgia State University, United States // mkim120@gsu.edu

ABSTRACT: 

The pedagogical shift into online learning, accelerated by COVID-19, has necessitated increased support to help K-12 students remain academically motivated. Motivation is a key driving force in effective online education, as it positively influences various learning outcomes such as academic performance, engagement, and course completion. Nevertheless, online learning environments, where there is a physical absence of instructors and peers, often cause challenges for students in maintaining their motivation, making it more difficult for teachers to offer support. Despite prior efforts to enhance online education and measure students’ emotions or motivation, existing approaches remain insufficient. Acknowledging that detecting students’ real-time motivation is crucial for identifying ideal learning content, activities, or instructional methods, this study designed and developed the Artificial Intelligence-Augmented Motivation Indicator (AIMI) system. AIMI uses machine learning algorithms and big data related to students’ facial expressions, emotions, and motivation. The system achieved an accuracy of 74.69% and a precision of 77.53%. Furthermore, to assess AIMI’s accuracy against the Instructional Materials Motivation Survey (IMMS), we compared AIMI-generated motivation estimates with IMMS scores from 1,073 students ranging from kindergarten to 12th grade in the southeastern United States. The results showed a strong positive correlation between motivation estimates and IMMS scores, supporting AIMI’s validity and potential applicability for real-time motivation assessment. This study contributes to the emerging field of AI in education and psychology, demonstrating the potential of AIMI to improve responsive and personalized online learning.

Keywords: 

Artificial intelligence, Emotion, Facial expression, Motivation, Online learning