Hello everyone.
In this article, I will introduce the paper we read in this English seminar and share my thoughts on it.

Paper Title: Leveraging Process-Action Epistemic Network Analysis to Illuminate Student Self-Regulated Learning with a Socratic Chatbot
Journal: Journal of Learning Analytics
Year of Publication: 2025
Volume, Issue, and Pages: 12(1) and 32-49
Authors: Joel Weijia Lai, Wei Qiu, Muang Thway, Lei Zhang, Nurabidah Binti Jamil, Chit Lin Su, Samuel S. H. Ng, Fun Siong Lim

Amidst the rapid proliferation of generative AI, this study aims to clarify the process of Self-Regulated Learning (SRL) using interaction data between students and an educational chatbot. We developed a Socratic chatbot and conducted a three-week experiment with 34 university students to analyze the differences in behavioral patterns between high-achieving and low-achieving students.
This study is based on the theoretical models of Zimmerman and Pintrich, specifically adopting the process-action framework model. This model consists of four phases: “Definition,” “Exploration,” “Engagement,” and “Reflection,” each containing specific actions (e.g., problem identification, goal setting, information seeking, review, and self-evaluation).

As for the research methodology, an experiment was conducted with 34 university students enrolled in an introductory statistics course. It began with the development of the Socratic chatbot. This chatbot was developed in Python and is based on GPT-4. The system prompt was set with the role: “As an AI tutor, break down students’ questions step-by-step and guide them to understand the statistics course materials.” When a student asks a question, the chatbot is designed not to provide a direct answer, but to promote the student’s own thinking through guiding questions. The experiment was conducted over three weeks in a blended learning format, with the first week on-site and the following two weeks online. After taking a pre-test, students were required to watch relevant online videos and interact with the chatbot at their own pace. A post-test was conducted after three weeks to evaluate learning outcomes.

In the data analysis, all conversations between students and the chatbot were coded based on the aforementioned process-action framework. Three researchers performed the tagging, and the inter-rater reliability was evaluated using Krippendorff’s alpha, confirming sufficient reliability with α = 0.693. Students were classified by prior knowledge level (A1 group with 50 points or less, and A2 group with more than 50 points) and learning outcomes (G1, G2, and G3 groups based on score differences), and their behavioral patterns were compared using Epistemic Network Analysis (ENA) and Ordered Network Analysis (ONA).

The analysis yielded interesting findings. Regarding differences in prior knowledge levels, students in the A2 group with high prior knowledge tended to engage more in goal setting and organizing materials. On the other hand, students in the A1 group with low prior knowledge focused more on information seeking. In particular, the ONA analysis suggested that students in the A1 group tended to review before searching for materials, focusing on integrating knowledge before exploring new content. From the perspective of learning outcomes, students in the G3 group (high learning outcomes) mainly engaged in information seeking regarding basic concepts, while students in the G1 group (low learning outcomes) engaged more in self-evaluation activities. Interestingly, 11 out of 12 students in the G3 group belonged to the A1 group (low prior knowledge), indicating that their learning outcomes improved significantly through interaction with the chatbot.

The following are my thoughts. I feel this study is highly significant in that it provides a concrete analytical framework for the currently high-profile issue of how generative AI can be utilized in education. Specifically, regarding the design of the Socratic chatbot, I feel it has educational value distinct from a mere information-provision tool by promoting the learner’s thinking through guiding questions rather than providing direct answers. The fact that educational dialogue is possible through appropriate prompt design provides important implications for future AI utilization. Furthermore, the method of systematically analyzing chatbot interaction data from the perspective of Self-Regulated Learning contains many aspects that will be useful for future research. I also found the application of the process-action framework model interesting. The approach of integrating multiple SRL theories into four phases and utilizing them for the analysis of dialogue data is something I could apply to my own research. In particular, I think the perspective of capturing learner behavior in the stages of “Definition,” “Exploration,” “Engagement,” and “Reflection” is useful for understanding the learning process. I also feel that the fact that different learner groups exhibit different learning behavior patterns is a factor that should be considered in the design of personalized learning support. One point that caught my attention while reading is that it might have been even more interesting if the chatbot’s responses could have been analyzed in more detail. I look forward to future research clarifying not only student behavioral patterns but also what types of questions or prompts promote specific SRL behaviors. Overall, I feel this study demonstrates the potential of educational support utilizing generative AI and provides an effective method for analyzing the learning process in detail. In my own future research, I would like to refer to this analytical approach to gain a deeper understanding of the interaction between AI tools and learners.

Author: Xuewang Geng