Hello everyone.

In this article, I would like to introduce a paper I read in the most recent English Literature Seminar and my thoughts on it.

• Paper Title: Using multimodal learning analytics to model students’ learning behavior in animated programming classroom

• Journal: Education and Information Technologies

• Volume: 29(6)

• Pages: 6947-6990

• Year of Publication: 2024

• Authors: Abdullahi Yusuf, Norah Md Noor, Shamsudeen Bello

Learning processes have often been identified by measuring them through methods such as questionnaires and interviews, which rely on students’ self-reports. However, with advances in technology, there has been an increasing interest in objective behavioral indicators of students. Two behavioral indicators in particular have drawn attention: behavioral attention and behavioral engagement. Behavioral attention and engagement can be understood through the following three theoretical models.

First is the Cognitive Psychology Model. In this model, attention and engagement are positioned as filtering mechanisms that determine interest and curiosity and facilitate the amount of information entering working memory. Therefore, attention and engagement are considered extremely important for students to select appropriate activities. Second is the Engagement Model. This model explains that attention and engagement are interrelated. They can be cognitive factors influencing learning and information processing, or emotional factors such as excitement, boredom, curiosity, and anger. Third is the Instructional Curiosity Model. In this model, attention and engagement are treated as indicators showing the status of students’ engagement with tasks and the provision of appropriate support by teachers.

To measure behavioral attention and engagement, student behavior in the classroom is often analyzed. Specifically, the following four aspects are examined:

• Gaze Direction: Paying attention to the teacher’s instructions is considered an essential prerequisite for learning. Interestingly, it has been reported that short fixation durations indicate high attention, while long saccades indicate significant attentional transitions.

• Sitting Posture: It is suggested that people who often sit upright are more likely to recall positive memories or think about pleasant things compared to those who sit facing backward or sideways.

• Note-taking Behavior: According to the information processing model, note-taking increases the likelihood of students retaining the teacher’s lecture content, leading to improved learning outcomes.

• Behaviors that Disrupt the Lesson: Major disruptive behaviors include excessive talking, inappropriate walking, loud voices, and cell phone use. These behaviors lead to decreased learning outcomes and reduced engagement.

However, previous research on student behavior analysis in the classroom has tended to focus on specific actions, often centering on single indicators such as eye movements or operational tasks. Therefore, this study aims to analyze learning behaviors from multiple perspectives by collecting data on gaze direction, sitting posture, note-taking behavior, and disruptive behaviors in a programming classroom, and to predict learning outcomes using machine learning algorithms.

The study was conducted at a university in Nigeria, targeting 35 second-year students majoring in computer science and computer education. The participants included 27 males and 8 females, with an average age of 19.8 years. For programming learning, Alice 3, a block-based programming tool, was used. This environment features the ability to learn basic programming concepts through animations and games. The lessons were conducted over eight weeks; in each lesson, students first watched a 20-minute animated video and then worked on programming exercises. The learning content included fundamental programming elements such as variable declaration and assignment, data types, control structures, and string handling. A post-test was also conducted to measure programming comprehension. Three video cameras were installed in the classroom to record student behavior: one in the center and one on each side, allowing for observation of the students from various angles. Two coders respectively classified the recorded videos into the four behaviors, such as gaze direction and sitting posture. The Kappa coefficient was 0.91, suggesting high coding reliability. The observed behavioral streams were analyzed using a Hidden Markov Model (HMM), and the three best-fitting patterns were identified based on the Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC). Furthermore, multiple linear regression, Random Forest, Naive Bayes, and Support Vector Machine (SVM) were used to predict participants’ learning outcomes from the identified behavioral characteristics.

The analysis revealed that student behavior could be broadly classified into three patterns. The first is the “Active Learning State,” accounting for approximately 43% of the total. Students in this state tended to gaze steadily at the screen, maintain correct posture, and take notes diligently. The second is the “Inactive Learning State,” which was the most frequently observed at approximately 49%. In this state, gazes frequently drifted, postures were often poor, and students were seen engaging in conversations unrelated to the lesson. The third is the “Passive Learning State,” representing about 9%. Students in this state looked at the screen but showed little active behavior such as note-taking. A particularly noteworthy point was the frequent observation of transitions from the “inactive state” to the “passive state.” This might be influenced by the characteristics of animated teaching materials. While animation attracts and maintains students’ attention and engagement, it contains more information than text and can cause cognitive overload in working memory. Regarding learning outcomes, it was found that eye movements and disruptive behaviors were significant factors in predicting grades. In particular, it was confirmed that students with longer gaze durations on the screen and fewer disruptive behaviors tended to have higher learning outcomes.

Several important implications for learning support in animated programming environments were derived from the analysis results. First, from the teacher’s perspective, understanding student behavior patterns would enable more effective support. For example, for students whose gazes frequently drift, it might be effective to give specific instructions on which part of the animation they should focus on. Also, for students whose posture tends to deteriorate, concentration might be maintained by incorporating breaks at appropriate times. Implications were also obtained from the perspective of learning environment design. Animated materials certainly have the effect of attracting students’ visual attention, but they can simultaneously impose a cognitive load. Therefore, when explaining complex concepts, techniques such as adjusting the animation speed or emphasizing key points are necessary.

The above is an overview of the paper, and as for my personal thoughts, I believe this study provides a very interesting perspective. In particular, I think the analysis of multimodal data such as gaze and sitting posture in programming learning provides significant implications for future research. Such detailed analysis of classroom behavior is expected to lead to the development of more effective educational environments. On the other hand, some questions remain. For example, I feel that the explanation of how behavioral attention and behavioral engagement relate to these behavioral data is not sufficient. I also believe that including measurements of prior knowledge levels might enable a more accurate evaluation of learning outcomes. Regarding gaze data, I think more objective measurement will become possible in the future by using devices such as eye trackers.

By: Geng Xuewang