Hello everyone.
In this article, I will introduce the paper I read for this English seminar and share my thoughts on it.

Paper Title: Uncovering students’ problem-solving processes in game-based learning environments
Journal: Computers & Education
Publication Year: 2022
Authors: Tongxi Liu, Maya Israel

This study aims to clarify the problem-solving processes students face in game-based learning environments and connect them to personalized support measures in educational settings. Conventional surveys based on tests or self-reporting have struggled to capture learners’ real-time cognitive activities and have faced issues with the high time costs associated with large-scale implementation. On the other hand, digital games generate vast amounts of detailed log data, holding the potential to visualize processes such as students’ trial-and-error and strategy selection.
In this study, we focus on learners’ behavior during puzzle games within the educational game “Zoombinis” and propose a method that integrates Continuous density Hidden Markov Models (CHMM) with sequence mining. The research questions are as follows:
1. What data mining techniques can be applied to Zoombinis to quantitatively clarify students’ problem-solving processes?
2. How do students develop their processes from trial-and-error to the generalization of solutions?

As for the theoretical background and previous research, problem-solving is considered a cognitive activity that involves exploring, integrating, and transforming information in the face of unknown tasks, and has been discussed in mathematics and science education in terms of four phases: “decoding,” “representation,” “processing,” and “implementation.”
Previous research on game-based learning has proposed that repeated trials and feedback in games contribute to learning promotion. In particular, it is suggested that the process of applying efficient strategies through trial-and-error and generalizing solutions across multiple problems is key.
In this study, we set four phases for students’ problem-solving behavior: “trial-and-error,” “systematic testing,” “execution of solutions,” and “generalization of solutions.” These were defined as unplanned attempts, sequential testing, presenting solutions in one or all dimensions, and pattern extraction through repetition, respectively.

As for the research method, 30 students in grades 3 to 7 (approximately 8 to 12 years old) recruited from local schools and extracurricular programs were asked to play the “Pizza Pass” game mode in Zoombinis. No external support was provided other than a one-hour test play, and all logs were acquired.
Zoombinis records detailed in-game action information such as student ID, timestamps, and selections/removals for each attempt. Based on previous research, all attempts were labeled into the aforementioned four phases based on these actions.

As a data analysis method, initial probabilities, transition matrices, and observation distribution parameters were estimated based on a multivariate Gaussian mixture model using a “Continuous density Hidden Markov Model (CHMM).” The log-likelihood of each attempt was compared across the four-phase model, and the phase showing the maximum likelihood (the most plausible value) was identified.
Next, through sequence mining, the selection/removal action sequences of all 906 attempts were encoded, and frequent sequences were extracted using PrefixSpan. Through this, we quantitatively grasped the three major strategy patterns: “trying one by one,” “adding,” and “filtering.”

As a result of the CHMM analysis, it was revealed that the most frequent behavioral path is “trial-and-error → systematic testing → execution of solutions → generalization of solutions.” This path shows a typical process of moving from initial exploration through planned verification to attempting solutions, and finally generalizing patterns. In addition, cases were observed where some students jumped directly from “systematic testing” to “generalization of solutions,” suggesting that differences in individual problem comprehension and logical construction skills may be influential.
As a result of strategy pattern extraction by sequence mining, the following three major strategy patterns were quantitatively confirmed:
・”Trying one by one”: A basic strategy to reduce unplanned selections and efficiently transition to the execution of solutions.
・”Fixing correct elements and adding a new one (adding)”: A key strategy that promotes the generalization of solutions, effective when verifying multiple dimensions sequentially.
・”Removing and narrowing down after placing all elements (filtering)”: Functions as a bridge from systematic testing to the execution of solutions, forming a foothold for the generalization of solutions.
These patterns were ranked based on student support (s-support) and trial frequency (i-support) in each phase, and it was shown that “trying one by one” has the highest contribution to the “execution of solutions” phase.
Furthermore, when comparing the time spent in each phase per student using the CHMM estimation results, a clear difference was recognized between the group of students who transitioned efficiently to the next phase and the group who remained in “trial-and-error” for a long time. The former had an average of about 12.3 attempts, while the latter had about 25.7, a large gap, suggesting that early application of strategies contributes to overall learning efficiency improvement. Also, although no significant differences were seen between genders or grades, it is possible that familiarity with games or logical thinking skills influence the speed of progress.

As a discussion of the results, CHMM is considered to make a significant contribution to the use of educational data in that it can quantitatively estimate problem-solving phases and their transition probabilities, which are difficult to visualize from multidimensional log data, and explicitly visualize “where students stumble” and “where they make steady progress.” Also, by combining it with strategy pattern extraction, it becomes easier to identify successful patterns that students use autonomously within the game. This leads to the design of systems that automatically provide scaffolding in adaptive learning systems at the timing when strategies such as “filtering” or “adding” are needed.
For example, providing concrete and step-by-step hints to students who cannot get out of “trial-and-error” to guide them to the “systematic testing” phase, or taking advantage of the fact that the “adding” strategy promotes generalization by intentionally providing multi-stage partial feedback at the puzzle design stage, are considerations for lesson design.

Below are my thoughts.
I read this paper because I was curious about how to analyze learning behavior during games to evaluate learner behavior, especially in fields that are not subject-based learning. The analytical framework of identifying learning models based on transition probabilities, including those not directly observed, and predicting important elements for problem-solving was very helpful. On the other hand, a point of concern is that the subjects this time were aged 3 to 7, and there was variation in age, and internal factors such as cognitive/non-cognitive abilities and motivation were not considered. I believe there is potential for further multifaceted analysis, such as analysis that takes into account learning behavior during games and individual characteristics, as well as the presence or absence of long-term effects. Also, this study focused solely on “problem-solving behavior during games.” In contrast, I would like to conduct future research on how learning during games influences real-world complex problem-solving situations by incorporating games as an element of lesson design such as PBL.

Responsible for the text: Kohei Ozaki