Hello everyone.
In this article, I will introduce the paper read in our recent English seminar and share my thoughts on it.
Paper title: Game-based learning analytics for supporting adolescents’ reflection.
Journal: Journal of learning analytics,
Publication year: 2021
Volume/Issue: 8(2).
Authors: Cloude, E., Carpenter, D., Dever, D., Azevedo, R., & Lester, J.

This study examined the impact of the quantity and quality of “reflection” performed by learners on knowledge acquisition and problem-solving within the game-based learning environment Crystal Island.
As background, while PBL and GBLE (Game Based Learning Environments) that foster 21st-century skills are becoming widespread, it has been pointed out that learning outcomes are unstable due to the lack of systematic reflection support within games. Reflection is a process of critically reviewing and making sense of experiences, and it is reported that many learners have not acquired sufficient problem-solving skills because explicit instruction is rare in the American curriculum.

1. Theoretical Framework
This study adopted the model by McAlpine et al. (1999) and visualized reflection in the following four stages:
(1) Awareness (becoming aware of goals and the current situation), (2) Explanation (verbalizing the experience), (3) Relating (connecting to past knowledge or experience), and (4) Integration (reflecting the gained insights into future actions).
The key point is that this framework allows for the quantification of the depth of reflection from the perspective of “alignment with learning goals.”

2. Method
120 students were randomly selected from a public middle school in the United States and asked to play Crystal Island, a mystery-solving GBLE based on microbiology. The learning goals were “understanding microbiology concepts” and “identifying the pathogen and presenting a solution.”
As reflection prompts, three types were presented after important actions in the game: (1) What is the important information you learned? (2) What approach are you currently considering for the solution? (3) How could you solve it using a different approach?
The quantity of reflection was based on the number of descriptions, and the quality was evaluated based on four elements: “observation, deliberation, examination, and hypothesis formation.” Indicators for learning outcomes included pre- and post-knowledge test scores and the number of successful problem-solving instances during the game (correctness of pathogen identification and solution presentation). Behavioral logs were also collected and analyzed from multiple perspectives.

3. Results
We analyzed the correlation between the quantity and quality of reflection written by participants and the changes in knowledge test scores and the number of successful problem-solving instances (during the game).
As a result, a significant positive correlation was found between the quantity of reflection and changes in knowledge test scores, but no relationship was found with the problem-solving success rate. On the other hand, the higher the quality of reflection, the better the performance in both knowledge scores and problem-solving. In particular, descriptions involving hypothesis formation were the strongest predictors. In the analysis by prompt, the question “think of an alternative approach” induced the deepest reflection and had the greatest positive impact on learning outcomes.

4. Discussion
From the results of this study, the following points are suggested:
(1) Regarding reflection, quality is a more important factor than quantity for the effect on problem-solving.
It was shown that simply increasing the number of reflections is not enough; problem-solving ability does not improve unless accompanied by strategic deliberation and hypothesis formation directly linked to goals.
(2) The design of questions (prompts) is key.
From the result that prompts regarding alternative approaches encouraged deep deliberation, it is understood that the design of “when and what to ask” determines the depth and outcome of reflection.
(3) Regarding visualization and personalization through Learning Analytics.
By using multimodal Learning Analytics that integrates behavioral logs, speech, and eye-tracking data, there is a possibility to personalize reflection support in real-time.

When applying these research results to class design, teachers are required to intentionally incorporate activities and questions that support reflection so that learners can act toward their goals. It is also pointed out that reflection support should not be viewed merely as a cognitive activity but should be designed in coordination with metacognition and Self-Regulated Learning, and that more effective support is possible by capturing how reflection is linked to behavior in interactive situations such as game environments.

As for limitations and future tasks, the following points are noted. First, the evaluation of the quality of reflection is based on written text and does not necessarily reflect the learner’s actual thinking process. Regarding this, multimodal analysis combining speech, eye-tracking, and behavioral logs is required in the future.
Second, in this study, the timing of reflection was automatically presented based on specific events in the game, but it is also necessary to examine the effectiveness of free-writing formats where learners reflect at their own timing, or self-selected prompts.
Third, personalization of reflection support based on individual differences of learners (e.g., metacognitive ability, motivation, learning style, etc.) is also mentioned as a future research task.

5. Conclusion
This study demonstrated that “quality of reflection” determines the success or failure of learning and problem-solving in a game-based learning environment. The key to encouraging deep reflection lies in prompt design that directly links to learning goals and forces the generation of alternatives, as well as appropriate presentation timing. Providing mere opportunities for reflection is insufficient, and it is suggested that strategic thinking of learners can be fostered by combining “support to improve quality” and “visualization using multimodal data.” In future research, optimization of reflection support according to individual differences and the development of real-time reflection support technology in learning environments are considered important tasks.

Below are my thoughts.
I referred to this paper because I wanted to learn about how reflection relates to learning outcomes and how it is effective to incorporate reflection into games.
What I found particularly helpful was that the quality of reflection is related to learning outcomes and that the quality of reflection is influenced by the design of reflection prompts. Regarding this point, I would like to incorporate reflection into my own research not just as an evaluation, but as an effective means to improve problem-solving ability.
On the other hand, while I understood the relationship between the quality of reflection and learning outcomes, questions remain about what influences the improvement of reflection quality other than prompts. Although there was a mention of “collecting learning behavior logs” in the text, the relationship between learning logs and reflection or learning effects was not deeply mentioned. I would like to consider the possibility that the quality of reflection changes depending on behavior during learning. I am also curious about how the timing and frequency of giving reflection instructions affect it. I would like to continue to learn about the pros and cons of designing from multiple perspectives, such as the load on the learner and the characteristics of the learner.

Written by: Kohei Ozaki