Hello everyone.

In this article, I would like to introduce a paper I read for the English seminar and share my thoughts on it.

Paper Title: Predictive Student Modeling in Game-Based Learning Environments with Word Embedding Representations of Reflection

Authors: Geden, M., Emerson, A., Carpenter, D., Rowe, J., Azevedo, R., & Lester, J.

Volume/Pages: 31(1), 1-23.

Year of Publication: 2021

Journal: International Journal of Artificial Intelligence in Education

Background

This study examines whether it is possible to predict post-test performance at an early stage by quantifying short reflections written by students during game-based learning, including the linguistic context, and combining them with learning logs and pre-test scores. In classes and games, the timing of support significantly influences learning effectiveness. If teachers can estimate “who needs what kind of support and to what extent” at an early stage, they can make more accurate decisions, such as providing hints or adjusting group compositions. While traditional approaches have focused on behavioral logs (such as click counts, movement, and conversations) and prior knowledge, this study attempts to improve the speed and stability of predictions by capturing “the students’ own words” through contextual representations.

Method

The subjects were middle school students (K-8) in the United States, and the data analyzed were from classroom settings using the science inquiry game “CRYSTAL ISLAND.” In the game, behavioral logs such as movement, conversations with characters, viewing materials, and task completion are automatically recorded, and students are required to write short reflections at key points in the story about “what they learned” and “what they will do next.”

Four types of data were collected: (1) pre- and post-test scores, (2) game behavioral logs, (3) questionnaires on emotions and values regarding learning, and (4) the text of the reflections. After preprocessing, the text was quantified using two embedding methods. One is “GloVe,” which captures the closeness of words statically, and the other is “ELMo,” which can capture the fact that the meaning of a word changes depending on the context. The vectors of each reflection were averaged, and the cumulative average from the start of play to an arbitrary time was used as the “linguistic features at that point.” Predictions were updated every two minutes after the start of play, and the post-test performance was classified as high or low using all available data (behavioral logs, pre-tests, and reflections up to that point).

Results

Models that represented reflections contextually using ELMo showed stable predictions from a very early stage of learning, and in particular, when combined with simple decision rules (linear SVM), stable predictions were possible at an early point. Furthermore, using an “ensemble” method, which makes judgments by combining the results of multiple models, reduced the variance in results compared to predicting with a single model, further increasing stability.

On the other hand, when GloVe was used, the results were sometimes equivalent to cases where reflections were not used at all, suggesting the superiority of being able to “read context” in reflections that contain short sentences and diverse expressions.

Overall, it is concluded that even with short free-text responses, incorporating contextual representations adds clues that are difficult to capture through behavioral logs or pre-tests, enabling predictions that can be used for support decisions quickly and with confidence.

In addition, regarding evaluation metrics, various metrics such as F1 score (precision, recall) and Accuracy were used this time, but the optimal model sometimes differed between metrics. This indicates a trade-off between the “speed” and “reliability” of predictions, suggesting the importance of evaluation using diverse metrics.

Discussion

The significance of this study lies in the fact that it quantified words from learners, which are difficult to pick up from learning logs and pre-tests, by delving into their context, and linked them to reliable predictions from an early stage. Even with short reflections, using “context-reading” representations like ELMo served as an effective clue, raising both the speed and certainty of predictions. In particular, the ensemble method of combining multiple models increases stability in the initial stages, while the use of linear SVM functions reliably in terms of final accuracy, demonstrating a clear division of roles that is easy to understand when used in educational settings.

From the perspective of evaluation design, in addition to “predictive accuracy” such as Accuracy, F1, and AUC, it is important to use both “when a stable judgment was reached (standardized convergence point)” and “to what degree it was stable (convergence rate).” In practice, this makes it easier to identify learners who are expected to perform poorly in the first half of a unit and connect them to specific interventions such as material guidance, hint presentation, and review of pair compositions.

On the other hand, since there are a certain number of learners whose early predictions are unstable, a design that makes judgments by layering additional factors such as learning logs is necessary. Overall, this study provides practical guidelines for repositioning predictive models from post-hoc evaluations after class to “driving forces for teacher decision-making during class.”

Limitations

Limitations of this study include the following points.

First, because the post-test is binarized, elements such as the range of improvement or partial understanding are not sufficiently reflected. Methods such as performing predictions using regression with the scores themselves as continuous values are desired in the future.

Second, because reflections are represented by average vectors, sequential information such as word order and the development of arguments is lost. Since word order information is important for shorter sentences, models that can handle sequences (such as convolutional neural networks) should also be considered.

Third, free-text responses are easily influenced by motivation and vocabulary size, and the amount of information is insufficient for learners who write short or formulaic sentences. It is necessary to design research that also takes into account the original learning motivation and other influences.

Fourth, from the perspective of model interpretability, it is difficult to see “which expressions were effective for prediction.” By using embedding representations, tagging can be done without human intervention, but it is not possible to distinguish whether the improvement in prediction accuracy is due to capturing the “essential characteristics of the reflection” or simply considering superficial features such as “the difficulty of the words used or the length of the sentences.”

Below are my thoughts.

I read this paper because I was interested in how to evaluate the relationship between reflection and learning outcomes. Using it as formative assessment, not just for quiz scores, but by quantifying reflections to predict learning outcomes, allows for a more multifaceted grasp of the learner’s situation and leads to scaffolding, which was a useful reference for lesson design. If I were to actually implement this in the field, I think it is an interesting field with potential for further expansion, such as combining it with learning logs and quiz scores, and verifying over a longer term how teachers use reflections to make decisions in class.

On the other hand, this study used short reflections and was a short-term practice of 2-3 sessions, so I wondered how the results would be affected by sentence length or in the case of long-term classes. For example, I am curious whether it can still serve as an aid for grade prediction and teacher decision-making for longer-term learning, such as conducting reflections on learning at the end of a unit and how that relates to learning outcomes in the next unit.

Also, this time it was game-based learning for acquiring biological knowledge, but I think there is potential for using even more diverse reflections if one wants to utilize it in classes where evaluation criteria (what can be called learning outcomes) are complex, such as “career education” or “inquiry-based learning.” I believe that using reflection for evaluation has a major advantage in that it can visualize the learner’s thoughts behind learning outcomes, such as scores, by capturing semantic features rather than just simple quantification. Since there are already various qualitative analysis methods, I would like to broaden my knowledge so that I can utilize reflections in a multifaceted way with diverse perspectives in the future.

Text by: Kohei Ozaki