Hello everyone. This is Li, a graduate student.
I would like to introduce a paper I read at the recent English seminar, along with my thoughts on it.
Paper Title: A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students’ Formative Assessment Responses in Science
Publication Year: 2024
Authors: Clayton Cohn, Nicole Hutchins, Tuan Le, Gautam Biswas
Journal: In Proceedings of the AAAI Conference on Artificial Intelligence
Volume: Vol. 38
Pages: 23182-23190
In recent years, with the development of STEM education, the importance of appropriately evaluating students’ understanding of scientific concepts and their problem-solving abilities has increased. In particular, formative assessment is an effective means of grasping students’ learning status and providing feedback at appropriate times, but it places a heavy grading burden on teachers and carries the risk of inaccurate evaluation. On the other hand, Large Language Models (LLMs), which have been attracting attention in recent years, are expected to be applied to automated grading and feedback generation through natural language processing technology. However, research on the evaluation of short-form free-text responses, especially in the field of science, and on the generation of explanations for grading remains limited.
In this study, we developed and evaluated a method for automated grading and feedback generation for short-form free-text responses, targeting formative assessment in middle school earth science classes by combining Chain-of-Thought Prompting (CoT) using GPT-4 with active learning methods.
Formative assessment is an important opportunity to support students’ learning processes and for teachers to grasp students’ understanding. However, grading free-text short answers is time-consuming and labor-intensive, making it difficult to implement in the field. While research on automated grading is progressing in the field of natural language processing, the target tasks are biased toward structured formats such as mathematics and computer science, and are not sufficiently capable of handling free-text responses with diverse expressions in science education. Furthermore, problems such as a lack of learning data and class imbalance also hinder automation.
In this study, we attempted to use LLMs to grade and generate reasoning based on learning objectives for free-text short answers that evaluate scientific concepts and reasoning. In addition, to overcome the challenge of data scarcity, we introduced an active learning framework to achieve effective model improvement.
As the research subject, 270 students attending a public middle school in the United States were asked to work on a redesign task, “How to reduce water runoff in the school,” after a three-week unit of study on earth science. The questions consisted of the following three parts, each with clear evaluation criteria (rubrics) set:
1. “Explain the meaning of the size of the arrows shown in the diagram.”
2. “List two points that the diagram explains well scientifically.”
3. “Point out two errors contained in the diagram.”
All 270 responses were collected and graded by teachers based on the rubrics.
The LLM used in this study was GPT-4, which was the latest at the time. The evaluation prompt included 4-5 representative student answer examples, evaluation criteria, and scoring rationales, and was structured so that GPT-4 would reason about the reasons for point deductions and additions. In the active learning framework, humans analyzed cases where the model was prone to misjudgment, such as errors in reasoning or misapplication of evaluation criteria, and added new prompts containing these to be used for retraining. This allows for efficient improvement of model accuracy even with limited labeled data.
The dataset was split into 80% for training and 20% for testing. When an initial evaluation was performed with the training data, there were cases where significant differences were observed between the LLM and human grading results, and these were treated as “LLM evaluation errors.” For example, in the third question, there was a case where the model gave full marks to the incorrect answer, “All arrows should be changed.” To address such errors, we added prompts that emphasized the correct answer example, “Only the runoff arrow should be changed,” and gradually improved the model’s reasoning accuracy. This process was repeated until no further performance improvement was confirmed in the test data, or conversely, until accuracy decreased, and finally, the optimal prompt design was completed. As a result, the combined use of Chain of Thought (CoT) and active learning improved overall evaluation accuracy, with notable improvements observed particularly in items related to conceptual understanding.
On the other hand, this study also revealed the challenge of overfitting, where the model reacts too strongly to specific words or overly detailed reasoning. It was also shown that evaluation by LLMs can serve as a clue to discovering ambiguities in questions and rubrics, potentially leading to improvements in task design in educational settings. Furthermore, there are occasional cases where the model does not provide sufficient reasons when presenting grading results, which is a challenge for future improvement.
This study showed promising results for the automation of formative assessment in middle school science education by utilizing GPT-4 and Chain of Thought (CoT). By introducing this method, it is thought that it will be possible not only to grasp students’ level of understanding but also to provide useful feedback that encourages the next learning step. In the future, it is necessary to deepen collaboration with teachers and continue empirical research to adapt LLMs to educational settings. In addition, ethical issues inherent to LLMs, such as privacy, bias, and model hallucinations, require continued careful consideration. Furthermore, there is no clear insight at this time regarding how reasoning by CoT affects the model’s judgment, which remains a subject for future research. Also, when evaluation items are simple, rule-based methods may be more effective than LLMs, suggesting once again the importance of selecting methods according to the application.
The following are my thoughts on this paper. Currently, many researchers often use existing LLM (Large Language Model) APIs as they are, and there is not much research on designing models from scratch or performing fine-tuning. Against this background, it is noteworthy that this paper combines the Chain-of-Thought (CoT) reasoning prompt introduced last time with active learning methods to significantly improve the evaluation accuracy and explanatory ability of LLMs for students’ free-text short-answer questions in the field of science education. Such a method is highly novel and has very high practical value in educational settings, so I think it is worth recommending. In addition, this study adopts an actual middle school earth science curriculum (SPICE) as a concrete experimental environment, and the research content is closely linked to educational practice in the field. For this reason, the research results are of high reference value for teachers in the field and developers of educational support technology, and are useful for evaluation activities in class and providing individual feedback to students. Furthermore, this paper effectively addresses the problems of data scarcity and data distribution imbalance, which are frequently seen in educational data, through active learning methods. This method can be widely applied in other educational evaluation situations, and it is expected to promote deeper application and wider deployment of artificial intelligence technology in the field of education.
