Hello everyone. This is Ozaki, a second-year master’s student.
In this article, we introduce a paper on the design, implementation, and evaluation of a collaborative chatbot developed to maintain “productive dialogue” among peers in “inquiry-based learning,” where students formulate their own questions and seek answers.

Authors: Adelson de Araujo, Pantelis M. Papadopoulos, Susan McKenney, Ton de Jong (2024).
Title: A learning analytics-based collaborative conversational agent to foster productive dialogue in inquiry learning
Publication: Journal of computer assisted learning
Vol. 40, No. 6, pp. 2700-2714.
Publication Year: 2024

1. Background and Objectives
In modern education, collaboration is gaining attention as an essential skill for surviving in the 21st century. In inquiry-based learning in particular, it is expected that students will deepen their thinking and cultivate scientific reasoning skills through dialogue with their peers. However, maintaining “productive dialogue” in actual classroom settings is not easy. In online learning, it is an even more difficult challenge to encourage students to share their thoughts and engage deeply with each other’s opinions. Furthermore, with many groups working simultaneously, there are limits to how much a teacher can intervene and provide support for every dialogue in real time.
Therefore, this study aims to design a chatbot named “Clair” to facilitate dialogue among learners and to evaluate how this chatbot was able to promote productive dialogue.

2. What is the collaborative learning agent “Clair”?
In this study, we developed and introduced an agent called Clair (Collaborative Learning Agent for Interactive Reasoning). Clair analyzes students’ chat logs in real-time and provides “prompts to encourage dialogue” at appropriate moments.
Clair’s Prompt Design
Clair’s intervention strategy is based on the educational theory of “Accountable Talk.” This is a dialogue framework for discussing logically based on evidence while respecting one another. Clair utilizes eight different “talk moves” as follows.
Recapping: Prompting a summary of the discussion so far (e.g., “Could you briefly summarize what we have discussed so far?”).
Add-on: Prompts students to add new information to others’ opinions.
Paraphrasing: Explaining the other person’s opinion in your own words to confirm understanding.
Agree, disagree: Clarify your position.
Linking contributions: connecting individual ideas to the main theme.
Association with prior knowledge/exemplification: Requesting the presentation of concrete examples.
Expanding inference: Asking for an explanation of the reason, such as “Why do you think so?”
Design of Clair’s Decision-Making During Dialogue
Clair grasps the state of dialogue using 12 indicators, such as whether the student’s utterances are “related to the learning content (domain),” “whether they are small talk (off-task),” or “whether they explain reasons (logical).” This data is processed using a mechanism called a “fuzzy expert system.” This is a system that determines interventions based on flexible rules close to human judgment, such as “if a student is focused on learning but is not using specific keywords much, prompt them with an add-on.”

3. Research Methodology: Practices in Brazil and the Netherlands
To verify the effectiveness of Clair, we conducted experiments in two educational settings: Brazil (equivalent to 8th grade, 19 pairs) and the Netherlands (equivalent to 9th grade, 28 pairs).
Experimental Design: The experiment was conducted in two phases.
Phase 1: All groups work without an agent (baseline measurement).
Phase 2: Activities are divided into an “experimental group” where agents intervene and a “control group” where they do not.
The learning materials used were inquiry-based learning programs on photosynthesis and the digestive system.
Evaluation Metrics
We evaluated how “productive” the dialogue became from the following two aspects.
(1) Sequential Pattern Mining: We extracted behavioral patterns from chat text and counted the frequency of patterns corresponding to four goals (FGPDs): “sharing one’s own ideas,” “listening and guiding,” “deepening reasoning,” and “engaging with others’ reasoning.”
(2) Subjective evaluation: We conducted a questionnaire survey on how the students themselves felt about aspects such as “whether their dialogue was productive.”

4. Analysis Results
Emerging Effects and Limitations As a result of the objective analysis of changes in dialogue patterns, it was found that the presence of Clair clearly influences the dialogue behaviors of FGPDs in alignment with their goals.
Regarding the promotion of “sharing thoughts,” in both studies, the groups with Clair showed a significant increase in behaviors of sharing one’s own thoughts from Phase 1 to Phase 2. On the other hand, such changes were not observed in the groups without Clair.
Regarding the improvement of “engagement in others’ reasoning,” a study in the Netherlands showed that Clair’s intervention increased behaviors such as refuting or questioning others’ opinions.
Next, regarding the impact on “deep inference,” no clear difference was observed based on the presence or absence of Clair in terms of actions to further deepen one’s own inferences.
Finally, regarding the students’ subjective perceptions, surprisingly, the questionnaire results (dialogue productivity) showed no significant difference based on the presence or absence of Clair. It seems that the students felt they had “collaborated sufficiently” regardless of whether the AI was present or not.

5. Why did it not reach “deep inference”?
The results of this study revealed that while Clair was highly effective in broadening the entry points for dialogue, such as “trying to speak up first” and “reacting to others,” there are challenges in reaching a level that deepens the content. The following points are considered to be the factors behind this.
・Limitations of content-independent intervention: While Clair reads the “intent” of utterances, it does not delve into the specific “content” (such as scientific accuracy). Therefore, the current general-purpose prompts were insufficient for resolving specific conceptual misunderstandings or posing deeper, more specialized questions.
– Timing of intervention: As a result of manual log inspection, cases were observed where students felt that interventions by the system at inappropriate times were an “unnecessary burden” and ignored them.
– Motivation and sense of responsibility toward the system: Compared to feedback from humans (teachers or peers), prompts from a system without emotions may have had limited power to ignite students’ motivation or sense of responsibility.

6. Conclusion and Future Work
The Potential of Generative AI (LLM) This research demonstrated that chatbots using Learning Analytics can achieve certain results as a “scaffolding for dialogue” in inquiry-based learning. In particular, their ability to support the first step of collaborative learning—externalizing one’s own ideas (sharing thoughts)—was consistently confirmed. Moving forward, the following improvements will be key to realizing “deep inquiry.”
– Utilization of LLMs (Large Language Models): By integrating LLMs such as ChatGPT, it becomes possible to provide “content-dependent” interventions that delve deeply into the learning material.
・Adding a human touch: It is desirable to design agents with functions for praise and natural follow-up so that students can feel more familiarity and responsibility.
– Collaboration with teachers: It is also necessary to consider building a “hybrid” support system where AI does not replace everything, but instead provides teachers with analytical data collected by AI, allowing teachers to focus on more advanced interventions.
Inquiry-based learning is a process of confronting questions that have no single correct answer. This paper argues that as technology supports such dialogue, we are approaching a future where more students can enjoy deep learning.

Feedback
The following are my impressions.
In my research, I deepened learners’ thinking through a promising method called “Inquiry-Based Learning.”
I wanted to learn methods for scaffolding the learning process, so I read this paper
I chose this. It was very helpful to learn about methods such as appropriately integrating theory into the design when using chat, and utilizing those patterns for evaluation.
On the other hand, how can we further develop learning outcomes through classes that utilize the system?
We believe there is still room to consider this further. Regarding the point of “deepening inference,” which was one of the goals of this study, it is considered an important aspect of deep inquiry-based learning; however, it was particularly interesting that no significant difference was found in this research. While this paper suggests the possibility of leading to deep inference by utilizing LLMs to delve into the content of inquiry-based learning, I would like to study more deeply in the future how to promote the inquiry-based learning process—including the design of components for the entire class outside of the system—to achieve important learning outcomes such as deep inference.

Author: Kohei Ozaki