Hello everyone, I am Chu, a research student.

I would like to introduce a paper I read in a recent English Literature Seminar and my thoughts on it.

• Paper Title: VizChat: Enhancing Learning Analytics Dashboards with Contextualised Explanations using Multimodal Generative AI Chatbots

• Journal: Artificial Intelligence in Education

• Year of Publication: 2024

• Authors: Lixiang Yan, Linxuan Zhao, Vanessa Echeverria, Yueqiao Jin, Riordan Alfredo, Xinyu Li, Dragan Gašević & Roberto Martinez-Maldonado

The following is an overview of the content of the paper.

Overview With the development of technology, the utilization of educational data is deepening. Dashboards that visualize this data are called Learning Analytics Dashboards (LADs). LADs allow students to monitor their own learning progress and promote self-regulated learning, while also providing teachers with a function to grasp students’ learning status. However, with progress in multimodal learning analytics and educational data mining, the collection of diverse and complex data such as text, audio, images, and video is advancing. Since these data differ significantly in format and characteristics, they must be visualized using appropriate methods for each. Consequently, it has become difficult to visualize these data in a unified, simple, and easy-to-understand form.

There are concerns that as visualizations become more complex, the cognitive load on learners and educators will increase. In particular, for those who struggle with understanding charts and graphs, the practicality of LADs may decrease. Thus, providing support for such individuals is a challenge. This study developed software called “VizChat.” This software generates text explanations for complex LADs, helping teachers and learners who struggle with interpreting charts and graphs to improve their understanding of LAD content.

Accordingly, the following Research Questions (RQs) were set for this study:

1. What kind of content is appropriate for this software to present to readers?

2. What technology is required to process visual data (LAD) and text data simultaneously?

Regarding Question 1, the authors propose a method called “data storytelling.” Data storytelling is the concept of building a compelling narrative based on complex data and analysis to effectively convey intended content to a specific audience. By using this method, they state that appropriate support can be provided to people using LADs by combining text and visualization, thereby deepening their overall understanding of the LAD.

Regarding Question 2, natural language processing (specifically, Large Language Models capable of handling multimodal data like GPT-4V) serves as a solution for processing complex multimodal data and generating appropriate data storytelling. These models provide accurate, contextually relevant explanations based on customizable and predefined learning content, ensuring that the information is rich and relevant to the learning objectives. To address the issue of Large Language Models generating incorrect information (hallucinations), this study utilizes Retrieval-Augmented Generation (RAG) technology. RAG is a technology that adds information retrieval capabilities to generative AI models. This allows the model to refer to a specified set of documents when answering user questions, supplementing information from its own vast training data. By setting up a dedicated database, incorrect generation is prevented.

System Design The VizChat prototype is built as a Chrome extension on a web-based LAD, featuring screenshot capture and chat functionality with users.

• Database: Creating a knowledge database containing all relevant contextual information is critical to minimizing the risk of generative AI providing hallucinations or inappropriate explanations that could mislead learners or educators. LAD administrators and educators can upload documents related to Learning Analytics, such as course and task descriptions.

Case Study The case study in this research, “Design of a Clinical Simulation Unit in Undergraduate Nursing Education,” demonstrates how VizChat is actually utilized with an LAD. Through three dialogue examples, it explains how VizChat supports students’ understanding and reflection on visualized data. VizChat remembers user questions and provides specific answers. It can also integrate multiple visualization tools to explain the data collection and analysis process. These examples were selected from difficulties students faced when using LADs, demonstrating the potential of VizChat.

While not a formal evaluation, research is facilitated by the fact that VizChat is provided as open source. In actual use, VizChat appears on the LAD interface as a Chrome extension and supports student reflection. The LAD system is used in clinical simulations for undergraduate nursing education, helping students understand complex visualized information in practical applications through visualization tools such as timelines, priority charts, ward maps, and social network diagrams.

Thoughts and Problems Overall, VizChat is software that leverages Large Language Models and RAG to support LADs, offering many interesting functions. This paper mentions the issue of visual perceptibility in current LADs and points out that it affects the actual usefulness of the LAD, which I felt was a very important point.

This is also relevant to my own research, and I am likely to face several of the constraint issues mentioned in this paper. For example, the cost issues of paid models like ChatGPT-4 and the fact that text quality is extremely important when providing a RAG database. Therefore, I chose this paper.

In this paper, evaluations and experiments for the software have not been conducted, and while the idea is wonderful, I believe many questions remain. First, in the original text, those who struggle with understanding charts and graphs are referred to as individuals with low “visualization literacy.” However, the question remains: what is “visualization literacy” and how is it defined? I feel this is a very important issue. If a user’s visualization literacy cannot be accurately judged, it becomes difficult to recommend subsequent functions accurately. In my own research, there is also the question of what kind of LAD recommendation is optimal for the user. If an inappropriate recommendation is mistakenly made to a teacher or student with high visualization literacy, this software might instead hinder their judgment. Therefore, I feel it is necessary to design a mechanism to filter out people who do not require these prompts beforehand. However, I think implementing this in an actual educational setting would be quite difficult. They would only be able to use it after going through a filtering process, and many users might give up on using the software.

Regarding the discussion on generative AI, RAG was mentioned broadly, but I would like to know how to create a more specific RAG database. I am also curious about how to ensure the quality of data used in this database and what the selection criteria are. Clear criteria are necessary; otherwise, issues regarding the generated content of Large Language Models cannot be fundamentally solved, and there is a possibility that inaccurate content will be generated.

Finally, while this paper talks only about the use of GPT, it lacks a comparison with other local open-source models, such as LLaMA. I believe such comparisons are necessary, as the cost problem of Large Language Models, which the authors cited as a limitation, still remains.