Hello, everyone. I am Higuchi, a first-year Master’s student.

In this article, I will review a paper that I introduced in the English Literature Seminar.

Paper Information

• Title: A teacher-facing learning analytics dashboard for process-oriented feedback in online learning

• Publication: In LAK21: 11th International Learning Analytics and Knowledge Conference

• Pages: 482-489

• Year of Publication: 2021

• Authors: Dourado, R. A., Rodrigues, R. L., Ferreira, N., Mello, R. F., Gomes, A. S., & Verbert, K.

• Link: https://doi.org/10.1145/3448139.3448187

During my time working as an ICT support staff last year, I occasionally had the opportunity to assist in classes, and I found it truly difficult to grasp how children were learning. I believe that for teachers, a tool that provides information and analysis results regarding students’ learning progress and behavioral patterns would certainly be useful in their instruction.

The paper introduced this time focuses on online vocational courses provided by public schools in Brazil. It reports on the design and evaluation of a Learning Analytics Dashboard (LAD) that visualizes student behavior using logs from Moodle, a Learning Management System (LMS).

Challenges The authors noted that while LADs are utilized for providing continuous feedback regarding student progress and learning regulation, there are few empirical LAD studies focusing on “process-oriented feedback”—a type of feedback provided during the learning process, prior to formal assessment or teacher intervention. Therefore, they designed, developed, and evaluated an LAD for this purpose.

Research Methods This study follows a “design research” process, which aims for an improved form by repeatedly iterating through design and verification based on academic findings. The development of the LAD was based on a theoretical framework called the “Design Activity Framework for Visualization Design,” following the flow of “Understand” → “Ideate” → “Make” → “Deploy.” However, this paper describes the details up to the stages before “Deploy.”

Furthermore, this study consists largely of two stages, named the first and second iterations (diagrammed in Figure 1 of the paper). An iteration is a development process where functionality is added by repeating the development of small steps.

1. First Iteration: Corresponds to the “Understand” and “Ideate” stages. Researchers interviewed teachers in charge of the courses, analyzed audio and video data, and created paper prototypes for evaluation by teachers.

2. Second Iteration: Includes the “Ideate” and “Make” stages. Following the recreation of the prototype, a web application was actually developed. The usability of the app was evaluated by experts (doctoral students) in HCI (Human-Computer Interaction) using the Forsell & Johansson’s heuristics set questionnaire.

Results and Discussion The following summarizes the design requirements derived from the interviews, the LAD developed in this study, the usability evaluation results, and discussion points. The actual LAD interface can be viewed in Figures 2 and 3 of the paper.

1. Design Requirements: The authors analyzed the teacher interview data and examined challenges, requests, and information to be visualized. Specifically, they identified challenges such as teachers being unable to know where students are struggling until they ask for help. They also noted requests for information regarding the learning process utilizing Moodle logs. Based on these and preceding studies, they extracted data obtainable within Moodle (Table 1) and the corresponding visualization content (Table 2).

2. Dashboard Design (Letters indicate areas in the figures of the paper):

   • A) Overview: Provides information on class participation (A1) and grades (A2). Teachers can specify a particular student (A3) and check their learning progress on the Details screen (C).

   • B) Pattern-discovery panel:

     • a. Real-time tab: Allows teachers to check learning progress and verify if students are following the instructor’s directions.

     • b. Retrospective tab: By selecting an assignment within the course, teachers can discover behavioral patterns leading up to its submission. The group’s lines are colored based on the passing rate, and the lines are thicker when more students followed that path.

   • C) Details Panel: This screen allows teachers to check each student’s learning process chronologically. Each action is visualized with an icon (refer to Table 3 for icon types). Additionally, using the filter area (right side) and the Legend panel (D) within the Details panel, teachers can narrow down or reorder events.

3. Dashboard Evaluation: The results of the LAD evaluation by HCI experts were as follows (H01, etc., refer to question numbers; results are in Figure 4):

   • [Positive Evaluations]: The LAD evaluation was generally positive, with particularly high ratings in “Understandability without needing to recall (meaning of usage/visualization) (H10),” “Exclusion of redundancy (few wasted elements, H09),” and “Data reduction (unnecessary data is efficiently removed, H05).”

   • [Negative Evaluations]: Evaluations for “Flexibility (H03)” and “Spatial Arrangement (H05)” were not very high. Regarding flexibility, some argued that easily changing the app’s appearance might not be good for teachers unfamiliar with the tool. For spatial arrangement, suggestions were made to make each panel collapsible.

   • [Issues/Improvements]:

     • Overall LAD: There was feedback that having three tables of student names on the screen made it unclear which graph referred to which table’s conditions.

     • Overview Screen: Mostly positive, but it was suggested to place the screen for identifying a student and checking their progress (A3) elsewhere.

     • Real-time Tab: Almost no issues were raised.

     • Retrospective Tab: Many evaluators pointed out that the visualization method was difficult to understand. However, while pointing out the difficulty, they agreed on the usefulness of the content, such as seeing the paths students took before submission and which paths had higher passing rates.

     • Details Screen (C): Actions on Moodle are plotted via icons. Concerns were raised that for very active students, an excessive number of icons might be plotted. The authors responded that they would address this through efficient data selection.

Limitations The authors identified the following two limitations of this study:

1. While the developed LAD app was evaluated by HCI experts, it was not evaluated by the teachers who would actually use it upon implementation.

2. The Moodle logs utilized as a data source consisted only of clickstream data (page transition logs), which places potential limits on the information that can be visualized.

Based on the issues identified in this LAD evaluation, the authors plan to design an application that is easier for teachers to use and move toward practical implementation.

Reasons for Choosing and Personal Thoughts Since I am developing a teacher-facing LAD, I chose this paper to explore examples of similar research overseas. As mentioned in the limitations, because this LAD’s visualization uses only Moodle clickstream data, the granularity is slightly coarse compared to e-books, and the visualization design was somewhat difficult to understand at times. However, the types of visualization and usability evaluation methods introduced were very helpful. Additionally, I found the perspective of visualizing the “change in process,” including past actions rather than just showing the current state from device logs, to be very interesting. I am currently working hard on my own development, and I intend to develop an LAD that is as helpful as possible by carefully listening to the opinions and requests of teachers in the field.