Hello everyone. I am Naohiro Higuchi, a second-year master’s student.
In this English seminar, I will introduce a highly cited paper in the fields of Educational Technology and Learning Analytics (LA) that models the process of “how teachers interpret data and translate it into decision-making (instruction).”
I am currently developing a system that provides feedback to teachers based on classroom data, but the question, “Does showing data really change a teacher’s behavior?” is always a major challenge. This paper is a very interesting study that delves into this black box.

Paper Information
・Title: Teaching with analytics: Towards a situated model of instructional decision-making
(Teaching with analytics: Towards a situated model of instructional decision-making)
・Authors: Alyssa Friend Wise & Yeonji Jung
・Journal: Journal of Learning Analytics, 6(2), 53-69.
・Year: 2019
・DOI: https://doi.org/10.18608/jla.2019.62.4

Introduction/Background
In recent years, “Learning Analytics (LA)” has been attracting attention in educational settings, aiming to analyze log data obtained from LMS (Learning Management Systems) and digital learning materials to improve education.
However, this paper states that simply showing teachers “impressive data” or “beautiful graphs” does not easily lead to actual instructional improvement. It points out that the process by which teachers themselves convert the information provided by a system into meaningful “knowledge” in the field and translate it into “actions” such as classroom instruction is actually very complex and difficult.
In this study, the authors attempted to clarify how teachers engage with LA in real educational settings by integrating the following three existing perspectives:
Teacher Inquiry (TI): The process where teachers themselves raise questions, collect data, and reflect.
Learning Design (LD): Redesigning learning materials and lesson plans based on data.
Orchestration: Adjusting instruction on the spot while observing real-time data.
By combining these, this study aimed to model the series of processes of “what teachers feel, how they think, and how they act when looking at data.”

Method
This study was conducted at a private university in the United States. Five instructors (from both science and humanities backgrounds) used an LA dashboard (a learning status visualization tool) developed by the university’s IT team for one semester.
This dashboard allows users to check information such as which materials students accessed, how much of a video they watched, and their quiz accuracy rates.
After the semester ended, the research team conducted interviews with the teachers, asking them to look back at the actual screens and reflect on questions such as “Where did you look on the dashboard?”, “What did you think when you saw that?”, and “Did you do anything in class as a result?” (Retrospective interviews).

Results/Discussion
Analysis of the interview results revealed interesting realities regarding the process by which teachers utilize data, following the five research questions (RQs) below.

RQ1: What kind of “questions” do teachers have when looking at data?
Textbooks often suggest that “one should first formulate a clear question before looking at data,” but in reality, the teachers did not necessarily have clear questions from the start.

・Vague curiosity: It seems that teachers often start looking at data out of vague curiosity, such as “Are the students really reading the materials?” or “Is there anyone slacking off?”
・Questions born from dialogue: An interactive process was observed where, while tinkering with the data, teachers would notice something like, “Oh? Access is low only here?” and a new question would emerge from that.

RQ2: How do teachers “interpret” data?
After looking at the data, the teachers performed two steps: “reading” and “explaining.”

1. Reading Data:
・Finding an entry point: Rather than just looking at the whole, they try to understand the whole by using specific data that catches their attention (e.g., students with 0 access) as an “entry point.”
・Need for benchmarks: Even if told “10 accesses,” they don’t know if that is good or bad. They strongly sought reference points such as “How does this compare to other students? (relative evaluation)” or “Is it 0? (absolute evaluation).”
2. Explaining Patterns:
・Fleshing out with context: They give meaning to the data using their contextual knowledge of the class, such as “Access increased because the assignment was difficult this week.”
・Emotional response: Looking at data is not just an analytical task. Emotions (Affect), such as “I’m happy that everyone watched the video!” or “I’m disappointed that it wasn’t watched at all…”, significantly influenced their interpretation.

RQ3: How did teachers “react (act)” after seeing the data?
This is the highlight of this study. As a result of looking at the data, the teachers showed the following reactions:

・Support for the whole class: Judging that “everyone doesn’t understand this,” they provide supplementary explanations in the next class.
・Support for specific students: Noticing that “this student hasn’t done anything,” they intervene by sending an email or similar action.
・Revision of course design: Reviewing materials, thinking, “No one looked at this material, so I’ll change it next year.”
In addition to these, an important reaction of “not acting” was also discovered.
・Wait-and-See: A decision to intentionally observe the situation, thinking that they are not confident in intervening immediately after seeing the data, or that it might be a temporary trend. This is also a valid form of decision-making.
・Reflection on pedagogy: Even if it did not lead to specific actions, it served as an opportunity for deep self-reflection on their own educational views, such as “What does ‘participation’ actually mean?” or “Is my way of teaching okay?”

RQ4: Did they check the results (impact) of their actions?
Surprisingly, almost no teachers checked “what happened as a result” after taking action based on the data. It was suggested that for busy teachers, completing the “C (Check)” of the PDCA cycle is quite difficult without systematic support.

RQ5: What are other important issues?
・Transparency and privacy: Teachers harbored conflicts such as “Won’t students dislike it as if they are being monitored?” or “Won’t they try to outsmart me by ‘pretending to watch’?”
・Dialogue with colleagues: It was shown that discussing with other teachers, “What do you think about this?” (collaborative interpretation) is more effective for deepening understanding of data than looking at it alone.

Conclusion/Limitations
As a conclusion of this study, the authors propose a “Model of Instructor Analytics Use.”
This is a model that views data utilization not as a linear process, but as a cyclical one that moves back and forth between “sense-making” and “pedagogical response.” In particular, it is groundbreaking that it incorporates human aspects into the model, such as “starting from curiosity,” “being accompanied by emotions,” and “‘wait-and-see’ also being a type of reaction.”
As for limitations, the number of participants is small at five, and since it is a retrospective interview, there is a possibility of memory distortion. However, the detailed analysis in a real context provides many suggestions for future system design.

Reason for selection and impressions
Reason for selection:
This was a paper I had wanted to read for a long time, and it is a fundamental document related to “Teacher Inquiry (TI),” which is the foundation of my research theme, “Teacher Reflection Support System.”

Impressions and how to apply it to future research:
First, what reassured me while reading was that the concept of “Teacher Inquiry,” which I use as the basis for my research, was treated as a matter-of-course premise in the context of LA. Literature I usually refer to, such as Avramides et al. and Sergis & Sampson, was also cited, and I felt that I could continue my research with confidence based on this theory.
What was particularly interesting was that the “Wait-and-See” attitude was viewed positively. As a system developer, one tends to think, “I have to make them act (intervene) immediately after seeing the data!” but for teachers, the judgment of “not moving now” also seems important. Also, the point that LA tools have value just by promoting “reflection,” not just behavioral change, is something I would like to incorporate into my perspective on system evaluation.
In the future, I am thinking of designing evaluation indicators for the system I am currently developing (TiTela) by referring to the model proposed in this paper. I would like to conduct evaluations not just on “whether the functions were used,” but also from perspectives such as “whether it stimulated the teacher’s curiosity” and “whether it led to deep reflection.”

Responsible for text: Naohiro Higuchi