Hello everyone! I am Naohiro Higuchi, a second-year master’s student.
Dashboards that visualize classroom data are becoming increasingly common in schools, but even when looking at the same screen, “what they notice and how they think” varies greatly depending on the teacher. In this post, I would like to introduce a paper that carefully codes how middle school mathematics teachers and “instructional coaches” who support lesson development react when faced with a classroom data dashboard, revealing the differences based on their roles. I am currently conducting research on conducting reflections with teachers while looking at dashboards, and this paper has greatly broadened my analytical perspective.
Paper Information
・Title: Making Sense of Sensemaking: Understanding How K–12 Teachers and Coaches React to Visual Analytics
・Authors: Fabio C. Campos, June Ahn, Daniela K. DiGiacomo, Hoang Nguyen, & Marisa Hays
・Journal: Journal of Learning Analytics, 8(3), 60–80. ・Year: 2021
・DOI: https://doi.org/10.18608/jla.2021.7113
1. Introduction/Background: Multiple steps grouped under the single term “sensemaking”
In K–12 schools in the United States, the ability to use classroom data for instruction has become an essential competency for educators, and dashboards that display data are becoming widespread. However, it is not yet fully understood how educators in different positions construct interpretations from that data.
The authors point out two issues with existing research. One is the lack of granularity in models. In LA (Learning Analytics: a research field that analyzes learning and instructional data to help improve education) and HCI (Human–Computer Interaction), conceptual models have been used to explain how visualization tools affect instruction and learning. However, these models group different activities—such as reflection, noticing, generating insights, and interpretation—together as “sensemaking” (the process of giving meaning to data) and do not capture the detailed steps of each. The other issue is the oversight of differences based on roles. The authors believe that tools that do not distinguish between users, assuming that all educators read data in the same way, cannot sufficiently support the improvement of instruction.
Therefore, this study describes in detail how teachers and instructional coaches (experienced educators trained to provide subject-specific instructional support) make sense of classroom data on a dashboard. The paper includes a diagram of an existing LA process model (Fig. 2: p.64) that organizes this awareness of the issue, which is helpful to refer to in order to understand where the research focuses.
2. Theoretical Framework: Viewing sensemaking as a “process of transcribing meaning”
This study incorporates two insights from outside LA/HCI. One is organizational sensemaking research (Weick, 1995), which considers sensemaking not as the “discovery” of hidden meanings, but as a process of transcribing reality while being influenced by one’s own frame of reference and emotions, with interpretation being the product of this. The other is educational research dealing with interactive sensemaking in K–12 (Horn et al., 2015). The relationship where a teacher is helped by a coach to interpret data is unique to K–12. From this standpoint, this study observes educators’ reactions, including emotions, and excludes actions from the scope of analysis as a separate (though related) process. This is because actions are also influenced by school policies and team dynamics. Previous educational research has classified reactions to data as “conceptual,” “symbolic,” or “instrumental,” but the authors believe these are too coarse as a common language for discussing the differences between teachers and coaches.
The research question (RQ) set is: “What types and patterns of reactions do K–12 educators exhibit toward visualized data?”
3. Research Methodology: Turning reactions collected through think-aloud into a codebook
This study was conducted within PMR2 (Practical Measures, Routines and Representations), a network consisting of four Research-Practice Partnerships (RPP) between universities and school districts. Teachers were presented with the results of about 8 questionnaire items asking how students experienced classroom discussions, displayed in a way that did not identify individual students on the “Edsight” dashboard, which was co-designed by the research team and educators. Participants were 9 middle school math teachers and 9 instructional coaches selected from 74 educators in the network.
Teachers here are in a position to interact directly with students and provide instruction in their own math classes. On the other hand, an instructional coach is a position appointed by the school district from among educators with extensive teaching experience, trained to provide continuous, subject-specific instructional support to teachers within the school. In U.S. K–12, they are widely adopted as a role to support teachers in interpreting data and connecting it to instructional improvement. The average years of experience were 10 years for teachers and 17 years for coaches, and teachers had no experience using visualization tools, while only one coach did. In a session of about one hour, they used the think-aloud method to interpret the data while speaking their thoughts, and semi-structured interviews were conducted, with each person looking at about 15 visualizations. 12 people looked at data from their own classes, and 6 looked at anonymized example data. To elicit cognitive processes directly, they were only prompted with, “What do you notice?” and “What can you tell from this data?”
In the analysis, transcripts were divided into excerpts for each “act of interpretation,” and reactions were organized through inductive open coding, resulting in a codebook consisting of three dimensions: emotion, analysis, and intention. Final coding was performed by two or more people, and the inter-rater reliability (Cohen’s κ) was 0.82. For comparisons between roles, the Wilcoxon rank-sum test, which tests the difference in the median number of occurrences, was used.
4. Results/Discussion: 25 reaction types and differences in how teachers and coaches read data
Since there is one RQ in this study, I will organize it according to the paper’s framework: reaction types, differences by role, and co-occurrence patterns.
Reaction Types: Three dimensions of emotion, analysis, and intention
As a result of coding, 25 reaction types, including “no reaction,” were organized.
・The emotion dimension includes surprise, satisfaction, empathy, distrust, responsibility, etc., and the analysis dimension includes questioning, recalling, attribution, filling in the gaps, connecting the dots, predicting, etc.
・The intention dimension consists of “intent to act” (willingness to improve but without a plan), “plan” (mentioning specific actions), and “information seeking” (seeking information outside the dashboard for action).
・To supplement the terminology of the analysis dimension, “attribution” refers to seeking the cause of a result in something outside the data, and “filling in the gaps” refers to supplementing the gaps in the data with one’s own opinions, beliefs, and educational views.
Differences by Role: Teachers “narrate,” Coaches move to “action”
Reactions with significant differences in the Wilcoxon test are as follows (numbers are counts of occurrences).
・Reactions common to teachers: Distrust (9 vs 1), Responsibility (7 vs 0), Recalling (21 vs 4), Attribution (47 vs 21)
・Reactions common to coaches: Connecting the dots (21 vs 7), Intent to act (25 vs 11), Information seeking (19 vs 2), Total of intention dimension (50 vs 13)
Teachers tended to describe the situation in their own words and attribute causes, while coaches tended to look for information from the data and think from the perspective of educational action. For example, coaches brought up the district’s social-emotional learning policy and presented their own explanatory framework of “class culture,” which was not in the data or the questionnaire items. On the other hand, teachers constructed specific explanations such as “they lack confidence and don’t want to proceed on their own” based on the students’ behavior they saw in past classes. Note that the number of times they asked questions was more than twice as high for coaches than for teachers, but this was not statistically significant.
Co-occurrence Patterns: Which reactions occur as a set
The number of times two codes were attached simultaneously within one excerpt was also analyzed (Fig. 4: p.71). First, what stands out is that the combination of attribution and “no intention” reached 62 times, confirming that even when looking at data, it does not lead to action, which is the most common reaction. Based on that, the following patterns connecting analysis and intention were highlighted.
・Questioning × Information seeking (17 times): Mostly coaches. Questions about the data lead to plans for collecting information outside the dashboard, such as classroom observation or interviews.
・Filling in the gaps × Intent to act (13 times): Those who supplement explanations with values or external information are more likely to express an intention to improve.
・Attribution × Recalling (14 times): Mainly teachers. They recall past events and attribute causes.
Discussion: Extended LA process model and “over-fitting”
The authors interpret the role differences as “teachers read with experiential knowledge (scenes) about people and classes as parties to the event, while coaches read with propositional knowledge such as pedagogical issues.” They then proposed an extended LA process model that depicts the entire sensemaking process, distinguishing it from the interpretation that is its product (Fig. 5: p.75). It is important that educational knowledge and reflection are born from this process, even if it does not lead to action. Furthermore, the authors called the risk of forcing new information into existing frameworks and repeating superficial interpretations “over-fitting,” borrowing a term from machine learning. The co-occurrence of attribution/recalling and “no intention,” which was common among teachers, is the basis for this. Since this risk can change depending on the design of the visualization, they state that it is a particularly important issue for LA research.
5. Conclusion and Limitations of the Study
Contributions of this study: (1) Types of reactions as a common language that educators, researchers, and designers can share, (2) Clarification that reactions differ by role, such as teachers using recalling and attribution, and coaches using information seeking, and (3) An extended process model. The authors state that this can be applied to training where coaches promote sensemaking and to the design of visualizations that suppress over-fitting.
Limitations of the study: The number of participants is small, and generalizable causal claims are not intended. Also, the stimulus presented was only a questionnaire about the teacher’s instruction from the student’s perspective, and the context of teaching experience and school districts was not aligned. Furthermore, collaborative sensemaking, where educators create meaning through dialogue with each other, was not captured, and the typology is considered only a first step.
6. Reasons for choosing this paper and how to apply it to my research
I am also conducting research where I have high school teachers look at dashboards and reflect on them, but I chose this paper because I had never seen research that coded the reactions of teachers who looked at dashboards in such detail, and I wanted to use it as a reference.
In my research, I repeatedly conduct reflection sessions where I talk with teachers while looking at the dashboard, and I record and film them. I felt that careful consideration is necessary regarding the examination of the elements of the codebook itself raised in this study and the way of dividing the unit of analysis called “one act of interpretation,” but I felt that by analyzing based on this codebook, it would lead to a contribution to this research field. Furthermore, since my research tracks the same teacher’s reflections multiple times and also collects system operation logs, there is potential for development, such as temporal changes in sensemaking, more detailed analysis of actions, and analysis focusing on the text written by teachers during insights and reflections. Also, while this study focuses on the part where teachers make sense of visualized data, I record everything from the point of setting up questions to analyze my own classes to the reflection, so I would like to challenge myself to analyze with a wider range of observations.
Text by: Naohiro Higuchi
