Hello everyone! I am Naohiro Higuchi, a second-year master’s student at Yamada Laboratory.
With the spread of ICT, the field of Educational Technology has seen a surge in research focusing on Data-Driven Decision Making (DDDM), which utilizes “data” such as test scores and learning logs to improve classroom instruction. However, in the field, there are many challenges, such as the observation that “even with data, teaching styles do not fundamentally change.” Why is it that despite having vast and detailed data, classes do not change?
In this post, I will introduce a paper that sheds light on the “psychological processes” teachers undergo when engaging with data, offering hints to overcome this limitation. This paper has provided significant insights into the evaluation design of my own research theme, “Support for Teachers’ Instructional Improvement.”

Paper Information
・Title: Metrics Matter: How Properties and Perceptions of Data Shape Teachers’ Instructional Responses
・Authors: Caitlin C. Farrell & Julie A. Marsh
・Journal: Educational Administration Quarterly, 52(3), 423-462.
・Year of Publication: 2016
・DOI: https://doi.org/10.1177/0013161X16638429

1. Introduction/Background: Overconfidence in Rational Models and the Reality of the Field
Recent educational policies strongly urge school leaders to demonstrate leadership and teachers to analyze and utilize learning data to improve instruction. Underlying this approach is a linear, rational mindset that “data is an objective and neutral fact, and if data is provided, teachers will naturally arrive at the optimal teaching method.”
However, previous studies investigating actual classrooms have revealed that no matter how much data is provided, it does not directly lead to fundamental qualitative changes in instruction (such as changes in how students access knowledge or how teachers deliver it).

2. Theoretical Framework: “Sensemaking Theory” Connecting Objective Data and Subjective Interpretation
The “sensemaking theory” adopted as the theoretical framework for this study refers to the cognitive and social processes by which individuals or groups “give meaning” to situations when faced with uncertainty or new information.
According to sensemaking theory, data is not an “objective fact” unconditionally accepted by teachers. When looking at data, teachers do not approach it with a blank slate; they view it through the lens of their own “existing knowledge of teaching methods and students, educational beliefs, and values.” Consequently, they engage in active interpretation, focusing strongly on specific data while ignoring data that is inconvenient or difficult to understand.
In other words, subsequent decision-making is determined by how a teacher, upon seeing the data, makes sense of “which part of my class has issues” or “how should I change my teaching method.” This is the crossroads between whether it leads to a fundamental “change in delivery” or ends in superficial repetition, such as “teaching the same way again.”

3. Research and Analysis Methods: “Object Probes” to Eliminate Facades and a One-Year Longitudinal Study
This study conducted a detailed case study over one academic year, targeting five “high-need” middle schools located in three school districts in a U.S. state, where many households are low-income and academic standards are not being met.The data collected was highly multifaceted, as follows:

・Interviews with district leadership (13 sessions)
・Interviews with principals, instructional coaches, and case study teachers (73 sessions)
・Focus groups with 24 other teachers (6 sessions)
・Observations of Professional Learning Community (PLC) meetings among teachers (20 sessions)
・Monthly web-based activity logs

In the interviews, a qualitative interview method called “Object Probes” was adopted to eliminate teachers’ “facades” and vague memories, and to capture the reality of data utilization. Object Probes is a method where teachers are asked to bring “actual physical data (student essays, corrected work, quiz results, class-specific grade graphs output by analysis tools, seating charts, etc.)” to the interview. With these items in front of them, they are asked to speak retrospectively about “how they interpreted this data and what they specifically changed in their class.” Also, to ensure they did not just associate “test data” with specific state-wide tests, “data type cards” listing various types of data were presented to broaden their perspective.
The vast amount of qualitative data (such as speech logs) collected in this way was coded in detail using qualitative analysis software (NVivo). The analysis team extracted 255 specific cases as “instances of data use,” where “which data source was used” and “what instructional response was taken” were clearly paired. By quantifying these through cross-tabulation, they scientifically analyzed patterns of which data is likely to induce which actions.

4. Results and Discussion: Teacher Responses Differ by Data Type
In this study, the teacher’s specific actions (instructional responses) to data are treated as the dependent variable and are classified and analyzed into the following five patterns.

1. Change in delivery: Fundamentally restructuring how students access content or how the teacher presents information (e.g., introducing new instructional strategies).
2. Reteach, retest: Teaching the same standards or units that had low comprehension levels again, using the same teaching style.
3. Small groups: Dividing students into pairs or groups based on academic level or needs.
4. Student analysis: Having students reflect on their own test results and set their next goals.
5. Out of the classroom support: Providing additional support such as individual tutoring or remedial lessons outside of class hours.

In the overall analysis, “Change in delivery,” which fundamentally reconsiders teaching methods, remained at only 16%, with the majority being superficial responses such as “Reteach” (53%) or “Small groups” (20%). However, the proportion of these responses differed dramatically depending on the “type (characteristics) of data” the teacher engaged with.

(1) State tests (State-wide standardized tests)
Because results are obtained at the beginning of the year, they functioned for student grouping or small group organization (42%), but because the data was old and had little relevance to current classes, it did not lead to instructional improvement.

(2) District benchmark tests (Externally created tests)
“Reteach” accounted for 56%, while instructional changes were 10%. Due to the discrepancy between the tests and the classes, teachers had strong distrust, resulting in nothing more than “defensive sensemaking” as follows.

[What is defensive sensemaking?] An interpretive process where teachers exercise psychological self-defense to protect their own teaching style, avoiding self-reflection that “there is a problem with the way of teaching” in response to poor test results. It refers to a psychological tendency to attribute responsibility externally, such as “the wording of the test questions is bad,” or to remain at superficial measures like “mechanically re-teaching only the units where scores were low.”

(3) Common grade-level assessments (Tests created jointly with colleagues)
There was a high response of 64% for “Reteach.” Although reliability was high because they created them themselves, the awareness as “creators” became too strong, and there was a tendency for PLC discussions to focus on improving the tests themselves, such as “revising the wording of test questions,” rather than “improving teaching methods.”

(4) Classroom assessments/Student work (Daily quizzes, essays, etc.)
The rate of “Change in delivery” was the highest (about 30%), strongly promoting instructional improvement. This is because they are timely indicators designed by the teachers themselves and have a strong connection to the instruction just provided. In particular, qualitative “soft data” such as essays visualize students’ thinking processes and stumbling blocks, serving as a powerful catalyst for teachers to face “misunderstandings caused by their own teaching” and fundamentally reconsider their teaching methods.

5. Conclusion and Limitations of the Study
This study clearly rejected the rational model in DDDM that “if objective data exists, instruction will mechanically improve.” It is argued that what promotes substantial improvement in classes is not the quantity or objectivity of data, but the sensemaking process where characteristics such as the “timeliness” of data, the “agency” of teachers being involved themselves, and the “format (qualitative soft data)” that makes students’ thinking processes visible, are deeply intertwined with the “subjective perception” of teachers who trust them. While standardized numerical data is useful for school and district accountability and classification, it was formative and soft data generated within daily classes that changed the learning of students right in front of them.
Limitations of the study: A major limitation of this study is that the content reported by teachers as “having changed their class” relies on “self-reported data” from interviews and the like. Although specificity was increased through Object Probes, there is a lack of direct micro-observational data on how teaching methods were actually implemented in the classroom.

6. Reasons for Choosing This Paper and How I Will Apply It to My Research
I am currently developing a system (TiTela) that helps teachers reflect on their classes and use them for improvement. This paper has given me great confidence and backing for its “evaluation method” and “system design philosophy.”
First, as an evaluation method, I will utilize the findings of the Object Probes from this paper and combine not only questionnaires (self-reporting) but also “objective behavioral logs such as operation logs and descriptive content within the system (mixed methods research)” for analysis. By doing so, I plan to overcome the “dependence on self-reported data,” which was a limitation of this paper, and scientifically verify the transformation process of practice.
Also, as a design philosophy, TiTela handles safe, formative data for one’s own “instructional improvement,” separated from external “accountability.” I believe this can free teachers from defensive biases and encourage deep reflection (sensemaking).
Like the argument in this paper, I also believe that data does not change classes, but rather the “meaning-making” through data is what changes them. I will deepen my research further toward developing a system that allows teachers to proactively engage in instructional improvement!

Text by: Naohiro Higuchi