Hello everyone!
I am Naohiro Higuchi, a second-year master’s student at Yamada Laboratory. In the field of education, there are high expectations for “data-driven decision-making,” where classes are reviewed based on assessment data and learning logs. However, in reality, Teacher Inquiry (TI)—where teachers regularly question their own practices based on data—has not yet taken root in schools. In this post, I will introduce a paper that identifies the cause of this difficulty as the models being “too theoretical” and proposes and validates an inquiry model equipped with concrete steps and hints that are easy for teachers to use themselves. This paper is deeply connected to the design of “TiTela,” a class improvement support system I am developing.

Paper Information
・Title: Towards Data-Informed Teaching Practice: A Model for Integrating Analytics with Teacher Inquiry
・Authors: Marge Saar, María Jesús Rodríguez-Triana, Luis P. Prieto
・Journal: Journal of Learning Analytics, 9(3), 88–103.
・Year of Publication: 2022
・DOI: https://doi.org/10.18608/jla.2022.7505

1. Introduction/Background: Why can’t teachers sustain “data-based inquiry”?
As times and student needs change, teachers are required to review their daily instructional methods and try out new teaching strategies. Teachers should have an abundance of information available to them, such as assessment data, classroom observations, and data regarding their own teaching methods. However, “Teacher Inquiry (TI),” in which teachers regularly and systematically examine their own practices based on such data, has yet to take root in schools. The background to this includes a lack of access to relevant data that meets teachers’ needs and a workload that is disproportionate to the results obtained. In addition, many existing TI and data utilization models focus more on student learning than on the teacher’s teaching, and because they lack concrete practical examples, they tend to appear “too theoretical” to teachers in the field. In fact, a prior study reviewing Learning Analytics (LA) related papers from 2011 to 2018 reported that most dealt only with student data, and interest in “teaching analytics,” which focuses on teachers’ teaching, only began to emerge after 2018. This study aims to overcome this barrier to adoption by developing the “Analytics Model for Teacher Inquiry (AMTI),” which provides concrete guidance to teachers at each step of the inquiry process.
* Note: The paper includes a conceptual diagram (p. 97, page 10 in the PDF) that summarizes each step and component of AMTI on a single page. Since the entire study is structured toward the construction and refinement of this diagram, it is very helpful to refer to it as well.

2. Theoretical Framework: Integrating existing models and designing to start from “purpose” rather than “problems”
This study stands on the position of connecting Teaching and Learning Analytics (TLA: a framework that guides the teacher’s process of reflecting on their own educational design and implementation based on educational data involving both the teacher and the learner) with TI (an activity in which teachers take the lead in engaging in the professional development of their own practice based on evidence). The approach here is that data does not unilaterally “drive” decision-making, but rather “informs” it by providing material for judgment. AMTI was constructed by integrating three representative existing models. First, because existing TI models lacked guidelines on how to interpret data, the interpretation stage was divided into “sense-making” (the stage of finding patterns in data) and “interpretation” (the stage of adding pedagogical knowledge to give meaning). Second, based on the idea that inquiry does not necessarily mean dealing with “problems” but can also have the goal of positive improvement, the starting point of inquiry was placed on “purpose” rather than “problem identification.” Third, among the four perspectives shown by existing reference models for data utilization (what data, what techniques, who, and why), we excluded “who,” which is self-evident in a classroom setting, and incorporated the perspectives of “what happens after” (how to act) and “why inquire” in addition to “what and how.”

3. Research Methodology: Three-stage development and validation using Design-Based Research
This study adopted “Design-Based Research (DBR),” which designs solutions to real-world problems and aims to bridge the gap between theory and practice, and developed and validated AMTI through three iterations. In Iteration 1, multiple existing TI models were compared to extract common steps, and the data utilization process and types of data meaningful to teachers were organized with the help of expert advice to build the first version of AMTI. In Iteration 2, 10 teachers from four secondary schools in Estonia, ranging from less than one year to over 20 years of teaching experience, were selected to conduct 30–45 minute think-aloud sessions using eight questions asking about each step of AMTI and four open-ended questions asking about their orientation toward data utilization. In Iteration 3, the model was validated through a “Nominal Group Technique (NGT)” session with seven experts with over 10 years of experience, including university researchers, head teachers, and senior teachers, and two teacher-researchers collected data using AMTI in actual classes to verify its applicability.

4. Results/Discussion: Answers to the three research questions
This study set the following three research questions (RQs).

RQ1: How can a TLA process model be designed that teachers feel they can use in their practice?
Participants evaluated AMTI as logical and easy to apply. From this, four design principles were derived: (1) it should inform teaching practice in general rather than just “problems,” (2) it should direct teachers’ attention to inquiry questions, (3) it should show a path to supplement student data with data regarding the teachers’ own teaching, and (4) the analysis process should be divided into multiple logically connected steps. However, it was shown that the two stages of “sense-making” and “interpretation” are difficult for teachers to distinguish, as pattern exploration and pedagogical interpretation tend to proceed simultaneously without being consciously separated, and support from colleagues, literature, or training is necessary when interpretation skills are lacking.

RQ2: What do teachers’ perceptions of each step of AMTI reveal about their understanding and orientation toward data utilization?
Teachers’ motivation to analyze their own teaching was linked not to a sense of duty, but to an internal desire to “be a good teacher” and “meet students’ needs.” On the other hand, regardless of their level of experience, some teachers hesitated to face their own weaknesses or situations that did not go well, and novice teachers in particular expressed anxiety about the lack of clear criteria for “what constitutes effective teaching.” It was also found that means of data collection and analysis were often limited to questionnaires and personal observation, indicating a need for access to tools that collect data from the physical space of the classroom and guidelines on how to use them.

RQ3: What do teachers cite as factors hindering the spread of TLA, and what incentives do they consider possible?
Factors hindering adoption were broadly divided into two categories. One is factors that can be removed by changing the environment surrounding teachers (workload, habits, the feeling of being “evaluated”). The other is factors that can be supported by improvements in the inquiry model itself (simplification of the process, presentation of guidelines and concrete examples, and presentation of technical means for data collection and analysis). Above all, if teachers themselves cannot feel the benefits of introducing TLA, they will not act, as they weigh the “effort” against the “effect” before adopting new methods. While not only intrinsic motivation but also external support such as school policy is necessary, it is also pointed out that if TLA becomes a tool for evaluating and auditing teachers’ practices, it will undermine a positive attitude toward inquiry.

5. Conclusion and Research Limitations
Through the three iterations, it became clear that even if teachers set the improvement of their teaching as a goal, they do not actually try to collect data for that purpose (teaching data) and tend to focus their inquiry questions on students rather than their own teaching. The authors argue that teaching data is still a new concept, and that in the future, it is necessary to place more emphasis on teaching analytics to correct the bias toward learning data, and to promote the development and dissemination of tools and technologies tailored to the context of practice. Research Limitations: The findings of this study are based on a small number of participants, and there are limits to generalization. In addition, the validity of the think-aloud questions based on AMTI has not been sufficiently verified, and it has been pointed out that it may have caused confusion for teachers, such as the difference between “motivation” and “purpose.”

6. Reasons for choosing this paper and how I will apply it to my research
I chose this study because it directly addresses the relationship between Teacher Inquiry (TI) and Teaching and Learning Analytics (TLA), and it was very close to the awareness of the issues in “TiTela,” which I am developing as a framework to integrate the two. The fact that it goes beyond proposing a theory to actually surveying the perceptions of 10 teachers and delving into their understanding of the model and obstacles to its spread is a useful reference for my own research, which is in the empirical stage. TiTela relies on a different prior model than this study and sets the first step of inquiry as “problem identification,” but the finding presented in this paper that “inquiry is not limited to dealing with problems and can start from a purpose” was something I had also felt in my own prototype evaluation sessions. This seems to be material for reviewing that design. Also, while teachers’ data collection methods in this study’s survey were limited to questionnaires and personal observation, the fine-grained eBook logs such as page transitions and markers handled by TiTela can be positioned to technically compensate for these limitations. Furthermore, while the emphasis on dividing the meaning-making of data into “sense-making” and “interpretation” is common with this study, I also felt that TiTela could claim uniqueness in that it can track the process of the teacher’s teaching itself. I would like to continue to apply the direction of TLA, which connects both the teacher’s teaching and student learning, to my own research in the future.

Report by: Naohiro Higuchi