Hello everyone. This is Higuchi, a second-year master’s student.
I was on an exchange program in Turkey from last September, but I returned in June, so I will be working hard on my research at Yamada Laboratory again. Thank you for your support.
In this article, I will review a paper that I introduced in the English seminar.
Paper Information
Title: Mediating Teacher Professional Learning with a Learning Analytics Dashboard and Training Intervention
Publication: Technology, Knowledge and Learning
Pages: 981-998
Year: 2023
Authors: Khulbe M & Tammets K
Background
In recent years, the digitalization of schools has been progressing in Japan due to initiatives such as the GIGA School Program. Internationally, there is also active movement toward utilizing digital technology in educational settings to collect diverse data for learning and instruction, leading to increased demand and expectations for data-informed teacher decision-making.
Amidst this trend, the authors of this paper focus on the fact that “for effective data utilization, teachers must acquire new pedagogical, technical, and data-related knowledge and skills.” They conducted an eight-month digital literacy training program for active high school mathematics teachers in Estonia. Based on the insights gained, they propose an effective design for a LAD (Learning Analytics Dashboard).
Research Questions
This study focuses primarily on two points: “data literacy” and “student engagement.” For data literacy, the definition used is “the ability to transform information into actionable instructional knowledge and practices by collecting, analyzing, and interpreting all types of data to help determine instructional steps (Mandinach & Gummer, 2016).” The reason for focusing on student engagement is that it is recognized as having a strong connection to predicting and achieving learning outcomes. Furthermore, among engagement types, they consider four categories: 1) behavioral (learning time, task completion, etc.), 2) cognitive (deep understanding, use of appropriate strategies for learning goals, etc.), 3) emotional (feelings toward learning), and 4) agentic engagement (active participation in learning activities).
The research questions in this paper are the following three:
RQ1: To what extent did the training program support teachers’ improved understanding of the importance of educational data in measuring student engagement in mathematics classes?
RQ2: To what extent did teachers’ data literacy change throughout the training period?
RQ3: What design principles should be considered when designing data literacy training for teachers?
Method
To verify RQ1 and RQ2, an eight-month training program for mathematics teachers called “the Teacher Innovation Laboratory” was conducted. Results were compiled through surveys on data utilization (quantitative analysis) and references to reflections (qualitative analysis). Twenty-one active high school teachers participated in this program. In the monthly training sessions, instructors gave lectures to improve teachers’ digital literacy, and teachers collaborated with others to create lesson plans aimed at improving engagement. Between sessions, each teacher implemented the lesson plans they created, and findings were shared at the next session. A survey app called “LAPills” was used to check student engagement, allowing teachers to distribute surveys, collect responses from students, and check the results. For RQ3, discussions on design were held based on the insights gained from the training program.
Results and Discussion
Report on Teachers’ Perception of Data
As a result of the questionnaire survey on teachers’ perceptions of the usefulness of data in teaching, a significant improvement was observed in the pre- and post-test scores (Pre M=3.75, Post M=4.02, t(11)=-2.59, p=.02.). For qualitative analysis, teachers provided free-text responses to the question, “To what extent did you agree with the LAPills results, and what did you learn from them?” The analysis confirmed that teachers gained useful insights that they would have missed without using LAPills, such as “students showed more interest than usual during the implementation of the newly planned lessons.”
Teachers’ Data Literacy Skills
Results and discussions regarding teachers’ literacy skills through the training program were conducted from the following four perspectives:
1. Data selection and collection skills: As a result of investigating whether teachers became able to select and collect appropriate data through the training program using questionnaires, a significant increase in numerical values was confirmed for both selection and collection (Selection: Pre M=2.35, Post M=3.12, t(16)=-3.05, p=.007.; Collection: Pre M=2.29, Post M=3.35, t(16)=-4.24, p<.001.).
2. Data interpretation skills: Teachers’ reflections were analyzed to evaluate their ability to interpret data and translate it into teaching activities. As a result, it was confirmed that most teachers used data to accurately identify student engagement, with a particularly high focus on cognitive engagement.
3. Practice of data-informed teaching behavior: While most teachers only identified challenges in cognitive engagement, two teachers mentioned specific tactics to promote deeper learning experiences based on the engagement challenges they discovered (e.g., incorporating concept rephrasing more frequently, incorporating discussions for conceptual understanding at the end of class).
4. Obstacles faced by teachers: Regarding the use of LAPills for checking student engagement, most teachers were able to interpret the collected data, but some teachers complained about the difficulty of interpretation (2 people), low visibility of data display (1 person), and low customizability (1 person).
LAD Design Based on Insights from the Training Project
From the results of the training project, the authors discovered the challenge that “the ability to link new pedagogical knowledge with information extracted from data to plan future instruction could not be sufficiently confirmed (it remained limited to focusing on specific data).” To solve this, they proposed a theory-based LA tool design to scaffold the development of teacher expertise by explicitly connecting pedagogical design elements (e.g., student engagement) with pedagogical knowledge learned in training and data actually obtained in the classroom. As an LA tool, the focus is particularly on LADs (Learning Analytics Dashboards), which are increasingly being used for a deep understanding of student learning, and the following three design considerations are proposed:
1. Usefulness of theory-based LA tools in scaffolding teacher learning:
To support the acquisition of effective teaching and data utilization skills, it is stated that LA tools should provide tactically staged scaffolding. In the authors’ study, it is noted that information related to student engagement, which is important in teaching, is explained, and teaching practice proposals based on educational theory are notified to aim for the development of pedagogical knowledge and data utilization skills.
2. Theoretical basis of LA tools:
It is argued that when designing LA tools that utilize teacher scaffolding and educational data (data related to engagement in this study), the foundation must be relevant theory.
3. Use of multiple data sources:
Although the main data source in this study was student self-reporting via LAPills, the use of more diverse data is recommended for a deeper understanding of the complex phenomenon of student engagement.
Conclusion
In this study, a collaborative training intervention program incorporating classroom data and practice was conducted for K-12 mathematics teachers regarding the challenges of data utilization. As a result, while most teachers showed a positive attitude toward data use, challenges remained regarding how to display data and the fact that the majority of teachers could not reach the stage of selecting appropriate educational actions based on findings from the data. Based on the insights gained from practice, the authors also proposed a LAD that serves as a learning scaffold for teachers to connect educational knowledge and skills with data obtained in the classroom to move to appropriate teaching practices based on educational theory.
Reflections
Since the research I plan to conduct involves developing a LAD that promotes teacher reflection based on data, I chose this paper to review very similar research and deepen my knowledge of theoretical backgrounds and evaluation methods in research.
As for my thoughts on the review, I felt it was a wonderful, field-rooted study in that they set a long period of eight months and continuously conducted practice with teachers in the field. On the other hand, although there was a design proposal for a LAD, the actual research only went as far as the training intervention, so it was difficult to get a concrete image. Also, in the proposed LAD, it was said that information to focus on would be displayed and appropriate teaching actions would be proposed based on it, considering data obtained from the classroom and related pedagogical theories. However, I felt that it would be very difficult to generalize and define these actions in advance because the actions proposed within the LAD are greatly influenced by factors such as classroom context, constraints of schools/municipalities, and other resources, even if there is an underlying educational theory. I look forward to future research on how to balance theory and teaching actions and introduce them into the field.
Report by: Naohiro Higuchi
