I will introduce the paper discussed in this  English seminar.
The paper presented at LAK in 2022 is titled “How do Emails Improve Homework Completion Rates? – A Case Study Using A/B Comparison.”

Title: How can Email Interventions Increase Students’ Completion of Online Homework? A Case Study Using A/B Comparisons
Authors: Angela Zavaleta-Bernuy, Ziwen Han, Hammad Shaikh, Qi Yin Zheng, Lisa-Angelique Lim, Anna Rafferty, Andrew Petersen, and Joseph Jay Williams
Conference: LAK22 (12th International Learning Analytics and Knowledge Conference), March 21–25, 2022, Online, USA. ACM, New York, NY, USA, 12 pages）
Link: https://doi.org/10.1145/3506860.3506874

Procrastination on homework is a common issue across all countries, isn’t it?
One of the most popular methods to prevent this is the use of reminders.

This paper examines “what types of email reminders are effective” in getting students to start their homework early, and “how to create effective reminders”. The effectiveness of various email reminders was tested using the method of “randomized A/B comparisons,” while interviews regarding the effectiveness of the reminders were conducted with both teachers and students. This experiment demonstrated that “randomized A/B comparisons” are a valuable method for exploring how to design email reminders.

Previous research has reported that specific prompts and reminder designs can improve engagement with homework. The authors of this paper combined prompts that enhance the start time and performance of homework to create multiple email reminders. For example, they modified the sender and subject of the reminders and added to the content written within them.

To examine how the created reminders affect homework grades and start times, and how they are perceived by students and teachers, the following four research questions were established:

RQ1: What impact do reminder messages have on the start times and grades of homework?
RQ2: What impact does the design of the reminders have on the start times and grades of homework?
RQ3: How do students perceive the reminders?
RQ4: How do teachers’ intuitions about the reminders differ from the experimental results?

In this study, the effectiveness of various reminder designs in improving students’ homework performance was tested using the A/B comparison method. The A/B comparison method, also known as a randomized controlled trial (RCT), compares two or more groups that are randomly assigned to evaluate the effect of the subject under study. The participants in this study were 946 American college students enrolled in a computer science course, and the research aimed to help them start their weekly homework earlier.

The college students’ homework consists of solving Python-related problems before and after class. The assignments are made available four days before the deadline, and students complete the tasks on their computers. Students can attempt the homework multiple times, with the highest score being recorded as the final result.

Reminders were sent from the university’s Learning Management System (LMS), Canvas, at 84, 51, 40, and 47 hours after the assignment deadlines. In designing multiple reminders, five different prompts were established, and each week a different prompt was randomly applied to test the effectiveness of various prompts.

1. Sender: The sender of the email is randomly assigned as either “Course Instructor” or “Teaching Assistant.”

2. Information: The message includes randomly chosen information explaining why it is important to start homework early.

3. Subject Line: There are two types of subject lines: “Deadline Summary,” which includes the homework deadline, and “Prompt to Plan Homework,” which encourages planning for the homework (e.g., “When will you do this week’s homework?”).

4. Recommendation: Descriptions on how to start homework early, such as “Try just starting with five minutes to get going.”

5. Advice: Messages that ask students to think of advice for other students on how to start their homework early (e.g., “Think about what advice you would give to another student to start their homework early.”).

Students were randomly assigned each week to either the experimental group (with reminders) or the control group (without reminders). The intervention took place from the fifth to the eighth week of the course. To monitor homework completion and submission, timestamps were collected, and both the start times and completion rates of the homework were analyzed. Additionally, to understand how students perceived the reminders and how they acted after receiving them, a 7-point Likert-scale questionnaire and open-ended responses were collected post-intervention. Furthermore, two instructors were asked to predict the impact of the reminders on the homework start times and completion rates.

Here are the results:

1. Effect of Reminders
The impact of reminder messages over four weeks was analyzed using a panel data regression model. The results showed that the reminders significantly increased the number of students starting their homework by about 4 percentage points, and students who received reminder messages saw a 4% improvement in their homework grades. However, the reminders did not have a statistically significant effect on the time students started their homework (start time).

2. Differences in Effects Based on Reminder Design
The differences in the distribution of start times across various variables were analyzed using the Kolmogorov-Smirnov test. Significant differences in the distribution of start times were observed between the sender (control group and “instructor”), recommendation (control group and “no prompt”), and advice (control group and “no prompt”). For the variables of information (“control group” and “with prompt”) and advice (“control group” and “with prompt”), minor significant differences (0.05 ≤ α < 0.1) were observed. Regarding the distribution of start times by subject line, no statistical significance was observed under any condition. Defining the completion rate as a Bernoulli variable and assessing the impact of reminders on completion rates via the Z-test, significant increases in completion rates were found in the categories of recommendation (“control group” with and without prompt) and advice (“control group” and “with prompt”).

3. Student Perceptions of Reminders
Most students felt that their motivation for homework was enhanced by the information (63%) and recommendations (63%) provided in the reminders. Regarding actions taken after receiving the reminders, more than half of the learners (57%) reported a positive change in behavior. Of those who provided reasons, the majority (20% of the total) stated that the messages helped reduce procrastination with their homework. On the other hand, the most common reason for not changing behavior after receiving a message was “already having a study plan or having started the homework.”

4. Teachers’ Predictions About Reminders
When two instructors were asked to predict the effects of the reminders, the lead instructor was generally able to accurately forecast which designs would be effective for homework completion. However, it was also found that teachers tend to overestimate the effectiveness of some conditions.

Here are the authors’ reflections based on the results:
First, it appears that sending reminder emails does not advance the start time for homework. However, the number of students who began their homework increased, resulting in an improvement in the average homework grades. From the qualitative data collected from students, more than half reported that the reminders were helpful, while some expressed concern that the teachers might be too involved in monitoring their learning. Regarding the usefulness of instructors’ predictions, it was found that instructors tend to overestimate the effectiveness of emails. Using data from randomized A/B testing can help in supporting optimal decision-making by instructors. Additionally, the design of the reminders suggests that including reasons “why to start homework early” and “methods or advice on how to begin homework promptly” could potentially motivate students to start their assignments earlier.

Here are my reflections after reading this paper:
The study proposed several reminder message designs and clarified which designs were effective in promoting early start and enhanced performance on student homework. I believe this serves as a useful precursor study in proving the effectiveness of the A/B comparison method and the designs of effective reminders. From the perspective of Learning Analytics (LA), it would have been beneficial to have data on whether students opened the reminder emails or how long they viewed the reminders. This is because, among students who thoroughly read the reminder texts, an increase in homework completion rates and early starts could presumably be observed. Furthermore, had the reminder messages included data about the time spent on performing the homework, it might have helped students in planning their homework execution.

The purpose of creating reminders in this study is to encourage students to carry out their homework in a planned manner. A potential design for such reminders could involve prompting students to reflect on the time spent on previous assignments and using that reflection to assist in planning future tasks. For future research, it would be beneficial to conduct experiments that incrementally present prompts while identifying specific challenges in challenges in following planned sequences for homework execution, thereby verifying the effectiveness of such strategies while carrying out their homework.

By: Saki Hirata (M2 student)