What I’m about to say is just my “personal” feeling regarding the peer review of journals and international conference manuscripts in the field of educational technology. Recently, I’ve found myself increasingly thinking that the quality of review comments has declined. This might just be within the scope of what I’m seeing, and I’m not sure about the recent situation within domestic (Japanese) journals. I feel this way not only as a paper submitter but also because, as a reviewer myself, I can see the results provided by other reviewers.
Peer review is a research activity that contributes to the development of an academic field. It involves explaining how a submitted paper or manuscript can be improved and engaging in scientific and logical communication with the author. Lately, I feel there are many review comments lacking logic and scientific rigor.
Since my work involves learning analytics, I often receive review requests related to that area. A common comment I see in recent learning analytics paper reviews is simply: “The sample size (N) is small.” Pointing that out is easy; it’s the kind of comment even a master’s student could make in a seminar. However, the role of a reviewer is to explain how the paper or conference manuscript can be improved and to request that the authors address those points. When N is small, researchers often utilize non-parametric tests or qualitative research methods. The reviewer must explain, in light of the research objectives written by the author, why those choices are insufficient. If one points out that “N is small, so the conclusions are limited,” the essential thing to convey is not just the small N, but a logical and persuasive explanation of why non-parametric tests or qualitative research are inadequate for achieving the stated research objectives. I feel fewer researchers are providing such explanations lately. Just saying “N is small” only gives the authors a sense of despair. That isn’t good.
Another common type of comment is “The literature review is lacking.” To some extent, these comments occur because the author’s writing is poor. Since this relates to the originality of the paper, it is a crucial point. However, as a responsibility of the reviewer, if you point this out, you must explain why you think the literature review is lacking. For example: “If you are developing your argument along these lines, Study A is relevant to your work, but a review related to A is missing.” An explanation of why and what kind of literature review needs to be added is necessary. This concerns the very foundation of the paper’s originality and motivation, and it alone can be grounds for rejection. Therefore, I believe a polite and thorough explanation is required.
In the past, I served as an editor for a long time in Japan. Among the reviews I oversaw, there were reviewers who would almost always suggest an immediate rejection for any paper assigned to them, and those reviewers were often the ones writing things that made me want to say, “Wait, what on earth is this?!” I’ve felt that “having high expertise” and “being an excellent researcher” do not always coincide. Unfortunately, such cases exist in Japan as well. However, around the time I stepped down from that society’s editorial committee, they began deciding on review policies based on the type of paper, which I thought was a very good development.
Peer review is usually a volunteer activity. In Japan, many people perform reviews while juggling heavy administrative duties and their own research time. I have the utmost respect for them. However, while one’s own time is important, once you accept a review, there are people whose careers depend on those results. Regardless of the final acceptance or rejection decision, I believe logical communication with the submitter is required from the perspective of “how can this paper and research be improved?”
Through peer review, researchers not only level up their own skills but also need to learn what it means to contribute to an academic field. To boost the research field of educational technology, spread good research findings internationally, and create a strong research community, I want to convey the importance of these things to my students during research guidance, even if my individual contribution is small. It’s a modest effort, but I believe that such things spread and lead to the development of research across generations. I hope young researchers will actively engage in peer review to broaden their perspectives and hone their research abilities. This becomes the power to advance one’s own research and increases one’s presence in the research field.
As I approach the “age 50 wall,” I want to think about what I personally can leave behind in the academic field by fostering the next generation of researchers, however small that contribution may be.
