Also, the exploding nature of paper submissions leads to paper-vetting by less-experienced researchers. However, the majority of these efforts, which are targeted towards reforms on reviewers, might not be sufficient as there is an implicit human bias to invest less time in this voluntary yet critical job. There have been efforts in several venues to guide reviewers to write good reviews, curating detailed guidelines in reviewer forms, all aimed towards the objective and fair evaluation of submitted manuscripts, thereby restoring the faith in the widely accepted method of scholarly communication. Alarmed by some of these glaring problems at the core of the review process, coupled with the exponential increase in research paper submissions, the larger research community (not only just Meta Science) felt the need to study the paper-vetting system and build proposals towards mitigating its problems. The meta-research and science of science communities have long invested in studying annals of the peer-review process. However, academia often criticizes the peer review system as non-transparent, biased, arbitrary and inconsistent, a flawed process at the heart of science, leading to researchers arguing with its reliability and quality. The peer-review process is the only widely accepted method of research validation. The fourth author is also a recipient of the Visvesvaraya Young Faculty Award and acknowledges Digital India Corporation, MEITY for supporting this research.'Ĭompeting interests: The authors have declared that no competing interests exist. Fellowship Award (# VISPHD-MEITY-2518) from the Ministry of Electronics and Information Technology (MEITY), Government of India. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.ĭata Availability: All the codes and data are available on GitHub ( ).įunding: The first author thanks Visvesvaraya Ph.D. Received: NovemAccepted: OctoPublished: January 27, 2022Ĭopyright: © 2022 Ghosal et al. Northwestern Polytechnical University, CHINA Ĭitation: Ghosal T, Kumar S, Bharti PK, Ekbal A (2022) Peer review analyze: A novel benchmark resource for computational analysis of peer reviews. Our dataset and associated codes are available at.

We believe our dataset would provide a benchmark experimental testbed for automated systems to leverage on current NLP state-of-the-art techniques to address different issues with peer review quality, thereby ushering increased transparency and trust on the holy grail of scientific research validation. We also present baseline experiments and results for the different tasks for further investigations. We further introduce four novel tasks with this dataset, which could serve as an indicator of the exhaustiveness of a peer review and can be a step towards the automatic judgment of review quality. We also annotate the sentiment of the reviewer (positive, negative, neutral) for the first two layers to judge the reviewer’s perspective on the different sections and aspects of the paper. Given a text written by the reviewer, we annotate: to which sections (e.g., Methodology, Experiments, etc.), what aspects (e.g., Originality/Novelty, Empirical/Theoretical Soundness, etc.) of the paper does the review text correspond to, what is the role played by the review text (e.g., appreciation, criticism, summary, etc.), and the importance of the review statement (major, minor, general) within the review. Paper Section Correspondence, Paper Aspect Category, Review Functionality, and Review Significance. In this work, we present a first of its kind multi-layered dataset of 1199 open peer review texts manually annotated at the sentence level (∼ 17k sentences) across the four layers, viz. Peer review texts could serve as a rich source of Natural Language Processing (NLP) research on understanding the scholarly communication landscape, and thereby build systems towards mitigating those pertinent problems. These problems could also be due to the lack of studies with the peer-review texts for various proprietary and confidentiality clauses. However, academia often criticizes the peer review system as non-transparent, biased, arbitrary, a flawed process at the heart of science, leading to researchers arguing with its reliability and quality. Peer Review is at the heart of scholarly communications and the cornerstone of scientific publishing.

Benchmark research critique part 1