Strengthening ethical review in natural language processing: insights, best practices, resources and paths forward

by Luciana Benotti, Snigdha Chaturvedi, Sunipa Dev

Summary

This text offers practical guidance and highlights valuable resources, such as tutorials and guidelines, designed to help *CL authors and reviewers navigate ethical questions with greater clarity and responsibility. As EMNLP 2024 ethics chairs we highlight the lessons learned and advice for future review cycles.

Insights

Ethical reviewing in Natural Language Processing (NLP) conferences and proceedings is a critical responsibility that ensures that our research practices align with societal values and minimize potential harm. When reviewers flag ethical concerns, it’s not just to inform authors but also to assist program and ethics chairs in determining whether the paper warrants a deeper ethics review. A well-justified ethical flagging provides essential context, helping ethics chairs understand the gravity of the issue and decide on the next steps.

Best practices

Technical reviewers are often the first to detect potential risks in research, such as biased data, harmful applications, or privacy concerns. When flagging these issues, it’s important to explain the ethical concerns clearly. This helps both authors and ethics chairs understand why the concern was raised and how it may affect the broader impact of the work. A clear justification helps the authors in understanding the ethical implications of their work which they might have (mistakenly) overseen. It helps them improve their work. A clear justification also helps the ethics chairs to decide if the concern can be addressed within the scientific process or requires further ethical scrutiny. The tone and clarity of ethical feedback are crucial. Phrases like, “This section would benefit from a more thorough consideration of potential biases in the data. For example, all annotators hired in this study were from a specific demographic group which might bias the data…” or “Further reflection on the societal impact of deploying this model on [vulnerable social group] would strengthen the paper. For example, outputs from such a model might negatively portray people from lower socio-economic groups.” encourage constructive engagement rather than defensiveness. By framing ethical flagging as a collaborative process, reviewers support both the authors in improving their work and the ethics chairs in ensuring thorough ethical oversight. If the reviewer identifies actionable recommendations for authors—such as improving transparency in the methodology, mitigating bias in data, clarifying annotator guidelines and payment strategies or compute usage, or considering the societal impact of the model—these can be included in the scientific review. This allows authors to address these issues directly during the revision process, potentially resolving ethical concerns without needing an additional ethics review. For more contextualized pointers to what constitutes a clear justification see resources below.

Resources and paths forward

To further support *CL paper reviewers who are interested in deepening their understanding of the ethics review process, we are providing links to valuable training materials. These resources include a tutorial that was presented at the 2022 EACL conference, which covers best practices for identifying and addressing ethical concerns in NLP research. Future tutorials will be announced here: https://ethics.aclweb.org/. Also, there are the guidelines for technical reviewers for flagging papers written by ARR chairs and the guidelines for ethics reviewing used by ARR. These materials are designed to help reviewers become more familiar with the ethical dimensions of their role and offer practical guidance on how to navigate complex ethical questions in the papers they review.

Thanks

We would like to thank all those involved in the ethics review process for their time, effort, and commitment. They include but are not limited to, the ethics reviewers, technical reviewers, area chairs, senior area chairs, the program chairs, the ARR chairs, the ACL ethics committee, ethics chairs from previous conferences, and the OpenReview support staff. They are typically experts in their fields, volunteering their time and expertise to uphold the integrity of the research community. By thoroughly evaluating papers, they ensure that both the technical and ethical dimensions of research are held to the highest standards. Their dedication fosters a culture of trust, rigor, and responsibility in the field.