SECOND CALL FOR PAPERS

Perspectivist Approaches to Disagreement in NLP (NLPerspectives)

Until recently, the dominant paradigm in natural language processing (and other areas of artificial intelligence) has been to resolve observed label disagreement into a single “ground truth” or “gold standard” via aggregation, adjudication, or statistical means. However, in recent years, the field has increasingly focused on subjective tasks, such as abuse detection or quality estimation, in which multiple points of view may be equally valid, and a unique ‘ground truth’ label may not exist. At the same time, as concerns have been raised about bias and fairness in AI, it has become increasingly apparent that an approach which assumes a single “ground truth” can erase minority voices.

Strong perspectivism in NLP (Basile et al., 2021) pursues the spirit of recent initiatives such as Data Statements (Bender and Friedman, 2018), extending their scope to the full NLP pipeline, including the aspects related to modelling, evaluation and explanation.

The “Perspectivist Approaches to Disagreement in NLP” (NLPerspectives) workshop will explore current and ongoing work on the collection and labelling of non-aggregated datasets, and approaches to modelling and including these perspectives, as well as evaluation and applications of multi-perspective Machine Learning models. We also welcome opinion pieces and literature reviews, e.g. in the context of fairness and inclusion.

A key outcome of the workshop will be to build on the work begun at https://pdai.info/ to create a repository of perspectivist datasets with non-aggregated labels for use by researchers in perspectivist NLP modelling.

Authors are therefore invited to share their LRs. In particular, when submitting a paper from the START page, authors will be asked to provide essential information about resources (in a broad sense, i.e. also technologies, standards, evaluation kits, etc.) that have been used for the work described in the paper or are a new result of your research.

Moreover, ELRA encourages all LREC authors to share the described LRs (data, tools, services, etc.) to enable their reuse and replicability of experiments (including evaluation ones).

Submissions

We accept two types of submissions, long papers and short papers (e.g., demonstration papers, dataset and resource papers, short focused contributions), all following the LREC template: https://lrec2022.lrec-conf.org/en/submission2022/authors-kit/

Long papers may consist of up to 8 pages of content, plus unlimited references. Short papers may consist of up to 4 pages of content; shorter versions are also welcome. Final versions will be given one additional page of content so that reviewers' comments can be taken into account. Submissions should be sent in electronic forms, using the Softconf START conference management system. The submission site is https://www.softconf.com/lrec2022/NLPerspectives/.

We invite original research papers from a wide range of topics, including but not limited to:

Non-aggregated data collection and annotation frameworks

Descriptions of corpora collected under the perspectivist paradigm

Multi-perspective Modelling and Machine Learning

Evaluation of multi-perspective models/ models of disagreement

Multi-perspective disagreement as applied to NLP evaluation

Fairness and inclusive modelling

Applications of multi-perspective modelling

Computing with (dis)agreement

Perspectivist Natural Language Generation

Foundational aspects of perspectivism

Opinion pieces and reviews on perspectivist approaches to NLP


Submissions are open to all, and are to be submitted anonymously. All papers will be refereed through a double-blind peer review process by at least three reviewers with final acceptance decisions made by the workshop organizers.

The workshop will take place at LREC, and is scheduled to last for half a day on June 20th 2022.

Important Dates

* Friday April 15, 2022: Paper submission

* Friday May 6, 2022: Notification of acceptance

* Friday May 20, 2022: Camera-ready papers due

* Monday June 20, 2022: Workshop

Workshop organizers:

Gavin Abercrombie, Heriot-Watt University

Valerio Basile, University of Turin

Verena Rieser, Heriot-Watt University

Sara Tonelli, Fondazione Bruno Kessler

Alexandra Uma, Queen Mary University of London

Contact us at g.abercrombie@hw.ac.uk if you have any questions.

Website: https://nlperspectives.di.unito.it/

Founded in 1821, Heriot-Watt is a leader in ideas and solutions. With campuses and students across the entire globe we span the world, delivering innovation and educational excellence in business, engineering, design and the physical, social and life sciences. This email is generated from the Heriot-Watt University Group, which includes:

Heriot-Watt University, a Scottish charity registered under number SC000278
Heriot- Watt Services Limited (Oriam), Scotland's national performance centre for sport. Heriot-Watt Services Limited is a private limited company registered is Scotland with registered number SC271030 and registered office at Research & Enterprise Services Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS.

The contents (including any attachments) are confidential. If you are not the intended recipient of this e-mail, any disclosure, copying, distribution or use of its contents is strictly prohibited, and you should please notify the sender immediately and then delete it (including any attachments) from your system.