DOI: 10.36190/2021.14

Published: 2021-06-01
Detecting Anti-Jewish Messages on Social Media. Building an Annotated Corpus That Can Serve as A Preliminary Gold Standard
Gunther Jikeli, Deepika Awasthi, David Axelrod, Daniel Miehling, Pauravi Wagh, Weejoeng Joeng

Hate speech detection in online environments faces numerous challenges. One of them is that hate speech has fundamental target-specific elements. Although certain characteristics are common to many forms of hate speech, forms directed against one group, such as Jews, can be very different from forms directed against Muslims, Roma, members of the LGBTQ+ community, and bullying victims. Due to the heterogeneity of hate forms, we suggest approaching forms piecemeal and building labeled datasets that are specific to target groups. These datasets can then be combined into an aggregate dataset that increasingly reflects the diversity of hate speech found for a given language. Another challenge is the subjectivity of annotators and heterogeneous labeling. We created a labeled dataset of 4,159 antisemitic and non-antisemitic tweets, using a detailed definition and a specially designed annotation portal. The annotation was done by expert annotators who discussed their disagreements of each tweet. The dataset is built on representative samples of tweets containing more common keywords (such as "Jews") and keywords most likely to be used in antisemitic contexts (such as the term "kikes"). The dataset will be made available to the scientific community with the publication of this paper and will be updated with additional tweets and labels as the project continues. The paper describes the dataset, the labeling process, the infrastructure that was built for this project, some of the challenges that we faced, and an evaluation of the inter-coder reliability. The goal is to provide a detailed description of the labeled dataset to serve as a preliminary gold standard and a model for creating similar datasets.