Open Source Software (OSS) platforms like GitHub thrive on large-scale collaboration, yet hostile interactions can erode participation and undermine community health. This motivates the need to identify and profile hostile user-generated content on GitHub. In this paper, we employ a multi-faceted hostility definition that encompasses user behavior that deters collaboration or participation by other users, including toxic, insulting, and threatening behaviors. We then present ASTRO, a systematic framework for identifying and profiling user-generated hostile content on GitHub at scale. Given a user or repository and its associated comments, we generate a multi-dimensional hostility vector that captures activity across each hostility attribute. Algorithmically, we introduce and combine two layers: 1) the PerspectiveAPI functionality, and 2) an appropriately-prompted LLM to find a good balance between accuracy and computational cost. We collect and then deploy ASTRO on three datasets: (a) 2,344 hostile users, (b) 1,253 hostile repositories, and (c) 100 benign random users. First, we show that our two-layer method achieves good performance and cost: it achieves an F1 score of 0.85 on our ground truth, while reducing the cost of using only the LLM layer by a factor of roughly 29. Second, we study the relationship between different dimensions of hostility and the distribution of hostile behavior across users and repositories, e.g., hostile users post much more than non-hostile ones. Finally, we shed light on communities that appear more conducive to hostile behavior, e.g., projects related to video game development. Overall, our work demonstrates that a nuanced understanding of hostility and the ability to detect it effectively can help model it and assist the development of approaches to mitigate its adverse effects on GitHub and other OSS platforms.

PDF