Workshop Proceedings of the 18th International AAAI Conference on Web and Social Media

Workshop: REAL-Info 2024: First Workshop on Reliable Evaluation of LLMs for Factual Information

DOI: 10.36190/2024.33

Published: 2024-06-01
Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case Study on Reddit
Parker Seegmiller, Joseph Gatto, Omar Sharif, Madhusudan Basak, Sarah Masud Preum

Large language models (LLMs) have been shown to be proficient in correctly answering questions in the context of online discourse. However, the study of using LLMs to model human-like answers to fact-driven social media questions is still under-explored. In this work, we investigate how LLMs model the wide variety of human answers to fact-driven questions posed on several topic-specific Reddit communities, or subreddits.We collect and release a dataset of 409 fact-driven questions and 7,534 diverse, human-rated answers from 15 r/Ask{Topic} communities across 3 categories: profession, social identity, and geographic location. We find that LLMs are considerably better at modeling highly-rated human answers to such questions, as opposed to poorly-rated human answers. We present several directions for future research based on our initial findings.