The spread of multimodal election misinformation, where text and images jointly convey misleading narratives, seriously threatens democratic integrity. This emerging challenge demands better automated detection and deeper insight into how such narratives are formed and propagated. Yet, little work has systematically examined the characteristics of such content or evaluated how well Large Language Models (LLMs) can classify it across nuanced categories. We introduce a large-scale, annotated multimodal dataset of election-related social media posts from X.com (formerly Twitter), spanning the 2024 U.S. presidential election. Our dataset captures temporal trends, includes text-only and image-based posts, and is labeled across five nuanced misinformation categories: Conspiracy, Sensationalism, Hate Speech, Speculation, and Satire. Given the high cost and time demands of manual annotation, scalable solutions are essential. To address this, we explore automated labeling in our dataset using six LLMs of varying complexity. We compare three lighter-weight models against three full-scale models using a majority vote mechanism and human validation. Results show that lighter-weight models exhibit higher internal agreement and stability, particularly in classifying subjective categories such as Satire and Speculation. In contrast, larger models demonstrate more variance and lower inter-model reliability. We also inspect where models diverge and identify potential causes of disagreement, such as ambiguous tone, sarcasm, and exaggeration. Our preliminary findings indicate that LLM-driven annotation produces labels that are reliable enough to serve as a usable ground truth, especially for large-scale studies. We aim to facilitate future research in multimodal misinformation detection and annotation-efficient learning.

PDF