Social perception, i.e. how people form impressions from language, forms the basis for subjective NLP tasks like sexism detection. Yet the mechanisms driving annotator disagreement remain unexplained. We propose that Theory of Mind (ToM) provides this mechanism: annotators perform cognitive ToM (inferring speaker intent) and affective ToM (estimating target impact), producing structured disagreement when these inferences diverge. Analysing the EXIST~2025 dataset (7,958 tweets, 4,044 memes), we find a detection-interpretation dissociation. Annotators who agree content is sexist show near-total disagreement about speaker intent and sexism type. This gap between detection and interpretation replicates across text and image modalities. Perceiver gender does not affect detection for text but does for memes (where even detection is ToM-demanding), and selectively shapes harm-related categorization (misogyny in text, extit{objectification} in memes) while leaving abstract categories unaffected. These patterns replicate established social psychology findings at computational scale and demonstrate that social perception operates through dissociable ToM processes that ,qny current NLP systems collapse.

PDF