Workshop Proceedings of the 18th International AAAI Conference on Web and Social Media

Workshop: REAL-Info 2024: First Workshop on Reliable Evaluation of LLMs for Factual Information

DOI: 10.36190/2024.31

Published: 2024-06-01
Using GPT-4 for Text Analysis: Insights from English and German Language News Classification Tasks
Viktor Suter, Miriam Meckel

Large language models are rapidly becoming an essential tool for social scientists. In particular, they have the potential to completely change the way researchers approach text analysis. In this study, we use the GPT-4 model to classify the content of newspaper articles and assess their sentiment. To do this, we collect headlines and leads from U.S. and German newspapers (n = 1,629) on how generative AI is represented in major news media outlets in both countries and inductively develop coding instructions based on this data. We then feed the data and instructions to GPT-4 and a human coder to compare their outputs and assess validity. We find that the coding procedure is highly reliable, with substantial to near perfect agreement between the human coder and GPT-4. We also find that reliability decreases for more complex constructs and is modestly lower for classification tasks performed in German than in English. Based on this analysis, we argue that LLMs offer powerful new approaches to text analysis that cross methodological divides between qualitative and quantitative approaches to empirical text analysis.