Large language models are rapidly becoming an essential tool for social scientists. In particular, they have the potential to completely change the way researchers approach text analysis. In this study, we use the GPT-4 model to classify the content of newspaper articles and assess their sentiment. To do this, we collect headlines and leads from U.S. and German newspapers (n = 1,629) on how generative AI is represented in major news media outlets in both countries and inductively develop coding instructions based on this data. We then feed the data and instructions to GPT-4 and a human coder to compare their outputs and assess validity. We find that the coding procedure is highly reliable, with substantial to near perfect agreement between the human coder and GPT-4. We also find that reliability decreases for more complex constructs and is modestly lower for classification tasks performed in German than in English. Based on this analysis, we argue that LLMs offer powerful new approaches to text analysis that cross methodological divides between qualitative and quantitative approaches to empirical text analysis.

PDF