Evaluating Ratings from Large Language Models: A Simulation-Based Workflow

Large Language Models (LLMs) are increasingly used as research tools to facilitate fast and automated ratings of text features. In psychological studies, they have been used to quantify the degree to which stimulus materials reflect certain psychological constructs. However, the application of LLMs entails a high degree of flexibility regarding prompt design (e.g., instruction details and examples) and model specification (e.g., model family, size, and configuration), which can produce divergent results and threaten the robustness and generalizability of conclusions. To navigate the multiverse