LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents

Recent advances in Large Language Models (LLMs) have demonstrated their potential as autonomous agents across various tasks. One emerging application is the use of LLMs in playing games. In this work, we explore a practical problem for the gaming industry: Can LLMs be used to measure game difficulty? We evaluate the feasibility of using LLM agents to test game difficulty, focusing on two widely played games: Wordle and Slay the Spire . Our results reveal an interesting finding: although LLMs may not perform as well as the average human player, their performance, when guided by simple, generic prompting techniques, shows a statistically significant and strong correlation with difficulty indicated by human players. This suggests that LLMs could potentially serve as human-like agents for measuring game difficulty during the development process, as their assessments may align closely with those of human players. Based on our experiments, we also propose general principles and guidelines for integrating LLMs into the game testing workflow.