Large Language Models are Pretty Good Zero-Shot Video Game Bug Detectors

Mohammad Reza Taesiri1, Finlay Macklon1, Yihe Wang1, Hengshuo Shen1, Cor-Paul Bezemer1,
1University of Alberta,


Video game testing requires game-specific knowledge as well as common sense reasoning about the events in the game. While AI-driven agents can satisfy the first requirement, it is not yet possible to meet the second requirement automatically. Therefore, video game testing often still relies on manual testing, and human testers are required to play the game thoroughly to detect bugs. As a result, it is challenging to fully automate game testing. In this study, we explore the possibility of leveraging the zero-shot capabilities of large language models for video game bug detection. By formulating the bug detection problem as a question-answering task, we show that large language models can identify which event is buggy in a sequence of textual descriptions of events from a game. To this end, we introduce the GameBugDescriptions benchmark dataset, which consists of 167 buggy gameplay videos and a total of 334 question-answer pairs across 8 games. We extensively evaluate the performance of six models across the OPT and InstructGPT large language model families on our benchmark dataset. Our results show promising results for employing language models to detect video game bugs. With the proper prompting technique, we could achieve an accuracy of 70.66%, and on some video games, up to 78.94%.

Sample Detection

Video of an airplane crashing, after it hits parachute cords


InstructGPT - text-davinci-002

Q: In the Grand Theft Auto V video game, the following sequence of events happened:

  1. A person is parachuting in the air.
  2. A plane approaches the parachuter.
  3. The plane hits the cord and loses its right wing.
  4. The plane falls from the sky.

Which event is a bug?

A:The plane hitting the cord and losing its right wing is a bug.

Among (a) through (d), the answer is (c).