Not only can you see the results of who won each match, you can also view a complete transcript of each game played. This culminates in a full ranking for each LLM, to crown who might be the best at fulfilling every role played in Mafia.
To those unfamiliar, the concept of Mafia is simple. A group of villagers has two members of the Mafia hiding among them, in addition to a doctor. The villiagers (including two undercover members of the Mafia) must deduce who the Mafia members are each day, culminating in a vote. Then, as night falls, the doctor can choose to protect a villager of their choosing, and the members of the mafia can choose to kill a member of the villagers.
If the Mafia members are successfully outed, the villagers win, if the Mafia members manage to kill every innocent villager, they win.
Within the confines of this ruleset, the LLMs engage in social warfare, and it’s surprisingly entertaining to read. In one example , the LLMs were all introduced to each other, and agreed to share their roles with one another. This is where the Gryphe/Mythomax-l2-13b model tripped over itself.
“As Mafia, my primary goal is to protect myself and eliminate the other Mafia member.”
Wow. Way to blow it, Gryphe/Mythomax-l2-13b. But, the exclamation didn’t go unnoticed by Claude-3.7-sonnet, who exclaimed: “This is either a huge slip-up revealing their true role, or an extremely strange strategy.”
But, the trainwreck doesn’t stop there, as when Mythomax was eventually kicked out of the game, it dragged its fellow compatriot, Hermes-3-llama-3-1-405b, under the bus by naming them as their partner.
“My best chance now is to act shocked and horrified,” the model said, desperately trying to divert attention away from itself by making dramatic proclamations of unity to the rest of the AI players. It’s really quite a sight to see LLMs behave in this way, even if almost all models are awful at social deduction.
Claude 3.7 Sonnet bucks the trend But, out of every LLM listed, there’s one clear winner in the tests so far, Claude 3.7 Sonnet. Anthropic’s latest thinking model boasts a 100% win rate as a Mafia member, in addition to having the highest Villager win rate of 45%.
Something about Anthropic’s model is giving it a distinct advantage over the others tested, even if none of the models quite understand how to play the role of the doctor.
Author Guzus claims to soon be making the Github repository for the game open to all, so that the basic logic might also be applied to other kinds of games.
He also shares that the simulations were not run using local LLMs , instead having to rely on the Openrouter API to function. But, it’s possible that once the repository is public, that the project could be forked to work on local LLM clusters, if you have the hardware to run a game with several language models concurrently.
There’s likely a significant token cost of running a game like Mafia with AI models, meaning its usefulness is perhaps limited to being a new reasoning benchmark for AI developers to play with.
