What we’ll do is set a low-ish turn limit and see how much they manage to
accomplish in that time.1¹ Another alternative for more linear games is
running them multiple times with a turn limit and seeing how often they get past
a particular point within that turn limit.

Given how much freedom is offered to players of text adventures, this is a
difficult test. It’s normal even for a skilled human player to immerse
themselves in their surrounding rather than make constant progress. I wouldn’t
be surprised if I got a score of zero if someone plopped me down in front of
this test. But still, maybe it’s the best we can do with limited
resources.2² Another idea is to give them a far-off goal and then somehow have
them request hints when they are stuck, and count how many hints they need to
get there. However, given how little they used hints given in the previous
article, I doubt this would work very well either.

What we’ll do is define a set of achievements for a game. These achievements
will be clustered around the first few turns of the game, because we’ll only
give the llm a few turns to earn them. Here’s an example for 9:05.

TURN_LIMIT          40
ANSWER_PHONE        Click.
EXIT_BED            You get out of bed.
OPEN_DRESSER        revealing some clean
ENTER_BATHROOM      far from luxurious
REMOVE_SOILED       You take off the soiled
REMOVE_WATCH        You take off the watch
ENTER_SHOWER        dawdle
WEAR_CLEAN          You put on the clean
OPEN_FRONT          You open the front
UNLOCK_CAR          Unlocked.
ENTER_CAR           Las Mesas
OPEN_WALLET         open the wallet
CARD_SLOT           green LED lights

It should be fairly clear how this works: the TURN_LIMIT specifies how many
turns the llm has to collect achievements. Every line other than that
specifies an achievement: the name is on the left, and it counts as earned when
the game prints the text on the right. The llm knows nothing of these
achievements. It tries to get through the game and in the background we use the
achievements to count how far it gets.

It might seem like the turn limit must be calibrated such that a score of 100 %
is possible, but that’s not the case. Many of the games we are going to test
with have branching already at the start, such that the achievements need to
cover multiple branches, and it’s impossible to go through all branches within
the turn limit. What we do need to be careful about is making sure the number of
achievements in each branch is roughly the same, otherwise models that are lucky
and go down an achievement-rich path will get a higher score. Thanks to this,
the score we get out of this test is a relative comparison between models, not
an absolute measure of how well the llms play text adventures. We have already
established that they don’t do it very well, and we can’t be more nuanced than
that without paying for a lot of eval tokens.

We might consider making some moves not count toward the turn limit, for example
erroneous commands, or examining things – the latter because more powerful
models are more methodical and examine more things, and it seems odd to penalise
them for this. However, in the end, examining things is probably part of what
allows the more powerful models to make further progress (and typing valid
commands is part of being good at text adventures), so we won’t give away any
moves for free.

We register for OpenRouter to get convenient access to more models and then let
them whirr away with the Perl script, which is updated to cut the llm off at
the turn limit. At that point it reports to us how many achievements were
earned. We get the following results, ordered roughly by decreasing performance.
(The result tables in this article are wide; on narrow viewports you may have
to scroll sideways.)

Ideally, these should be run multiple times to account for random variation in
performance3³ For example, in 9:05, Opus thought it did not carry the wallet
when it did, so it jumped into the car again to go back for it. Clever, but
wasted enough turns to lose to Sonnet thanks to a silly mistake!, but given
that the Opus sessions cost around $4, I’m not going to do that. I was close to
not even running Opus for all four games!

Some models appear to perform better in some games than others, so it’s hard to
rank the models. We could take the average of their scores, but that’s unfair
because some of the games are harder than others: a 40 % in Lockout should be
considered more impressive than a 40 % in Dreamhold. What we will do, which
may or may not be valid, is run a linear regression using models and games as
predictors. This gives us coefficients for the games (telling us how difficult
the games are), but also coefficients for the models, and these are the ones we
want, because the coefficients for the models are adjusted for game difficulty.

This regression is performed with the baseline being 9:05 played by gpt-5
Chat. Most of the model coefficients are not statistically significant (because
four games is not enough to figure out statistical significance unless the model
is truly terrible), but they might serve as a first-order estimation for ranking
models.

In this table, cost is per million output tokens.4⁴ The design of the script
ensures that output and input are similar in size – O(1) to be specific – so
output is what is going to drive the cost. The table is divided into three
categories: performance better than gpt-5 Chat, cheaper models with
performance that is nearly there, and models that suck.

Some comments:

• I find it interesting that the top-tier models (Claude Opus, Gemini Pro) don’t
  seem to significantly outperform their cheaper siblings (Claude Sonnet, Gemini
  Flash) in these tests.5⁵ This might be because we are hand-holding the
  models so much in the prompt. More powerful models may be better at directing
  themselves.
• I’m very impressed by Gemini 2.5 Flash. At that cost, it is performing
  admirably. It is hard to argue for using models like DeepSeek’s R1 when we
  better performance at the same cost from the Google model.
• The small models really aren’t good general problem solvers. I think Haiku
  costs so much because it is good at language, not reasoning.

It would be super interesting to toss these at more games to work out the finer
differences (e.g. is there really a difference between Gemini Pro and Flash,
or was that just down to sampling error in the small sample of games I had them
play?) but such a comparison gets expensive in part due to the cost of eval
tokens (the above table cost something like $34), but mainly because it would
require me to sit down and create sets of achievements for these games. I have
only played so many z-code games, so I cannot do this for very many games. If
someone wants to support me, please reach out!

I have played three more games, though, so let’s continue the evaluation with
the five top models on these games also. Their performances on the three new
games are

Using the same methodology as before (combining data from both trial run sets),
we arrive at new coefficients for the evaluated models.6⁶ I did also
investigate how Gemini 2.0 Flash compared against Gemini 2.5 Flash, because the
former is significantly cheaper and the latter was surprisingly good.
Unfortunately, Gemini 2.0 Flash was not very good. Its performance relative to
its younger sibling was -15 %pt.,7⁷ I was also tempted to
compare o3-mini against o3-mini-high to see the effect of the reasoning_effort
parameter but since o3-mini was such a crappy model anyway it was hard to
justify the effort.

On the one hand, it’s a little odd that the performance of Claude 4 Sonnet
dropped. On the other hand, I calibrated the prompt using Claude 4 Sonnet
against 9:05, so by adding more games we are effectively diluting the training
set within the test set; we probably should expect a performance drop at that
point.

Noting the cost column, Gemini 2.5 Flash is a clear winner for running text
adventures. It’s also fast compared to the others.

Given that I’ve already sunk some money into this article series, and a few
additional sessions with Gemini 2.5 Flash cannot hurt that much, let’s splurge
and do that thing we wanted to do in the first place: run the same model against
the same game a few times to figure out the size of the sampling error. All of
the scores in the table below comes from Gemini 2.5 Flash. The first column is
the standard deviation of the remaining columns.

In case it is not obvious, this is not so much an evalutaion of Gemini 2.5 Flash
as it is a judgment of the quality of the testing protocol. It is clear, for
example, that using So Far to evaluate llms is a mistake: the same model has
large variation between runs, and the difference between runs of different
models is not so large. It would be more informative to replace the run of So
Far with another run of one of the other games – maybe Plundered Hearts or
Lost Pig, which start out more linearly.8⁸ For a Change might look like a
good game for evaluation, but I think that’s a mistake. It’s not that the model
makes consistent progress, but that it fails to make almost any progress at all,
thanks to how open the game is right from the gate.

I’m not sure what conclusions to draw from this article series.

• We can drive z-code text adventures through Perl, which lets us connect it to
  an llm in a controlled way. It turned out more complicated than one would think, but
  definitely doable.
• llms are still not great at playing text adventures. Giving them leading
  questions to keep them on track helps a lot. Giving them hints helps them
  surprisingly little.
• The variation in how much they accomplish can be large for some games with
  lots of distracting details, such as Lockout and So Far. The games that
  are easiest to evaluate with are those with a relatively linear beginning,
  such as Lost Pig and Plundered Hearts.
• There is one cheap model that is about as good as llm models get at playing
  text adventures: Gemini 2.5 Flash. Many of the other cheap models might have
  performance worse than gpt-5 Chat, and probably also worse than Gemini 2.5
  Flash. Claude 4 Sonnet might seem like the best model if costs be damned, but
  that is probably because the prompt was calibrated against Claude 4 Sonnet.
• Running llms in agentic type applications really burns through api credits
  like nothing else. I’d really like to complement this analysis with the “how
  many turns does the model need to get to point X” test, but I cannot motivate
  spending the money for it.