{"id":662700,"date":"2025-04-15T01:51:45","date_gmt":"2025-04-14T22:51:45","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/debates-over-ai-benchmarking-have-reached-pokemon\/"},"modified":"2025-04-15T01:51:45","modified_gmt":"2025-04-14T22:51:45","slug":"debates-over-ai-benchmarking-have-reached-pokemon","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/debates-over-ai-benchmarking-have-reached-pokemon\/","title":{"rendered":"Debates over AI benchmarking have reached Pok\u00e9mon"},"content":{"rendered":"
\n

Not even Pok\u00e9mon is safe from AI benchmarking controversy. <\/p>\n

Last week, a post on X<\/a> went viral, claiming that Google\u2019s latest Gemini model surpassed Anthropic\u2019s flagship Claude model in the original Pok\u00e9mon video game<\/a> trilogy. Reportedly, Gemini had reached Lavendar Town in a developer\u2019s Twitch stream; Claude was stuck at Mount Moon as of late February.<\/p>\n

\n

Gemini is literally ahead of Claude atm in pokemon after reaching Lavender Town<\/p>\n

119 live views only btw, incredibly underrated stream pic.twitter.com\/8AvSovAI4x<\/a><\/p>\n

\u2014 Jush (@Jush21e8) April 10, 2025<\/a><\/p>\n<\/blockquote>\n

But what the post failed to mention is that Gemini had an advantage.<\/p>\n

As users on Reddit<\/a> pointed out, the developer who maintains the Gemini stream built a custom minimap that helps the model identify \u201ctiles\u201d in the game like cuttable trees. This reduces the need for Gemini to analyze screenshots before it makes gameplay decisions.<\/p>\n

Now, Pok\u00e9mon is a semi-serious AI benchmark at best \u2014 few would argue it\u2019s a very informative test of a model\u2019s capabilities. But it is<\/em> an instructive example of how different implementations of a benchmark can influence the results.<\/p>\n

For example, Anthropic reported<\/a> two scores for its recent Anthropic 3.7 Sonnet model on the benchmark SWE-bench Verified, which is designed to evaluate a model\u2019s coding abilities. Claude 3.7 Sonnet achieved 62.3% accuracy on SWE-bench Verified, but 70.3% with a \u201ccustom scaffold\u201d that Anthropic developed.<\/p>\n

More recently, Meta fine-tuned a version of one of its newer models, Llama 4 Maverick, to perform well on a particular benchmark, LM Arena. The vanilla version of the model scores significantly worse on the same evaluation.<\/p>\n

Given that AI benchmarks \u2014 Pok\u00e9mon included \u2014 are imperfect measures to begin with, custom and non-standard implementations threaten to muddy the waters even further. That is to say, it doesn\u2019t seem likely that it\u2019ll get any easier to compare models as they\u2019re released.<\/p>\n<\/div>\n