{"id":655518,"date":"2025-03-04T04:35:11","date_gmt":"2025-03-04T01:35:11","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/people-are-using-super-mario-to-benchmark-ai-now\/"},"modified":"2025-03-04T04:35:11","modified_gmt":"2025-03-04T01:35:11","slug":"people-are-using-super-mario-to-benchmark-ai-now","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/people-are-using-super-mario-to-benchmark-ai-now\/","title":{"rendered":"#People are using Super Mario to benchmark AI now"},"content":{"rendered":"
\n

Thought Pok\u00e9mon was a tough benchmark for AI? One group of researchers argues that Super Mario Bros. is even tougher. <\/p>\n

Hao AI Lab, a research org at the University of California San Diego, on Friday threw AI into live Super Mario Bros. game<\/a>s. Anthropic\u2019s Claude 3.7 performed the best, followed by Claude 3.5. Google\u2019s Gemini 1.5 Pro and OpenAI\u2019s GPT-4o struggled.<\/p>\n

It wasn\u2019t quite the same version of Super Mario Bros. as the original 1985 release, to be clear. The game ran in an emulator and integrated with a framework, GamingAgent<\/a>, to give the AIs control over Mario.<\/p>\n

\"Super
Image Credits:<\/strong>Hao Lab<\/span><\/figcaption><\/figure>\n

GamingAgent, which Hao developed in-house, fed the AI basic instructions, like, \u201cIf an obstacle or enemy is near, move\/jump left to dodge\u201d and in-game screenshots. The AI then generated inputs in the form of Python code to control Mario.<\/p>\n

Still, Hao says that the game forced each model to \u201clearn\u201d to plan complex maneuvers and develop gameplay strategies. Interestingly, the lab found that reasoning models like OpenAI\u2019s o1, which \u201cthink\u201d through problems step by step to arrive at solutions, performed worse than \u201cnon-reasoning\u201d models, despite being general<\/a>ly stronger on most benchmarks.<\/p>\n

One of the main reasons reasoning models have trouble playing real-time games like this is that they take a while \u2014 seconds, usually \u2014 to decide on actions, according to the researchers. In Super Mario Bros., timing is everything. A second can mean the difference between a jump safely cleared and a plummet to your death.<\/p>\n

Games have been used to benchmark AI for decades. But some experts have questioned the wisdom<\/a> of drawing connections between AI\u2019s gaming skills and technological advancement. Unlike the real world, games tend to be abstract and relatively simple, and they provide a theoretically infinite amount of data to train AI.<\/p>\n

The recent flashy gaming benchmarks point to what Andrej Karpathy, a research scientist and founding member at OpenAI, called an \u201cevaluation crisis.\u201d<\/p>\n

\u201cI don\u2019t really know what [AI] metrics to look at right now,\u201d he wrote in a post on X<\/a>. \u201cTLDR my reaction is I don\u2019t really know how good these models are right now.\u201d<\/p>\n

At least we can watch AI play Mario.<\/p>\n<\/div>\n

If you liked the article, do not forget to share it with your friends. Follow us on\u00a0Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n

\n

If you want to read more like this article, you can visit our Technology<\/a><\/span> category.<\/strong><\/p>\n<\/blockquote>\n

Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

Thought Pok\u00e9mon was a tough benchmark for AI? One group of researchers argues that Super Mario Bros. is even tougher. Hao AI Lab, a research org at the University of California San Diego, on Friday threw AI into live Super Mario Bros. games. Anthropic\u2019s Claude 3.7 performed the best, followed by Claude 3.5. Google\u2019s Gemini…<\/p>\n","protected":false},"author":1,"featured_media":655519,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/techcrunch.com\/wp-content\/uploads\/2020\/09\/mario-35-jump.jpg?resize=1200,713","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[77337,153713,80,10751,38291],"class_list":["post-655518","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology","tag-ai","tag-benchmarks","tag-games","tag-gaming","tag-super-mario-bros"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/655518","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=655518"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/655518\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/655519"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=655518"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=655518"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=655518"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}