{"id":658724,"date":"2025-03-25T11:25:16","date_gmt":"2025-03-25T08:25:16","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/a-new-challenging-agi-test-stumps-most-ai-models\/"},"modified":"2025-03-25T11:25:16","modified_gmt":"2025-03-25T08:25:16","slug":"a-new-challenging-agi-test-stumps-most-ai-models","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/a-new-challenging-agi-test-stumps-most-ai-models\/","title":{"rendered":"A new, challenging AGI test stumps most AI models"},"content":{"rendered":"
\n

The Arc Prize Foundation, a nonprofit co-founded by prominent AI researcher Fran\u00e7ois Chollet, announced in a blog post<\/a> on Monday that it has created a new, challenging test to measure the general<\/a> intelligence of leading AI models.<\/p>\n

So far, the new test, called ARC-AGI-2, has stumped most models.<\/p>\n

\u201cReasoning\u201d AI models like OpenAI\u2019s o1-pro and DeepSeek\u2019s R1 score between 1% and 1.3% on ARC-AGI-2, according to the Arc Prize leaderboard<\/a>. Powerful non-reasoning models including GPT-4.5, Claude 3.7 Sonnet, and Gemini 2.0 Flash score around 1%.<\/p>\n

The ARC-AGI tests consist of puzzle-like problems where an AI has to identify visual patterns from a collection of different-colored squares, and generate the correct \u201canswer\u201d grid. The problems were designed to force an AI to adapt to new problems it hasn\u2019t seen before. <\/p>\n

The Arc Prize Foundation had over 400 people take ARC-AGI-2 to establish a human baseline. On average, \u201cpanels\u201d of these people got 60% of the test\u2019s questions right \u2014 much better than any of the models\u2019 scores.<\/p>\n

\"\"
a sample question from Arc-AGI-2 (credit: Arc Prize).<\/span><\/figcaption><\/figure>\n

In a post on X<\/a>, Chollet claimed ARC-AGI-2 is a better measure of an AI model\u2019s actual intelligence than the first iteration of the test, ARC-AGI-1. The Arc Prize Foundation\u2019s tests are aimed at evaluating whether an AI system can efficiently acquire new skills outside the data it was trained on.<\/p>\n

Chollet said that unlike ARC-AGI-1, the new test prevents AI models from relying on \u201cbrute force\u201d \u2014 extensive computing power \u2014 to find solutions. Chollet previously acknowledged this was a major flaw of ARC-AGI-1.<\/p>\n

To address the first test\u2019s flaws, ARC-AGI-2 introduces a new metric: efficiency. It also requires models to interpret patterns on the fly instead of relying on memorization.<\/p>\n

\u201cIntelligence is not solely defined by the ability to solve problems or achieve high scores,\u201d Arc Prize Foundation co-founder Greg Kamradt wrote in a blog post<\/a>. \u201cThe efficiency with which those capabilities are acquired and deployed is a crucial, defining component. The core question being asked is not just, \u2018Can AI acquire [the] skill to solve a task?\u2019 but also, \u2018At what efficiency or cost?\u2019\u201d<\/p>\n

ARC-AGI-1 was unbeaten for roughly five years until December 2024, when OpenAI released its advanced reasoning model, o3, which outperformed all other AI models and matched human performance on the evaluation. However, as we noted at the time, o3\u2019s performance gains on ARC-AGI-1 came with a hefty price tag.<\/p>\n

The version of OpenAI\u2019s o3 model \u2014 o3 (low) \u2014 that was first to reach new heights on ARC-AGI-1, scoring 75.7% on the test, got a measly 4% on ARC-AGI-2 using $200 worth of computing power per task.<\/p>\n

\"\"
Comparison of Frontier AI model performance on ARC-AGI-1 and ARC-AGI-2 (credit: Arc Prize).<\/span><\/figcaption><\/figure>\n

The arrival of ARC-AGI-2 comes as many in the tech industry are calling for new, unsaturated benchmarks to measure AI progress. Hugging Face\u2019s co-founder, Thomas Wolf, recently told TechCrunch that the AI industry lacks sufficient tests to measure the key traits of so-called artificial general intelligence, including creativity.<\/p>\n

Alongside the new benchmark, the Arc Prize Foundation announced a new Arc Prize 2025 contest<\/a>, challenging developers to reach 85% accuracy on the ARC-AGI-2 test while only spending $0.42 per task.<\/p>\n<\/div>\n

If you liked the article, do not forget to share it with your friends. Follow us on\u00a0Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n

\n

If you want to read more like this article, you can visit our Technology<\/a><\/span> category.<\/strong><\/p>\n<\/blockquote>\n

Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

The Arc Prize Foundation, a nonprofit co-founded by prominent AI researcher Fran\u00e7ois Chollet, announced in a blog post on Monday that it has created a new, challenging test to measure the general intelligence of leading AI models. So far, the new test, called ARC-AGI-2, has stumped most models. \u201cReasoning\u201d AI models like OpenAI\u2019s o1-pro and…<\/p>\n","protected":false},"author":1,"featured_media":658725,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/techcrunch.com\/wp-content\/uploads\/2024\/03\/GettyImages-1708266672.jpg?resize=1200,1029","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[77337,151633],"class_list":["post-658724","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology","tag-ai","tag-agi"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/658724","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=658724"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/658724\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/658725"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=658724"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=658724"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=658724"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}