{"id":646389,"date":"2024-12-10T06:10:16","date_gmt":"2024-12-10T03:10:16","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/a-test-for-agi-is-closer-to-being-solved-but-it-may-be-flawed\/"},"modified":"2024-12-10T06:10:16","modified_gmt":"2024-12-10T03:10:16","slug":"a-test-for-agi-is-closer-to-being-solved-but-it-may-be-flawed","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/a-test-for-agi-is-closer-to-being-solved-but-it-may-be-flawed\/","title":{"rendered":"#A test for AGI is closer to being solved \u2014 but it may be flawed"},"content":{"rendered":"
\n

A well-known test for artificial general<\/a> intelligence (AGI) is closer to being solved. But the tests\u2019s creators say this points to flaws in the test\u2019s design, rather than a bonafide research breakthrough.<\/p>\n

In 2019, Francois Chollet, a leading figure in the AI world, introduced the ARC-AGI benchmark, short for \u201cAbstract and Reasoning Corpus for Artificial General Intelligence.\u201d Designed to evaluate whether an AI system can efficiently acquire new skills outside the data it was trained on, ARC-AGI<\/a>, Francois claims, remains the only AI test to measure progress towards general intelligence (although others<\/a> have been proposed.)<\/p>\n

Until this year, the best-performing AI could only solve just under a third of the tasks in ARC-AGI. Chollet blamed the industry\u2019s focus on large language models (LLMs), which he believes aren\u2019t capable of actual \u201creasoning.\u201d<\/p>\n

\u201cLLMs struggle with generalization, due to being entirely reliant on memorization,\u201d he said<\/a> in a series<\/a> of posts on X in February. \u201cThey break down on anything that wasn\u2019t in the their training data.\u201d<\/p>\n

To Chollet\u2019s point, LLMs are statistical machines. Trained on a lot of examples, they learn patterns in those examples to make predictions, like that \u201cto whom\u201d in an email typically precedes \u201cit may concern.\u201d <\/p>\n

Chollet asserts that while LLMs might be capable of memorizing \u201creasoning patterns,\u201d it\u2019s unlikely that they can generate \u201cnew reasoning\u201d based on novel situations. \u201cIf you need to be trained on many examples of a pattern, even if it\u2019s implicit, in order to learn a reusable representation for it, you\u2019re memorizing,\u201d Chollet argued<\/a> in another post.<\/p>\n

To incentivize research beyond LLMs, in June, Chollet and Zapier co-founder Mike Knoop launched a $1 million competition<\/a> to build open source AI capable of beating ARC-AGI. Out of 17,789 submissions, the best scored 55.5% \u2014 ~20% higher than 2023\u2019s top scorer, albeit short of the 85%, \u201chuman-level\u201d threshold required to win.<\/p>\n

This doesn\u2019t mean we\u2019re ~20% closer to AGI, though, Knoop says.<\/p>\n

\n

Today we\u2019re announcing the winners of ARC Prize 2024. We\u2019re also publishing an extensive technical report on what we learned from the competition (link in the next tweet).<\/p>\n

The state-of-the-art went from 33% to 55.5%, the largest single-year increase we\u2019ve seen since 2020. The\u2026<\/p>\n

\u2014 Fran\u00e7ois Chollet (@fchollet) December 6, 2024<\/a><\/p>\n<\/blockquote>\n

In a blog post<\/a>, Knoop said that many of the submissions to ARC-AGI have been able to \u201cbrute force\u201d their way to a solution, suggesting that a \u201clarge fraction\u201d of ARC-AGI tasks \u201c[don\u2019t] carry much useful signal towards general intelligence.\u201d <\/p>\n

ARC-AGI consists of puzzle-like problems where an AI has to, given a grid of different-colored squares, generate the correct \u201canswer\u201d grid. The problems were designed to force an AI to adapt to new problems it hasn\u2019t seen before. But it\u2019s not clear they\u2019re achieving this. <\/p>\n

\"ARC-AGI
Tasks in the ARC-AGI benchmark. Models must solve \u2018problems\u2019 in the top row; the bottom row shows solutions. <\/span>Image Credits:<\/strong>ARC-AGI<\/span><\/figcaption><\/figure>\n

\u201c[ARC-AGI] has been unchanged since 2019 and is not perfect,\u201d Knoop acknowledged in his post.<\/p>\n

Francois and Knoop have also faced criticism<\/a> for overselling ARC-AGI as benchmark toward AGI \u2014 at a time when the very definition of AGI is being hotly contested. One OpenAI staff member recently claimed<\/a> that AGI has \u201calready\u201d been achieved if one defines AGI as AI \u201cbetter than most humans at most tasks.\u201d<\/p>\n

Knoop and Chollet say that they plan to release a second-gen ARC-AGI benchmark to address these issues, alongside a 2025 competition. \u201cWe will continue to direct the efforts of the research community towards what we see as the most important unsolved problems in AI, and accelerate the timeline to AGI,\u201d Chollet wrote in an X post<\/a>.<\/p>\n

Fixes likely won\u2019t come easy. If the first ARC-AGI test\u2019s shortcomings are any indication, defining intelligence for AI will be as intractable \u2014 and inflammatory<\/a> \u2014 as it has been for human beings.<\/p>\n<\/div>\n