{"id":664051,"date":"2025-04-21T02:31:44","date_gmt":"2025-04-20T23:31:44","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/openais-o3-ai-model-scores-lower-on-a-benchmark-than-the-company-initially-implied\/"},"modified":"2025-04-21T02:31:44","modified_gmt":"2025-04-20T23:31:44","slug":"openais-o3-ai-model-scores-lower-on-a-benchmark-than-the-company-initially-implied","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/openais-o3-ai-model-scores-lower-on-a-benchmark-than-the-company-initially-implied\/","title":{"rendered":"OpenAI&#8217;s o3 AI model scores lower on a benchmark than the company initially implied"},"content":{"rendered":"<div>\n<p id=\"speakable-summary\" class=\"wp-block-paragraph\">A discrepancy between first- and third-party benchmark results for OpenAI\u2019s o3 AI model is <a rel=\"nofollow\" target=\"_blank\" rel=\"nofollow\" href=\"https:\/\/www.reddit.com\/r\/singularity\/comments\/1k2lap5\/epoch_ai_has_released_o3_o4mini_gpt41_gpt41_mini\/\">raising questions about the company\u2019s transparency<\/a> and model testing practices.<\/p>\n<p class=\"wp-block-paragraph\">When OpenAI unveiled o3 in December, the company claimed the model could answer just over  a fourth of questions on FrontierMath, a challenging set of math problems. That score blew the competition away \u2014 the next-best model managed to answer only around 2% of FrontierMath problems correctly.<\/p>\n<p class=\"wp-block-paragraph\">\u201cToday, all offerings out there have less than 2% [on FrontierMath],\u201d Mark Chen, chief research officer at OpenAI, <a rel=\"nofollow\" target=\"_blank\" rel=\"nofollow\" href=\"https:\/\/www.youtube.com\/watch?v=SKBG1sqdyIU\">said during a livestream<\/a>. \u201cWe\u2019re seeing [internally], with o3 in aggressive test-time compute settings, we\u2019re able to get over 25%.\u201d<\/p>\n<p class=\"wp-block-paragraph\">As it turns out, that figure was likely an upper bound, achieved by a version of o3 with more computing behind it than the model OpenAI publicly launched last week.<\/p>\n<p class=\"wp-block-paragraph\">Epoch AI, the research institute behind FrontierMath, released results of its independent benchmark tests of o3 on Friday. Epoch found that o3 scored around 10%, well below OpenAI\u2019s highest claimed score.<\/p>\n<blockquote class=\"wp-block-quote twitter-tweet is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">OpenAI has released o3, their highly anticipated reasoning model, along with o4-mini, a smaller and cheaper model that succeeds o3-mini.<\/p>\n<p class=\"wp-block-paragraph\">We evaluated the new models on our suite of math and <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/sciencee\/\" data-internallinksmanager029f6b8e52c=\"5\" title=\"Science\" target=\"_blank\" rel=\"noopener\">science<\/a> benchmarks. Results in thread! <a rel=\"nofollow\" target=\"_blank\" rel=\"nofollow\" href=\"https:\/\/t.co\/5gbtzkEy1B\">pic.twitter.com\/5gbtzkEy1B<\/a><\/p>\n<p class=\"wp-block-paragraph\">\u2014 Epoch AI (@EpochAIResearch) <a rel=\"nofollow\" target=\"_blank\" rel=\"nofollow\" href=\"https:\/\/twitter.com\/EpochAIResearch\/status\/1913379475468833146?ref_src=twsrc%5Etfw\">April 18, 2025<\/a><\/p>\n<\/blockquote>\n<p class=\"wp-block-paragraph\">That doesn\u2019t mean OpenAI lied, per se. The benchmark results the company published in December show a lower-bound score that matches the score Epoch observed. Epoch also noted its testing setup likely differs from OpenAI\u2019s, and that it used an updated release of FrontierMath for its evaluations.<\/p>\n<p class=\"wp-block-paragraph\">\u201cThe difference between our results and OpenAI\u2019s might be due to OpenAI evaluating with a more powerful internal scaffold, using more test-time [computing], or because those results were run on a different subset of FrontierMath (the 180 problems in frontiermath-2024-11-26 vs the 290 problems in frontiermath-2025-02-28-private),\u201d <a rel=\"nofollow\" target=\"_blank\" rel=\"nofollow\" href=\"https:\/\/epoch.ai\/gradient-updates\/how-much-energy-does-chatgpt-use\">wrote<\/a> Epoch.<\/p>\n<p class=\"wp-block-paragraph\"><a rel=\"nofollow\" target=\"_blank\" rel=\"nofollow\" href=\"https:\/\/x.com\/arcprize\/status\/1912567067024453926\">According to a post on X<\/a> from the ARC Prize Foundation, an organization that tested a pre-release version of o3, the public o3 model \u201cis a different model [\u2026] tuned for chat\/product use,\u201d corroborating Epoch\u2019s report.<\/p>\n<p class=\"wp-block-paragraph\">\u201cAll released o3 compute tiers are smaller than the version we [benchmarked],\u201d wrote ARC Prize. <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/general\/\" data-internallinksmanager029f6b8e52c=\"3\" title=\"General\" target=\"_blank\" rel=\"noopener\">General<\/a>ly speaking, bigger compute tiers can be expected to achieve better benchmark scores.<\/p>\n<p class=\"wp-block-paragraph\">OpenAI\u2019s own Wenda Zhou, a member of the technical staff, <a rel=\"nofollow\" target=\"_blank\" rel=\"nofollow\" href=\"https:\/\/www.youtube.com\/watch?v=sq8GBPUb3rk\">said during a livestream last week<\/a> that the o3 in production is \u201cmore optimized for real-world use cases\u201d and speed versus the version of o3 demoed in December. As a result, it may exhibit benchmark \u201cdisparities,\u201d he added.<\/p>\n<p class=\"wp-block-paragraph\">\u201c[W]e\u2019ve done [optimizations] to make the [model] more cost efficient [and] more useful,\u201d Zhou said. \u201cWe still hope that \u2014 we still think that \u2014 this is a much better model.\u201d<\/p>\n<p class=\"wp-block-paragraph\">Granted, the fact that the public release of o3 falls short of OpenAI\u2019s testing promises is a bit of a moot point, since the company\u2019s o3-mini-high and o4-mini models outperform o3 on FrontierMath, and OpenAI plans to debut a more powerful o3 variant, o3-pro, in the coming weeks.<\/p>\n<p class=\"wp-block-paragraph\">It is, however, another reminder that AI benchmarks are best not taken at face value \u2014 particularly when the source is a company with services to sell.<\/p>\n<p class=\"wp-block-paragraph\">Benchmarking \u201ccontroversies\u201d are becoming a common occurrence in the AI industry as vendors race to capture headlines and mindshare with new models.<\/p>\n<p class=\"wp-block-paragraph\">In January, Epoch was criticized for waiting to disclose funding from OpenAI until after the company announced o3. Many academics who contributed to FrontierMath weren\u2019t informed of OpenAI\u2019s involvement until it was made public.<\/p>\n<p class=\"wp-block-paragraph\">More recently, Elon Musk\u2019s xAI was accused of publishing misleading benchmark charts for its latest AI model, Grok 3. Just this month, Meta admitted to touting benchmark scores for a version of a model that differed from the one the company made available to developers.<\/p>\n<p class=\"wp-block-paragraph\"><em>Updated 4:21 p.m. Pacific: Added comments from Wenda Zhou, a member of the OpenAI technical staff<\/em>.<\/p>\n<\/div>\n<p><script async src=\"\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMN63nwsw68G3Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/category\/technology\/\" target=\"_blank\" >Technology<\/a><\/span> category.<\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/techcrunch.com\/2025\/04\/20\/openais-o3-ai-model-scores-lower-on-a-benchmark-than-the-company-initially-implied\/\" target=\"_blank\" >Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>A discrepancy between first- and third-party benchmark results for OpenAI\u2019s o3 AI model is raising questions about the company\u2019s transparency and model testing practices. When OpenAI unveiled o3 in December, the company claimed the model could answer just over a fourth of questions on FrontierMath, a challenging set of math problems. That score blew the&#8230;<\/p>\n","protected":false},"author":1,"featured_media":664053,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/techcrunch.com\/wp-content\/uploads\/2025\/04\/GettyImages-2206295463.jpg?resize=1200,800","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[77337,153721,141199],"class_list":["post-664051","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology","tag-ai","tag-o3","tag-openai"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/664051","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=664051"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/664051\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/664053"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=664051"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=664051"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=664051"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}