{"id":661305,"date":"2025-04-07T23:05:19","date_gmt":"2025-04-07T20:05:19","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/meta-exec-denies-the-company-artificially-boosted-llama-4s-benchmark-scores\/"},"modified":"2025-04-07T23:05:19","modified_gmt":"2025-04-07T20:05:19","slug":"meta-exec-denies-the-company-artificially-boosted-llama-4s-benchmark-scores","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/meta-exec-denies-the-company-artificially-boosted-llama-4s-benchmark-scores\/","title":{"rendered":"Meta exec denies the company artificially boosted Llama 4&#8217;s benchmark scores"},"content":{"rendered":"<div>\n<p id=\"speakable-summary\" class=\"wp-block-paragraph\">A Meta exec on Monday denied a rumor that the company trained its new AI models to present well on specific benchmarks while concealing the models\u2019 weaknesses.<\/p>\n<p class=\"wp-block-paragraph\">The executive, Ahmad Al-Dahle, VP of generative AI at Meta, <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/x.com\/Ahmad_Al_Dahle\/status\/1909302532306092107\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">said in a post on X<\/a> that it\u2019s \u201csimply not true\u201d that Meta trained its Llama 4 Maverick and Llama 4 Scout models on \u201ctest sets.\u201d In AI benchmarks, test sets are collections of data used to evaluate the performance of a model after it\u2019s been trained. Training on a test set could misleadingly inflate a model\u2019s benchmark scores, making the model <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>ear more capable than it actually is.<\/p>\n<p class=\"wp-block-paragraph\">Over the weekend, <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/substack.com\/@recodechinaai\/note\/c-106642739?r=5erp\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">an unsubstantiated rumor<\/a> that Meta artificially boosted its new models\u2019 benchmark results began circulating on X and Reddit. The rumor appears to have originated from a post on a Chinese <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/social-mediaa\/\" data-internallinksmanager029f6b8e52c=\"1\" title=\"Social Media\" target=\"_blank\" rel=\"noopener\">social media<\/a> site from a user claiming to have resigned from Meta in protest over the company\u2019s benchmarking practices.<\/p>\n<p class=\"wp-block-paragraph\">Reports that Maverick and Scout <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/x.com\/kimmonismus\/status\/1909245779136348590\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">perform<\/a> <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/x.com\/zimmskal\/status\/1908638551048138798\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">poorly<\/a> on <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/x.com\/ChaseBrowe32432\/status\/1908989296163299352\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">certain tasks<\/a> fueled the rumor, as did Meta\u2019s decision to use an experimental, unreleased version of Maverick to achieve better scores on the benchmark LM Arena. Researchers on X have\u00a0<a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/x.com\/TheXeophon\/status\/1908900306580074741\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">observed stark<\/a>\u00a0<a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/x.com\/TheXeophon\/status\/1908900306580074741\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">differences in the behavior<\/a>\u00a0of the publicly downloadable Maverick compared with the model hosted on LM Arena.\u00a0<\/p>\n<p class=\"wp-block-paragraph\">Al-Dahle acknowledged that some users are seeing \u201cmixed quality\u201d from Maverick and Scout across the different cloud providers hosting the models.<\/p>\n<p class=\"wp-block-paragraph\">\u201cSince we dropped the models as soon as they were ready, we expect it\u2019ll take several days for all the public implementations to get dialed in,\u201d Al-Dahle said. \u201cWe\u2019ll keep working through our bug fixes and onboarding partners.\u201d<\/p>\n<\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMN63nwsw68G3Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/category\/technology\/\" target=\"_blank\" >Technology<\/a><\/span> category.<\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/techcrunch.com\/2025\/04\/07\/meta-exec-denies-the-company-artificially-boosted-llama-4s-benchmark-scores\/\" target=\"_blank\" >Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>A Meta exec on Monday denied a rumor that the company trained its new AI models to present well on specific benchmarks while concealing the models\u2019 weaknesses. The executive, Ahmad Al-Dahle, VP of generative AI at Meta, said in a post on X that it\u2019s \u201csimply not true\u201d that Meta trained its Llama 4 Maverick&#8230;<\/p>\n","protected":false},"author":1,"featured_media":661307,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/techcrunch.com\/wp-content\/uploads\/2022\/02\/meta-brands-glitch.jpg?resize=1200,675","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[77337,122306,155369],"class_list":["post-661305","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology","tag-ai","tag-meta","tag-llama-4"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/661305","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=661305"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/661305\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/661307"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=661305"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=661305"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=661305"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}