{"id":662202,"date":"2025-04-11T13:20:20","date_gmt":"2025-04-11T10:20:20","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/ai-models-still-struggle-to-debug-software-microsoft-study-shows\/"},"modified":"2025-04-11T13:20:20","modified_gmt":"2025-04-11T10:20:20","slug":"ai-models-still-struggle-to-debug-software-microsoft-study-shows","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/ai-models-still-struggle-to-debug-software-microsoft-study-shows\/","title":{"rendered":"AI models still struggle to debug software, Microsoft study shows"},"content":{"rendered":"
\n

AI models from OpenAI, Anthropic, and other top AI labs are increasingly being used to assist with programming tasks. Google CEO Sundar Pichai said in October<\/a> that 25% of new code at the company is generated by AI, and Meta CEO Mark Zuckerberg has expressed ambitions<\/a> to widely deploy AI coding models within the social media<\/a> giant.<\/p>\n

Yet even some of the best models today struggle to resolve software bugs that wouldn\u2019t trip<\/a> up experienced devs.<\/p>\n

A new study<\/a> from Microsoft Research, Microsoft\u2019s R&D division, reveals that models, including Anthropic\u2019s Claude 3.7 Sonnet and OpenAI\u2019s o3-mini, fail to debug many issues in a software development benchmark called SWE-bench Lite. The results are a sobering reminder that, despite bold<\/a> pronouncements<\/a> from companies like OpenAI<\/a>, AI is still no match for human experts in domains such as coding.<\/p>\n

The study\u2019s co-authors tested nine different models as the backbone for a \u201csingle prompt-based agent\u201d that had access to a number of debugging tools, including a Python debugger. They tasked this agent with solving a curated set of 300 software debugging tasks from SWE-bench Lite.<\/p>\n

According to the co-authors, even when equipped with stronger and more recent models, their agent rarely completed more than half of the debugging tasks successfully. Claude 3.7 Sonnet had the highest average success rate (48.4%), followed by OpenAI\u2019s o1 (30.2%), and o3-mini (22.1%).<\/p>\n

\"Microsoft
A chart from the study. The \u201crelative increase\u201d refers to the boost models got from being equipped with debugging tooling.<\/span>Image Credits:<\/strong>Microsoft<\/span><\/figcaption><\/figure>\n

Why the underwhelming performance? Some models struggled to use the debugging tools available to them and understand how different tools might help with different issues. The bigger problem, though, was data scarcity, according to the co-authors. They speculate that there\u2019s not enough data representing \u201csequential decision-making processes\u201d \u2014 that is, human debugging traces \u2014 in current models\u2019 training data. <\/p>\n

\u201cWe strongly believe that training or fine-tuning [models] can make them better interactive debuggers,\u201d wrote the co-authors in their study. \u201cHowever, this will require specialized data to fulfill such model training, for example, trajectory data that records agents interacting with a debugger to collect necessary information before suggesting a bug fix.\u201d<\/p>\n

The findings aren\u2019t exactly shocking. Many studies have shown<\/a> that code-generating AI tends to introduce security vulnerabilities and errors,\u00a0owing to weaknesses in areas like the ability to understand programming logic.\u00a0One recent evaluation of Devin<\/a>, a popular AI coding tool, found that it could only complete three out of 20 programming tests.<\/p>\n

But the Microsoft work is one of the more detailed looks yet at a persistent problem area for models. It likely won\u2019t dampen investor enthusiasm for AI-powered assistive coding tools, but with any luck, it\u2019ll make developers \u2014 and their higher-ups \u2014 think twice about letting AI run the coding show.<\/p>\n

For what it\u2019s worth, a growing number of tech leaders have disputed the notion that AI will automate away coding jobs. Microsoft co-founder Bill Gates has said he thinks programming as a profession<\/a> is here to stay. So has Replit CEO Amjad Masad<\/a>, Okta CEO Todd McKinnon<\/a>, and IBM CEO Arvind Krishna.<\/p>\n<\/div>\n

If you liked the article, do not forget to share it with your friends. Follow us on\u00a0Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n

\n

If you want to read more like this article, you can visit our Technology<\/a><\/span> category.<\/strong><\/p>\n<\/blockquote>\n

Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

AI models from OpenAI, Anthropic, and other top AI labs are increasingly being used to assist with programming tasks. Google CEO Sundar Pichai said in October that 25% of new code at the company is generated by AI, and Meta CEO Mark Zuckerberg has expressed ambitions to widely deploy AI coding models within the social…<\/p>\n","protected":false},"author":1,"featured_media":662203,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/techcrunch.com\/wp-content\/uploads\/2024\/07\/microsoft-logo-office.jpg?resize=1200,798","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[77337,70937,155543,70286,61514],"class_list":["post-662202","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology","tag-ai","tag-artificial-intelligence","tag-debugging","tag-microsoft","tag-research"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/662202","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=662202"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/662202\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/662203"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=662202"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=662202"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=662202"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}