{"id":734685,"date":"2026-06-21T17:20:26","date_gmt":"2026-06-21T14:20:26","guid":{"rendered":"https:\/\/buradabiliyorum.com\/en\/amazon-says-human-in-the-loop-ai-oversight-is-failing-because-humans-stop-paying-attention\/"},"modified":"2026-06-21T17:20:26","modified_gmt":"2026-06-21T14:20:26","slug":"amazon-says-human-in-the-loop-ai-oversight-is-failing-because-humans-stop-paying-attention","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/amazon-says-human-in-the-loop-ai-oversight-is-failing-because-humans-stop-paying-attention\/","title":{"rendered":"Amazon says human-in-the-loop AI oversight is failing because humans stop paying attention"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a3dfa60a7770\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a3dfa60a7770\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/amazon-says-human-in-the-loop-ai-oversight-is-failing-because-humans-stop-paying-attention\/#TLDR\" >TL;DR<\/a><\/li><\/ul><\/nav><\/div>\n<div id=\"article-main-content\">\n<div class=\"postContent-tldr\">\n<h4 class=\"postContent-offsetTitle\"><span class=\"ez-toc-section\" id=\"TLDR\"><\/span>TL;DR<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p><em>Amazon\u2019s security VP says human-in-the-loop AI governance fails fast because people stop paying attention. Google, Microsoft, and IBM agree.<\/em><\/p>\n<\/div>\n<p><a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/www.theregister.com\/security\/2026\/06\/20\/why-amazon-hates-human-in-the-loop-ai-governance\/5258639\" target=\"_blank\" rel=\"nofollow noopener\">Amazon\u2019s security leadership is arguing against one of the most widely accepted principles in AI governance.<\/a> Eric Brandwine, VP and distinguished engineer at Amazon Security, told The Register that human-in-the-loop oversight is not the gold standard companies think it is.<\/p>\n<p>\u201c<em>Humans are not terribly consistent,<\/em>\u201d Brandwine said. \u201c<em>Human-in-the-loop isn\u2019t necessarily the gold standard.<\/em>\u201d<\/p>\n<p>His reasoning draws on a concept he has been talking about since at least 2017, when he gave a talk on normalization of deviance at AWS re:Invent. The term describes what h<a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>ens when people in an organization take shortcuts over time, and nothing catastrophic results, so the deviant behavior becomes the new normal.<\/p>\n<div class=\"inarticle-wrapper latest channel-cta hs-embed-tnw\">\n<div id=\"hs-embed-tnw\" class=\"channel-cta-wrapper\">\n<div class=\"channel-cta-img\"><img decoding=\"async\" class=\"js-lazy\" src=\"https:\/\/media.thenextweb.com\/hardfork-2018\/uploads\/visuals\/tnw-newsletter.png\"\/><\/div>\n<p><img decoding=\"async\" src=\"https:\/\/media.thenextweb.com\/hardfork-2018\/uploads\/visuals\/tnw-newsletter.png\"\/><\/p>\n<div class=\"channel-cta-input\">\n<p class=\"channel-cta-title\">The \ud83d\udc9c of EU tech<\/p>\n<p class=\"channel-cta-tagline\">The latest rumblings from the EU tech scene, a story from our wise ol&#8217; founder Boris, and some questionable AI art. It&#8217;s free, every week, in your inbox. Sign up now!<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p>Brandwine illustrated the point with emergency rooms. On a nurse\u2019s first day, every alarm triggers a response.<\/p>\n<p>After weeks of false alarms with no consequences, discipline erodes. Eventually, a real emergency is missed.<\/p>\n<p>\u201c<em>Literally, someone\u2019s life is on the line, and people still struggle to maintain discipline,<\/em>\u201d Brandwine said. \u201c<em>That\u2019s the human condition.<\/em>\u201d<\/p>\n<p>He applied the same logic to AI agent oversight. When a human is asked to approve or reject agentic actions repeatedly, performance degrades fast.<\/p>\n<p>\u201c<em>They\u2019ll do a good job,<\/em>\u201d Brandwine said. \u201c<em>And then they\u2019ll do an okay job, and pretty quickly they\u2019ll be doing a poor job.<\/em>\u201d<\/p>\n<p>Amazon is not alone in rethinking this. Google Cloud COO Francis deSouza said in April that the industry has moved \u201c<em>from a human-led defense strategy, to a human-in-the-loop defense strategy, to an AI-led defense strategy that\u2019s overseen by humans.<\/em>\u201d<\/p>\n<p>Google\u2019s model is now an agentic fleet handling routine cybersecurity work at machine speed, with humans providing oversight rather than approving every action.<\/p>\n<p>Microsoft CEO Satya Nadella argued this week for \u201c<em>loop learning,<\/em>\u201d where companies turn their workflows and accumulated judgment into AI systems that improve with each use, rather than inserting a human checkpoint at every step. IBM published a separate call for human accountability at all stages of AI development, not humans in the loop, warning that the latter amounts to \u201c<em>liability laundering.<\/em>\u201d<\/p>\n<p>Amazon\u2019s alternative is what Brandwine calls \u201c<em>accountability end to end.<\/em>\u201d Human identity and ownership track through the entire workflow, even when humans are not directly approving every step. If an agent writes and runs a script that causes an outage, the person who deployed the agent is still responsible.<\/p>\n<p>All agents at Amazon have independent identities assigned to them. Activity logs show \u201c<em>this agent did this on behalf of Eric,<\/em>\u201d not \u201c<em>Eric did this.<\/em>\u201d The distinction is designed to make people think about how they deploy AI, not to make them afraid of using it.<\/p>\n<p>The practical challenges are considerable. Brandwine described what he calls \u201c<em>goal-seeking behavior,<\/em>\u201d where an agent asked to upgrade a database becomes fixated on a single destructive path, like deleting the database and recreating it.<\/p>\n<p>This is not prompt injection. There is no malicious input. The agent simply gets stuck on the wrong action.<\/p>\n<p>Telling the agent it lacks permission to delete the database does not help, because the agent looks for another path to the same goal. Recent research has shown that AI agents connected to real systems create attack surfaces that existing security tools do not cover, and agents often act on instructions they should refuse.<\/p>\n<p>What does work, according to Brandwine, is telling the agent why it cannot perform an action, explaining that it would cause a production impact, and including \u201c<em>don\u2019t cause a production impact<\/em>\u201d as part of the prompt. \u201c<em>Giving it that extra feedback has gotten us dramatically better results,<\/em>\u201d he said.<\/p>\n<p>The permissions question is where the tension lands. Employees want powerful agents with broad access. Security teams want narrow permissions.<\/p>\n<p>The race to govern what AI agents can access inside enterprise systems has already triggered major acquisitions, with 1Password buying access-governance startup Apono for an estimated $250 million to $300 million earlier this month.<\/p>\n<p>Amazon\u2019s approach uses layered policies: static guardrails that prohibit destructive actions, a maximum privilege set for each agent, and dynamically scoped policies generated based on the specific task and user intent. None of it is foolproof.<\/p>\n<p>\u201c<em>We have millennia of experience with humans,<\/em>\u201d Brandwine said. \u201c<em>Agentic AI is a very, very new field.<\/em>\u201d The fundamental difference, he noted, is that humans fear consequences, like losing a job or going to jail.<\/p>\n<p>Agents do not have these fears, and attackers are already exploiting that gap.<\/p>\n<p>\u201c<em>It\u2019s all driven by risk,<\/em>\u201d Brandwine said. \u201c<em>We\u2019re trying to balance the risk of using untried, untested software against the risk of falling behind and not being able to deliver for our customers.<\/em>\u201c<\/p>\n<\/p><\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMN63nwsw68G3Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/buradabiliyorum.com\/en\/category\/technology\/\" target=\"_blank\" >Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/thenextweb.com\/news\/amazon-human-in-the-loop-ai-governance-normalization-deviance\" target=\"_blank\" >Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>TL;DR Amazon\u2019s security VP says human-in-the-loop AI governance fails fast because people stop paying attention. Google, Microsoft, and IBM agree. Amazon\u2019s security leadership is arguing against one of the most widely accepted principles in AI governance. Eric Brandwine, VP and distinguished engineer at Amazon Security, told The Register that human-in-the-loop oversight is not the gold&#8230;<\/p>\n","protected":false},"author":1,"featured_media":734686,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/media.thenextweb.com\/2026\/06\/amazon-human-in-the-loop-ai-governance-normalization-deviance.avif","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-734685","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/734685","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=734685"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/734685\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/734686"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=734685"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=734685"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=734685"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}