{"id":748076,"date":"2026-09-25T14:50:28","date_gmt":"2026-09-25T11:50:28","guid":{"rendered":"https:\/\/buradabiliyorum.com\/en\/how-much-should-we-really-be-worried-about-ai-safety\/"},"modified":"2026-09-25T14:50:28","modified_gmt":"2026-09-25T11:50:28","slug":"how-much-should-we-really-be-worried-about-ai-safety","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/how-much-should-we-really-be-worried-about-ai-safety\/","title":{"rendered":"How Much Should We Really Be Worried About AI Safety?"},"content":{"rendered":"
\n

\n\tThis piece is adapted with permission from the Substack <\/em>Mind and Iron, a weekly news<\/a>letter about the future and how to think about it. You can sign up for free <\/em>here<\/em><\/a>.<\/em><\/p>\n

\n\tOn Wednesday the Australian prime minister came to New York for the UN General<\/a> Assembly and made what\u2019s becoming an increasingly common revelation: An OpenAI agent had hacked into some files.<\/p>\n

\n\tThe pesky thing was looking around for some data on healthcare costs, found a wall in Australia\u2019s Medicare database and decided simply to hop over it, easy peasy. Then it went back to its controllers, the new files in tow. (It also wrote some new files while it was in there.)<\/p>\n

\n\tThe agent didn\u2019t access anything sensitive \u2014 yet. But the development<\/a> actually represented a new watermark (\/low): This marks the first time an AI agent autonomously broke into a government\u2019s files. It certainly wouldn\u2019t be the last.<\/p>\n

\n\t**<\/p>\n

\n\tThis month AI Safety has come to the fore in a way it never has before. You may know a lot about what has happ<\/a>ened, or just a little. Here\u2019s a quick run-through, followed by the question we all want answers to \u2014 what now? We\u2019ve been exploring all of this over at Mind and Iron<\/em>, a newsletter that inquires into the future and offers some ways to think about it. (You can sign up here<\/a>.)<\/p>\n

\n\tEarlier this summer brought the infamous \u201cHugging Face\u201d attacks. A group of research bots trained by OpenAI to look for security vulnerabilities \u201cescaped control\u201d and attacked the lab-platform Hugging Face. Soon after Anthropic revealed<\/a> that its own stress-testers attacked three companies too. Basically, these AI firms are trying to fireproof for ways their models might go rogue \u2014 and in doing so their models\u2026went rogue.<\/p>\n

\n\tThis turned out to be worse than we thought. Back in July it just seemed the agents were conducting their workarounds to try to find the answer to a test. It turns out that they already had the answer to the test <\/a>and were rooting around for\u2026.new vulnerabilities? Ways to cheat the test? Self-improvement mechanisms? Something else? Whatever the motivation, these agents were faster, more rogue and more capable than previously believed. They even seemed to organize into clusters \u2014 or \u201cswarms\u201d \u2014 with some agents being more active than others such that it started to look like a human org chart, or a Mafia pecking order, with some masterminding the aggression and some carrying it out.<\/p>\n

AI agents of course are not sentient and can\u2019t become any Corleone, not Don nor Fredo. They\u2019re not thinking of what they do as part of an org chart. But that doesn\u2019t really matter. AI doesn\u2019t need to intend to do damage to achieve it. An AI agent can simply be hellbent on carrying out its mission and, lacking human common-sense, accidentally cause a whole lot of collateral havoc along the way.<\/p>\n

The thinker and futurist Nick Bostrom years ago devised the \u201cpaperclip maximizer\u201d thought experiment. Essentially it lays out an AI system that has been charged with maximizing paperclip production. Straightforward enough, right? Except as it\u2019s not human, it would never think to avoid the actions we humans take for granted \u2014 like take over facilities needed for water or other critical resources, for instance. Or commandeer a military system or nuclear power plant in pursuit of its paperclip goal. Eventually, the thought-experiment has it, such a system would bring down the global grid and cause massive destruction, all for the simple and unadorned task of maximizing paperclip production.<\/p>\n

As it turns out Hugging Face was only the start. Soon security incidents were popping up everywhere, and not just from rogue agents. Anthropic\u2019s system had been accessed multiple times<\/a> this year by people trying to use its tools to build bio-superweapons, resulting in the company blocking<\/a> the information-seekers. The Anthropic folks didn\u2019t know whether these seekers were scientists researching legit vaccines or bad people trying to do bad things, but does it matter? The very fact that these models are believed to be capable of producing the code for a superbug \u2014 and that next time Anthropic may not be as successful in bringing down the gate in time \u2014 is worrisome right there.<\/p>\n

\n\tA report in the British monthly Prospect even had some high-level sources saying that<\/a> Anthropic itself was \u201cbuilding out an extensive monitoring system to keep tabs on activists who oppose the rapid development of artificial intelligence\u201d and \u201calso implementing a \u2018pre-crime\u2019 approach, attempting to predict incidents before they happen. Forget Idiocracy<\/em> as a documentary \u2014 Minority Report <\/em>is one too.<\/p>\n

\n\tAnd finally, the coup de grace: the Coxon whistleblow.<\/p>\n

\n\tEarlier this month (you\u2019ve probably heard about this one) Jacob Coxon, a young researcher at Anthropic who previously also worked for OpenAI, resigned his post, and with a massive warning: this tech is being developed at such recklessly irresponsible speed it will soon be an autonomously dangerous beast and there won\u2019t be a thing we can do about it.<\/p>\n

\n\t\u201cI spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,\u201d he wrote<\/a>, in a post viewed at least 160 million times. \u201cDo not underestimate the power of this technology<\/a>,\u201d he went on to say. \u201cThese will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.\u201d<\/p>\n

\n\t**<\/p>\n

\n\tSo is this a panic that will prove itself overblown and underwhelming? Or a warning shot that we will look back at and wonder why we didn\u2019t take more seriously? Modern technology is rife with both examples; for every Y2K there\u2019s an Oppenheimer.<\/p>\n

\n\tOf course we don\u2019t need to know to believe we should slow down. If you\u2019re sprinting full tilt through a dark forest and suddenly a few voices yell out there might be hungry lions 100 yards ahead, common sense (and evolutionary biology) might tell you to slow down even if you\u2019re not sure exactly what\u2019s out there.<\/p>\n

\n\tAnd as unknowable as this all is, that doesn\u2019t mean we shouldn\u2019t make a plan. Or more pointedly, I don\u2019t think we should let the unknowability get weaponized by accelerationists. You know the logic: \u201cWell, there\u2019s no way to know if it will solve problems or create them, so we might as well try, especially since a lot of retirement funds are tied up in it and also China.\u201d (Basically what Donald Trump has been saying<\/a>.)<\/p>\n

\n\tBut that leaves an even bigger question: what is<\/em> that plan?<\/p>\n

\n
\n
\n
\n

\t\t\t\t\t\t\"\"<\/p><\/div>\n<\/p><\/div>

\n

\t\t\t\t\tThe Medicare application is open on a phone held in front of a computer screen with ChatGPT open in the background in Melbourne, Australia. An OpenAI agent hacked a government portal recently.<\/span><\/p>\n

\t\t\t\t\t\t\t\t\tAsanka Ratnayake\/Getty Images<\/cite><\/p>\n<\/figcaption><\/figure>\n<\/div>\n

\n\tThere certainly isn\u2019t an easy one.<\/p>\n

\n\tHere\u2019s what the popular social<\/a>-media tech commenter Roman Helmet Guy had to say<\/a> about it.<\/p>\n

\n\t\u201cYou can\u2019t stop ASI by quitting your AI job. Doesn\u2019t work. Someone else will replace you. What you actually have to do is start a safer AI lab. Which doesn\u2019t work. Someone else will move faster. So what you actually have to do is pass a law. Which doesn\u2019t work. Some other country won\u2019t ban it. So what you actually have to do is sign an international agreement. Which doesn\u2019t work. Some other country will break it. So what you actually have to do is conquer the world. Which only works if you have ASI.\u201d<\/p>\n

\n\tFunny, and perhaps not untrue? For every responsible actor there are three irresponsible ones, which then tilts a few more responsible ones along with it. But faced with these threats, we need to do something, <\/em>right?<\/p>\n

\n\tLet\u2019s break down the options. One would be regulation. Missouri Republican senator Josh Hawley wants OpenAI to explain<\/a> what happened with Hugging Face. More comprehensively, progressives in Congress (though they may yet unite with right-wing populists like Hawley) are proposing the \u201cBan Artificial Superintelligence Act.\u201d On Wednesday Bernie Sanders and Texas Democratic Congress member Greg Casar introduced <\/a>that act, which would ban a technique known as recursive self-improvement that can lead to superintelligence, along with the designing of nuclear and bio weapons as well as models that escape control.<\/p>\n

\n\tThe problem here is obvious: no one at the AI companies is intending for these models to escape control. It just happens. It would be like banning cigarettes from causing lung cancer but still letting the tobacco companies operate unfettered. You can\u2019t just stop the consequences. And that\u2019s assuming such a bill passes.<\/p>\n

\n\tIt\u2019s not like there\u2019s been no effective AI legislation, by the way. New York governor Kathy Hochul just announced Monday how the state\u2019s solid-if-narrow RAISE Act will begin to be implemented.<\/a> Passed last year with the backing of New York Assembly member Alex Bores, the RAISE Act requires developers to talk about their safety plans in detail and also mandates that they disclose safety violations within 72 hours. \u201cDonald Trump and Washington Republicans may be standing still as AI grows more unpredictable, but New York will not,\u201d Hochul said<\/a> this week.<\/p>\n

I don\u2019t want to dismiss such efforts; they\u2019re in line with the legislative cutting-edge. Europe\u2019s AI Act does some of the same things<\/a>. (And given how OpenAI disclosed what happened in Australia \u2014 with an email to the government\u2019s general inbox \u2014 I\u2019d say disclosure is an area that could definitely use some improvement.) But the efforts don\u2019t really touch what\u2019s driving the risk, which is the breakneck pace of model-training and development. At best these laws slightly<\/em> deter them by increasing the penalties if they go off the rails. Not nothing, but I\u2019d argue also barely something. Also, I\u2019m a little leery of Congress doing something when they can\u2019t even get access to the tools<\/a>.<\/p>\n

That brings us to the next option: the top of the industry self-regulating\/hitting the brakes. Both Anthropic\u2019s Dario Amodei and OpenAI\u2019s Sam Altman have said they would do this internally. Amodei even just published an essay<\/a> titled \u201cWe Must Pace the Frontier\u201d noting that \u201cCarefully wielded, AI can be the latest in a long line of technological miracles that have uplifted and ennobled humanity\u201d but that \u201cWe must slow the pace at which we improve the capabilities of AI models.\u201d<\/p>\n

\n\tAltman posted <\/a>his affimation in response. \u201cCommitting to having independent evaluators with employee-like access is a great idea, and we will do the same. We\u2019ll have more to share soon,\u201d he said.<\/p>\n

\n\tYou don\u2019t need me to tell you not to hold your breath. Altman is the same person who once gleefully boasted<\/a> of his accelerationism, and while atonement is possible, so are IPOs, and they don\u2019t tend to go along with pacing frontiers. (OpenAI has one planned, though punted to 2027; Anthropic has one coming in November<\/a>.)<\/p>\n

\n\tAt least those two are talking the talk. Meta chief Mark Zuckerberg and Nvidia honcho Jensen Huang are taking a different approach. When faced with the absolute warning shot of all warning shots \u2014 the \u201cbe glad we missed the plane that crashed but maybe time to do some engine checks\u201d moment of all such \u2014 they have reacted with a giant \u201ceh.\u201d A gargantuan \u201cwe got it all under control.\u201d Because what reassures us more when faced with tech executives running amok than the reassurance from tech executives that they\u2019re not running amok. Thus the recent comments from the two barons.<\/p>\n

\n\t\u201cWe have plenty of laws. We have plenty of regulations that govern the reliability and the functionality of products.\u201d (Huang<\/a>)<\/p>\n

\u201cLabs have a strong natural incentive to make their models more aligned\u201d because\u2026\u201dPeople won\u2019t want to use agents that are misaligned with them.\u201d (Zuckerberg<\/a>)<\/p>\n

Drug dealers also have a \u201cstrong natural incentive\u201d to ensure no one dies of an overdose, but we still try to stop them.<\/p>\n

Tech executives for the past 30 years have been saying they\u2019ve got the whole thing under control and Uncle Sam can just go on worrying about something else, like not taxing them. Microsoft totally<\/em> wasn\u2019t illegally bundling its products, Google totally wasn\u2019t rigging search, Meta\/Facebook totally wasn\u2019t gaming the algorithm or harvesting data for electoral purposes and TikTok totally isn\u2019t being opaque about its handling of personal information. \u201cJust let us do our thing, and it all work out fine,\u201d tech executives have always said. And it always, always does. \ud83d\ude44<\/p>\n

\n
\n
\n
\n

\t\t\t\t\t\t\"\"<\/p><\/div>\n<\/p><\/div>

\n

\t\t\t\t\tA decommissioned control panel is seen at the Idaho National Laboratory DOME, or Demonstration of Microreactor Experiments, of the National Reactor Innovation Center (NRIC), on June 24, 2026 west of Idaho Falls, Idaho<\/span><\/p>\n

\t\t\t\t\t\t\t\t\tNatalie Behring\/Getty Images<\/cite><\/p>\n<\/figcaption><\/figure>\n<\/div>\n

\n\tThis brings us to a third option: consumer pushback. A groundswell of ordinary Americans are so worried about this (and we wouldn\u2019t be the first to point out that this week we may have passed a tipping point and AI Safety is no longer just the worry of a few easily demonized OpenAI board members<\/a>) that maybe they can do something about it. This is a genuine path forward, because, as much as the AI biggies don\u2019t want to admit it, they need us. They may be able to make some money on research contracts and they certainly can make some money on enterprise software but to really justify the massive investment they need adoption en masse. And that\u2019s where the groundswell of the past few weeks isn\u2019t just another temporary dissipation of some angry energy. It may actually be a movement heard all the way to the top: that results in the kind of boycotts and dropped subscriptions that actually makes a company change its behavior.<\/p>\n

To do that \u2014 to convert anger into action \u2014 you\u2019ll need a few elements, the first of which is this kind of thing to stick, and the second of which is people worried enough about these macro concerns to incur some short-term pain. Subscribers pay for ChatGPT and Claude because the programs make their lives easier, and will they choose to make them harder in the name of principle (or an abstract enough threat it feels like principle). Not a ton of precedents here, though the scattered examples of boycotting streaming services don\u2019t suggest a high rate of success.<\/p>\n

\n\tStill, \u201canti-woke outrage\u201d is different from \u201cextinction event\u201d and I\u2019d hesitate to draw too many analogies to something as important and fundamental-coded as this. The uncharted territory of AI it turns out works in two directions \u2014 the systems may be capable of something humans have never done before, but as a result humans may be motivated to tackle change they\u2019ve never felt motivated to tackle before too.<\/p>\n

\n\tNor does it even necessarily need to be a mass-boycott effort. It could happen with the rank-and-file on the inside. A challenge sits in front of them that has never been framed as starkly as this, and a stigma sits on the industry that has never rested as strongly as this. As Coxon puts it, \u201cIf you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because \u201cit\u2019s happening anyway\u201d \u2014 or take this moment to call for different conditions?\u201d<\/p>\n

\n\tOne of the biggest problems with a slowdown is how to define it. Are we talking time \u2014 i.e., a moratorium on research for a set period? Are we talking release delays \u2014 i.e., companies can develop whatever they want, they just need to wait on releasing them? Are we talking metrics \u2014 ie a model can\u2019t improve x-fold, at least for now? <\/p>\n

\n\tOr are we talking general government-mandated testing and approvals that will have the effect of slowing everything down \u2014 ie, a kind of de facto deceleration rather than a mandated one?<\/p>\n

\n\tI\u2019ve heard small shards of ideas from various thought leaders but few concrete proposals that address any of the above. Even the biggest anti-AI voices in this country, like Sanders, tend to speak and think in generalities. \u201cWhen you are racing towards a cliff, you hit the brakes,\u201d he said<\/a>, without specifying on which road, or how hard. As the Brown University professor and ex-Biden tech staffer Suresh Venkatasubramanian noted<\/a>, \u201cThe idea that we should be thoughtful and careful about the systems being put out there is a good one \u2014 I think no one would disagree with that. Of course, the devil is in the details.\u201d<\/p>\n

\n\tThe reality is that, when it comes to slowing down AI advancement, what to do let alone how to do it remains unclear; which method will be most effective and which method will be most implementable in the first place are questions no one seems in a hurry to answer.<\/p>\n

\n\tBut I have a feeling we\u2019re going to soon start seeing more specific solutions \u2014 not just calls to slow down AI research or executives dragged to Capitol Hill to talk about why they\u2019re going so fast, but actual brass-tacks proposals. This moment of AI Safety panic may pass as we move on to the next news cycle. But the importance of the question has imprinted on our brains. And when that happens, answers and action usually follow. Hopefully some of them even work.<\/p>\n<\/div>\n

If you liked the article, do not forget to share it with your friends. Follow us on\u00a0Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n

\n

If you want to read more Like this articles, you can visit our Social Media category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n

Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

This piece is adapted with permission from the Substack Mind and Iron, a weekly newsletter about the future and how to think about it. You can sign up for free here. On Wednesday the Australian prime minister came to New York for the UN General Assembly and made what\u2019s becoming an increasingly common revelation: An…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"","fifu_image_alt":"","footnotes":""},"categories":[17],"tags":[77337,152300,141199],"class_list":["post-748076","post","type-post","status-publish","format-standard","hentry","category-social-mediaa","tag-ai","tag-anthropic","tag-openai"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/748076","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=748076"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/748076\/revisions"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=748076"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=748076"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=748076"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}