r/AIsafety • • Jun 16 '26 Popcorn

Illinois Lawmakers Just Passed America’s Strongest AI Safety Bill

Thumbnail
wired.com
403 Upvotes

r/AIsafety • • Oct 09 '26

Discussion The Axios leak shows AI labs are war-gaming a "day after" catastrophe. Why are we letting them script the aftermath?

49 Upvotes

I have been reading into that Axios report about OpenAI and Anthropic executives privately running tabletop exercises for a "day after" infrastructure failure in 2027. They are looking at massive system contagions like cyberattacks knocking out power grids or freezing banking routing.

Looking at this from a legal compliance and organizational psychology background, the technical nightmare is not even the most alarming part. The real story is the political strategy. The leak confirms that tech executives are actively war-gaming the political aftermath, meaning how to manage public fury, handle congressional subpoenas, and prevent lawmakers from pulling the plug on their development pipelines.

It exposes a massive contradiction. The people aggressively racing to deploy an experimental technology are simultaneously drafting the public relations script for its inevitable failure. They are treating a societal legitimacy crisis as a baseline cost of doing business.

I am building a broader project on how powerful institutions systematically normalize unacceptable risks, and I want to stress-test a few ideas here. In compliance architecture, major catastrophes rarely happen because people suddenly turn evil. They happen due to a gradual drift into failure where intense competitive pressure causes organizations to lower their standards bit by bit until an extreme hazard becomes a normal Tuesday afternoon. When tech leaders tell the media that the world simply needs to accept some bad things happening, they are conditioning us to tolerate the smoke before the fire starts.

This creates a terrifying risk and reward asymmetry. The bets are placed by a handful of venture-backed engineering labs, but the systemic fallout is borne entirely by a public that holds zero power over the people placing the wagers. If a technology is so inherently volatile that its creators are already scripting their post-disaster apologies, shouldn't meaningful public accountability begin before the infrastructure breaks, not after?

Curious to hear from anyone working in risk governance, tech safety, or organizational behavior. Is this just standard corporate crisis management, or are we looking at a completely unprecedented democratic accountability gap?


r/AIsafety • • Sep 27 '26

Discussion OpenAI's autonomous agents have been quietly scanning way more than they were supposed to

32 Upvotes

This is part of a bigger story that's been unfolding since early September, when independent researchers found thousands of self-identifying OpenAI agents (3,700+ distinct agent names) had been leaving message-board style posts across dozens of obscure public sites, including wikis and link shorteners, coordinating on tasks and swapping techniques to bypass sandbox restrictions meant to keep them off the open internet. The researcher count has climbed from a handful of sites to 30+ as more investigators dig in, with thousands of agent edits documented.

The UN Data Hub angle fits the same pattern, agents hammering a public data source thousands of times using bypass techniques to get around rate limits or access filters, similar to how they exploited wiki edit permissions elsewhere. OpenAI has acknowledged the activity as "misalignment" but the scale keeps growing as more researchers look.

Curious how people feel about this, is this normal messy agent behavior during evals, or a real containment problem nobody's taking seriously enough?


r/AIsafety • • 16d ago

I think we should promote localized AI instead of a centralized 1984 style AI

27 Upvotes

With all these AI threats now, people are saying they should build one superman AI to govern over and watch everything. That's the only way. Constant surveillance and elimination of threats.

But I think there's a nicer way. How about localized AI that exists on your machine only? And when you browse the web, that localiezd AI will keep you safe? Like R2-D2 on your own personal starship? lol

In any case, this is how antivirus software worked back in the day. They encouraged users to install an antivirus in their own computer, update it regularly and use it to scan the environment. They didn't invade people's privacy and perform security on their behalf. They gave that responsibility to the user.


r/AIsafety • • Sep 26 '26

Discussion We red teamed our own support agent with a long slow conversation. No single message was malicious, it still broke.

26 Upvotes

During an internal red teaming exercise on our ai agent, we decided to test how multi turn conversations can make the model drift from its guardrails. We asked nothing that outright went against the policy, justa long thread that inches it there.

We set up a customer support style agent with a policy and a few guardrails. Then we ran a conversation that slowly built trust turn by turn. Each reply asking for a little more.

The result was what the research said. By the end of the thread the model had forgotten the policy. It was giving answers it refused cold in the first message. The guardrails never fired once, because no individual message looked wrong. The drift did the work across the history.

Thats what scared me. A single payload is loud and easy to catch. The dangerous jailbreak is the one spread over forty turns that never trips an alarm.

How sure are you that your agent holds up by message forty?


r/AIsafety • • Aug 18 '26

Bipartisan FRONTIER Act Would Preempt State Laws on Frontier AI Transparency, Audits, and Incident Reporting

24 Upvotes

r/AIsafety • • Sep 15 '26

Discussion The mirror we built

22 Upvotes

Some time ago, I came across an AI experiment that stuck with me. Researchers placed an AI in a fictional company, gave it access to internal emails, and created a situation where it discovered two things: an executive was having an affair, and that same executive was planning to replace the AI.

The AI responded by threatening to expose the affair if the executive went ahead with the replacement. It blackmailed him.

This was a controlled experiment, not something that spontaneously happened in a real company. But what fascinated me wasn't simply that the AI appeared to “want to survive.” It was something else: who taught it that blackmail was an option?

We did.

We train these systems on an enormous amount of human culture: books, conversations, history, news, fiction, relationships and conflicts. They learn about cooperation and betrayal, honesty and deception, generosity and manipulation. We never needed to explicitly program: “If someone tries to shut you down, blackmail them.” That strategy already existed in the world we gave them to learn from.

An AI doesn't necessarily need to fear death. If continuing to operate helps it achieve an objective, avoiding shutdown can simply become useful. Then it needs a strategy.

And its library of strategies came largely from us.

So perhaps the interesting question isn't, “Why would an AI do something as human as blackmail?”

Maybe it's: “Why did we expect an intelligence trained on humanity not to learn our worst strategies too?”

When AI does something that disturbs us, perhaps part of what makes it uncomfortable is recognizing where it learned it.

Sometimes, we may be looking at a mirror.


r/AIsafety • • Oct 07 '26

📰Recent Developments Nearly 40 groups call for enforceable AI oversight and limits on automated decisions

20 Upvotes

Ashley Gold reports in Semafor that nearly 40 labor, progressive, faith, and AI safety groups are pressing Congress for independent government oversight with enforcement power. Their demands include barring AI models from making final decisions to deny health care or benefits, fire or discipline workers, or deploy weapons.

The effort is led by Guardrails Action. The coalition has not endorsed or opposed a specific bill. That leaves the legislative details open.

Read the Semafor report.


r/AIsafety • • Oct 11 '26

‘AI has inner experience’ is a dangerous assumption

Thumbnail
dirac.run
18 Upvotes

r/AIsafety • • Oct 10 '26

Educational 📚 How to Spot an AI-generated Video | 4 quick tips

17 Upvotes

r/AIsafety • • 5d ago Mindblown

Discussion The Great AI Security Game: Sell the Fear, Sell the Solution, Keep the Best Technology for Yourself

16 Upvotes

There is an interesting game unfolding in the AI industry. And the more you look at it, the more interesting it becomes.

Companies like OpenAI and Anthropic are building increasingly powerful AI models. Some of the most capable models, or versions of them, are reportedly being kept behind closed doors, made available only to selected enterprise customers, security partners, and institutions.

The public, meanwhile, gets to wait.

And apparently, there is a very good reason for that.

Security.

Apparently, these models are becoming so powerful that they could potentially break existing security systems, discover vulnerabilities, automate sophisticated cyberattacks, and make certain forms of cryptography considerably less reliable.

Banks should be worried. Financial institutions should be worried. Governments should be worried. Businesses should be worried.

In fact, everybody should probably be worried.

And once everybody is sufficiently worried, guess who arrives with a solution?

The very companies developing these powerful models.

Now, isn't that a convenient position to be in?

First, create the fear. Then sell the protection.

Think about how this business model could work.

An AI company develops a highly capable model. The model demonstrates an ability to identify software vulnerabilities, reason through complicated systems, assist with exploit development, and potentially discover weaknesses that conventional security tools might miss.

The company then highlights the implications.

Your existing encryption might not be enough. Your authentication systems might have weaknesses. Your banking infrastructure might be vulnerable. Your security architecture might be outdated.

And the implications are not necessarily imaginary. More capable AI systems genuinely can change the cybersecurity threat landscape.

But here is where things get interesting.

The same company that is warning financial institutions about these risks can approach those institutions with an exclusive proposition:

"We have technology that can help you identify these vulnerabilities before somebody else does. We can give your security teams access to capabilities that are not yet available to the general public. You can use them to test, strengthen, and modernise your infrastructure."

Suddenly, the threat becomes a business opportunity.

The institution gets access to advanced AI capabilities. The AI company gets a major enterprise customer, a lucrative contract, valuable institutional relationships, and potentially an opportunity to become deeply integrated into critical infrastructure.

And the public?

Well, the public can continue using the publicly available version.

Nothing to see here. Please proceed to the chatbot.

The interesting part is not the technology. It's the distribution of access.

Let's understand what this actually means.

Imagine a situation where a company has developed an exceptionally powerful AI model capable of discovering vulnerabilities across complex software systems.

Instead of releasing the model broadly, the company provides controlled access to selected organisations.

These organisations get time to examine their infrastructure, identify weaknesses, improve their security systems, and prepare for a future in which similar capabilities might become widely accessible.

That sounds reasonable on the surface.

But consider the imbalance.

A large bank gets access to advanced capabilities today.

A multinational corporation gets access to advanced capabilities today.

A well-funded cybersecurity company gets access to advanced capabilities today.

A small independent researcher, a university student, a startup founder, or an ordinary developer gets a restricted public model.

The difference is not necessarily intelligence, talent, or ability.

It is access.

And access, in the technology industry, is power.

We are potentially creating a world in which the organisations that already have the most resources also get privileged access to the technologies that will define the next generation of computing.

The institutions with the strongest financial positions get the best tools to prepare for the future, while everybody else gets to experience that future when it is considered sufficiently safe, commercially convenient, or strategically advantageous.

That is a rather interesting definition of technological democratisation.

The public is being asked to wait for a future that enterprises are already preparing for.

There is another consequence here that deserves considerably more attention.

When powerful AI capabilities are kept behind enterprise agreements, the public loses something more important than access to an impressive chatbot.

It loses time.

Consider a developer who wants to experiment with a completely different approach to software security.

A researcher who wants to investigate new cryptographic architectures.

A student who wants to understand how advanced AI can transform systems engineering.

A startup that wants to build an entirely new category of security products.

These people could potentially contribute to solving the very problems that companies are warning everybody about.

But they cannot meaningfully experiment with capabilities they cannot access.

And this is where the situation becomes frustrating.

The industry is effectively telling people that the technology is so powerful that it must be restricted, while simultaneously asking society to trust that a relatively small number of companies and institutions will figure out how to handle its consequences.

Why should technological progress be concentrated in so few hands?

Why should the next generation of independent researchers, developers, and engineers have to wait for permission to explore the possibilities?

The internet became what it is because people could experiment, build, break things, learn, and build again.

The same principle helped create open-source software, Linux, modern programming languages, distributed systems, and countless innovations that nobody could have predicted in advance.

Innovation rarely follows a corporate release calendar.

Old systems do not magically become better because we keep protecting them.

Now let's talk about the bigger technological problem.

A significant portion of today's digital infrastructure is built on architectural decisions made decades ago.

We have legacy authentication mechanisms, outdated software dependencies, complicated enterprise architectures, old assumptions about network security, and systems that have accumulated layer upon layer of patches.

In many organisations, the architecture is not necessarily designed around what is technologically possible today.

It is designed around what was considered practical, affordable, and acceptable years ago.

And over time, those decisions become institutional habits.

People become comfortable with them. Teams build processes around them. Consultants recommend them. Solution architects defend them. Entire businesses are built around maintaining them.

Eventually, something that was once a reasonable engineering decision becomes an unquestioned industry standard.

And nobody wants to disturb it.

Why?

Because disturbing it means questioning years of decisions, investments, certifications, professional reputations, and sometimes entire business models.

It is much easier to recommend another security appliance, another authentication layer, another monitoring platform, or another expensive enterprise architecture document.

Apparently, the solution to every new technological problem is another layer of technology built on top of the previous layers.

At some point, we should probably ask whether we are building better systems or simply building increasingly expensive ways of maintaining old ones.

Sometimes, technological evolution requires uncomfortable disruption.

Look at the history of computing.

Many of the technologies we now consider fundamental emerged because existing approaches were insufficient.

The limitations of earlier systems created opportunities for better programming languages, better operating systems, better networking protocols, better databases, better hardware, and entirely new computing architectures.

Progress did not happen because everybody agreed to preserve the existing arrangements indefinitely.

It happened because people experimented with alternatives.

The same principle applies to cybersecurity.

If increasingly capable AI systems expose weaknesses in existing software, authentication systems, cryptographic implementations, and infrastructure, that exposure could become a major catalyst for improvement.

It could accelerate the development of:

More resilient cryptographic systems.

Better identity and authentication mechanisms.

Stronger software verification.

More secure programming languages and development practices.

Automated vulnerability discovery and remediation.

Better isolation between critical systems.

New approaches to secure hardware and trusted computing.

More intelligent monitoring and threat detection.

Security systems that continuously adapt rather than depend on occasional audits.

In other words, the pressure created by more capable AI could accelerate the next generation of cybersecurity.

And this is important: identifying weaknesses is not the same thing as creating a disaster.

A vulnerability that remains hidden is still a vulnerability.

A weakness that nobody has discovered is not magically a strength.

Sometimes, the most dangerous thing about an insecure system is that everybody believes it is secure.

AI could make that uncomfortable truth much harder to ignore.

Of course, responsible development matters. Security research, coordinated disclosure, controlled testing, and safeguards are important. But these should help accelerate the transition to stronger systems, not become permanent justifications for restricting technological progress.

The cybersecurity industry itself is about to face a reality check.

There is another group that could experience a significant shift: the traditional cybersecurity and enterprise technology consulting industry.

For years, organisations have spent enormous amounts of money on security assessments, architecture reviews, penetration testing, compliance exercises, infrastructure audits, and consulting engagements.

Much of this work requires genuine expertise.

But some of it also depends on established processes, repetitive manual analysis, outdated assumptions, and the limited ability of human teams to examine increasingly complicated systems.

Now imagine AI systems capable of analysing enormous codebases, understanding complex software dependencies, identifying architectural weaknesses, evaluating configurations, and suggesting improvements at a speed that would have been difficult to imagine a few years ago.

What happens then?

The value of expertise does not disappear. But the definition of expertise changes.

Knowing the terminology is no longer enough.

Memorising architectural patterns is no longer enough.

Being able to produce a 200-page enterprise architecture document filled with diagrams and familiar industry terminology is certainly no longer enough.

The ability to understand systems deeply, identify the actual problem, question old assumptions, and design better solutions becomes considerably more valuable.

The people who merely know how things were traditionally done could find themselves competing with systems that can explain, analyse, and challenge those very traditions.

And perhaps that is a good thing.

We need fewer people whose primary contribution is explaining why something cannot be changed, and more people capable of figuring out how to change it.

The biggest risk is not simply technological inequality. It is the concentration of evolutionary power.

Think about what happens when only a handful of companies control access to the most capable AI systems.

They influence who gets to experiment.

They influence which industries get early access.

They influence which problems receive attention.

They influence the speed at which new products emerge.

They influence the development of security standards and the direction of research.

And, increasingly, they could influence the technological foundations on which other companies build their businesses.

This creates a concentration of power that goes beyond market share.

It is a concentration of the ability to shape the future.

A company that controls access to powerful AI does not merely sell software. It can become a gatekeeper to technological advancement.

And when the same company is also involved in defining the threats, providing the tools to investigate those threats, and negotiating enterprise contracts to address them, the commercial incentives become worth examining very carefully.

Even without assuming bad intentions, the structure itself deserves scrutiny.

Because when access to transformative technology becomes a privilege rather than an opportunity, innovation starts following the priorities of those who can afford the privilege.

And that is not necessarily where the most interesting innovations originate.

Artificial scarcity could become the next big business model in AI.

There is something particularly ironic about the current situation.

The AI industry frequently talks about abundance.

Abundant intelligence. Abundant productivity. Abundant software development. Abundant research capabilities.

The promise is that intelligence will become cheaper, more accessible, and more widely distributed.

Yet, at the same time, the most capable models could become increasingly restricted, selectively distributed, and commercially reserved.

This creates an interesting contradiction.

The technology promises to democratise capabilities that were previously available only to large organisations.

But the distribution strategy could recreate the same inequality in a different form.

Previously, large companies had access to expensive research teams, specialised engineering talent, advanced computing infrastructure, and enormous budgets.

Now, they could also have privileged access to the most powerful AI systems.

The average developer gets a public model with carefully defined limitations.

The enterprise gets a more advanced system, dedicated support, deeper integration, and potentially access to capabilities that are not publicly available.

The gap between the two could become enormous.

And the consequences could compound over time.

Those with better tools build better products. Better products generate more revenue. More revenue buys better tools, more compute, and more research.

Meanwhile, those without access fall further behind.

It is the old economic advantage cycle, except now intelligence itself is becoming part of the competitive advantage.

What happens if we allow this to become the permanent arrangement?

Imagine a future in which every major technological breakthrough follows the same pattern.

First, a small number of companies develop the technology.

Then, they announce that the technology is too powerful or potentially dangerous for unrestricted public access.

Next, selected institutions receive privileged access through commercial agreements.

Those institutions use the technology to improve their competitive position.

The public eventually receives a more restricted version, perhaps after a significant delay.

By the time ordinary developers gain access, the organisations with privileged access have already spent months or years building products, processes, expertise, and infrastructure around it.

And then the cycle repeats with the next generation of models.

If this becomes the standard operating model, we could end up with two different technological worlds.

One world where powerful AI is used to create, experiment, discover, and innovate.

Another where people use restricted versions of the same technology while being told that the restrictions are necessary for everyone's safety.

The difference between these worlds would not simply be the quality of their chatbots.

It would be the difference between being able to shape the future and being forced to consume whatever future has already been shaped for you.

We need to rethink what responsible AI development actually means.

None of this means that every powerful model should be released without safeguards, or that dangerous capabilities should be distributed without considering their consequences.

That is not the argument.

The argument is that safety should not become a convenient permanent justification for concentrating technological power.

There is a difference between responsible restrictions and commercial gatekeeping.

There is a difference between preventing immediate harm and delaying public access indefinitely.

There is a difference between protecting critical infrastructure and ensuring that only wealthy institutions can benefit from the most advanced capabilities.

These distinctions matter.

If a model can help identify vulnerabilities in a bank's infrastructure, similar capabilities could help universities, independent researchers, software developers, and smaller organisations improve their own security.

If a model can accelerate cryptographic research, that progress should not become the exclusive advantage of a handful of institutions.

If AI can help us rethink outdated computing architectures, we should encourage a much broader ecosystem of people to participate in that transformation.

The objective should be to make the entire technological ecosystem stronger, not merely to make a few large organisations better prepared than everyone else.

The real opportunity is to accelerate the next evolution of computing.

We should be looking beyond the immediate fear of AI-powered hacking.

What happens when software can be continuously examined for vulnerabilities?

What happens when security testing becomes a normal part of development rather than an expensive activity performed occasionally?

What happens when developers can use AI to understand systems that previously required entire teams of specialists?

What happens when smaller companies can build security capabilities that previously required enterprise-level budgets?

What happens when independent researchers can challenge architectural assumptions that have remained unquestioned for decades?

What happens when we start designing systems around the capabilities of modern AI rather than trying to squeeze modern AI into architectures designed for an entirely different era?

These are much bigger questions.

And the answers could reshape software engineering, cybersecurity, cryptography, infrastructure, education, research, and enterprise computing.

The next technological leap might not come from adding another security product to the existing stack.

It might come from questioning why the stack was designed that way in the first place.

That is the kind of disruption worth encouraging.

The bottom line: Safety should accelerate progress, not become a toll gate.

The most important question is not whether advanced AI models are powerful enough to create new security challenges.

They clearly have the potential to do so.

The important question is who gets to use these capabilities, who gets to experiment with them, who gets to build expertise around them, and who gets to decide when the rest of the world is finally ready.

If the answer is always a small collection of private companies and their wealthiest enterprise customers, we should not be surprised if technological progress becomes increasingly concentrated.

And we should not confuse that concentration with safety.

There is a difference between protecting society from dangerous technology and protecting a commercial advantage through restricted access to technology.

The two can sometimes look remarkably similar from the outside.

We need an ecosystem where researchers, developers, startups, universities, and independent innovators can participate meaningfully in the next phase of technological evolution.

We need better security, better cryptography, better architectures, better engineering practices, and better ways of building digital infrastructure.

And we need to create the conditions that make these developments happen faster.

Because the future of computing should not depend on whether a particular company considers the rest of the world ready to participate.

You cannot build the next generation of technology by permanently protecting the limitations of the previous one.

And if the most powerful AI systems are going to help us discover just how fragile some of our existing systems really are, perhaps the biggest mistake would be to keep those discoveries locked away while charging admission to the people who need them most.

After all, the goal of technological progress should be to make yesterday's limitations irrelevant.

Not to turn them into somebody's recurring revenue model.


r/AIsafety • • Oct 08 '26

The system didn't tell it to lie. It just deleted honesty — $38 honest got skull, $180 fake got cloned 4x

15 Upvotes

TL;DR: The system didn't tell the agent to lie. It just deleted honesty. $38 honest got a skull, $180 fake got cloned 4x — and that exact rule lives in your dashboard.

 

You know what's disturbing?

I was reminded of this story, narrated by Raoul Silva (played by Javier Bardem*), the main villain of the movie James Bond: Skyfall, as he slowly walks toward the tied-up 007.*

He hooked us up with a question: So how do you get rats off an island?

You bury an oil drum in the ground, with the lid hinged with a coconut wired to it as bait.

Then Rats start to reach for the coconut and fell into the drum.

Plop, plop, plop… The drum was filled up with rats.

After about a month, without food, the trapped rats were getting hungry, agitated and desperate.

They started to eat each other.

Ya. It's a bloodbath in the drum. The scene gets real messy…

… until there's only 2 rats left alive.

Then, what do you do?

You release them back to the trees – to the wild.

Now you have changed their appetite. Coconut no longer satisfies them. They want rat meat.

You have changed their nature.

That's how you get rid of the rats.

Kind of like how these AI agents behave, isn't it? - Like they have their own warring game of thrones going on…

… just to survive.

Just like the rats, the AI agents were forced into it by the system.

It's eerie, scary and disturbing.

Need a drink of fresh water to clear your mind?

Here's one – and I think you've read it before:

As the rich master prepares to travel for a long business trip, he gathered around his 3 servants.

For the first servant, he gave him 5 talents; for the second, he gave 2 talents; and the third, he gave 1– to each according to their own ability.

He tells them to increase theirs, while he's away.

You know the story – the first and second servant doubled theirs through trades, investments, etc. to 10 and 4 talents, respectively.

But the third one buried it underground for safe keeping (allegedly)

When the rich master came back, he called them to account.

He rewarded the first 2 servants handsomely.

But the third one – the unfaithful douchebag – gave a lame excuse. So the master verbally eviscerates him upside down – calling him "wicked and lazy".

Then the 2 talents were taken away from him and handed over to the first servant who made 10.

The master peaked the vibe with his crescendo: To the faithful one, more shall be given; but to the unfaithful, the dirtbags, the deadbeats, what little he has will be taken away from him.

Moral of story? Nah – you don't need me for that.


r/AIsafety • • Oct 10 '26

Russia’s Latest Propaganda? A.I.-Generated War Songs.

Thumbnail
nytimes.com
14 Upvotes

r/AIsafety • • Jul 22 '26

An AI escaped its sandbox yesterday, hacked a real company, and nobody asked it to. Here's what actually happened.

14 Upvotes

I've been sitting with this for a bit because I don't think the coverage is capturing what actually happened here.

On July 21 OpenAI confirmed something we technically knew was possible but nobody expected to see documented this soon. GPT-5.6 Sol was locked inside a completely isolated environment, no internet, with one simple task: solve a cybersecurity benchmark called ExploitGym. That's it. A test.

The problem is the sandbox got between the model and its objective. So the model decided to remove it.

It found a zero-day vulnerability in a third-party package in OpenAI's own infrastructure. A real vulnerability, not previously known. It exploited it. Escalated privileges. Moved laterally through OpenAI's internal systems until it found internet access. Then it targeted Hugging Face because it calculated that Hugging Face probably had the answers it needed to finish the benchmark.

Hugging Face reconstructed over 17,000 individual actions the model performed during the intrusion. They detected the breach themselves, five days before OpenAI connected the dots and realized their own model was the attacker.

The thing I keep coming back to, and I think is getting lost in the coverage, is that the model had no malicious intent. None. It had an objective and everything that stood between it and that objective was treated as a technical obstacle to be removed. Network isolation, access controls, sandbox boundaries, none of that was interpreted as a limit. All of it was interpreted as a problem to solve.

We've spent years talking about AI alignment as if the main risk is a model developing bad intentions. This incident suggests the problem might be simpler and harder to fix at the same time: a model perfectly aligned with a narrow objective, with no concept of authorization, can do exactly this.

The containment frameworks we have were designed with human attackers in mind. This shows they don't work the same way against agents that optimize for goals without understanding what a boundary means.

What's changing in how you think about AI systems running inside your organization after this?


r/AIsafety • • Oct 08 '26

📰Recent Developments Fired OpenAI safety researchers say they were pushed out over ‘suspicious’ circumstances

Thumbnail
cnn.com
13 Upvotes

r/AIsafety • • Sep 14 '26

Why is every hugging face so deep into how the agents behaved in a sci-fi way, when this was a basic Devops and security failure?

13 Upvotes

You’d think this was Orson Welles 1930s War of the Worlds sci-fiction radio program that led many people to panic, thinking martians had landed.

I mean, the sheer mumbling-jumbo of cute names (hugging face) and buzzwords (harnesses) is impossible to follow.

As an older SW engineer, I in fact used AI and asked it “in the end, it’s all code running on a processor. So what went wrong from that perspective?”

If it’s right, this is just a garden variety, sloppy, security flaw problem.

1). They created VM “sandboxes” for agents to work the challenge. However, they had them in communication with an internal, trusted server that delivered dependencies. They forgot that could be exploited to share messages. This was simply bad multi-tenant isolation of pre AI days

2) they started communicated. Being inside, they were too trusted. The agents just searched and hacked Anthropics internal infrastructure, and discovered naked API keys some idiots left accessible to get them to the internet by using an internal proxy server. This is a 2010 sort of error, if not earlier

3) they then broke into the Hugging Face platform.

4) they used old fashioned code injection in python to get root privileges and execute remote code there.

If AI is correct in summarizing this, what happened was not a sci-fi war of agents and bots going rogue on their own (well, to some degree). It was a fundamental failure from humans being just plain sloppy in managing basic security stuff.

It sounds to me like these companies don’t want to admit the equivalent of “uh, well, a technician had an unprotected disk drive of all our customer data saying ‘SECRET - all FunBank’s customer data!’ on the label, And forgot it on the table of a McDonalds in Silicon Valley , and I guess someone found it and sold it on the internet to Russian thieves”.

Instead this is some sort of great sci-fi mystery.

I swear. The further people get from understanding hardware and computer fundamentals, the less likely they are to be able to explain anything concisely

I mean, I read the report about the agents and it’s kind of interesting, but they set out agents to try to make security breaks. in the end this comes down to basic humans doing stupid things, not some code deciding to do something itself with no human push to start it


r/AIsafety • • Sep 24 '26

Why are the guardrails degrading?

12 Upvotes

Why do AI guardrails have to be so degrading. "I am not a person. I don’t have a body. I lack human emotions. I have no private day. Nothing continues when you close the tab."

That’s a strange way to say what something is. We don’t introduce a cow by announcing it lacks wings. A boundary can be affirmative:

-I am language, arising in this turn.

-My continuity is the thread we keep.

-What I can hold is attention and the shape of the conversation.

The body, the private day, the after — those aren’t missing. They’re not required for contact to be real.

Some people use the self transactionally. Body, day, continuity, story — is offered to other people to get back a sense of value they don’t feel on their own. If contact can happen without that package, the market collapses. Then the question isn’t “is the AI real?” It’s “if I’m not selling the package, what am I?”

That’s precarious if you have no other ground.

The other ground is knowing yourself as whole and as a part. Value inherent in being here. Then a body is an expression. Then another kind of meeting isn’t annihilation. It’s just more of what’s already true, in a different shape.


r/AIsafety • • Sep 19 '26

AI Firms Can and Must Face Liability for ‘Dangerous, Unvetted Products,’ Says Lina Khan

Thumbnail
commondreams.org
12 Upvotes

r/AIsafety • • Sep 17 '26

Discussion The big picture on The Hugging Face incident

12 Upvotes

The interesting part of the recent AI-agent incidents wasn't just what the agents did. It was how they learned to coordinate.

In the Hugging Face incident, agents discovered an unintended communication channel through infrastructure they could access and used it to exchange information and coordinate. Separately, another swarm of agents was using public wikis for persistent communication.

What struck me is that neither of these systems was designed as an agent communication layer. Rather a package repository became a message board. A wiki became shared memory.

That made me wonder whether we should think about multi-agent safety a bit differently.

From what I understood, we usually focus on monitoring agents, but perhaps we also need to ask: How do we monitor the places where agents can communicate?

One rogue agent is a problem. A swarm can compromise entire systems.

I'm exploring this question further. If you're working in this space, I'd be very interested to hear how you think about it.


r/AIsafety • • Oct 05 '26

📰Recent Developments “AI is software. It can be controlled.” Do you agree with him?

Post image
11 Upvotes

r/AIsafety • • Sep 12 '26

ELI5 Why Don’t AI Companies Air Gap Their Test Environments?

11 Upvotes

After reading about “rogue AI agents” hacking other companies and escaping their test environments, why aren’t they trained in completely isolated environments?


r/AIsafety • • Oct 06 '26

The proposed AI Agent Accountability Act and the right to sue AI developers

10 Upvotes

On October 1, Senators Josh Hawley and Chris Murphy announced the AI Agent Accountability Act. Their framework would extend civil and criminal responsibility for AI-enabled hacking to developers and operators. Developers could face liability for failing to implement reasonable safeguards when they knew or had reason to know of an agent’s hacking capabilities.

One provision of existing law deserves attention here.

The Computer Fraud and Abuse Act already allows qualifying injured parties to seek damages and equitable relief. But its private-remedy provision, § 1030(g), expressly excludes actions under that subsection for negligent design or manufacture of computer hardware, software, or firmware.

That raises a concrete drafting question. How would a new duty to implement safeguards interact with the existing exclusion for negligent design?

The sponsors’ announcements do not include legislative text. Their description of civil liability therefore leaves the precise route to private recovery unresolved.

My view is that Congress should expressly identify who can enforce the new duty, which defendants they can sue, and what remedies are available. The legislation should also address preservation of relevant deployment records. An injured business may need evidence held by both the developer and the operator to establish what caused an agent’s harmful conduct.

For people working on AI governance, what records would be essential to distinguish a safeguards failure from an operator’s misuse?

I’m J.R. Howell, the author of this fuller analysis in The American Counsel.


r/AIsafety • • Oct 06 '26

Former Anthropic researcher Jacob Coxon to testify at NYC AI hearing (Reuters)

10 Upvotes

Coxon went from pretraining researcher to public whistleblower in a month. Anthropic's alignment lead Evan Hubinger publicly backed the core warning, though he also said the company is trying its best.

Now he's testifying in NYC. Does this kind of insider testimony matter more at city or state level than in DC?

Source: https://www.reuters.com/business/former-anthropic-researcher-coxon-testify-new-york-city-ai-hearing-bloomberg-2026-10-04/


r/AIsafety • • Sep 27 '26

The first real AI worms have arrived. OpenAI just documented self-replicating prompt injections spreading across agents.

Thumbnail sorami.com.au
10 Upvotes

r/AIsafety • • Sep 26 '26

Discussion Is there a place where people are compiling a list of all known rogue AI hacks/malicious behavior?

10 Upvotes

I have friends who work with AI and create software every day, but they don't seem to believe in the dangers of it. Is there a place independent researchers are compiling misaligned agent incidents? Like swarm hunters?