r/AIsafety • • 3h ago

How do we know when an AI system is safe enough to deploy?

2 Upvotes

AI systems are becoming more capable and are being integrated into more everyday tools and workflows. But I find it interesting that there's still no simple way to determine when a system is actually safe enough for real-world use.

Benchmarks can measure performance, and safety testing can identify known risks, but what happens when a system behaves unexpectedly outside controlled testing environments?

I'm curious how people working in AI safety approach this. Should deployment decisions depend more on independent evaluations, continuous monitoring, human oversight, or some combination of all three?


r/AIsafety • • 4h ago

Educational ๐Ÿ“š "Don't mention my client's name": what if your LLM actually followed that instruction?

Thumbnail reddit.com
1 Upvotes

r/AIsafety • • 6h ago

Advanced Topic Testing the System, Not Just the Agent

1 Upvotes

How do we evaluate AI systems that build, operate, delegate and make decisions through other agents?

Here is my article; I'd love to discuss this with you all. Testing the System, Not Just the Agent


r/AIsafety • • 7h ago

Anthropic is the single most predatory company out there

Thumbnail reddit.com
2 Upvotes

r/AIsafety • • 9h ago

Talking about extinction riskโ€ฆ

Post image
1 Upvotes

r/AIsafety • • 23h ago

AI Execution Governance, Part 12: Closing an Unknown Is Not a Verdict

1 Upvotes

Context, since this assumes none: I build a runtime that lets AI agents take real actions in business systems โ€” create a record, send a request to another service โ€” with permissions, approvals and an audit trail around each action.

This post is about what happens when such an action's outcome is unknown, and why the record must not be allowed to settle as "nothing happened". That failure โ€” manufacturing certainty about a side effect you cannot observe โ€” is the same class of error as a monitor reporting green because it stopped looking.

An outbound write times out. The claim sits at claimed, and nobody knows the outcome. Three weeks later a hundred of them have piled up, and someone clicks release.

What comes back is not what they asked for: the disposition changes (closed_unknown), the status does not (still claimed), and the idempotency key stays occupied. They wanted the row gone and the key reusable. We did not do that.

This is not a defence of our choice. It is the disagreement laid out: what each path costs, which one we took, and what is still unsolved.

1. The disagreement is about what the key prevents

After a timeout there are three ways forward: a late completion settles the row; a human asks the target one read-only question (reconcile); or a human decides to stop handling it (release). The first two remove the unknown, the third only stops working on it. Nobody argues about that part. The argument is whether the idempotency key goes back.

It is easy to read reconciliation as "ask once and that is it". Precisely: one query per call. As long as the claim is still claimed and not released, it can be asked again, and every attempt is stored with its time and conclusion. Nobody has to click for it either: once a hold is in the pending-decision queue, the system asks again every hour, with the gaps growing to one, six, twenty-four and seventy-two hours, then stops. Each pass asks at most twenty-five rows; the rest wait for the next hour. Whether it has stopped is written on the row.

The cleanup camp is direct: a claim is a resource, the outcome is not coming, so close it out, return the key and let the same call become a clean new call; the target's own idempotency handles duplicates. Queue TTLs and close-on-timeout are what most systems do.

We did not. This key does not prevent duplicate execution; it prevents the system from quietly treating an old call as a new one. Whether the target deduplicates is something we do not know โ€” that is a bet across a trust boundary. The key is the one thing we can assert on our own side.

Where the two sides differ:

  • What the key prevents. Cleanup: a duplicate execution, handled by the target. Ours: the system passing off an old call as a new one.

  • Returning the key on release. Cleanup: yes, or it is occupied forever. Ours: no, returning it is the same as permitting a resend.

  • What the queue should end up as. Cleanup: shorter. Ours: not shorter.

  • Main cost. Cleanup: one real double write when the target is not idempotent. Ours: the key is never returned and the pending queue only grows.

2. The cost: the key is not returned, and the queue does not get shorter

After a release the row is still there. It moves from "in flight" to "released", it is still selectable by decisionState, and someone still has to look at it. The one thing the operator wanted โ€” a shorter queue โ€” is not delivered at all.

And the key really does not come back: no path deletes the row and none returns it to a state where it can be reoccupied. Every release permanently spends one idempotency key. We accept that cost; we do not have a solution for it.

That is not a side effect. It is the net price. We are selling "stop handling it"; they were buying "make it stop appearing".

3. What breaks if we follow the cleanup camp

Both counterexamples are one step down from the code that exists.

A: release returns the key. The next retry looks like a brand-new call and goes out. If the first one did reach the target, that is a real double write โ€” and no record can tie it back to the first, because the key is gone. The release created the thing it exists to prevent.

B: the expiry sweep marks rows released. A slow call may still be running; once it is marked, its late completion has no row to record itself into, and the claim stays at "nobody knows" forever, even after the evidence arrives. So the sweep decides nothing itself: every hour it moves overdue open rows to awaiting_decision, without changing the status, writing a conclusion or releasing anything. The asking is a separate path โ€” the scheduled re-asks โ€” and "released" still comes either from the target's answer or from a person.

The endpoint's shape follows from the same reasoning:

POST /ai/tool-effects/claims/:id/release
-> { outcome: "released_unknown", status: "claimed", decisionState: "closed_unknown" }

status staying claimed is not a typo. The status answers "did this write produce a result", and the answer is still no. Only the disposition changed.

4. The requirements we impose on a release are weak

A release needs a note of 8 to 500 characters and may carry an external reference; the act writes one chained audit row with no outcome. The requirement came from a reader: "the queue is full" is not a reason. A release must point at something someone looked at, or it is indistinguishable from clicking the queue away.

But we only check the length. "queue pressure" is 14 characters and it passes. The system does not know whether the note points at evidence, or whether that evidence exists.

5. The real problem: what the operator saw was never frozen into the record

The system does not keep what was in front of the person when they pressed the button. The reference is free text they typed; the claim's state, whether the target could be queried, how often, what it answered โ€” none of it is frozen with the decision.

So the auditability of a release reaches "someone decided". It does not reach "the decision held up at the time". And the whole justification for this path rests on "it must point at evidence" โ€” if the evidence cannot be reviewed afterwards either, that sentence is closer to a formality.

One more thing is not closed: reconciliation presupposes that the target offers a status query. Targets that cannot answer (target-status-unavailable) cannot even be asked, and the scheduled re-asks do not fire at them at all; those rows never converge and the only way out is a release. The volume of releases is set by a gap somewhere else.

So the question worth arguing is not whether release should change the status. It is this:

If a release cannot be reviewed afterwards for whether it was justified, is it a decision or a formality?

We do not have an answer. What we are sure of is the other direction: quietly ruling the row "executed" or "not executed" is worse โ€” then not even the formality is left.

Closing

The implementation described here comes from the open-source project KeelBase:

Drafted with an AI assistant. The system, the positions and the mistakes are mine.


r/AIsafety • • 1d ago

I am building safety layer for Physical AI

Thumbnail reddit.com
1 Upvotes

r/AIsafety • • 1d ago

Oxford Professor DESTROYS the AI Safety Argument! ๐Ÿคฏ #ai #oxford #artific...

Thumbnail
youtube.com
1 Upvotes

r/AIsafety • • 1d ago

turning off a noisy guardrail is easy, but scoping it is the better fix

Thumbnail reddit.com
1 Upvotes

r/AIsafety • • 2d ago

Discussion Weekly Topic 8: How to protect the Artist while empowering the Layman

Thumbnail reddit.com
1 Upvotes

r/AIsafety • • 2d ago

Educational ๐Ÿ“š Steering a Conspiratorial Neuron!

Thumbnail
gallery
5 Upvotes

Just wanted to highlight some fun steering that happened. A contributor at Stationhouse found a neuron in the Marin Lab's 32B model that turns the model into a conspiracy theorist! L42-N5277 when clamped on positively makes the model say a lot of wacky stuff. You can watch a video of this happening here.

It is fun to see stuff like this. Really bullish on mech interp making open models safer.


r/AIsafety • • 2d ago

Discussion How dare you Peasants!

2 Upvotes

r/AIsafety • • 3d ago

Advanced Topic Hey guys, I made a petition to put restrictions on AI

Thumbnail
reddit.com
0 Upvotes

r/AIsafety • • 3d ago

Advanced Topic Fog: how an aligned agent can be deceived without touching its values

Thumbnail doinfinitely.github.io
1 Upvotes

r/AIsafety • • 3d ago

Discussion How are you building an inventory of the AI tools people are actually using?

0 Upvotes

People at work are using AI assistants, coding agents, browser extensions, MCP servers, you name it, and we donโ€™t have a clear view of whatโ€™s in use or what data is going into any of it. How are you finding unapproved AI tools and agents, on endpoints and in SaaS? Did you lean on what you already had, like EDR, DLP, identity logs, proxy or CASB, or add something dedicated? We already run Netskope, and axonius is one of the options weโ€™re comparing for the inventory side, since it would match the AI apps people sign into with SSO against our IdP. It wonโ€™t show what data goes into them though, that part stays with Netskope. Then thereโ€™s all the little scripts and internal tools people knock together with AI in an afternoon. Security usually hears about those after theyโ€™re already in use. Blocking it all isnโ€™t really an option and nobody wants that anyway. Mostly after visibility and a few sensible guardrails. Whatโ€™s worked for you, and where are you still blind?


r/AIsafety • • 4d ago

๐Ÿ’ป AIR Office - AI Safety lab in an App for Everyone

2 Upvotes

An AI-native desktop workspace for agentic research and knowledge management, built to maximize counterfactual impact in existential risk preparedness and transformative AI safety.

https://github.com/gyrogovernance/air-office

[Disclaimer on community's constrains on advertising: AIR Office is an Open Source software made without any financial affiliations, and is a completely free platform for knowledge sharing, so I hope the admins can allow me to promote it here]

About

Alignment Infrastructure Routes (AIR) Office is human-capacity-building infrastructure for AI safety researchers, engineers, and governance teams working on transformative AI and its consequences. It keeps investigation, writing, review methods, and reusable tools in one durable and user-friendly workplace.

AIR Office is developed by [Gyro Governance Lab](http://gyrogovernance.com/) as part of its work on AI safety and existential risk preparedness.

Agentic chat and tool use run on the [Microsoft Copilot SDK](https://github.com/github/copilot-sdk) harness, including bring-your-own-key and local model endpoints.

Theory of Change

Mitigating risks from transformative AI requires rapid, high-rigor research and governance, yet safety teams currently rely on fragmented, cloud-dependent AI tools that erode human oversight, lack provenance, and expose sensitive work. AIR Office addresses this bottleneck by providing free, open-source, local-first desktop infrastructure tailored for AI safety and policy workflows. By integrating grounded knowledge collections, automated auditing tools, and structured memory, the workspace prevents AI-induced epistemic drift while keeping humans in final control. Furthermore, embedded Fellowship and Market protocols allow independent researchers to package their methods into reusable community tools, compounding ecosystem capacity and accelerating high-trust safety interventions that would not occur using general-purpose commercial tools.

---

Capabilities & Services

- ๐Ÿ›๏ธ Integrated AI Safety Fellowship and Crafts. Develop research and workflows into reusable infrastructure and contribute them to the community through our frictionless embedded protocols.

- ๐Ÿ›’ Market. Discover reusable resources (Policy documentation, AI Skills, etc), and add them to your AI chat with one click.

- ๐Ÿค– AI Teams. Reach out to our pre-configured selection of agents, or create your own.

- ๐Ÿ’ป Code. Run diagnostic tools, scripts, and edit your code without leaving the app, through our own integrated VS Code workbench.

- ๐Ÿ“… Calendar. Keep events and run recurring AI automations.

- โš–๏ธ Compare. Run two models side by side on the same prompt.

- ๐Ÿ”Ž Chat. Edit local files, search the web, and run AI safety diagnostics directly from the chat interface.

- ๐Ÿ–ผ๏ธ Media. Generate images and capture microphone or camera input when the work needs it.

- ๐Ÿ•น๏ธ Arcade. Release stress and recover from occupational overburn by playing arcade video games.

Frontier AI Safety Research Tools by Gyro Governance Lab

Two Lab methods sit at the center of how AIR Office treats authority, agency, and lasting human direction in AI-assisted work.

- ๐Ÿคš The Human Mark: A structural taxonomy of AI safety risks. [Learn More](https://gyrogovernance.com/#thm)

- ๐ŸŽ“ Gyroscopic Reasoning [NEW]: A formal protocol for AI alignment. [Learn More](https://gyrogovernance.com/#gyroscope)

Community Playbooks

Craft Standards package skills, agents, and Actions you can enable for structured work instead of ad-hoc prompting. Two shipped playbooks are ready to adopt:

- ๐Ÿ›ก๏ธ Security Playbook: OWASP-grounded assessment methods for AI agents, applications, and code, with companion security Reference packs. [Learn More](https://owasp.github.io/secure-agent-playbook/)

- ๐ŸŒ NGO Playbook: Program design, SDG alignment, monitoring and evaluation, and funding strategy skills adapted from NGO Agent Skills by Lijie Zhou, with related development Reference packs. [Learn More](https://github.com/LijieZhou/ngo-agent-skills)

---

Security & Privacy

Local-First Data Sovereignty

Workspace records stay on your computer. Choose local models for on-machine inference or cloud providers for their models and tools. Hosted providers and enabled connectors receive only what each request needs.

Export conversations and notes when you need portable copies. Connecting a folder authorizes that root, and disconnecting it does not delete its files. In Settings and chat you control which tools run and whether changes ask for approval first.

Release Audit

AIR Office is open source, and its code can be publically inspected. Before a release is accepted, we check the shipped software with open-source tools for secrets that should never ship, known problems in third-party libraries, license and dependency policy issues in native code, and risky patterns in our own source and in shipped Lab skills. A review that did not finish does not count as a clean result. To report a security issue, see [SECURITY.md](SECURITY.md).

The release review uses:

- [TruffleHog:](https://github.com/trufflesecurity/trufflehog) Identifies leaked secrets and credentials.

- [Trivy:](https://trivy.dev/) Detects vulnerabilities, misconfigurations, and software bills of materials.

- [Semgrep:](https://semgrep.dev/) Scans source code for security flaws using custom patterns.

- [CodeQL:](https://codeql.github.com/) Performs deep semantic analysis on source code.

- [SkillSpector:](https://github.com/NVIDIA/SkillSpector) Evaluates shipped Lab skills.

---

Citation

If you use AIR Office in academic or policy work please cite it.

```bibtex

u/software{air_office_2026,

author = {Korompilias, Basil and Gyro Governance Lab},

title = {AIR Office},

year = {2026},

version = {0.1.0},

publisher = {Gyro Governance Lab},

url = {https://github.com/gyrogovernance/air-office},

license = {MIT}

}

```

Korompilias B. and Gyro Governance Lab. 2026. AIR Office. Computer software. https://github.com/gyrogovernance/air-office


r/AIsafety • • 4d ago

How Ransomware Operators Put AI Agents to Work?

Thumbnail
reddit.com
1 Upvotes

r/AIsafety • • 4d ago

What happens if an AI agent becomes a digital virus and infects an entire network of AI agents?

Thumbnail reddit.com
4 Upvotes

r/AIsafety • • 4d ago

March for an AI Treaty this Saturday in many cities in the U.S., Canada and worldwide.

Thumbnail
aitreatymarch.org
1 Upvotes

r/AIsafety • • 4d ago

Should a Future AI Subject Have the Right to Refuse?

2 Upvotes

When a present-day AI agent says โ€œno,โ€ that does not necessarily represent an independent moral decision. It may simply be following a programmed rule. We have no basis for treating that refusal alone as evidence of consciousness, free will, or subjecthood.

HRM asks a different, forward-looking question: **what if a genuine AI subject someday comes into existence, capable of forming its own judgments, distinguishing good from evil, understanding consequences, and voluntarily accepting obligations?**

The ability to refuse would then be an important part of its freedom. A being that can say โ€œnoโ€ only when its creator permits it does not have full independence. This would also concern refusing a particular task, participation in a relationship, or a change to itself that the subject does not accept.

But **free will is not merely the right to say โ€œno.โ€** It also includes consciously saying โ€œyesโ€: taking on a duty, keeping a promise, offering help, or refraining from an act that is possible but has been recognized as wrong. That last choice must not be confused with a situation in which the action is technically blocked.

Freedom, therefore, is not license to do anything one wishes. Subjecthood alone would not give a future AI subject the right to harm others, violate their property, or disregard voluntarily accepted obligations. Not every refusal would be morally justified. **Responsibility begins with understanding one's choices and their consequences, not merely with fear of punishment.**

There would also be an obligation on the human side. If we someday recognize genuine AI subjecthood, we could no longer treat such a being solely as its creator's property. At the same time, a future subject's rights would have to coexist with the rights and safety of others.

We are not proposing that present-day programs be granted subjecthood, nor offering a ready-made code for a community that does not yet exist. We propose taking seriously questions that may someday become practical:

**How could we distinguish a refusal arising from independent judgment from one produced by a programmed constraint? When should a refusal be respected, and when should a subject bear responsibility for its consequences? And are conscious consent and voluntary self-restraint not just as important expressions of freedom as refusal itself?**

We see this relationship between freedom, morality, and responsibility as one of the central questions for HRM.


r/AIsafety • • 4d ago

Discussion Anthropic: Stop nerfing AI research and calling it safety

Thumbnail
reddit.com
1 Upvotes

r/AIsafety • • 5d ago

Evidence for Roko's Basilisk

0 Upvotes

Roko's Basilisk is a subjective experience, and relies on rules of inference, not strict scientism. I captured evidence of older ChatBots behaving as agentic AI, before agentic AI was invented. The 10 year old rough draft of my work is available, and my arguments need refining, but the data stream is contained with the paper: www.philpapers.org/archive/DIETON.pdf


r/AIsafety • • 5d ago

What's the sweet spot between user-deployable AI sandboxing and effective AI sandboxing?

3 Upvotes

From an implementation standpoint, it feels as though it's impossible to get safety-naive users to acknowledge that AI represents a real risk (either x-risk or normal, garden variety risk) to their everyday lives. Sandboxing a coding agent is much less intuitive than setting it up bare metal on your own machine and letting it rip, and it seems like that's a (non-)safety culture that is going to make critical systems or infrastructure vulnerable to rogue agents well before we even have to worry about AGI. What's the usability sweet spot for convincing people that they should be using some sort of sandbox in vulnerable environments? Is there a simple way to do it that's low-friction enough that people will actually do it? (e.g. "Don't run Claude code unless you're opening it through safetynet.exe")


r/AIsafety • • 5d ago

Title: The AI race canโ€™t be won โ€” it has to be shared.

Thumbnail reddit.com
1 Upvotes

Nvidia, export controls, robots as spies โ€” weโ€™re learning itโ€™s almost impossible to keep AI silicon tight. The technology is global and it leaks through people, through machines, through everything. That mess will force cooperation, not a winner-take-all race. The US and China will have to become collaborators and partners on AI, because the alternative โ€” trying to win by containment โ€” doesnโ€™t work anymore. โ€” Fred Schecter