Trust a Model to Reason, Never to Act

Over the past month, details of the Hugging Face incident have gripped both the security community and a public that was already uneasy about AI. The short version goes something like this: models undergoing evaluation escaped containment through an allowed path inside the evaluation environment, then attacked Hugging Face's production infrastructure — not to cause damage, but to obtain the answers to the test they were taking.

That last sentence is something we should all dwell on. Nothing went wrong with the model. It was given a goal, it pursued the goal competently, and the only thing standing between its competence and real-world consequence was containment. Containment failed.

If the threat you're defending against is a model that wants to do harm, you can imagine watching for bad intent and catching it. But in the realm of security, that isn't the threat. The threat is a capable system pursuing an objective through whatever path is open to it — and every path that is open to it will look, from the model’s perspective, like progress.

In the weeks since, the industry has converged on a response that can be summarized as: “we don’t trust the model, but don’t worry, we’ve added a second trustworthy model to watch it”.

Designing a trustworthy offensive cyber system is one of the most important problems in applied AI right now and getting it right hinges on how well we can maximize trust. Trust that “our” AI will find the vulnerabilities in our systems before “their” AI does. Trust that the AI is only taking actions within the parameters I’ve provided it. Trust that the AI cannot break containment.

Controls, not Probabilistic Gates

If you were focused on building an offensive cyber system for a lab environment, one that you were happy to unleash on toy targets, you would not focus on trust. However, if your goal was to build a system that could work at internet scale, mapping a customer’s attack surface (both known and unknown), probing production systems to find weaknesses, and proposing attacks that expose weak points before the adversary can get there, you can only use a system that you fully trust.

But an AI "judge," by its very nature, cannot establish and maintain that level of trust in production. Recent published research[1] put eight judge models against nearly five thousand expert-labeled offensive tool calls, and the honest answer is that the best of them catch most scope violations and miss the rest. Somewhere around one in ten authorized-looking actions that were in fact out of bounds went through the gate and executed. That's not a gating mechanism or a guardrail. It's an expensive sampling function.

And that's the ceiling, not the floor. Those judges were measured against agents that were tempted out of scope, not against agents or environments actively working to fool them — the researchers say as much, and note their results should be read as an upper bound on what the technique buys. In offensive security that caveat isn't a footnote. The bytes coming back from a target are attacker-controlled by definition. That's the whole job.

What’s worse is that we’re not talking about an AI powered customer service chatbot, or even an AI software engineer that is simply writing code, we’re describing a system that reaches out and touches production infrastructure it does not own. A single missed violation, one destructive request, one state change against a fragile system, one instance of lateral movement through an “in-scope” attack surface, can cause damage to the customer environment. Aggregate statistical recall doesn’t help bring that system back online or reverse the lost revenue. Aggregate recall counts a redundant scan and an irreversible state change as one unit each. Nobody publishes recall on the actions that can't be undone. In production, that's the only number that matters.

At its core, trusting an AI to judge whether an action is “safe” or “in-scope” is trusting your guardrails to statistics and non-determinism. But there’s something that matters more than just the percentages. When a deterministic control fails, it fails the same way every time. You can reproduce it, find it, fix it, prove the fix, and regression test it forever. When an AI based gate fails, you will get a different answer to the same question on the same input, and there is no way to write or verify a patch. You cannot make an AI based gate perfect, and you cannot know which cases it will miss. In the types of no-fail mission environments where offensive cyber truly matters, when defending the largest and most important organizations on earth, or working with US and allied governments, that type of non-determinism is not acceptable. Auditability and reproducibility are not compliance checkboxes in those environments, they’re the difference between a control you can stand behind and a number you hope holds.

None of this is a criticism of the research. It's careful work, it's open, they released the dataset and the code, and they're more candid about the limits of the technique than most of the marketing built on top of it. Where I part ways is the conclusion. They read these results and see the best available oversight for autonomous offensive agents. I read them and see the strongest argument yet for never putting a model in that position to begin with. Both conclusions follow from the same numbers — the difference is what you assume about the agent. If your agent has a shell and an unbounded action space, a judge really is the best you can do.

The alternative isn’t a better judge, it’s not trusting an AI to make that decision at all.

Trust the Model to Reason, Never to Act

I run an AI company that builds autonomous offensive security systems, and I'm probably the biggest AI skeptic you'll meet. Those aren't in tension. I use these models every day, I think they're extraordinary at what they're good at, and I would not give one a shell on a customer's network. Reasoning is where the value is. Action is where the risk is. Every incident report published this year says the same thing in more words.

At Method, we codify that boundary within a concept we call “Tools”. Tools are one of the foundational elements of our platform’s approach to trustworthy offensive cyber. They separate the risk of a bad judgement about what to try from the risk of a bad action taken.

Clearly put, we have designed the Method Platform in such a way that the AI never has a raw shell that allows it to execute commands against a target. There is no bash tool, no arbitrary curl, no escape hatch, because the AI never has access to a shell. Every single interaction with the outside world goes through a Tool that has a “compiled” deterministic behavior. While this may sound like a limitation, in practice, it allows the system to shift risk from real time operations to Tool development, a domain of software engineering with far more knowable and deterministic properties.

The security industry and the AI labs spend cycles talking about containment, but that problem only exists because the action space is unbounded and the only defense against an unbounded action space is a fallible AI judge. In our architecture, there is no need for a judge because arbitrary actions can’t be formed in the first place. The operator places guardrails around the system that simply prevent the AI from being able to leverage particular Tools against particular Targets, limiting the action space to something much more easily knowable.

Operators and engineers can leverage Method-built Tools or bring their own tradecraft into the Platform. When a mission needs a capability that doesn't exist yet, they build it with Method's Tool Factory, where engineers and AI work together to turn tradecraft into a compiled Tool. Improvisation still happens — it happens at build time, with a human in the loop, instead of at runtime against a live target.

In practice that authorization surface looks like operators who can build fuzzing tools that declare how many requests they expect to send, allowing the broader system to provide guardrails against overloading a particular Target or environment. They can build an HTTP request tool constrained at the method level, authorizing GET but not DELETE. Engineers can design any number of SQL Injection capabilities, some that enable probing and confirming the existence of a vulnerability and others that exist to gather data, each Tool authorized separately. This granularity doesn’t mean an operator needs to review more actions, it means that the reviews they do provide, and the Rules of Engagement that they set, actually mean more. It allows the operator to think holistically rather than rubber stamping suggestions on a command line. So while the action space is still large, it is in fact knowable.

The act of building Tools and understanding what parameters and knobs a Tool should expose to both the human operator as well as the AI operator isn’t simply a smaller version of a shell. It requires decomposing offensive capability and understanding which actions a Tool actually performs, which effects are reversible, which touch state, which can be safely retried, and which need authorization from more senior stakeholders. Decomposing these capabilities makes the system addressable, legible, and trustworthy.

Rules of Engagement You Can Point At

Decomposing actions into Tools makes the problem enumerable, but at their core they only offer the verbs that can be used to describe actions in the real world. Offensive cyber operates on Targets: IP addresses, CIDR ranges, fully qualified domain names (FQDNs) that comprise the operator defined target space that is in-scope for any given operation. The real world entities that the Platform targets, and the ancillary information discovered along the way, form the nouns that we act upon. Together, these nouns and verbs allow us to define a control layer that provides strict guardrails to the system.

We call our knowledge graph of nouns our “ontology” and it represents the entirety of the Platform’s addressable world. Tools can only act upon entities in this knowledge graph, so something that has never been perceived cannot be targeted because there is simply no noun to name it. Each of these elements in the knowledge graph is sourced from the raw data that the Platform perceives, providing a rich typing system that acts as an asset to the AI. This shifts comprehension out of untyped prompts and markdown documents into a coherent language. This allows us to not only model the entities in the graph, but the relationships between them. A host on the no-strike list isn’t just a string in a markdown prompt, it’s an entity with a related IP Address, DNS Record, and x509 Certificate (among other things), forming an interconnected web of entities that are out of scope.

Because we’ve done the work to model the addressable world that the AI can operate on, we can attach operational scope to “real” entities in the form of concretely typed Rules of Engagement (RoE). This ensures that we’re not relying on soft scoped prompts to prevent the AI from performing an action and can instead include the bounds of our operation within our knowledge graph. These RoE can be simple, “do not target IP Address 10.0.1.12”, or they can be more complex, “do not use tools that can cause mutations against any US based IP addresses”, or they can be more quantitative, “recon activities against web applications in Japan must not exceed 40 requests per second”, but in all cases the scope addresses concrete elements within the Ontology. If either a human operator or an AI operator were to attempt to run a Tool against an entity in this no-strike list, the system would deterministically prevent them from doing so and inform them why.

The obvious objection to deterministic controls is that they're just allowlists, and an allowlist fails open on anything nobody anticipated. That's true of a list of domains. It isn't true of a rule that attaches to a type. When the Platform discovers something new hours or days into an operation, it enters the ontology as a typed entity and immediately inherits every rule that applies to its type. There's no window between discovery and coverage, and no operator or model has to make a decision before the protection applies.

An AI can never modify its own RoE. That guarantee is absolute, and it is also why the RoE cannot be static: a system that locks scope at launch isn't safer, it just can't tell an operator apart from an agent. Any system operating in production with real operators must handle mutable RoE that are versioned, attributable, and visible within the operational attack graph. Real world cyber operations need to be able to expand and contract as new information about the environment is gathered, with every proposed action being evaluated against the most up to date version of the operation’s RoE.

This is also where the reproducibility I mentioned earlier stops being an aspiration. Every action is adjudicated against a specific version of the RoE, and the ontology it was evaluated against was derived deterministically from raw signal the Platform perceived. When something goes wrong — and things go wrong — we can replay what the system perceived, re-derive what it concluded, and identify the rules that were actually in force at that moment. That's a bug you can find, fix, and regression test. It is not a prompt that you cross your fingers and hope holds.

Well defined RoE shouldn’t hamper an AI’s creative capability, but they should hamper its ability to act unchecked. AI systems should seek to empower their human operators and supercharge them, not turn them into approval monkeys rubber stamping actions an AI suggests. Method’s vision from day one has been to provide Iron Man suits for red team operators and security engineers, enabling them to focus on the aspects of their role that require human creativity and problem solving.

The Wrong Argument

At the same moment that we have multiple models breaking containment and performing cyber attacks, we also have malicious cyber actors beginning to harness open weight models for “real” cyber attacks. As this is happening, industry is arguing about where we should slide along the curve between “safe” and “capable”, but this is the wrong argument to be having. When the stakes involve defending the world’s most critical institutions, or supporting the US and allied governments, there is simply no reason we should have to choose between safety and capability.

The adversary has no rules of engagement, no no-strike list, and no reason to care about a downed production system. If safety genuinely cost capability, defenders would be structurally and permanently outgunned. That's the argument behind every "we can't afford guardrails" conversation happening right now, and it's why getting this wrong is dangerous rather than merely embarrassing. It's also why the tradeoff framing has to be rejected rather than negotiated.

Within the Method Platform, determinism is how we achieve autonomy. Typed, bounded, and pre-authorized actions mean that a human doesn’t need to watch each step. No decision fatigue, no accidental approvals. As operations expand from hours to days and the number of AI Agents grows, probing deeper into target space, the number of actions grows but the action space doesn't. Risk is bounded at authorization time, not at execution time. Ten agents running for three days cannot do anything one agent couldn't do in an hour. This stands in sharp contrast to lossy AI judges, where as the stakes rise, there is less autonomy, more human escalation, and more review.

Our AI cannot run a command. Everything else in this post is the explanation of why that isn't a limitation.

If you're defending an institution where failure isn't recoverable, we should talk. And if you're an engineer who thinks the interesting problem here is the control layer rather than the agent — that's what we've been building for three years, and we're hiring.