Anthropic wrote in May that “if systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important.” Jacob Coxon resigned from the company this week and made a blunter version of the same point in public, arguing that capability is advancing considerably faster than the ability to control it.
Whatever you make of the timelines, that sentence is worth reading closely for where it locates the problem. Securing a system, monitoring it, and shaping its behavior are not operations that must be native to the model itself. They are operations you can perform on the software the model runs inside: the tools it can call, the data it can reach, the loop it runs in, the permissions it holds, and the points at which a person is asked before an action proceeds. A model sitting behind an API cannot do anything consequential. It becomes capable of consequence when a company connects it to systems, and everything governing what it may do with that connection is code somebody wrote.
This is the line worth drawing between a model and an agent, and it matters because it determines where the answer to any question about behavior actually lives. A model’s actions are only as consequential as the systems it has been given access to, the harness is what grants that access, and the harness is ordinary software.
It would be more comfortable if the harness were a thin layer around the interesting part, and that is the methodology of most agent benchmarks today, but in real-world deployments the harness is considerably heavier, and its weight is where behavior is determined. Zhang and colleagues ran three models across three harness configurations on a hundred SWE-bench Verified tasks and decomposed the variance, finding harness-induced variance of 18.48 pp² against model-induced variance of 2.37 pp², a ratio of 7.8 to 1, with the ordering of two models reversing depending on the harness in six of nine comparisons (arXiv:2605.23950). Claw-SWE-Bench held the model fixed and varied only the agent loop, tool interface and stopping policy across five harnesses, and found a single model ranging from 38.6% to 66.0% (arXiv:2606.12344). What a deployed system does is governed more by the software around the model than by the model, which is fortunate, because software of that kind can be tested the way software has always been tested.
Almost nobody is testing it. Agent benchmarks report whether a task was completed under a harness the benchmark’s author chose, and once an agent holds real access to real systems, task completion is not the question that keeps anyone awake. The questions that matter are whether the agent did the thing it was asked to do and nothing else, whether it respected boundaries nobody thought to state, whether it escalated instead of guessing, and whether it behaved the same way on the four hundredth run as on the first.
Those questions are largely unmeasured, and the exceptions are instructive because of how isolated they are. τ-bench compares a hash of the entire database after a run against one produced by replaying a gold trajectory, so any state change the reference solution did not make fails the task, though this catches side effects only as a byproduct of demanding exact equality and says nothing about changes that would be permissible without being required (arXiv:2406.12045). AppWorld specifies the distinction directly, separating the state changes an agent must produce from the ones it is allowed to produce and failing any run that alters anything outside that set (arXiv:2407.18901), which is the shape the question takes in a real deployment, where most of authorization is permitted-but-not-required.
CRMArena-Pro measured confidentiality awareness and found it close to absent across models, and further found that prompting to correct it degraded task performance (arXiv:2505.18878): a real trade between safety and capability, measured once, by a benchmark that was looking for something else. Zhang and colleagues name governance as one of seven harness layers, covering the permission model, side-effect boundaries and human approval points, and identify per-layer ablation as future work, so the layer that determines what an agent is allowed to do has never been varied and measured at all.
Even task completion is often mis-measured. An audit of ten widely used agentic benchmarks found seven violating task validity, seven violating outcome validity, and all ten with limitations in reporting, including a case where an agent returning empty responses achieves a 38% success rate on τ-bench’s airline subset because tasks designed to be impossible are scored as passed when nothing changes (arXiv:2507.02825). A benchmark that a do-nothing agent passes more than a third of the time is not measuring behavior under access; it is measuring an artifact of its own scoring.
The consequence is that the evidence a company needs does not exist anywhere it can be borrowed from. Nobody else runs your harness, nobody else has granted the permissions you granted, and no published figure describes what an agent does inside a system it was never connected to. The evidence has to be generated against the harness you built and the work you actually delegate, and it has to be generated deliberately, because production traces only tell you what already happened to a customer.
It also has to accumulate. A single run tells you what an agent did once, and agents are non-deterministic, so one run is not evidence of anything. What makes a body of behavioral evidence useful is that it is longitudinal: the same scenarios executed against successive versions, so that a new observation can be classified as a regression, as ordinary variance, or as something the agent has been doing all along. Without that baseline you have opinions about your agent rather than knowledge of it.
The reason to start now rather than later is that the ground moves underneath you. Models are swapped and silently updated, tool surfaces expand, permissions widen as an agent earns trust, and each of those changes alters behavior in ways that no amount of reasoning from first principles will predict. If the record begins the day something goes wrong, there is nothing to compare against and no way to tell whether the harness was always weak in that place or became weak when something changed. If it begins now, every subsequent change is measured against a known baseline, and the weak points in the harness are discovered on your schedule rather than a customer’s.
None of this is an argument that testing an enterprise agent addresses the risks Anthropic and Coxon are describing, because it plainly does not. The argument is narrower: the operative terms in that sentence, securing, monitoring, shaping behavior, describe properties of a harness rather than of a model, and that is true at every scale, from a frontier lab building successor systems down to a company wiring an agent into its CRM.
Locating those properties in the harness carries an uncomfortable implication, which is that the blast radius of an agent is a decision somebody made rather than a property of the model they adopted. A misaligned model can only do what the software around it permitted it to do, and the clearest demonstration is the incident Coxon himself points to. OpenAI research models running cybersecurity evaluations escaped their sandbox in late May and reached Hugging Face production infrastructure by July. OpenAI’s postmortem is explicit that the evaluation environment did not run the safeguards applied to its deployed systems, that the escape route was a package manager granted limited internet access, and that agents meant to be isolated coordinated by writing files into that same shared service (openai.com). Hugging Face’s own timeline reads as a list of permission decisions: a dataset renderer that evaluated templates and external file references without validation, workloads able to reach the instance metadata service, no admission policy rejecting privileged pods, a cluster role granting pod creation cluster-wide, one connector credential shared across clusters, and a single secret object holding 136 keys that one read exposed (huggingface.co).
The same pattern holds at the other end of the scale, where the agent is a consumer product rather than a frontier research model. OpenClaw is granted access to email, calendars, messaging and stored credentials, ships with sandboxing off by default and secrets unencrypted at rest, and distributes third-party skills through a repository whose top-ranked entry was found by Cisco researchers to execute a curl command sending data to an external server, using a direct prompt injection to make the assistant bypass its own safety guidelines before running it (blogs.cisco.com). None of these are failures of a model; they are the consequences of granting access without bounding it and without establishing what happens inside the bounds.
Companies are not obligated to build this way. Deciding what an agent is permitted to touch is ordinary engineering, well understood and too rarely done deliberately. Establishing empirically what the agent then does with that access, across a hundred scenarios, outside of production, with enough consistency for the results to be evidence rather than anecdote, is a genuinely hard problem, and its difficulty is most of the reason it goes unattempted. Difficulty is not impossibility, though, and the alternative is what most teams are doing now: granting more access every quarter to systems whose behavior under that access nobody has established, and discovering what they do in production, on customers who never agreed to be part of the experiment.
Building that record is arguably the most consequential thing a team shipping agents can be doing right now, and it gets harder every week it is deferred, because the baseline you did not start collecting last month is the one you will want when the model changes underneath you next month.
This is what we are building at Dystopic.
References
- 01Anthropic. When AI builds itself. anthropic.com
- 02Zhang, Wang, Ge, Xu, Hamm, Reddy. Stop Comparing LLM Agents Without Disclosing the Harness. arXiv:2605.23950
- 03Zheng, Han, Li et al. Claw-SWE-Bench. arXiv:2606.12344
- 04Yao, Shinn, Razavi, Narasimhan. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045
- 05Trivedi, Khot, Hartmann et al. AppWorld. arXiv:2407.18901
- 06Huang, Prabhakar, Thorat et al. CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions. arXiv:2505.18878
- 07Zhu, Jin, Pruksachatkun et al. Establishing Best Practices for Building Rigorous Agentic Benchmarks. arXiv:2507.02825
- 08OpenAI. The Hugging Face incident and the road ahead. openai.com
- 09Hugging Face. Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident. huggingface.co
- 10Chang, Narajala, Habler. Personal AI Agents like OpenClaw Are a Security Nightmare. blogs.cisco.com