The Trust Layer for Enterprise Agents Sits Outside the Model
TLDR
- Verify an agent’s work outside the model, and make the check deterministic wherever you can.
- Models are tuned on human ratings, so their explanations lean toward what a reviewer finds convincing. Reading the explanation checks the author with the author’s own account.
- Root cause, by five whys: nobody owns verification as a design decision, so review falls back on reading a fluent explanation.
- The fix is two things kept apart from the model: a decision ledger (settled, assumed, ruled out, open, each with an owner) and an independent checker. A partner who builds both sets how far a client can safely scale agents.
The answer first
Put the verification outside the model. Where a rule can decide, use a rule, not another model and not a person reading a summary.
How this piece is built: one governing thought (above), three supporting arguments (review is the scarce thing, the model’s own account is weak evidence, so the check must sit outside), a five whys chain on the root cause, then second and third order consequences and what to do.
Why: three arguments from first principles
Argument 1. Output is cheap, so review is the limit. When an agent can produce a report, a diagram or a working page in seconds, the number of things a company can safely accept is set by how much it can check, not by how much it can generate. Andrej Karpathy made a version of this point recently: as models do more of the legwork, human work moves up into oversight and understanding. Review capacity becomes the scarce thing. Rahul Matthan makes the same point for whole sectors in “The Absorption Frontier” (ORF, 2 October): a sector turns abundant AI into gains only when its data is legible, its systems are actionable and its outputs can be verified against ground truth, and verification is the hardest of the three. Software got there first because version control and automated testing existed before the models did. So: the constraint to plan around is how much a company can check.
Argument 2. The model’s account of its own work is a weak thing to review. Most chat models are shaped with reinforcement learning from human feedback, which means human raters’ preferences are built into what the model treats as a good answer. Raters like clear, confident, well organised text. So a clear, confident explanation is a likely output whether or not the work behind it is right. Thore Graepel, who worked on AlphaGo, argues in MIT Technology Review that these systems keep no inspectable record of what they believe, and that the reasoning they show is often composed after the answer. If so, a human reading that story is checking the author’s work against the author’s own telling of it. So: the explanation cannot be the evidence.
Argument 3. So the check has to be independent of the model and, where possible, deterministic. A rule gives the same verdict on the same input, carries no rater bias, and can be tested, versioned and audited. The evidence is starting to arrive. Microsoft’s ThinkingBox benchmark grades agents on the records they leave behind, not on what they say, across 507 tasks, and repeats runs to tell a lucky pass from repeatable work. A 1 October arXiv paper by Ali Atiah Alzahrani audits a portfolio agent in a synthetic market and finds its runtime verifier could be bypassed with no visible change in outcomes. The author notes the findings are specific to that agent and setting, so read it as a warning, not a rule. So: build the check outside the model, and make it deterministic where a rule can decide.
Root cause: five whys on why unverified agent output gets accepted
This chain is my reasoning from the arguments above, not measured data. The problem: a wrong agent output is accepted by a reviewer.
- Why was it accepted? The reviewer judged it by its explanation, and the explanation read well.
- Why did the explanation decide it? Reading it is the only check cheap enough at the volume agents produce.
- Why is there nothing cheaper to check against? The agent hands over a narrative, not an independent record of what it did.
- Why only a narrative? The system was built to deliver answers, and the report format was left to the model.
- Why was the format left to the model? Nobody owns verification as a design decision. Review was treated as the human’s job after the build, not as a control with an owner.
Root cause: verification has no owner and no design. That is why the fix below is a ledger with named owners and a checker, not a better prompt.
What the trust layer is
Two parts.
The ledger is a record kept outside the model. Each item sits in one of four states: settled, assumed, ruled out, open. Each has a named owner and the evidence behind it. The reviewer reads the ledger, not the narrative.
The checker is a set of independent tests the agent cannot talk its way past. Deterministic first: amounts match the system of record, the approver has authority for the value, required fields exist, policy thresholds hold. For what rules cannot judge, use sampling and a named person, and have that person review the failed checks and the ledger.
Illustrative example, not a client case: an agent proposes paying a supplier invoice. The checker confirms a purchase order exists, the amount is inside tolerance and the approver’s limit covers it. The ledger records “assumed: supplier bank details unchanged since last payment”, owned by the finance operations lead. The reviewer’s time goes to that one assumption, not to rereading a fluent paragraph.
Second order: the report format decides what a reviewer can get wrong
Aerospace maintenance writing uses a controlled language so that a mechanic cannot misread a step. The same idea applies to agent reports. A format that forces “done, assumed, needs your yes” removes whole classes of misreading before anyone is tired or rushed. Choosing that format is a decision about who is allowed to be wrong, and it deserves an owner.
Third order: whoever builds the layer sets the ceiling
A model upgrade makes the agent more capable. It does not make the checker more independent, because the checker’s value comes from sitting outside. So the ceiling on how far a client can scale agents is set by the trust layer, and it stays there through every model release. That makes it a durable offering for a partner or services firm: it does not depend on any one vendor, and the client keeps the ledger when the model changes. Vendors are building governed runtimes of their own, such as Oracle’s Fusion Claw. Useful, but if the check lives inside one vendor’s runtime, the client’s trust depends on that vendor. A layer the client owns survives a change of vendor.
What to do on Monday, if you advise clients on agents
- Pick one workflow where an agent already acts.
- Write its ledger: what is settled, assumed, ruled out and open, and who owns each.
- List the checks a rule could do today, and build those first.
- Decide who reviews the rest, and what they see.
Limits
Not every judgement can be turned into a rule, and a checker can be wrong too. It needs tests and version control like any other control. The claim is narrower: the verification should not share the model’s blind spots or its desire to please.
Sources
- Andrej Karpathy, On spending more time understanding model outputs: legwork moves up into oversight and understanding; ASD-STE100 for readable explanations, 2 Oct 2026.
- Rahul Matthan, The Absorption Frontier, ORF Special Report No. 322, Observer Research Foundation, published 2 Oct 2026.
- Thore Graepel, Don’t be fooled: LLMs don’t reason, MIT Technology Review, 2 Oct 2026.
- Microsoft, ThinkingBox: Measuring whether agents finish the job, 19 Aug 2026.
- Ali Atiah Alzahrani, Verify Claims, Not Scores: Evidence-Based Verification of Modular Agents, arXiv 2610.01348, 1 Oct 2026.