AI needs a governance structure.
AI governance is broken. Not because the rules are wrong. Because the architecture is wrong.
Think about what actually happens when a foundation model is deployed in a regulated enterprise. International bodies have published guidance. The national regulator has issued rules. The sector agency — finance, health, transport — has its own AI framework on top of that. The deployer has an internal AI policy. And whoever wrote the system prompt made their own judgment calls about what the model should and shouldn't do.
Five tiers. Five sets of instructions. No agreed hierarchy between them.
Now ask yourself: when something goes wrong, which tier was authoritative? Which instruction should the model have followed? Which one can you actually point to in an audit?
Most organisations can't answer that. Not because they haven't tried. Because the system doesn't produce an answer. The signals conflict, and nobody owns the conflict.
I've been working through this with clients for a while now. The same pattern appears every time. Strong intent at the top. Earnest effort in the middle. A model at the bottom doing its best with whatever reaches it. And a compliance function that couldn't reconstruct the governance chain if it had to.
That's not a training problem. That's not a model problem. That's a structural problem.
---
There's a 1982 computer science paper that names this exactly. The Byzantine Generals Problem. The scenario: several generals must coordinate a battle plan by messenger. But some generals are traitors. They send different messages to different recipients — "advance" to one flank, "retreat" to another. No coordinated action is possible. The army loses. Not from attack. From inconsistent signals inside its own hierarchy.
The traitors in AI governance aren't malicious. They're structural. A national regulator with conflicting departmental mandates. A sector agency siloed from every other sector agency. A deployer applying local policy that contradicts the framework two tiers above it. Each sends a signal. Each signal is locally coherent. Together, they aren't.
The model at the bottom receives all of this simultaneously. It has no mechanism to determine which instruction is authoritative. Neither does your compliance team. Neither does your board, when it's asked to sign off on AI accountability.
Byzantine fault tolerance — the mathematical solution to the generals problem — requires three things. Authenticated messaging, so you know who sent what. Enough honest nodes to outvote the faulty ones. And a consensus protocol: an agreed mechanism for arriving at a single decision despite contradictory inputs.
Current AI governance has none of these between tiers. There is no chain of custody on a governance instruction as it travels from OECD guidance to a system prompt. There is no quorum mechanism across regulatory bodies. There is no arbitration protocol when sector frameworks conflict.
What this means in practice: you can have excellent explainability — clear model cards, solid logging, good post-hoc analysis of what the model did — and still be completely unable to demonstrate that what it did was governed. Explainability tells you what happened. It doesn't tell you whether the instruction that produced it was legitimate, consistent, or traceable to any accountable tier.
That gap — between explainable and auditable — is where accountability actually lives. And right now it's almost entirely unoccupied.
---
We've solved this kind of problem before. GDPR created the Data Protection Officer — not as a compliance checkbox, but as a fault-tolerant node in a previously Byzantine data governance chain. Someone who could receive signals from multiple tiers, identify contradictions, and resolve them into a single auditable decision. Someone with documented authority and documented accountability.
The DPO model worked because it introduced something the system was missing: a human node whose job was explicitly to hold the governance chain together.
AI governance needs the same structural fix. Not more guidance at the top. Not better model cards at the bottom. A function in the middle — call it an AI Integrity Officer, a Model Accountability Officer, an Algorithmic Governance Lead — whose explicit remit is inter-tier consistency. Who can show, in an audit, that the system prompt is traceable to deployer policy, that deployer policy is consistent with sector guidance, and that sector guidance is anchored in national regulation.
That's not a technology problem waiting for a better model. It's a governance architecture problem waiting for organisations to take it seriously.
The ones that do it first won't just be more compliant. They'll be more defensible. More trustworthy to clients and regulators. And faster to adapt when the frameworks inevitably shift — because they'll actually know what they're governing.
Everyone else is still hoping the generals sort it out.
