Executive summary
The trustworthy AI field has made literacy and governance its default answer to responsible adoption, and the EU AI Act has made both a legal duty. But training and policy try to fix in human behavior what should have been handled in the product. Build the safeguards into the design instead: guardrails, friction at the right moment, easy override, real explainability, built-in accountability. Once the design owns the how and the what, literacy is freed to explain how people's role and responsibilities change, trickled in just in time. Governance stops policing behavior and instead codifies those design practices, fills the thin compliance residue, extends the structures you already have, and stays at the top of the pyramid where values live. Both keep disappointing us because they speak to the conscious mind, while the failures they target are unconscious: bias, shortcutting, the path of least resistance. The same holds for AI systems, which run on defaults and cheap shortcuts a written instruction can no more correct than a sign can close a door. You cannot lecture or legislate a default away. You can only redesign it.
In a former life, I worked at a greenhouse company. Every growing season started the same way: from sterile in vitro plants, propagated under glass in tightly controlled, almost surgical conditions. Those plantlets were the genetic material for the entire season. If they were compromised, everything downstream was compromised with them. Protecting them was the difference between a successful growing season and bankruptcy.
So there was a problem that kept everyone up at night. One of the doors to the sterile greenhouse was always left open. An open door meant moving air, viruses, pests, contamination. A significant risk to the most valuable thing in the company.
We called a management meeting about it. Ideas started flowing. We should sensitize everyone. Run a training program on the importance of closing doors. Print pamphlets. Put signs on every door. Send a company-wide email, on a regular cadence, to keep it top of mind. Someone suggested badges for the people who followed the rules. Someone else proposed penalties for the ones who did not. Draft a policy, and have everyone sign that they had read it. It escalated quickly, and for my taste it went a bit far.
Ludo sat at the end of the table. Ludo is a bit of a legend, a West Flemish entrepreneur who built the whole company from nothing. He started out living in his trailer parked next to the company’s first greenhouse. He owns the place. He was quiet, but you could see him getting more irritated by the minute. After about twenty minutes he slapped his hand flat on the table. Silence.
I have never forgotten what he said. Pure farmer's common sense: "In my company, I want people to solve problems, not create new ones."
Then he turned his laptop around. On the screen was an overhead door closer from a local supplier. Price: 17.39 euro. Ludo exercised his executive power and pressed the purchase button. We fitted them to every door. The next morning, every door was, surprise, closed. No training. No pamphlets. No badges. No policy. No dissenters to punish. Just a problem, solved.
Spend any time in trustworthy AI, AI compliance, or AI change management right now and you will notice two fixations. The first is AI literacy. The second is AI governance. Both are treated as the route to responsible adoption. Teach people how to use the system, train them on its risks, certify them. Write the policies, define the controls, log the sign-offs, run the audits.
Neither is a fringe view, and neither is optional. Article 4 of the EU AI Act now writes AI literacy into law: since February 2025, providers and deployers must ensure their people have a sufficient level of it, with national enforcement powers arriving in 2027. The rest of the Act, the risk management, the documentation, the human oversight requirements, the conformity assessments, is the governance half of the same instinct. So the reflex is understandable, and the obligation is real.
But watch what is happening. We have an open door, and the room is filling up with pamphlets, signs, emails, badges, and a thickening binder of policy. We are trying to train our way and police our way out of what is, at root, a design problem.
This is the pitfall of Ludo's door: overcorrecting with behavior change, whether through literacy or through governance, to compensate for something the design should have handled in the first place. Every hour of training you schedule to stop people doing the wrong thing, and every policy you write to forbid it, is an hour and a rule you spent because the wrong thing was possible at all.
I propose a paradigm shift, starting from design instead of obligation. The principle is simple. Do not ask people to behave around a flawed system, and do not govern a flawed system into safety. Build a system that earns trust by design. Most of what we reach for literacy and governance to fix is, on closer inspection, a missing door closer.
So before you write the training plan or the policy, ask what the design should have done. A few examples.
Don't want people to do certain things? Build the guardrail, not the warning sign. A support agent that physically cannot issue a refund above its limit needs no training on refund limits, and no policy forbidding large refunds. A tool that cannot send to "all contacts" without an explicit approval step does not rely on anyone remembering the rule. Put the constraint in the system, not in the staff handbook.
Worried about overreliance? Engineer the pause. If you need people to stop and think before they accept an AI output, design for it. Combine explainable AI with deliberate cognitive friction. A few patterns worth knowing: hold the recommendation, where the reviewer records their own judgment before the model reveals its answer, then sees where the two diverge; show the alternatives, surfacing two or three ranked options rather than a single output, so the cognitive task shifts from approving to evaluating; and explain the counterfactual, showing not just what the system recommends but what would have to change for it to recommend something else. You have turned a rubber stamp into a real decision, and you did it with interaction design, not a memo about critical thinking. For a deeper look at why friction belongs in AI interfaces, read my blog post Why frictionless design is wrong for AI.
Want users to take action based on critical thinking? Make the right action the easy one. This is the opposite failure mode. A user who has spotted the problem, then meets an override buried behind three confirmation screens and a justification box, learns fast that the system does not want their input, and stops offering it. Here friction is the enemy. Overriding, adjusting, and escalating should each be a single move, and the system should show that the move landed.
Want people to take responsibility for what they do with AI? Make accountability part of the record. Responsibility is not a value you can train into existence, nor a clause you can legislate into being. It is a property you can design in. Attribute every AI-assisted decision to a named human, log who did what across both the AI and the people, and make that trail visible. When ownership is built into the workflow, you do not have to keep reminding anyone to own it.
Is the decision really that critical? Put the friction where the stakes are. Design a four-eyes check into the workflow, so a second person signs off before a high-stakes call goes through. Add a cooling-off delay, so an irreversible action does not execute the instant it is requested. And add a preview-and-confirm step, so the user sees exactly what is about to happen before it happens. None of these depend on the user having sat through a course, or on a policy being consulted at the moment of action.
None of this means literacy is useless. It means we have been aiming it at the wrong target. Once the design owns the how and the what, which buttons to press and which actions are off limits, literacy is freed to do the one thing design cannot: help a person understand their changed relationship with the work. Aim it at the why and the who, not the how and the what.
Frame it as a new relationship, not a new tool. The real shift is not "how do I operate this" but "what is my role now that this exists". The mechanics, guardrails, and explanations belong to the design. Spend literacy on how the role of a user changes in light of this new AI system. What still needs my judgment, what I am now accountable for, where the work has moved, what is expected from me. Name that relationship between human and AI out loud. People adopt AI responsibly when they understand how it changes their role, not when they have memorized its menu options or learned how to read a confidence interval.
Trickle it, do not dump it. Replace the one-off, two-hour course with evergreen, just-in-time onboarding that lives in the flow of work. Cut the material into small pieces and surface each one at the moment it becomes relevant, inside the interface. The first time someone meets a recommendation is the moment to explain how to read it. Training people receive exactly when they need it sticks. A session they sat through last quarter does not.
Make it a two-way loop. The people using the system are your best sensors for where it misleads. Give them a frictionless way to flag confusing outputs and surprises, and feed what comes back into both the training and the design. Literacy that only broadcasts goes stale. Literacy that listens keeps improving the door closer.
The same correction applies to governance, and it is the more uncomfortable one, because governance is where the field has invested its hopes for control. The reflex is to treat governance as a body of compliance requirements imposed on top of the system: a register of rules, controls, attestations, and audits that people must satisfy. The thicker the binder, the safer we feel.
But look back at the management meeting. Almost every idea on that table was a governance idea. Reward the compliant. Punish the rest. Write the policy. Collect the signatures. Every one of them was an attempt to govern behavior around a door that did not close by itself. And every one of them was made redundant by a single decision: every door gets a door closer.
That is the reframe. Governance should not be a catalog of rules for how people must behave around your AI systems. In a design-first world, it collapses to something much smaller and much stronger: codify the door closers. The good design principles you build into your systems become the standard you govern by. You do not write a policy that says "users must remember the refund limit". You write one that says "refund tooling must enforce its limits in the system". You stop governing the behavior and start governing the build.
Done well, this turns a hundred behavior-policing policies into a handful of design standards. Instead of rules telling people how to act around flawed systems, you have rules about how systems must be made. One real policy, in the spirit of Ludo: every door ships with a door closer.
This does not make governance disappear. It gives it a sharper and more honest job. Five tenets follow from it.
Start by codifying the design best practices. This is the bulk of the work, perhaps eighty percent of it. Take the guardrails, the friction, the explainability and the accountability trails you build into your AI systems, and turn them into the standard every system must meet. You do not write a policy saying "users must remember the refund limit". You write one saying "refund tooling must enforce its limits in the system". Govern the build, not the behavior, and most of what people imagine governance to be simply evaporates.
Then close the compliance residue with ‘hard governance’. Once the design standard is in place, look for what it cannot enforce: the legal obligations, the value judgments and the accountability questions that remain open. That residue, and only that residue, is where hard governance belongs. It is a thin layer on top of good design, not a parallel universe of its own.
Think in deltas, not greenfields. You almost never need a new, bulky AI governance apparatus bolted on beside the company. Nobody wants that, and most of it goes unread. Ask instead what your existing structures, your risk management, data governance, model approval and audit functions, need to absorb in order to handle AI. Govern the delta. Extend what already works rather than erecting something separate next to it.
Build it incrementally, use case by use case. This is the smarter route in practice and it falls out naturally from the delta approach. Let your AI governance grow as you ship real use cases, so it is shaped around the systems you are actually building rather than hypothetical ones. Each deployment teaches you which door closers matter and which gaps are real, and the governance accretes around lived reality instead of speculation.
Keep AI governance at the top of the pyramid. Freed from the operational detail that design now owns, governance can do the one thing only it can and should do: sit at the apex of the policy pyramid and set the tone. Values, ethics, accountability, the lines you will not cross and the responsibility you will carry. That is where governance is irreplaceable. Spend it there, not on cumbersome rules the system should be enforcing for you.
Read it this way: even the EU AI Act is less a behavior regime than a design-and-evidence one. Risk management, data governance, robustness, human oversight: these are properties you build into the system and then prove. The Act is asking for door closers, and for proof that they are fitted. The failure mode is answering it with a stack of attestations and a literacy certificate, while the door still swings open.
Here is what literacy and governance have in common, and why both keep underdelivering.
They both speak to the conscious mind. Literacy informs it: here are the risks, here is how the tool works. Governance instructs it: here are the rules, here are the consequences. Both assume the same model of a person, one who, having been told the right thing, will choose to do it.
But the open door was never a conscious choice. Nobody at that greenhouse decided to endanger the crop. The door stayed open because closing it was the effortful path and leaving it was the free one, and the free one won, every single time. That is not ignorance you can train away, and it is not defiance you can punish away. It is the ordinary physics of human attention. We run on defaults, habits, and shortcuts. The deliberate, effortful mind only shows up when something forces it to.
This is what cognitive bias actually is. Not stupidity, but brain economizing. We ‘satisfice’. We take the first answer that is good enough. We rubber-stamp the model's recommendation, not because no one taught us to think critically (they did, last quarter, for two hours) but because checking is work and accepting is free. You cannot lecture a default out of existence, and you cannot legislate it away either. You can only change which path is the default. That is the whole genius of the door closer: it makes the safe action the effortless one, so it happens whether or not anyone is paying attention. It fixes an unconscious problem at the unconscious level.
Take a concrete case: a bank scoring loan applications with an algorithm. The bad-governance version is depressingly familiar. The employee is handed a thick policy and a training course on double-checking the model's output. To do that checking, they are given the raw, messy data behind each decision and told to sift through it for mistakes, based on a training they took six months ago. And when they do spot something and want to override, the system fights them: three screens, a free-text justification box, a two-hundred-word minimum. Every layer here is a conscious-level fix. It assumes the employee will recall the training, summon the effort to interrogate messy data, and push through friction to act on their judgment. Under time pressure they will do none of these things. They will accept the score. The governance is heavy, the design is hostile, and the result is a rubber stamp with paperwork.
The good-governance version puts the work into the design. The model ships with real explainability: not a bare confidence score, but the reasons, and a handful of counterfactuals the employee can actually use with the client. "This application would be approved if monthly debt repayments were 150 euro lower." "Extending the term from fifteen to twenty years moves it into approval." "Reducing the requested amount by 8,000 euro flips the decision." Now the employee is evaluating something legible, not excavating a spreadsheet. Overriding is a single clean move with a light justification, and the system shows that the override landed. And literacy arrives where it matters: not a generic course on algorithmic risk half a year ago, but in-the-flow guidance on how to read a counterfactual and when their judgment should outrank the model. Same regulation, same risk, opposite result. One governs behavior. The other governs the build.
Think about your phone. When you buy an iPhone you do not book a two-hour training session before you are allowed to use it. You just know. It uses patterns you already recognize, so the right action is the obvious one. It confirms before it does anything destructive. It quietly makes the harmful paths hard and the safe paths easy. It barely appeals to your conscious mind at all, and that is exactly why it works.
And here is the part the field keeps missing. The systems we are governing have the same shape we do. An AI model also runs on defaults, shortcuts, and the path of least resistance. It pattern-matches. It answers confidently when it should hesitate. It reaches for the cheap output. You do not fix that by adding "please be accurate and unbiased" to its instructions, any more than you fix an open door with a sign that says please close it. Both are conscious-level corrections aimed at unconscious-level behavior, and both fail for the same reason. What actually works on a model is what works on a person: change the design so the good path is the default. Grounding instead of guessing. Constraints instead of reminders. Verification in the loop. Friction placed where it matters and removed where it does not.
So the symmetry is complete. On both sides of the interface, the human and the machine, the failures we worry about are rarely failures of knowledge or of rules. They are failures of default. Literacy and governance address the conscious mind. The problems live underneath it. That is why the training and the policy keep disappointing us, and why a 17.39 euro door closer keeps quietly outperforming both.
None of which makes literacy or governance worthless. Aim literacy at the why and the who. Aim governance at codifying good design first, then the thin residue of hard rules that design cannot cover, built as a delta on what you already have and kept at the top of the pyramid where values live. But both belong on top of good design, never underneath it as its foundation, and never as a substitute for it. The moment you catch yourself printing the pamphlets, handing out the badges, or drafting your fourth policy to stop people doing something the system should never have allowed in the first place, stop and look for the open door.
Remember Ludo's door.