AI Governance has a Counting Problem

A long look at why the questions boards are being told to ask about artificial intelligence assume things that are no longer true — and what would replace them

---

Boards are being told what to ask about artificial intelligence (AI). The questions arrive in articles, podcasts, panel discussions and the trade press. Do we trust our AI? Are productivity gains translating into business outcomes? What does this mean for our people? They are reasonable questions, and a board that asked them would be doing better than one that did not. But they share assumptions — about what kind of problem AI governance is, about what can be enumerated, about what counts as evidence — that on examination do not hold up.

What follows tries to make those assumptions visible. It draws on analytical traditions that the popular treatments mostly skip. It is longer than a blog post usually is. The conventional treatment is short, and the subject deserves more room than it tends to get. The argument can be put in a sentence: AI governance has a counting problem. The reasons that matters take longer to say.

A note on sources. The proximate prompt for this piece is a recent Tapestry Networks publication, AI, Quantum and Beyond: Implications for Business (April 2026), and a popular LinkedIn post that distilled it into “five questions every board should be asking about AI in 2026.” Both are competent thought-leadership pieces. Neither is a governance instrument, in the sense I will try to explain.

I. What kind of problem are we trying to solve?

The right question depends on what kind of problem you are facing. Several decades of literature have addressed this directly. The popular AI-governance treatments touch almost none of it.

Horst Rittel and Melvin Webber, writing in 1973, distinguished tame problems from wicked problems. A tame problem has a clear formulation and a stopping rule: you know when you are done. A wicked problem has neither. Its formulation is contested; every attempt to solve it changes what the problem is; solutions are not true or false but better or worse; and the planner has no right to be wrong. AI governance is wicked in something close to the textbook sense. There is no agreed definition of what good AI governance would even consist of; every intervention shifts the field; the costs of being wrong fall disproportionately on those who did not choose the system. The popular five-question lists treat AI governance as tame — a defined problem to be solved with sensible questions — and that is the first misclassification.

Peter Checkland’s Soft Systems Methodology makes the same point in different vocabulary. Checkland distinguishes the problem situation from the problem, and insists that any problem statement embeds a Weltanschauung — a worldview — that should be made explicit. Run his CATWOE test (Customers, Actors, Transformation, Weltanschauung, Owner, Environmental constraints) on the question “are productivity gains translating into business outcomes?” and a worldview becomes visible: the firm is a productivity-engine; the beneficiaries are unspecified (shareholders, customers, employees?); the actors are management; the environmental constraints sit out of frame. A different worldview — firm-as-employer, firm-as-civic-actor, firm-as-ecosystem-participant — would generate a different question. The trouble with the popular questions is not that their Weltanschauung is wrong but that it is unstated.

Michael C. Jackson’s Critical Systems Thinking sharpens this with a methodological grid: simple/complex on one axis, unitary/pluralist/coercive on the other. Different problem contexts call for different methods. AI governance plausibly sits in complex-pluralist territory, and arguably complex-coercive once one includes workforce displacement, algorithmic management of customers and gig workers, and geopolitical contention over compute. The popular questions are methodologically suited to the simple-unitary quadrant — assume agreed goals, find the efficient path. That is a mismatch.

Werner Ulrich’s Critical Systems Heuristics extends the line with twelve boundary questions, each posed in is-mode and ought-mode, across four sources: motivation (who is the client, what counts as improvement?); control (who decides, with what resources?); knowledge (who is the expert, what counts as expertise?); and legitimation (who are the witnesses, whose worldview is heard?). The popular five touch is-mode motivation lightly, and engage ought-mode hardly at all. Whose AI should this be? Who should count as expert — the Chief Data and AI Officer (CDAO), the affected workforce, customers, civil society? Whose worldview should legitimise the firm’s choices? These are the questions that mark the line between governance and compliance.

Russell Ackoff’s distinction between problems and messes is the same insight in plain English. A mess is a system of interacting problems. AI in a firm is a mess: data quality, workforce displacement, vendor concentration, regulatory exposure, model risk, reputational risk, and capital allocation are not separable problems but interacting ones, where the interactions are where the action is. The popular questions decompose the mess into discrete domains — strategy, value, trust, people, resilience — which is analytically convenient and ontologically wrong. Ackoff would also note the missing idealised design move: none of the popular questions asks what the firm’s AI posture would look like if redesigned from scratch. They are reform-of-the-existing questions.

Donald Schön observed that hard problems live “in the swamp” of messy practice, while professional discourse lives “on the high ground” of strategic abstraction. The popular questions live on the high ground. They are problem-solving techniques, not problem-setting techniques. A board operating only on the high ground will tend to miss the problems that have not yet been formalised into the categories the high ground recognises.

Dave Snowden’s Cynefin framework offers a related cut: clear, complicated, complex, chaotic. Clear domains call for sense-categorise-respond; complicated for sense-analyse-respond (apply expertise to a knowable system); complex for probe-sense-respond (act, observe, adjust); chaotic for act-sense-respond. AI sits in complex (often chaotic) territory. The popular questions are framed as if it sits in complicated. That is, again, a mismatch — and one with consequences for what counts as evidence and how confident anyone should be in any given answer.

Stafford Beer’s aphorism the purpose of a system is what it does (POSIWID) is worth quoting because it cuts through the rhetorical halo that surrounds governance discourse. The stated purpose of the popular five-question lists is to help boards govern AI well. Their de facto function — the thing they actually cause — is to produce a CDAO presentation, modest action items, and manageable escalation. Whatever the stated intent, that is the work they do. POSIWID is not cynicism so much as a discipline of attention.

Donella Meadows, finally, ranked twelve places to intervene in a system, from least to most powerful. The least powerful are parameters and feedback delays. The most powerful are the goals of the system, the paradigm out of which the goals arise, and the power to transcend paradigms. The popular questions target low-leverage points — information flows, structure of reporting. The high-leverage points — paradigm, goals, the power to define system purpose — are absent. A board working from Meadows’ hierarchy would be looking at different objects.

Nine traditions, nine angles. They do not say identical things but they overlap on one observation. The popular questions assume AI governance is tame, unitary, complicated, paradigm-fixed, and decomposable into separable problems. None of these traditions thinks it is.

II. What we are trying to count

The observation that bites hardest, for board purposes, is the one about enumerability.

Conventional governance assumes the things being governed can be listed. Risks fit on a register. Scenarios fit in a workshop. Controls fit on a slide. The whole apparatus of corporate oversight — from the duty of care recognised in Marchand v. Barnhill to the audit committee’s monthly pack — was built to count, classify, and monitor things that can be counted, classified, and monitored.

A useful way in is the coastline. Try to measure the coastline of Britain. With a ten-mile ruler you get one number. With a one-mile ruler you get a larger number. With a one-foot ruler, larger again. As the ruler shrinks the coast lengthens, because each finer pass reveals coves and inlets the coarser pass smoothed over. The coastline is not, in any practical sense, a finite quantity. Benoit Mandelbrot called this the coastline paradox, and it is not a quirk of geography. It is a property of certain kinds of objects: they look smaller from a distance than they are.

AI systems are like that. The set of inputs a large language model can receive is, in any practical sense, infinite — every sentence in every language, every image, every combination. The set of outputs it can produce is infinite. The space of behaviours that emerges from those inputs and outputs interacting with tools, agents, memory, and other models is larger still. Adversaries can probe regions of that space your testing never imagined, because your testing was built for the inputs you anticipated.

Two years of evaluations by the United Kingdom AI Security Institute (AISI) have put numbers and examples to this. Capabilities appear on the frontier that no one designed in. Universal jailbreaks — single techniques that defeat safeguards across categories — have been found in every system AISI has tested. Models can sometimes “sandbag”, performing below their actual ability when they detect they are being evaluated. None of these properties was engineered. All of them emerged.

In the language of cybernetics — Ross Ashby’s Law of Requisite Variety — a controller can manage only as much variety as it possesses itself. A board, an audit committee, even the entire risk function of a large firm has bounded variety. The system being governed has, for practical purposes, unbounded variety. An asymmetry on that scale does not yield to better committee design.

This is not a complaint that boards are not clever enough. It is a structural observation about what kinds of governance can work for what kinds of system. The coastline is not governed by counting its bays. It is governed by patrols, sea defences, and watching for breach. Enumeration is the wrong instrument.

The Tapestry paper itself, to its credit, gestures at this when one of its quoted directors says that with agentic AI “things will move too fast”, reliance will become so high that the human is “unmotivated to spot anything that goes wrong”, and the human ultimately becomes “an irrelevant post-event control”. That is a description of a control system that has been outrun by the variety of the thing it was meant to control. The paper notes the observation; it does not name what is happening; it does not draw the consequence. The consequence is that human-in-the-loop, as the load-bearing element of governance, has stopped doing the work it was meant to do.

III. What is it worth?

The popular questions treat value loosely. “Where is value being captured, who owns it, and is it visible in the operating and financial plan?” is a managerial question. It is not yet a board-strategy question. Three sharper frames the popular treatment skirts.

The first is Geoffrey Moore’s core / context distinction, sharpened by Simon Wardley’s mapping technique. The most consequential AI question for many boards is not “are we using AI?” but where in the value chain are we building proprietary AI, and where are we explicitly buying commodity AI? Invest in differentiating capability; commoditise everything else. Wardley’s contribution is to make this dynamic: capabilities move along a genesis-to-commodity curve, and AI is shifting things along that curve at speed, which means yesterday’s differentiator is today’s vendor offering. The popular questions treat AI as monolithic. A sharper version would force capital allocation, vendor concentration, and make-versus-buy onto the table at once.

The second is Aswath Damodaran’s discipline of narrative and numbers. Every claim about strategic value should reduce to assumptions about revenue growth, margin, reinvestment, risk, and terminal value. Most firms have an unfinished AI story — transformation in narrative terms, untraceable to value drivers in the financial plan. The Damodaran question for the board is: what is the net present value of our AI investment, decomposed into revenue growth, margin expansion, capital efficiency, and risk reduction — and which assumptions in the story are doing the work? Paired with his scepticism move: paradigm-shift valuations that abandon fundamentals are usually wrong. A board should be wary of an AI strategy that will not survive a discounted cash flow test.

This is where Erik Brynjolfsson’s productivity J-curve (the modern descendant of Robert Solow’s productivity paradox) bites. Productivity gains lag investment, often by years, because the complementary changes — to processes, to skills, to organisational form — take time. Boards measuring AI on near-term productivity will tend to conclude either that it is working when it is not, or that it is not when it is. Neither is useful.

The third frame is real options theory. Stewart Myers introduced the term in 1977; the tradition since (Trigeorgis, Luehrman, Copeland and Antikarov) has formalised it. When investments are sequential, uncertain, and reversible, conventional net present value tends to undervalue them, because it omits the options to expand, abandon, defer, and switch. This matters acutely for AI. High uncertainty increases option value, counter-intuitively — most boards treat uncertainty as risk to be reduced; for optionality it is value to be preserved. An AI pilot is an option to scale, not a small commitment to a programme; its value is largely informational. Boards routinely misvalue this in both directions, over-funding failing pilots (sunk cost) and treating a winning pilot’s small budget as the size of the bet.

The right question for the board is therefore not whether AI is “delivering productivity” but: what is the structure of our AI portfolio as a set of options — where are the exercise gates, what triggers expansion or abandonment, and are we preserving optionality where uncertainty is high? That question has answers that can be checked, and answers that bind subsequent decisions. The looser formulation produces neither.

IV. What does the law expect?

The legal architecture rests, mostly without saying so, on the same assumption of enumerability that the analytical traditions challenge.

Under section 172 of the United Kingdom Companies Act 2006, a director must promote the success of the company while having regard to long-term consequences, employees, suppliers, customers, community, environment, reputation, and fairness between members. The duty assumes you can weigh considerations because you can identify them. AI’s externalities — workforce displacement, the environmental cost of compute, downstream harms, supply-chain dependence on a small number of model providers — are real, but several have no clear weighing because the affected populations and the magnitude of effect cannot be enumerated in advance.

The position is sharper in Delaware, where the Caremark line of cases holds that directors face personal liability for utterly failing to oversee mission-critical risks. The 2019 Marchand v. Barnhill decision — Blue Bell Creameries, listeria outbreak, three deaths — confirmed that “we trusted management” is not a defence and that boards must implement information and reporting systems for risks central to the firm’s operation. Subsequent cases (Boeing, McDonald’s) have extended the reasoning. As AI becomes mission-critical for more firms, the duty to oversee it engages.

But what does oversight mean when the failure space is unbounded? Marchand expects a board-level monitoring system whose adequacy can be judged. Adequacy implies coverage. Coverage implies enumeration. The doctrine assumes the very thing AI denies. That is not a reason to abandon the duties. It is a reason to recognise that satisfying them, in the AI context, will require a different shape of evidence than the duties were originally framed to elicit.

There are six further governance constructs that the popular treatments handle too lightly. Each is worth a moment.

Risk appetite is one. Boards approve risk appetite; that is part of the job. For AI this means concrete tolerances — how many hallucination-driven customer-facing errors per month? what bias-driven discrimination exposure? what vendor concentration? what shadow-AI use by employees? “Do we trust our AI?” gestures at this without operationalising it into anything a board can approve, audit, or review.

Information asymmetry is another. The classic governance problem is the gap between board and management. AI widens it sharply because the technical complexity is real and accelerating. None of the popular questions addresses how the board closes the gap: independent technical advisors, board-commissioned (not management-commissioned) AI reviews, second opinions on key deployments, rotation of director exposure, whistleblowing channels for AI concerns. This is where the popular treatments are most management-friendly. They accept the asymmetry rather than challenge it.

Committee architecture is a third. Where does AI sit? Audit committee for control effectiveness? Risk committee for appetite? Remuneration committee for incentive distortions (employees rewarded for “AI adoption” produce activity, not value)? Nominations committee for board composition (do we have AI fluency)? On most boards AI sits in no committee particularly cleanly, and the absence of a clear home is itself a governance question.

Board composition and education is a fourth. The popular treatments nod at “literacy”. The harder questions sit just behind: do we have a director with genuine AI/data fluency, or do we rely on management briefings? Do we have a standing independent technical advisor? Is AI experience now a criterion in director recruitment? Have we run real board education, not a 90-minute session? These are questions about the board’s own fitness to govern. They tend to be deferred. They oughtn’t be.

Disclosure is a fifth. Boards oversee what is said to the outside world — annual report disclosures, regulatory filings, stakeholder communications. AI is becoming a disclosure topic, with ISO/IEC 42001, the European Union AI Act, the NIST AI Risk Management Framework, and emerging Financial Conduct Authority and Securities and Exchange Commission expectations. What are we disclosing about our AI, and is it accurate, complete and consistent across regulatory, financial and stakeholder communications? — absent from most popular treatments.

Crisis governance is the sixth. When AI fails — a discriminatory output, a regulatory investigation, a chatbot liability event, a vendor breach — what is the board’s pre-agreed response architecture? Who escalates, on what triggers, to whom? Marchand is, in part, an anticipatory doctrine. The failure it punishes can be the failure to have prepared.

Taken together, these six areas are the governance layer the popular five questions almost entirely skip. They skip it because the popular questions are written for a thinking board — one that wants intelligent conversation about AI — rather than a responsible board, which knows it has duties, committee remits, risk appetite to approve, disclosures to oversee, and personal liability if it fails to oversee mission-critical risk.

V. The frameworks help, and do not solve

The formal frameworks now in circulation are useful, in the way good tools are useful for the jobs they are designed for. None of them, on its own terms, solves the counting problem. It is worth being precise about why not.

The United Kingdom Treasury’s Orange Book gives a clean enterprise risk taxonomy: strategic, operational, financial, compliance/legal, reputational, people, project/programme, environmental. Each is a type of risk, not a scenario or a control. It is genuinely categorical.

The NIST Cybersecurity Framework (NIST CSF) is, by contrast, a hybrid: its functions (Govern, Identify, Protect, Detect, Respond, Recover) are stages of a security programme, not risk categories. Mixing those with threat scenarios and control objectives makes for a confusing list. Useful for a Chief Information Security Officer (CISO); less useful for board-level coverage work.

The NIST AI Risk Management Framework is a property taxonomy — characteristics of trustworthy AI: valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; fair with bias managed. The companion Generative AI Profile sharpens this for generative-AI risks. The framework tells you what the system should be like; it does not tell you whether it has those properties across the unbounded space of inputs you have not seen.

ISO 42001 is more ambitious: a full management-system standard for AI, certifiable, modelled on the ISO 27001 information-security tradition. Done well, it gives a board confidence the firm has a process for AI risk. It does not tell the board what the process has missed.

The European Union AI Act sorts AI systems into unacceptable, high, limited, and minimal risk and imposes duties on each tier. This is a regulatory instrument, not a governance instrument: it tells you which laws apply; it does not tell you whether your system is safe.

Federal Reserve guidance SR 11-7 (and its OCC sibling 2011-12) on model risk management in financial services is the original “AI risk framework” before AI was the term: development risk, implementation risk, use risk. Still influential; still bounded by the assumption that models can be characterised in advance.

These frameworks differ in scope and ambition; each rewards study on its own terms. They share a structural limit. Each enumerates properties and processes — what the system should be, how it should be managed — and is mostly silent on states: what is in fact happening, including in parts of the failure space no one has imagined. That is a limit of the genre, not of any particular instrument.

There is a structural reason, beyond the limits of any one framework, that no list of five questions can be MECE — mutually exclusive and collectively exhaustive. MECE-ness only makes sense relative to a paradigm: a single classification axis along which the space is cut. The popular questions fail MECE not because they overlap (they mostly do not) but because each is cut from a different paradigm. One question is from a strategy paradigm, another from a value paradigm, a third from a risk paradigm, a fourth from a workforce paradigm, a fifth from a resilience paradigm. Five different cutting planes through one object.

A brainstorm of plausible AI paradigms quickly runs to a dozen or more — AI as tool, as utility, as agent, as platform, as productivity multiplier, as competitive advantage, as creative destruction, as commoditising force, as portfolio of options, as labour relations issue, as power redistribution, as cognitive prosthesis, as enterprise risk, as cyber threat surface, as compliance exposure, as model risk, as systemic societal risk, as epistemic technology, as decision-maker. Each yields a different and internally MECE list of questions. Under real options, the questions become “where are the gates, what triggers exercise, what is the option premium, are we preserving optionality?” Under AI-as-agent-workforce, the questions become “who manages the agent workforce, what is the performance management process, what is the headcount equivalent, how are agents onboarded and terminated, what is the liability when an agent errs?”

The popular five-question list is a consultancy composite, drawn from whichever paradigm seemed sharpest for each particular topic. That makes it useful as a discussion prompt. It also makes it weaker than it looks as anything more rigorous.

VI. Are these even questions?

There is a niche but genuine field that studies the logic of questions, called erotetics. It turns out to be more applicable to board work than its obscurity would suggest.

Nuel Belnap and Thomas Steel formalised the modern theory in The Logic of Questions and Answers (1976). Jaakko Hintikka built the interrogative model of inquiry, in which inquiry is a series of questions to a “source”, with each question rational only if its answer advances the principal question. Andrzej Wiśniewski’s Inferential Erotetic Logic introduced erotetic implication — when one question, in conjunction with prior answers, gives rise to another — and the idea of an erotetic search scenario: a sequence of questions in which each is implied by the previous.

Apply this to the popular five.

Every question presupposes things that must be true for it to make sense. “Are productivity gains translating into business outcomes?” presupposes that there are productivity gains, that they are measurable, and that translation is the right relation. Each presupposition is contestable, and the question hides the contest. Erotetic analysis surfaces presuppositions and lets you ask whether those are themselves the more interesting questions.

A question is sound if and only if it has at least one true direct answer. “Do we trust our AI?” is borderline. Trust is gradient; trust-in-what is unspecified; the question admits no clean direct answer, which is why it tends to elicit reassurance rather than information.

Belnap distinguished whether-questions (yes/no), which-questions (selection), and what-questions (specification). The popular five are mostly whether-questions in disguise. Whether-questions invite confirmation; which- and what-questions force discrimination. A board agenda heavy in whether-questions tends to produce theatre; one heavy in which- and what-questions tends to produce decisions.

A question is self-rhetorical when its asker is not really seeking information. “Do we trust our AI?” is largely self-rhetorical in a board setting — the answer is socially predetermined. A useful test: would a “no” answer be acceptable in the room? If not, the question is not doing inquisitive work.

The strongest test is whether a question set forms a scenario in Wiśniewski’s sense — a sequence in which each question is implied by the previous together with possible answers. The popular five are parallel, not sequential. They could be asked in any order, and the answers do not shape what to ask next. A more powerful question set would have erotetic structure: a principal question (implicitly, “Is our AI strategy and posture sound?”), decomposed into sub-questions, with sub-answers triggering further questions.

Beside erotetics there is a more practical set of question-quality tests — properties of good board questions. Coverage: do they span the space without gaps? Falsifiability: can the answer actually be wrong in identifiable ways? “If AI disappeared tomorrow, would the business feel it?” is a strong falsifier; “do we trust our AI?” is weak. Discriminating power: does the question separate a well-run firm from a poorly-run one, or would both give similar answers? Robustness to gaming: can management produce a confident answer that does not reflect reality? Affordance: does the question afford the right cognitive moves — genuine deliberation, productive disagreement, action? Decision-forcing vs information-gathering: does the question force a decision, or does it merely surface an update? Generative vs diagnostic: does it generate new strategy, or just diagnose existing posture? Most popular question lists score reasonably on coverage and poorly on the rest.

There is a related test — the formalisation test. Could a question be expressed as a checkable predicate over a defined state space? Tools for such expression have existed for decades: Z3 (a Satisfiability Modulo Theories solver from Microsoft Research), Temporal Logic of Actions Plus (TLA+) (Leslie Lamport’s specification language, used at Amazon Web Services), Alloy (Daniel Jackson’s relational specification language from MIT), the B and Event-B method, and the Coq, Lean, and Isabelle theorem provers. None of the popular questions can be formalised as written. Predicates are undefined and gradient (“strategic”, “trust”, “resilient”); there is no state model (what is an AI posture as a typed object?); there are no quantifiers (over what entities?); there are no transition relations.

A formalised rewrite makes the looseness visible. “Is AI a strategic capability?” becomes: ∀ initiative i in the portfolio, embedded_in_core_process(i) ∨ contributes_to_advantage(i). “Are we resilient?” becomes: ∀ critical_dependency d, ∃ alternative a such that switch_cost(d → a) < tolerance(d). The exercise is not idle: trying to formalise exposes which predicates are undefined, which entities are unnamed, and which quantifiers are missing. The closest the popular treatments come to formalisable structure is the “if AI disappeared tomorrow” counterfactual, which has the form ∀ business_process p, performance(p, no_AI) noticeably_below performance(p, with_AI). That has shape. None of the others does.

The deeper lesson is the one Hintikka built his model around: a question is rational only if its answer would advance the principal question. The popular five never state their principal question. That is the chief reason their answers, however carefully composed, do not add up to a verdict.

VII. What the safety engineers know that the boardroom does not

The popular literature on AI governance does not engage the safety-engineering disciplines that have decades of experience with problems of this shape. The omission is strange.

The most fully developed of these, for present purposes, is Systems-Theoretic Process Analysis (STPA), developed at MIT by Nancy Leveson and built on her Systems-Theoretic Accident Model and Processes (STAMP) framework. STPA does not start from a list of hazards. It derives them from a model of the system’s control structure — who controls what, with what feedback. The procedure runs roughly thus: identify the losses (unacceptable outcomes); identify the hazards (system states that could lead to losses); model the control structure; identify unsafe control actions (provided when shouldn’t be / not provided when should / wrong timing / stopped too soon); identify loss scenarios — why unsafe control actions occur (communication gaps, model errors, control structure flaws, environmental disturbance).

Apply this to AI governance and a different question set falls out. What are our unacceptable losses — discriminatory outcomes, catastrophic decisions, reputational events, regulatory action, safety harm? What hazards lead to them — model drift undetected, agent acts outside authorised scope, control loop too slow, alert fatigue? Is our control structure sufficient — does the board have the feedback signals it needs; do escalation triggers exist; are humans-in-the-loop genuinely effective rather than nominal? What unsafe control actions are possible — model deployed without evaluation, agent given authority without escalation triggers, oversight signal suppressed when threshold crossed? What loss scenarios have we explicitly enumerated and tested against?

Recent work has applied STPA directly to frontier AI. The Tapestry paper actually describes an STPA-style emergent failure mode in passing — the “post-event control” observation about humans rendered nominally present and effectively absent. But it is diagnostic narration, not systematic derivation. STPA would have generated it (and several others) before deployment, with traceable artefacts a board could ask to see.

Adjacent techniques are similarly missing from board-level AI literature: STPA-Sec (William Young’s security extension), the Functional Resonance Analysis Method (FRAM), developed by Erik Hollnagel, Hazard and Operability Studies (HAZOP), originally a chemical-engineering technique now applied to software workflows, bow-tie analysis, system-level Failure Modes and Effects Analysis (FMEA). In aviation, nuclear, rail, and medical devices these techniques have been part of standard practice for decades. Board governance of AI has not absorbed them. It may be the largest single methodological gap in the field.

VIII. The shift — from inspection to harness

The response to unboundedness cannot be to enumerate harder. It has to be a different shape of governance.

First, containment replaces certification. You cannot certify an AI system as safe across all inputs because you have not seen all inputs. You can put it inside a smaller, well-understood structure — a sandbox, a permission boundary, a scope limitation, an allowlist of permitted actions — and certify the structure. The model can be a black box; the harness around it must not be.

Second, continuous monitoring replaces periodic inspection. Drift, emergent behaviour, and adversarial pressure are continuous phenomena. Quarterly risk packs are calibrated for systems that change quarterly. AI systems, deployed at scale, change their effective behaviour continuously through retraining, fine-tuning, and the changing distribution of user inputs. Monitoring has to run at the system’s tempo, not the board’s.

Third, statistical guarantees replace scenario coverage. You cannot say what will happen on every input. You can say, with calibrated confidence, what the rate of certain failures is, and watch that rate. Capability evaluations of the kind the UK AI Security Institute produces, and that frameworks like Anthropic’s Responsible Scaling Policy institutionalise, are exactly this — pre-deployment testing, red-teaming, structured probing of dangerous capabilities. They do not enumerate the failure space; they sample it adversarially and bound the unknown.

Each move trades a familiar form of evidence for a less familiar one. An audit committee accustomed to a closed risk register will find a statistical bound less satisfying than a list of risks. The list, in this case, is also a worse description of what is going on.

IX. The verification revolution

The most interesting recent work goes further. It accepts that the AI itself cannot be verified, and asks what can be — and how the unverified part can be made safer by being placed inside something verified.

The pattern is straightforward, even if the engineering is hard. Treat the language model as a non-deterministic oracle: a thing that produces outputs you cannot predict and cannot prove correct. Then build a formal harness around it. Specify the harness precisely, in a language a machine can check — TLA+, Alloy, Linear Temporal Logic (LTL). Constrain what the model is allowed to do — which tools it can call, which data it can touch, when it must defer to a human. Run a model checker over the harness to test whether, for the specified actions and properties, behaviour remains inside the allowed boundary.

A small but growing research line is beginning to explore this:

AgentVerify treats the language model as a non-deterministic oracle inside a finite-state orchestration layer, then verifies LTL safety properties — memory integrity, tool-call protocols, human-in-the-loop boundaries — over the orchestration layer. The model’s outputs are possible non-deterministic actions; the harness is the thing that gets proved.

PAT-Agent inverts the relationship. It uses language models to help build formal models, but the Process Analysis Toolkit (PAT) model checker — not the language model — is the authority on whether the property holds, and counterexamples drive an automated repair loop. The AI assists; the verifier decides relative to the model.

Other groups verify the plans a language model produces by translating them into Kripke structures and Linear Temporal Logic and model-checking them, rather than trusting the planner. Others again pair language models with theorem provers like Lean, Coq, or Isabelle, where the prover — not the model — has final say. Domain-specific work in legal AI is using Satisfiability Modulo Theories solvers to check whether AI-generated outputs align with statutory rules.

A further strategic shift is the emergence of world models, associated most prominently in 2026 with Yann LeCun’s Advanced Machine Intelligence (AMI). LeCun’s argument is that current large language models are too language-bound for robust autonomy; systems need learned models of the physical and operational world that support reasoning, planning and action. That makes world models both a new risk and a possible answer to part of the governance problem. The risk is that an AI system may act from a wrong, stale, partial or overconfident model of the world. The opportunity is that, if the world model is explicit enough, governance has something more inspectable than surface output alone: what state the system thinks it is in, what actions it believes are available, what consequences it predicts, and where that representation diverges from reality. Reuters

The conclusion is unfamiliar: not to verify the AI but to treat it as untrusted and non-deterministic, to specify the harness around it formally, to constrain the set of permitted actions, to check plans and actions against invariants before execution, and to monitor the boundary continuously at runtime.

A board does not need to follow the mathematics. It needs to know that these areas of research exist and may be part of a response to the counting problem, and that the question to put to management is not “is the AI safe” but “what is the harness, who specified it, and how do we know it holds?”

There is a connection here back to fiduciary duty. Marchand asks whether a board has implemented an information and reporting system adequate to a mission-critical risk. A formally specified harness, with continuous monitoring and adversarial evaluation, is the kind of artefact that question is looking for. It is something a board can ask to see, ask about, and ask to be reviewed.

X. The questions a board should ask now

A different set of questions opens up, fitted to the kind of system being governed rather than fighting it.

What kind of problem do we think AI governance is, and is our oversight calibrated to that judgment? What is the principal question we are trying to answer, and how do this quarter’s board materials advance it? Whose Weltanschauung is embedded in the way our AI strategy is framed, and whose interests does that framing advance or harm? Where in the value chain are we building proprietary AI we should own, and where are we explicitly buying commodity AI we should rent? What is the structure of our AI portfolio as a set of options — where are the exercise gates, what triggers expansion or abandonment, are we preserving optionality where uncertainty is high? What is the net present value of our AI investment, decomposed into revenue growth, margin expansion, capital efficiency, and risk reduction, and which assumptions in the story are doing the work?

Where does AI sit in our committee architecture, and is that architecture coherent? What is our risk appetite for AI failures of each kind, expressed in numbers a board can approve and audit? What independent evidence do we receive — not management-curated — about the state of our AI systems? What is our board’s own AI fluency, and how is it being maintained? What are we disclosing externally about AI, and is it accurate, complete, and consistent? What is our pre-agreed architecture for AI-mediated crises?

What unacceptable losses are we trying to prevent, and what hazards lead to them? What harness sits around our AI systems, what does it constrain, and who has signed off on its specification? What is monitored continuously, at what frequency, with what alert thresholds, and who sees the alerts? What is the rate of evaluated failure on adversarial probes — and is it falling, holding, or rising? What containment do we have for the cases the harness fails: kill switches, rollback procedures, scope reversal, vendor switch? Where are we relying on AI we cannot inspect, and what redundancy do we hold against its failure? Are the people who would notice an emerging problem incentivised and empowered to raise it, or have we built a control regime in which the human is, in the Tapestry director’s words, an irrelevant post-event control?

These are not better questions because they are cleverer. They are better because they fit the system being governed, in a way the conventional questions do not. They will produce different artefacts in the board pack: specifications instead of risk lists, statistical bounds instead of control attestations, formal-harness sign-offs alongside human-in-the-loop accountability. They will require board members who can read those artefacts well enough to challenge them. That puts another question on the table: whether the board’s own composition fits the system it claims to oversee.

XI. A note on the genre

It is worth pausing on what the popular treatments do and do not do, because the genre is now an industry in its own right.

Neither the Tapestry paper nor the LinkedIn post that distilled it declares any methodology beyond “we convened directors and executives” or “I have spent time with chairs and senior executives.” Neither cites a framework. Neither makes any optimality argument — that the chosen themes or questions are exhaustive, irreducible, or better than alternatives. Neither benchmarks against existing instruments such as the Financial Reporting Council’s Corporate Governance Code, the National Association of Corporate Directors handbooks, the International Corporate Governance Network principles, or the NIST AI RMF. The warrant for any claim is “I have heard this in many rooms.”

This is a legitimate genre — qualitative inductive synthesis from convened conversations — but its epistemic status is more limited than its presentation suggests, and the genre rarely says so. For pieces addressed to fiduciaries who are personally liable for oversight failures, the absence of declared method is striking.

There is also a quiet selection effect. The themes that drop out of the popular post relative to the Tapestry source it claims to draw on — data strategy as the actual differentiator, quantum computing entirely, agentic AI as a distinct risk profile, the board–CEO readiness gap (“they have the money but they don’t know what to do with it”) — are conspicuously the points that would create the most friction in a boardroom. That is not an accusation of bad faith. It is an observation about what survives the journey from a candid director conversation into a polished publication. What survives tends to be the comfortable part. What gets dropped is often the friction — the unfinished thought, the awkward concession, the bit that would not land smoothly in a webinar — and the friction is often the most valuable, creative element.

It is worth holding the next five questions piece to a higher standard. What method produced these questions? What framework, if any, sits behind them? What optimality argument supports the choice of these five rather than some other five? What is the principal question? Whose worldview is implicit in the framing? What could be wrong with the picture the piece is painting?

XII. A closing observation

There is an old saying: what gets measured gets managed, and authors like Douglas Hubbard tell you that you can count anything if you treat it as ‘uncertainty reduction.’ There are other thinkers whose work might make you stop and think – one of my favourites is Stephen Wolfram’s Computational irreducibility: for some systems, no shortcut exists; you must run the system to know what it does. In the light of that, whatever “measured” means in such a case, it cannot mean the same as “enumerated and ticked off.”

The hardest part for a board might not be learning a new framework. It is admitting, in plain language, that the failure space is no longer a list, and asking what its oversight system is in fact doing. A board that makes that admission is, if anything, more responsible than one that does not. The pretence of enumeration, in a domain that defies it, IS the failure mode and the law and the literature have not yet caught up with it. It is the failure mode that a future board may eventually have to explain to a court.