The mathematics of large language models sets a hard limit on the one thing enterprises most want from them – a guarantee. That limit will not be patched.

When an organisation expects deterministic, auditable, zero-defect behaviour from a large language model, the gap between what it expects and what it gets is not a temporary immaturity to be patched. It is a mathematical property of how these systems work. You cannot fine-tune a probability distribution into a guarantee.

That sentence is the whole argument; the rest of this piece is the proof. It matters to the board because most of the money now flowing into enterprise AI assumes the opposite – that with enough data, tuning, and time, a language model will eventually behave like the deterministic software it is replacing. It will not. Not because the technology is immature, but because the mathematics underneath it was built to do something else.

Start with what the model actually is

Strip away the branding and a large language model is a next-word predictor. Given the text so far, it computes a probability distribution across its entire vocabulary and samples the next token from it. Every answer, every summary, every apparently reasoned step is that one operation, repeated. This is not a description of today’s models awaiting improvement. It is the definition of the architecture – and three consequences follow from it, none of them a matter of opinion.

The error rate cannot be driven to zero

The first consequence is that the errors cannot be eliminated. In September 2025, OpenAI’s own researchers published the proof. Training a model to predict the next token is, mathematically, a density-estimation problem, and a calibrated model doing that job must place some probability on wrong answers. They bound it formally: the generative error rate is at least twice the rate at which the same model would misclassify whether a statement is valid. Even on perfectly clean data, generating a correct answer is harder than checking one. A model can be taught to say “I don’t know” more often, and its confident errors can be reduced – OpenAI argue exactly that – but reduced is not removed. There is a floor, and the floor is not zero.

The model cannot be verified the way software can

The second consequence is that you cannot verify it by parts. Professor Eerke Boiten of De Montfort University set this out in a BCS essay in late 2024, and though he offered the conclusion as a belief he would be glad to see disproved, the underlying property is not in dispute. Traditional software is compositional: you build it from parts, test each part, and trust the whole because you trusted the pieces. A neural network’s behaviour is emergent – distributed across billions of parameters, with no module that “does” a given rule. There is nothing to isolate, nothing to unit test, nothing to formally verify. Whole-system testing is the only tool left, and against an input space that size it can only ever be a drop in the ocean. Correct one error by retraining and the change is not local; it ripples, traditional regression testing is unavailable, and today’s fix becomes tomorrow’s undiscovered fault.

And no, you cannot simply “make it deterministic”

Here a good engineer objects: set the temperature to zero, decode greedily, and the same input returns the same output. True – and it changes nothing that matters. Making the output repeatable fixes the variance, not the validity. You now get the same answer every time, including the same wrong answer every time, and you still cannot decompose it, cover it, or prove it satisfies the rule. Determinism of output was never the guarantee anyone needed. The guarantee a balance sheet needs is that the answer is correct and can be shown to be correct – and that is precisely the guarantee the mathematics withholds.

The gap bites exactly where you can least afford it

For work that tolerates variance – drafting, summarising, first-pass triage, turning a messy request into structured form – a probabilistic tool is a superb fit, because being right most of the time is the job. For work that does not – pricing, policy enforcement, financial calculation, anything auditable or regulated – being right most of the time is not the same as being right, and that difference is where the liability lives. A widely held belief is that this gap is an implementation problem, closable with sharper prompts and more tuning. The mathematics says otherwise: in this class of work, the gap is a property of the tool, and it does not shrink with scale.

Put a number on it. Picture an insurer running two million automated quotations a year through a workflow with a language model applying pricing policy, in the expectation that it behaves like a rules engine. Suppose just 1% of outputs are subtly wrong – not broken, merely off in a way a human would have caught. That is 20,000 flawed decisions a year. At a conservative £40 each to detect, correct, and communicate, that is £800,000 a year in reconciliation before anything else. Add a retrofitted programme to wrap the model in the controls it should have had, and a single regulatory near-miss, and the saving from automating fifteen roles has not gone – it has reversed. The figures are illustrative; the mechanism is not. Probabilistic variance inside a deterministic workflow converts a one-off saving into a standing liability, because you are paying, in perpetuity, to catch the errors the maths guarantees will occur.

THE LEADERSHIP QUESTION “Where in our operation are we trusting a probabilistic system to deliver a deterministic guarantee – and would we notice if it quietly began to get it wrong?” Ask your teams to name the workflows where an AI output is acted on without a check. If they can point to the verification step, you are managing the risk. If they cannot, the mathematics is not on your side, and you are carrying it.

What the argument does not claim

Two clarifications keep this honest, and neither weakens it. This is not a prediction that AI will fail or stop improving; the models are getting better, and most enterprise pilots that stall do so for ordinary reasons – MIT’s 2025 research attributes the largest share of failures to approach and integration, not to model quality. And it is not the claim that a language model is useless. It is the narrower, more durable point: for deterministic, auditable, zero-defect work, no amount of scale, data, or tuning turns a probability distribution into a guarantee. Much about AI will change. That will not.

Match the tool to the task

This turns the boardroom question from whether to use AI into where, and how to govern it – and the design that follows from the maths is now well understood. Use language models for the probabilistic work they are built for, on the outside of the system, where they meet messy human language. Hand the calculation and the policy decision to deterministic components that can be tested the old-fashioned way, on the inside, where correctness cannot be negotiated. This is the premise of the hybrid, or neuro-symbolic, systems that Garcez and Lamb describe as the field’s third wave: probabilistic language wrapped around verifiable logic. It is why Gartner expects small, task-specific models to overtake general-purpose ones in usage by 2027, a direction the BCS has framed as starting smaller to work smarter. And it is Boiten’s own way out — current AI as a component in bounded contexts, where its errors can be reliably detected and managed. Scope the model. Verify the output. Keep the failure visible and recoverable.

Where does this leave us?

The organisations pulling ahead are not the ones buying the most AI. They are the ones that understood what they bought: a remarkable probability engine, valuable precisely where probability is acceptable, and dangerous precisely where a guarantee is required. Leadership’s task is not to wait for the mathematics to change, because it will not. It is to stop asking the tool for the one thing its construction cannot give.

Remember: You cannot fine-tune a probability distribution into a guarantee – so build, and govern, as though that were true. Because it is.

Sources

Boiten, E. (2024). ‘Does current AI represent a dead end?’ BCS, The Chartered Institute for IT, 6 November 2024.
Kalai, A. T., Nachum, O., Vempala, S. & Zhang, E. (2025). Why Language Models Hallucinate. OpenAI. (Proves a non-zero lower bound on generative error for calibrated models.)
MIT Project NANDA (2025). The GenAI Divide: State of AI in Business 2025 (July 2025). Preliminary and not peer-reviewed; cited only for its finding that most pilot failures track to approach and integration rather than model quality.
Garcez, A. d’Avila & Lamb, L. C. (2023). ‘Neurosymbolic AI: the 3rd wave.’ Artificial Intelligence Review, 56(11).
Bardell, A. (2025). ‘Rethinking AI: why starting smaller might be smarter.’ BCS, November 2025 (citing Gartner’s 2027 projection on task-specific model usage).