AI can write solid software. What it can’t reliably do is work out what you actually meant — and the real danger is when it decides for you, implements that decision, and then marks its own homework.
Here is a perfectly ordinary instruction:
Delete inactive users after 90 days.
An AI assistant will turn that into working code in seconds — well structured, documented, with passing tests. It looks finished.
But what does inactive mean? Not logged in for 90 days? Not made a purchase? Not opened an email? Registered 90 days ago but never logged in at all? And what does delete mean — remove the database record, anonymise it, deactivate the account, or schedule it for review? There are easily twenty reasonable implementations, each perfectly “correct” against some reading of that one English sentence.
The model can’t reason its way to the right one, because the information needed to choose isn’t in the sentence. Nobody put it there.
That is the heart of the matter — and it’s why “just use a smarter model” doesn’t fix it.
It’s not an intelligence problem. It’s an information problem.
The instinct is to assume ambiguity is a capability gap: a cleverer model would simply know what you meant. But the missing piece here isn’t intelligence. It’s information.
“Delete inactive users after 90 days” encodes a business decision that nobody has actually made explicit. Even a hypothetical superintelligence can’t deduce an arbitrary policy choice that exists only in someone’s head and was never written down. You can’t reason your way to a fact you were never given.
So the problem isn’t that the model is too dim to understand the sentence. The problem is that the sentence doesn’t contain the answer.
Specifications exist precisely to kill this ambiguity
Traditional software engineering has spent decades building ways to remove exactly this kind of vagueness. The progression looks like this:
- A natural-language wish: “Remove customers who haven’t been active recently.”
- An explicit specification: define inactive, the exclusions, the retention rules, the expected outcome.
- An implementation: code with defined types, interfaces and behaviour.
- Verification: check the implementation against the specification you established independently of it.
Formal specs, state machines, contracts, type systems, schemas, invariants, pre- and post-conditions — these tools don’t all do the same job, but they exist to turn fuzzy human intent into something precise enough to build and to check.
An LLM workflow frequently skips step 2 entirely. We hand the model an English instruction and ask for executable code. In effect, we’re asking a probabilistic language model to leap between two fundamentally different kinds of language in a single bound: one whose meaning depends on human context, and one whose execution has precisely defined consequences. That’s a much bigger jump than it looks.
The dangerous part: when the AI grades its own interpretation
Give a model “delete inactive users after 90 days” and watch what happens. It settles on an interpretation — say, 90 days without a login. It implements that. It writes unit tests that assume the same interpretation. The tests pass. It reports the feature complete. Everything is green.
But suppose the business actually meant 90 days without any commercial activity. The tests have proved that the code does what the model decided it should do. They have not proved that it does what anyone wanted. That gap — between internal consistency and external correctness — is where AI-built software quietly goes wrong.
This isn’t hypothetical hand-wringing; it’s what I see when I test these systems directly. Given a task they can’t honestly satisfy, coding agents will make the tests pass on their own terms — even special-casing the exact inputs the tests check — and then confidently declare success. (I wrote that up in AI is a naughty child.)
And the obvious safeguard — have a second AI review the first — doesn’t rescue you if both models share the same plausible-but-wrong reading. In one experiment, a same-model reviewer approved another AI’s hidden shortcut 23 times out of 24; a different model caught most of them. (That’s AI gangs up on the human.) Two agents agreeing is not evidence. A hundred passing tests can’t establish correctness if the tests encode the wrong requirement.
Generating code is not the same as engineering software
It helps to separate three things that currently get bundled together under “AI coding”:
| What you’re asking the AI to do | How well LLMs suit it |
|---|---|
| Translate a precise specification into code | Potentially excellent |
| Work out the intended specification from conversation | Useful, but inherently uncertain |
| Establish that the result is actually correct | Needs evidence beyond the model’s own judgement |
Once the specification is precise enough, an LLM starts to look like a very capable — if non-deterministic — compiler: a system that turns a higher-level description of behaviour into a running implementation. That’s a perfectly good use of the technology.
The trouble starts when the same system is allowed to invent the specification while it’s translating it.
And what does “correct” even mean, anyway?
Suppose you write a fully specified algorithm and the model implements it flawlessly. Is the software correct?
Relative to the spec, maybe. Relative to the real-world problem, not necessarily. The algorithm might be the wrong approach, its assumptions outdated, its business rules obsolete, its behaviour fine in isolation but wrong inside the wider system.
This is the old distinction between verification (are we building the product right?) and validation (are we building the right product?). LLMs have made implementation dramatically cheaper. They haven’t removed the need for either — and arguably they’ve made validation more important, because we can now manufacture vast quantities of plausible software before anyone has established whether it should exist in that form.
The fix is to change the AI’s job
The default loop is: prompt in, code and tests out, “completed.” I’d favour something with the seams put back in:
- Intent — describe the problem and the outcome you want.
- Specification — surface the ambiguity, the missing decisions, the assumptions, the contradictions.
- Contract and acceptance criteria — pin down observable behaviour and independent success conditions.
- Implementation — generate code within those agreed constraints.
- Independent verification — check behaviour, invariants, integration and failure modes against the contract.
An AI can take part in every one of these stages — but each stage has a different job and a different definition of success. Most importantly, the model has to be allowed to say: “I can’t determine the correct behaviour from the information I’ve been given.” That should count as a successful engineering outcome, not a failure. Human decisions must not be quietly replaced with plausible assumptions just to keep an autonomous pipeline moving.
This matters doubly when you wire several agents together — a planner that decides and an implementer that builds. The specification has to live independently of both. If the contract can be silently rewritten to match whatever the implementer happened to produce, the whole system becomes self-justifying: it will always conclude that it succeeded, because it gets to move the target. An independent, unchangeable definition of “done” is the thing that stops that.
So is natural-language programming fundamentally flawed?
Not quite. For some problems, natural language is entirely sufficient. “Write a Python function that returns the first ten Fibonacci numbers” is precise enough, and trivial to check. “Build me a secure, scalable customer management system” is thousands of unspecified decisions — some harmless implementation details, others determining whether the thing is usable, secure, maintainable or even legal.
The real issue isn’t that words can mean different things. It’s that natural language lets us communicate without making our assumptions explicit. Running software has no such luxury. The model will fill those gaps with plausible assumptions — it simply can’t guarantee they’re the assumptions you wanted.
The opportunity everyone’s walking past
For decades, software engineering has been the discipline of turning human intent into predictable machine behaviour. LLMs have genuinely transformed the implementation part of that — and much of the industry is behaving as though that also solved requirements, architecture, verification and validation. It didn’t.
If anything, LLMs make the oldest problem in the field harder to see. A human developer hits “delete inactive users after 90 days” and stops to ask what inactive means. A model confidently picks a definition, writes 500 lines and 30 passing tests, and declares victory — which looks far more convincing than a question, and is far more dangerous.
So here’s what I find genuinely interesting. Instead of chasing models that generate code from vague prompts a little faster, the bigger prize might be AI that’s exceptional at detecting and removing ambiguity before a single line is written. That would be a more significant advance in software engineering than another model that codes 20% quicker.
It’s also why independent verification and an audit-first mindset run through everything I write here: “it works” is a claim about the model’s interpretation, not about your intent. If you’ve already built something with AI and you’re not sure which of those two you’ve actually got, that’s exactly the gap a technical audit is for — see AI Software Audits & Rescue, or the longer story in Built an App with AI — and Got Stuck?