I have spent a lot of time getting frustrated with LLMs.

One minute they can design an architecture, explain unfamiliar code and suggest an approach I had not considered. Five minutes later they confidently do something that makes you wonder how anything so capable can be so bloody stupid.

I kept blaming the model.

Sometimes the model was wrong. Sometimes it really had produced rubbish. But often I was making a more basic mistake: I was asking a probabilistic system to behave like the deterministic systems I had spent my career building.

I was also judging its answer using information I had never given it.

That distinction changed how I build with AI.

A diagram comparing the rich project context an experienced developer carries with the smaller set of information supplied to an LLM. It shows that missing context can change a decision.
The developer and the model may be reasoning from very different versions of the problem.
Text version of the visual

A developer may know architecture decisions, business constraints, standards, fragile dependencies, rejected approaches, customer intent and the real definition of done. An LLM may receive only a prompt, ticket, selected files and partial state. That gap can affect its decision. Missing context is not the only reason a model can be wrong, and more context does not guarantee correctness.

The model does not know what you know

An experienced developer never sees only the ticket and the files open on screen.

We carry a hidden model of the system in our heads:

  • why an architectural decision was made six months ago;
  • which business rule looks wrong but absolutely cannot change;
  • which dependency is fragile;
  • which approach the team already tried and rejected;
  • what the customer probably meant by a vague requirement;
  • what our standards require;
  • and what “done” actually means here.

Much of that context is not in the code. Some of it is not written down anywhere.

Then we hand an LLM a short prompt, a ticket and a few selected files. When it makes a different decision from the one we would make, we call it stupid.

But we are not solving the same problem.

The developer is reasoning from years of experience and a rich picture of the current system. The model is reasoning from the context it was actually given.

That does not excuse a bad answer. Missing context is not the only reason models fail, and more context does not guarantee correctness. It does mean the first diagnostic question should be:

What did I know that the model did not?

Deterministic and probabilistic systems make different promises

A conventional program applies explicit rules to known inputs and state.

known inputs
+ explicit rules
+ known state
= predictable result

If the program and inputs stay the same, we expect the result to stay the same. When it does not, we call that a defect.

An LLM makes a different kind of promise. It generates a likely response from the context available to it and the patterns learned during training. That makes it powerful when the answer is not yet known.

It can explore architecture, explain unfamiliar code, draft a specification, compare options, critique an idea and create candidate solutions at remarkable speed.

That is why I think of an LLM as a design engine.

It is excellent in the messy part of engineering where judgement, synthesis and creativity matter. It is much less convincing as the final authority for a question that already has an exact answer.

Stop asking the model when a script can know

Suppose I need to check whether every object in a repository follows a naming rule.

I can ask an LLM to review the repository and tell me what it thinks. Or I can encode the rule and run a deterministic check.

for item in objects:
    assert follows_naming_rule(item)

The first approach gives me an opinion that sounds plausible.

These objects appear to comply.

The second gives me a result I can reproduce.

47 checked
47 passed
0 failed

The LLM can still help define the rule, find edge cases, write the first checker and explain a failure. But once the rule is stable, repeatedly asking the model to reason through it is slower, less reliable and harder to prove.

The useful move is to convert repeatable knowledge into executable checks.

Put probability where it earns its keep

This is not an argument against LLMs. It is an argument for using them where they are strongest.

Use probabilistic reasoning when

  • the problem is ambiguous;
  • you are exploring designs or alternatives;
  • you need synthesis or explanation;
  • you are generating candidates for review.

Use deterministic software when

  • the rule is known;
  • state must be preserved;
  • a schema or condition can be checked exactly;
  • you need reproducible evidence.

The boundary is not always obvious. A task may begin as an open-ended conversation and become deterministic once the rule is understood. That is the moment to stop paying for fresh reasoning and turn the lesson into a tool.

The engineering system belongs around the model

For a while, I kept asking how to make the AI smarter.

Now I ask different questions:

What am I asking the LLM to do that should never have been probabilistic in the first place?
What context am I using to judge the answer that the model never received?
What evidence would prove the result without trusting the model's confidence?

Those questions move responsibility away from the model and back into engineering.

The model should not own durable state. It should not be the final judge of its own work. It should not be expected to remember an undocumented decision from another session. And it should not be used as an expensive, inconsistent substitute for a script that can return a definite answer.

The LLM is one component in the engineering system. A powerful one, but still a component.

Use it to reason where the answer is uncertain. Give it the context required to make a useful decision. Move stable rules into deterministic mechanisms. Verify the outcome independently.

That distinction has become the foundation of what I am building now.

The next question is where the boundary should sit: when do you stop asking the LLM and build the tool instead?

More writing