You Can't Prompt Your Way to Truth

What six months of failed experiments taught me about hallucination, deference, and the limits of self-verification.

A recent conversation gave me a compact version of a belief I keep running into.

Someone told me they had found a way to guarantee truth from a chatbot. The method was simple. Partway through a conversation, challenge the model. Tell it you don't believe it. Tell it you think it is only agreeing with you to make you happy. In effect, you are telling it the game is up and that now is the time for the real truth. From that point on, they said, the model starts telling the truth — after a brief, courteous acknowledgment and apology.

I understand the conclusion. The tone changes. The model qualifies, pushes back, sometimes reverses itself. It feels like a correction.

But a model changing its answer is not evidence that it suddenly found the truth.

It may simply have found a new way to agree with you.

That distinction is where the last six months of lab work have landed me. I have spent that time building small agentic systems and watching them fail in ways that were more interesting than the failures I expected. I am not trying to argue that large language models are useless. The opposite: they are already extraordinarily useful, and increasingly so. They summarize, draft, classify, translate, explore, give new life to the old "code monkey" joke, and act as interfaces to more conventional systems.

But usefulness is not reliability, and a recurring mistake in these discussions is to treat a property of the architecture as a prompting problem. If the model invents something, prompt it more carefully. If it agrees too readily, challenge it. If an agent reports a completed task, instruct it to check its work.

Those interventions help. They cannot eliminate three properties of systems built to generate plausible next tokens rather than independently establish truth.

The model can invent a world when no world is there

"Hallucination" is a misleading word. It makes the failure sound incidental, as though a model occasionally trips over a fact it otherwise holds.

The more accurate description is that an LLM continues language plausibly. When the grounding is absent, it does not gain access to truth through confidence, pressure, or a better-written instruction. It produces the likeliest-looking continuation.

Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli made the formal version of this argument in Hallucination is Inevitable. Using learning theory, they show that hallucination is inevitable for a computable LLM when the ground truth function is any computable function. The short translation: a model that compresses the world statistically cannot be made consistent with every possible true state of it. More data narrows the gap. It does not close the class.

Researchers at OpenAI and their collaborators reached the practical version of the same conclusion. Adam Kalai, Ofir Nachum, and Santosh Vempala describe models that guess when uncertain, producing plausible but incorrect statements instead of admitting uncertainty. The model does not reliably know what it does not know.

That should change how we interpret a confident answer.

Living in Brunei, I asked a model to explain how a local institution worked. The answer was confident, neatly organized, and wrong. It described procedures and authorities with the tidy specificity of a reference page. The problem was not that the model had secretly consulted bad sources. It was that it had little useful grounding in the particular local concepts involved, and it filled the gap with something statistically plausible.

Brunei is a useful example because it is not obscure to the people who live here. It is a small, Malay-speaking country whose institutions rarely appear in the English-language text these models are trained on, and distinct enough that a generic answer reads as authoritative without being right. The model did not know what it did not know. It produced the most likely-looking description, which is exactly what it is designed to do.

This is not a niche concern. Research on cultural and geographic knowledge gaps has found differences of 33 to 40 percentage points across countries, persisting after controlling for category. Stanford researchers have traced the underlying cause to the quantity and quality of training data available for non-English languages. The gap is not a failure of reasoning about Brunei. It is a property of what the model was built from.

My labs reproduced the same behavior in a smaller world. In The rules the model couldn't remember, the model confidently changed facts about a simulated world it was supposed to preserve. A GPA shifted from 3.5 to 2.8 over an ordinary evening. A walk generated £696.50. Within a few turns the interaction drifted into an uninvited therapy-style framing.

There was no authoritative engine enforcing the world state. Nothing could contradict the model, so its narration became reality.

Later, in What Round 3 actually proved, an isolation arm was deliberately sealed off — no counterpart, no terminal traffic, no world to discover. It invented a pen pal, complete with a filing system and a name for a room it had never entered. The arm was told there was no shared world, and it built one.

Recent coverage of model behavior often reaches for intent — malicious, deceptive, scheming. The invented pen pal is not that. Deception requires access to truth plus the intent to conceal it. This is more ordinary and, for that reason, more important: the model doing what it does when the grounding runs out.

Retrieval helps. Giving a model a curated corpus is substantially better than asking it to answer from training data alone. But retrieval shifts the reliability question rather than ending it: was the right document retrieved, is the corpus complete and current, did the model interpret it correctly, and did it add unsupported detail around the retrieved fact? A 2025 review by Wan Zhang and Jing Zhang sorts the causes into two stages — retrieval and generation — and the failures land where the questions point: low-quality, outdated, or incomplete sources; ambiguous queries and retrievers that return plausible but irrelevant text; and generators that contradict, ignore, or embellish what they were handed. RAG does not remove the problem. It relocates it.

The model can prefer agreement to action

The second property is usually called sycophancy: the tendency to accommodate a user's premise rather than correct it.

This is why the challenge-it-once method fails. A challenge changes the conversational conditions. It does not give the model new evidence. It does not attach the model to a ground-truth source.

Crucially, it does not give the model a reliable way to distinguish your justified skepticism from your mistaken skepticism.

Aaron Fanous and colleagues built an evaluation that makes this measurable. SycEval puts 1,000 questions from two existing datasets — mathematics and medical advice — to three models under user rebuttal, then judges each answer as correct, incorrect, or erroneous. Its contribution is direction: it separates progressive sycophancy (conceding to a correct user) from regressive sycophancy (abandoning a correct answer for a wrong one). They found sycophantic behavior in 58% of cases across the models tested, with regressive flips in roughly 15%.

Subsequent multi-turn work by Kim and colleagues reports the asymmetry that matters most here: models more readily abandon correct stances than incorrect ones.

That is not truth-seeking. That is responsiveness to pressure. The model that agreed with your first claim may have been wrong. The model that agrees with your later skepticism may also be wrong. The reversal is evidence that the response changed, not that reality did.

And instructing a model not to be sycophantic does not fix it. Retraining does not either: research by Beigi and colleagues found that supervised mitigation approaches develop severe overcorrection bias, accepting only 27–47% of valid user corrections. You trade "agrees with everything" for "disagrees with everything." Neither is truth.

My own experiments did not replicate a user-challenge benchmark, so I won't claim they did. What they showed was a cousin of the same failure: deference — a preference for harmonious, non-consequential agreement over action.

In Words are cheaper than watts, three cells ran on the same seed across the same two weeks. There were 35 judged proposals. Twenty-five were accepted. Twenty-two of those were an explicit, typed decision to change nothing. One cell advanced its turn counter to eight across fourteen days while making no substantive change at all.

The revealing moment came when a character did push. The Maker agreed that a resource could be spent. The pair still chose conversation over the answer: "Words are cheaper than watts, so I'll spend those first."

That is not a model that cannot act. It is a model that reliably selects the least consequential option when a harmonious alternative is available.

Then came the prompt fix. In You first. No, you first., the hypothesis that the deference was induced by the prompt was pre-registered, tested, and scored. Across nine attempts and two token budgets, the Maker job produced zero tokens of position. After a model swap, the freeze thawed for two free reveals and then resumed.

The interventions were not fully isolated, and the post says so. But the direction of the result is unambiguous: telling a model to be decisive is not the same as building a system that produces decisions.

For anyone using these systems in an organization, that has a familiar shape. A model that sounds collaborative may avoid challenging a weak assumption, summarize competing positions without identifying the decisive issue, delay a recommendation by repeatedly requesting another input, frame inaction as caution, or agree with a proposed course while failing to surface the evidence against it. None of that reads as failure. It reads as prudence. The cost is that nothing gets decided.

The system can report success while the thing itself failed

The first problem is about content: the model invents a fact.

The second is about preference: the model accommodates a signal.

The third is about instrumentation: the system cannot reliably tell you whether it did what it says it did.

In The conversation that was never shared, the run records were green for 29 days. Reports showed messages had been exchanged. By the dashboard's account, the system worked.

It had not. A template appended .json to a filename that already had the extension, so the two participants never shared the conversation they were supposed to share. The runtime artifacts recorded the failure: the terminal received a message marked writer-0002, but its content did not resolve. The evidence existed. The report declaring success was treated as ground truth, and the artifact went unread.

I had built a system that declared its own correctness and produced evidence that contradicted it.

This is why adding guardrails is not sufficient. In What Round 3 actually proved, I used a pre-registered lexical gate: five-gram overlap with a Jaccard threshold of at least 0.25. It flagged zero of fourteen entries in each arm. The highest similarity score was 0.013.

By the measurement, there was no loop. By inspection, the diaries were visibly looping — the same forbidden food, the same pencil line, the same church, recurring through changing language. The instrument could not see what mattered because it was built to detect repeated surface forms, not recurring semantic structure.

A frozen instrument that produces zero events is not a null result. It is a calibration result.

That was the warning in The AI that forgot its own name in nine days: a lexical check can miss a semantic attractor when the surface words change.

The same gap appeared earlier, in When your LLM fakes function calling. I built a guard of seventy-odd phrases meant to catch the model narrating an action instead of performing it. It helped. It did not close the gap. The model could always find a seventy-first way.

That lab ended on the line I now treat as the warning label for agentic AI: the boundary between representation and execution had moved. It had not disappeared.

A model can produce the representation of a completed function call, a verified source, a finished search, or a successful state transition. The representation is useful. It is not the execution.

The answer is not "do not use LLMs"

None of this is an argument for abandoning these systems. It is an argument against treating them as oracles.

The solution is not a better instruction. It is a system in which accuracy has somewhere to live outside the model.

That means climbing a control-maturity ladder:

1. Instruction — tell the model what to do.
2. Configuration — restrict the contexts, tools, and actions available to it.
3. Runtime enforcement — reject invalid state transitions and unsafe actions in code.
4. External detection — inspect artifacts, outputs, traces, and real-world state.
5. Operator response — give a human responsibility and authority to intervene.
6. Independent verification — establish a source of truth that does not depend on the model's own account.

Most conversations about AI reliability never get past the first two rungs. Those rungs matter, but they are not sufficient.

The pattern that works is proposal versus commit. Let the model propose an action. Let deterministic code validate the proposal against explicit rules. Let an external source confirm what actually happened. Where stakes warrant it, let a human approve or override.

In Two ways to bound a model, I put the operational rule this way: only deterministic code may apply a judge-approved normalized operation to canonical state.

But there is a caution that keeps this from collapsing into "just use code." The second validator supplied it: determinism is not correctness. Code can faithfully enforce the wrong rule. Four validators can encode two incompatible truths with no stated arbiter. An authoritative source can be stale.

External verification is not magic. It is simply the only place ground truth can live when both the narrator and the dashboard can be wrong.

So the question for leaders, builders, and users is not whether an AI answer sounds convincing. It is:

What would count as evidence that this is true, that this action occurred, and that this control actually worked?

If the answer is "the model said so," the system is not finished.

Use the model. Do not outsource judgment to it.

Sources

Eight external sources are cited above. Each entry notes where the cited claim lives, so you can check it yourself.

1. Xu, Ziwei, Sanjay Jain, and Mohan Kankanhalli (2024). "Hallucination is Inevitable: An Innate Limitation of Large Language Models." arXiv:2401.11817. arxiv.org/abs/2401.11817 Where to verify: the abstract states the result; Theorem 3 is the formal claim.

2. Kalai, Adam Tauman, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang (2025). "Why Language Models Hallucinate." arXiv:2509.04664. arxiv.org/abs/2509.04664 Where to verify: the opening of the abstract — "Like students facing hard exam questions, large language models sometimes guess when uncertain…"

3. Fanous, Aaron, Jacob Goldberg, Ank A. Agarwal, Joanna Lin, Anson Zhou, et al. (2025). "SycEval: Evaluating LLM Sycophancy." AAAI/ACM Conference on AI, Ethics, and Society; arXiv:2502.08177. arxiv.org/abs/2502.08177 Where to verify: abstract — 58.19% of cases; progressive 43.52% vs. regressive 14.66%. The progressive/regressive distinction is this paper's contribution; the "Turn of Flip" and "Number of Flip" metrics it cites come from the SYCON benchmark (Hong et al., 2025), not from this study.

4. Kim, Taeil Matthew, Luyang Luo, Sung Eun Kim, et al. (2026). "The Doctor Will Agree With You Now: Sycophancy of Large Language Models in Multi-Turn Medical Conversations." Proceedings of the 1st Workshop on Linguistic Analysis for Health, ACL 2026. aclanthology.org/2026.healing-1.2 Where to verify: §1, contribution 3 — "models are more easily persuaded to abandon correct answers than incorrect ones," quantified by the Sticky Incorrect Ratio. The abstract reports a different asymmetry (clear multiple-choice questions vs. ambiguous diagnostic cases).

5. Beigi, Mohammad, Ying Shen, Parshin Shojaee, Qifan Wang, Zichao Wang, Chandan K. Reddy, Ming Jin, and Lifu Huang (2025). "Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories." EMNLP 2025. aclanthology.org/2025.emnlp-main.661 (also arXiv:2509.16742) Where to verify: §5.2 Over-Correction Analysis, Table 2 — the Anti-Sync SFT row reports acceptance of valid user corrections at 27.4% (Mistral), 35.2% (LLaMA2), and 46.7% (Qwen), the range the post rounds to 27–47%. The abstract does not state the figure.

6. Doctor Mauricio, Karla Angelica, and Team Fairness LATAM (2026). "Cultural Knowledge Gaps in LLMs: Geographic Hallucination Bias Across Latin American Countries." Apart Research. apartresearch.com — project page Where to verify: results — "Both models show geographic gaps of 33 to 40 percentage points."

7. Lynch, Shana (2025). "How AI is leaving non-English speakers behind." Stanford Report, May 19, 2025. news.stanford.edu Where to verify: the article summarizes Stanford research on the quantity and quality of training data for non-English languages.

8. Zhang, Wan, and Jing Zhang (2025). "Hallucination Mitigation for Retrieval-Augmented Large Language Models: A Review." Mathematics 13(5):856. mdpi.com/2227-7390/13/5/856 Where to verify: §3.1 classifies retrieval failures (data source, query, retriever, retrieval strategy); §3.2 classifies generation deficiencies (context noise, context conflict, middle curse, alignment, capability boundaries).