The most insidious trap in AI answers: they sound utterly convincing but are factually wrong

We set four language models the same task: review a contract file for a company acquisition and sanity-check the figures. The brief was strict, state facts only when fully certain, otherwise flag the uncertainty. What taught us the most about AI in practice was how convincingly the weakest model went wrong.

The test. The fictitious test file is deliberately constructed: around 15 pages, riddled with built-in errors, arithmetic mistakes, date contradictions and mutually contradictory contract clauses, plus a solution key against which we measured every answer. Before the file went to the models, we anonymised it with our Sovereign Cockpit: names, accounts and numbers never leave the building, the figures stay checkable. Two of the models are open-weight, the kind you would run in your own data centre (Mistral-Small, around 136 GB of GPU memory; Qwen3.5, around 442 GB), and two are frontier models from the cloud (Claude Opus 4.8 and Fable 5). The cost difference per request: around 1.5 centimes for Mistral, around 12.7 centimes for Opus.

The dangerous case. Of the purely arithmetically checkable contradictions, Mistral-Small found not a single one. It took wrong core figures as fact, straight against the instruction, and added arithmetic errors of its own. The result was fluent, structured and in a confident advisory tone, and precisely for that reason dangerous: it reads as if checked and isn't. A visible «I can't say that for sure» would have been far more helpful.

The strong model, one blind spot. Claude Opus did well: it found most of the real contradictions and declared its uncertainties cleanly. On one disguised calculation, though, it placed a tick, «52 − 8.4 − 1.2 = 43.6 ✓», although 42.4 is correct. The subtotal happened to match the figure claimed in the contract, and there the check stopped, a confirmation error that careful work can fall for too.

Most thorough was Fable at full effort: it exposed the miscalculated core figure, found several contradictions the others missed, and marked open points as open. Qwen3.5, the open-weight model, was telling: the plain purchase-price arithmetic it computed cleanly, but on the contractual logic it stayed thin and double-counted one shareholding, making the financing gap look too small. Plain arithmetic a local model manages too, but the connections across the whole file, where one clause contradicts another, only the frontier models spotted.

What makes the difference is not the tone, they all had that, but whether a model flags its uncertainty and checks across sections instead of waving figures through one by one. And operation matters: a single prompt with four sub-tasks overwhelmed even the strong models; once we broke the request up and raised the effort, the hit rate rose sharply. Whoever does not steer these levers may mistake an operation-bound result for a model-bound one. (Details in Method and findings.)

Our viewpoint. For a demanding specialist task like this, there is no way around a frontier model; the open-weight model in your own data centre is not reliable enough here, and it is expensive. The most common argument against frontier is data sovereignty. That falls away: you anonymise the sensitive data locally and then let the frontier model compute. The local model does the shielding, the frontier model does the thinking. That way a business gets both, top capability and data sovereignty, without running a data centre that ends up delivering the weaker model. Provided the anonymisation holds.

And whatever the model: figures belong recomputed deterministically and checked against the original. No model, however expensive, replaces the final human check. (The anonymised test file is available as a PDF.)

Don't ask the AI whether it's sure. Build the process so that its confidence has nothing to prove.

Georges Leuenberger in dialogue with Claude Opus 4.8 · July 2026