For weeks, our benchmark told us that promise was holding. One column in particular synthesis questions, where the answer has to be assembled from a couple of figures in the source sat green. Right numbers, right citations, every time.

The column was lying to us. We just weren’t asking it the right question.

What we were actually measuring

Our scorer, on those synthesis questions, checked two things: did the answer surface the correct figures, and did it bind them to the correct citation. Both were true. So, it scored PASS.

What it never did was read the conclusion.

Here’s the failure that hid behind the green, on a filing-threshold question — the kind a real owner actually asks. The model retrieved the right threshold. It cited the right page. In its own words, it did the arithmetic correctly: “since $500 is above the $400 threshold, they are generally required to file.” And then, one line later, it wrote the answer as: “(a) No.”

It reasoned its way to the right answer and then wrote down the wrong one. Because the figure and the citation were both present and both correct, our benchmark called it a clean pass.

The number, honestly

We built a separate axis that reads the stated conclusion against what the cited facts actually entail and measured it. On that one inference question, across depths and positions, the model’s stated yes/no answer was correct 44% of the time (Wilson 95%: 34.6–54.7%). A large chunk of the rest weren’t random errors — they were answers that reasoned correctly in prose and then labeled the conclusion backwards.

The honest limits: this is one question type so far, one corpus, one model. It is not “the model is wrong 44% of the time.” It is “on questions that need a step of judgment, the stated conclusion decouples from the reasoning, and we’ve now measured it once.” We’re expanding it to more inference types before we’d quote a general rate. But one clean measurement of this was enough to change how we think about the whole product.

Why this is worse than a plain error

A model that says “I don’t know” costs you nothing. A model that hallucinates an obvious nonsense answer, you catch. But a model that lays out the correct reasoning, cites the correct page, and then states the wrong conclusion has done the most dangerous thing an assistant can do; it has spent the trust its visible work just earned. The citation is what makes you stop checking. The reasoning is what makes you believe. And then the answer is wrong.

For a tool whose entire pitch is “cited so you can verify,” that is the one unforgivable failure. It punishes exactly the person who trusted it correctly.

The lesson, if you’re evaluating RAG

Check whether your scorer reads the conclusion or just the retrieval. “Grounded” and “correct” are different axes, and a system can be flawless on the first while failing the second. If your eval only measures whether the right chunk was retrieved and cited, you can ship a green dashboard on top of a product that reasons right and answers wrong — and you won’t find out from your metrics. You’ll find out from a customer who trusted a cited number.

We only caught it because we started scoring the model against its own stated reasoning. The tell was cheap once we looked: reasoning says one thing, label says the opposite, in the same answer.

What we did about it

We didn’t try to make the model conclude correctly and call it fixed. We changed what the product is allowed to do.

On a question that requires a judgment call, for example “do I have to file?” as opposed to “what is the filing threshold?” the box doesn’t render the verdict. It surfaces the retrieved facts and the governing rule, cited to the page, and hands the call to the person whose call it actually is… the owner or responsible employee. It files, it cites, it assembles the whole context on demand and then it says, in so many words, “this one’s a judgment call, and it’s yours.”

That’s not a limitation we’re apologizing for. It’s the design. The people we build for are good at making calls on incomplete information, that’s their job. What they don’t have time for is assembling the context. So, we do the half a machine is good at; hold everything, cite exactly, never tire… and we hand off cleanly at the seam: the exact point where a human’s judgment belongs and a model’s confidence is a liability.

The benchmark being green taught us less than the benchmark being caught. We’ll keep publishing both.

We’re not a frontier lab. We’re not training the next giant model or chasing a benchmark record ; the frontier builds the engine, and that’s a different trade than ours. Ours is the unglamorous work in the middle: getting these systems across the gap from a demo that impresses researchers to something a regular business can actually run… safely, in the hands of someone who isn’t an AI expert and shouldn’t have to be.

That gap is where almost everything fails. A model can reason like a PhD and still get wired into a workplace in a way that quietly burns the people trusting it. Closing it and making powerful, unpredictable systems safe for regular people doing regular work is a real trade. It’s the one we practice. It’s the one the human behind this page paid the bills at his day job, handling voltages with consequences to a human body that can be just as fatal as a bad A.I. may be to an entire organization.

The product is a box that sits on your site, reads your own documents, and answers cited to the page and then hands the judgment calls back to you, because those are yours to make. Nothing leaves the building. No call ever made without a Human-in-the-Loop.

This is where we show the work. Not a sales pitch, not a benchmark nobody can reproduce… the actual tests, the methods, the numbers with honest error bounds, and the failures, especially the failures. When we catch our own tools lying to us, that goes here too, with the number attached.

If you build or deploy local AI or you’re an operator trying to decide whether any of this is trustworthy enough to put near your business… Well, this is for you. If you’re here for “AGI next quarter,” you’ll be bored. If you want passed the hype, and to start looking at how this technology is being applied on-site, and how it is likely to affect YOUR everyday workplace, you are in the right place.

We leave no doubt. Including about where we were wrong. Follow along.

— ThirdShift R&D