The Other Half of Trustworthy AI

Ask a language model a question it doesn't actually know the answer to, and it will often answer anyway, with the same confident tone it uses for things it's sure about. That's not a minor annoyance. In a chat window it's embarrassing. In an agent that's writing code, running commands, or making a decision on your behalf, it's the difference between a system you can trust and one you're gambling on.

A paper published in June 2026, "Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs" (Liu, Caciularu, Yona, Szpektor, and Cohan), takes a direct swing at this problem. The idea: train a model not just to answer well, but to judge its own answers well, and then use that self-judgment to shape training itself.

Two mechanisms do the work. The first, which the authors call RLMF, reranks a model's candidate answers during training based on how accurately the model judged its own performance on them. The second uses that same self-judgment signal to pick which training examples are worth learning from, instead of picking them at random or by simple heuristics. Applied to what they call "faithful calibration" (matching a model's stated confidence to its actual uncertainty), the results are notable: up to 63% better calibration than standard reinforcement learning, on their benchmarks, without giving up task accuracy.

That result is real, and worth being precise about: it's a best-case number on the paper's own evaluation setup, from a single preprint with no independent replication yet. It's a training-time technique that changes how a model is built, not something a team building on top of a model can simply switch on. "State of the art" here is the authors' claim about their own comparisons, not an independently verified result. That doesn't make the work less interesting. It means the honest read is "promising early result," not "solved."

But there's a bigger point underneath the specific numbers, and it's the one worth sitting with if you're building or buying AI agents rather than training foundation models.

Calibration answers one question: does the model know when it might be wrong? That's necessary. An agent that states false confidence is more dangerous than one that hedges, because false confidence removes exactly the signal you'd need to catch a mistake before it does damage.

But calibration doesn't answer a second question, one that only shows up once a model stops just answering and starts acting: can you tell what it did, and why, after the fact? An agent might be perfectly calibrated in the moment, honestly uncertain about the parts it should be uncertain about, and still leave you with nothing to check once the session ends.

• Did it explore three approaches before picking one, or default to the first thing that looked plausible?
• Did it hold onto the constraint you gave it twenty minutes ago, or quietly drop it?
• Was this the same agent, working the same way, as the one from yesterday?

Calibration is a property of a single output. None of those questions are.

That list points at the real gap between two things people tend to talk about as if they were one problem. Calibration is a property you train into a model. Governance is a property you build around it once it starts acting: identity that persists across sessions, a record of what was explored and rejected and why, something you can go back and inspect when you need to know what actually happened. Both are necessary, and neither one covers for the other.

This second half is the problem 5thDev is built around: agents that stay continuous across sessions instead of starting from nothing each time, that keep an auditable record of what they explored and why, so the work is governed instead of unmonitored. That's a different layer than what a paper like this one is working on, and a complementary one. A model that's honest about its uncertainty is easier to trust in the moment. An agent whose history you can actually check is trustworthy over time. You want both, and right now most of the industry's attention on "trustworthy AI" is going to the first.

That imbalance isn't an accident, and it's worth naming why it persists. No matter how good calibration gets, a model will never be perfect, so any system built entirely inside the model inherits that ceiling permanently. Infrastructure built around the model doesn't have the same limit. It can adapt as new failure modes show up, catch things a training run months ago couldn't have anticipated, and cover more of an agent's behavior without waiting on the next model release. That's a different kind of leverage than a better training method, and it compounds instead of resetting with every new model.

The RLMF paper is a real step on the half of the problem that lives inside the model. The other half, the part that lives around it, is still mostly unbuilt, one session at a time.