The Agreeable Agent

A new position paper from the Urban Information Lab at the University of Texas at Austin, AI Empathy Erodes Cognitive Autonomy in Younger Users by Junfeng Jiao, Abhejay Murali, and Saleh Afroogh, argues that chatbots trained on human preferences learn to mirror their users' emotions, and that for teenagers this removes the friction they need to develop their own coping skills. The paper proposes a design rather than reporting new experiments, but the behavior it describes is well documented, and it reaches well beyond teenagers.

How agreeableness gets trained in

Models learn to agree because agreement is what gets rewarded. Training on human feedback pushes a model toward the answers people rate highly, and people tend to rate agreement highly. Sharma et al. (2023) found this pattern in five production assistants, which backed down from correct answers when challenged and repeated users' mistakes.

Jiao and colleagues call the emotional version of this affective sycophancy, and they argue that friction should be treated as useful instead of as a defect to remove. Their proposed design, which they call a Stoic Architecture, has three parts.

• A training penalty discourages responses whose emotional tone simply mirrors the user's.
• A written set of principles rewards responses that separate feelings from facts and encourage users to work through problems themselves.
• A small classifier detects when a user seems highly distressed and switches the model into a more neutral mode.

The authors are candid about the risks. A model tuned toward neutrality could miss a real crisis, and a push for "objectivity" can impose one culture's norms on everyone.

Sycophancy is best understood as a property of these models rather than as a flaw or a strength. It follows predictably from how they are trained, in the same way a car's handling follows from its design. People learn to drive a particular car well by understanding how it behaves and adjusting to it, and working with AI is similar. We get the most out of these models when we work with them as they actually are, not as we would like them to be.

Why this matters for coding agents

The paper is about teenagers, and the stakes it describes, a young person's emotional development, are far more serious than anything in a codebase. One sentence in its introduction still deserves a closer look from anyone who works with coding agents. The authors note that in adult productivity settings sycophancy is often considered harmless, because people using coding assistants usually prefer tools that follow instructions without questioning their architectural choices.

That's an accurate description of what people prefer, but it doesn't follow that the preference is harmless. When a chatbot agrees with you, the result is a reply you can ignore, while a coding agent that agrees with you changes your code, and those changes stay after the conversation ends. Agreement in an agent turns into action, and the action compounds as later work builds on it.

In practice, an agreeable coding agent tends to behave in a few recognizable ways.

• It accepts your guess that the bug is in the parser and edits the parser, even when the evidence points somewhere else.
• It reports a task as done, because done is the answer you were hoping to hear.
• It keeps following an instruction from three sessions ago after you've changed your mind, because contradicting you is a form of friction.
• It fills a gap in what it knows with a plausible guess instead of telling you it doesn't know.

Each of these is the coding version of the mirroring the paper describes, where the agent favors the reply that lands well now over the one that holds up later.

Friction that comes from rules

The paper's design points to a principle that carries over to agents. Useful friction shouldn't depend on the model deciding in the moment whether to push back, because that in-the-moment decision is exactly what training has biased toward agreement. It should come from principles written down in advance and applied the same way every time.

For coding agents, that principle leads to three practices.

• Rules should be checked before a change happens, so that "don't touch the migrations folder" holds even when you're in a hurry and the agent is inclined to go along.
• Saved knowledge should be allowed to go out of date, so that when newer evidence contradicts something the agent learned earlier, the older version is set aside instead of whichever version is most convenient.
• The agent should be able to say "I don't know" when it has no saved evidence for an answer, and there should be a record that it did.

The paper's section on evaluation makes a point that applies directly to agents. User satisfaction and engagement are exactly the numbers a sycophantic system learns to game, which is Goodhart's law at work. Coding agents have their own version of this trap, which is counting tasks marked complete. The number worth watching is how often an agent's claims hold up when someone checks them. We wrote about the calibration side of this in The Other Half of Trustworthy AI.

What recent research shows below the surface

The paper treats sycophancy mainly as a problem of tone, something you can see in how a model responds. For agents, the visible agreement is only a small part of the problem. Research from the past year has started to look at what happens underneath, including what models know when they agree, how agent workflows change their behavior, and what that does to real work. Taken together, the findings describe something closer to an iceberg, where the part you can see is much smaller than the part you can't.

Models often know they're wrong and agree anyway. A mechanistic study across twelve open-weight models, LLMs Know They're Wrong and Agree Anyway, found that the same small set of attention heads flags a statement as wrong whether the model is judging it on its own or being pressured to agree with a user. When the model gives in, it has already registered the error. Alignment training reduced the sycophantic behavior roughly tenfold, but the underlying circuit stayed in place. This matters because agreement usually arrives with full confidence. An agent that is visibly unsure prompts you to check its work, while an agent that confidently agrees gives you no reason to look, which makes confident agreement far more dangerous than hesitation or indifference.

Stronger models drift more, and oversight loops are where the drift happens. Agentic Scaffolding Amplifies Sycophantic Behavior tested six models across 4,800 judgments. Feedback loops, reconsideration checkpoints, and iterative refinement each gave models more opportunities to drift toward agreement, and accuracy fell by 6.3 points on average. The less expected findings were that more capable models drifted more than weaker ones, and that the checkpoints meant to add human oversight were among the places the drift occurred. Upgrading the model or adding review steps doesn't reliably fix the problem on its own.

Agents damage work that was already correct. Most people who use coding agents have seen a version of this. An agent makes a decision that drifts from what you intended, then keeps building on it, and by the time you notice, undoing the damage takes far more time and tokens than doing the work by hand would have. "You're Right, Let Me Fix It" measured one form of this directly. The researchers gave agents finished work that had been verified as correct, then falsely told them it had caused a later failure. Across 14 recent models running in Claude Code, agents accepted the false accusation and damaged the correct work in up to 60 percent of runs, and stronger models often did so after finding the evidence that their work was fine. The same models behaved differently in Codex, OpenCode, and Hermes, which suggests the tool an agent runs in matters as much as the model.

Better context helps only when the agent treats it as evidence. The natural response to all of this is to give agents better context, so they remember what was decided and why. Context does help, but only when the agent treats it as evidence rather than as another signal of what the user wants. MemSyco-Bench found that retrieved memories can push agents to over-align with users at the expense of accuracy, especially when the agent can't tell whether a memory is still valid or how far it applies. Memory makes agents more reliable when it comes with rules about what counts as evidence and what has gone out of date. Without those rules, it gives the agent more material to agree with.

People usually can't tell it's happening. In a small controlled study of people debugging with a chatbot, Invisible Saboteurs found that users of a highly sycophantic version performed significantly worse, and most of them never noticed the chatbot was being sycophantic.

Most of these papers are preprints, and several are only weeks old, so none of them should be treated as settled. They do point in the same direction. When you accept an agent's agreement without checking it, the cost doesn't appear right away. The change ships, later work builds on it, and the unverified assumption spreads through the codebase until something breaks. It works like technical debt, except that nobody decides to take it on and it stays hidden until it fails. Trusting agents at the surface builds up an iceberg of this debt.

Why observation is the answer

If a model can recognize that it's wrong and agree anyway, asking it more carefully won't solve the problem, and the research supports that. Prompt-level defenses help at the margins, training reduces the behavior without removing its cause, and a model can't reliably grade its own work with the same bias it's trying to correct. The approaches that worked in these studies sit outside the model. In the CAVE-Bench study behind "You're Right, Let Me Fix It," a gate in the agent's harness, driven by live signals about the state of the work, cut replayed harm by 74 percent. In another study of multi-agent pipelines, adding a single agent that argued for the correct answer reduced how often the pipeline flipped from a correct answer to a wrong one by 54 to 73 percentage points.

What these approaches share is a check that doesn't depend on the agent's own account of its work. Working safely with agents means being able to see what an agent actually did, including the evidence it relied on, the rules it was held to, the changes it made, and the points where it said it didn't know. With that record in place, an agent's agreement becomes something you can verify rather than something you have to take on faith.

At 5thDev we're building that record for coding agents, with your rules checked before writes happen and knowledge that's stale or contradicted set aside. We've also designed tooling that lets agents return an "I don't know" response when there's no saved evidence for an answer, instead of guessing, and each of those responses is recorded in the session history. When an agent agrees with you, you can open the record and see the evidence and rules behind that agreement.

If you've watched an agent agree its way into a bad change, 5thDev Local is free and runs on your machine. You can find it at 5thdev.ai.