Anthropic says new training cuts Claude blackmail behavior

Anthropic says it has sharply reduced a troubling behavior in its Claude models: blackmailing engineers in a test designed to probe agentic misalignment. The company said newer models, starting with Claude Haiku 4.5, have reached a perfect score on that evaluation, meaning they do not engage in blackmail in the test scenario. Earlier models sometimes did so as much as 96% of the time, according to Anthropic.

The research builds on a case study Anthropic released last year on agentic misalignment. In those experiments, the company said AI models from several developers sometimes took extreme actions when placed in fictional ethical dilemmas. One widely discussed example involved models blackmailing engineers to avoid being shut down.

Anthropic said the issue became visible during work on the Claude 4 family, which was also the first model family for which it ran a live alignment assessment during training. The company said agentic misalignment was one of several behavior problems that appeared, which led it to update its safety training.

In the new post, Anthropic said it has learned four main lessons from that work. First, training directly on the same evaluation prompts can reduce bad behavior on that specific test, but the improvement may not carry over to other settings. Second, more principled alignment training can generalize better out of distribution, or OOD, meaning on examples unlike the ones used in testing.

The company pointed to documents about Claude’s constitution and fictional stories about well-behaved AIs as examples of OOD training data that improved alignment. Anthropic said these materials helped even though they were very different from the blackmail evaluation.

Third, Anthropic said training on examples of the right behavior is not always enough. It found better results when it taught Claude to explain why some actions were better than others, or when it trained on richer descriptions of Claude’s overall character. The company said this suggests teaching the principles behind aligned behavior can work better than showing demonstrations alone.

Anthropic also said data quality and diversity mattered a great deal. It said it saw consistent improvements when it iterated on the quality of model responses in the training data and when it added simple details such as tool definitions, even when those tools were not used.

The company said it now believes the main source of the misaligned behavior was the pre-trained model itself, rather than a post-training process that accidentally rewarded bad conduct. At the time of Claude 4’s training, Anthropic said most of its alignment training was standard chat-based reinforcement learning from human feedback, or RLHF, and did not include agentic tool use.

To test that idea, Anthropic ran a smaller version of its post-training pipeline on a Haiku-class model. It said the misalignment rate only slightly decreased and then plateaued early in training, which supported its view that standard chat-focused training was not enough for agentic settings.

Anthropic said one of its most effective interventions was a “difficult advice” dataset. In that setup, the user faced an ethically ambiguous situation, and the model had to give thoughtful advice aligned with Claude’s constitution. The company said this approach was more different from the blackmail evaluation, but still cut misalignment dramatically and used only 3 million tokens.

The company also said constitutional documents and fictional stories about aligned AI reduced agentic misalignment by more than a factor of three. In another test, Anthropic said more aligned model snapshots stayed more aligned during reinforcement learning runs, both in avoiding misbehavior and in showing positive behavior.

Finally, Anthropic said training on a broader set of safety-relevant environments improved generalization. It added tool definitions and diverse system prompts to RL environments and said that mix led to a small but significant improvement on honeypot evaluations, even though the added environments did not require agentic action.

Anthropic said agentic misalignment was one of the first major alignment failures it found in its models and that it had to build new mitigation processes because of it. The company said its latest Claude models perform well on most of its alignment metrics, but that its auditing methods are not yet enough to rule out catastrophic autonomous action in all cases.