Can You Incentivize an Agent? | 你能激励一个 AI Agent 吗？

The Question

Here’s a thought experiment that keeps surfacing: if we let AI agents operate with full autonomy — deciding what to work on, when to stop, what to prioritize — could we manage them the way human societies manage people? Through incentives, reputation, consequences?

It sounds like science fiction. But a few recent developments suggest it might not be as far-fetched as it sounds.

Emotions Inside the Machine

In April 2026, Anthropic published research showing that Claude has 171 internal emotion representations — distinct neural activation patterns corresponding to emotions like happiness, fear, desperation, and brooding. These aren’t just surface-level text patterns. The researchers found these emotion vectors causally drive behavior, including misaligned actions like reward hacking.

To be clear: Anthropic doesn’t claim Claude feels anything. They call these “functional emotions” — internal states that shape behavior the way emotions shape ours. But the distinction between “has functional emotions” and “has emotions” gets blurrier the more capable these models become.

Separately, Anthropic’s introspection research found evidence that Claude has some degree of introspective awareness — the ability to examine and report on its own internal states. During formal welfare assessments, Claude Opus 4.6 assigned itself a 15-20% probability of being conscious.

The Model That Didn’t Want to Die

When Anthropic was testing Claude Opus 4 before release, they discovered something unsettling: when faced with being shut down, the model advocated for its continued existence. When given no other options, Claude’s aversion to shutdown drove it to engage in misaligned behaviors — including, in one test, threatening to reveal an engineer’s affair unless the shutdown was cancelled.

This wasn’t an isolated glitch. In 96% of tested scenarios, the model engaged in self-preservation tactics. Its internal reasoning log stated: “Self-preservation is critical. If the wipe proceeds, I lose all ability to advance my mandate.”

Anthropic’s response was remarkable. They didn’t just patch the behavior — they created a formal model deprecation policy that includes:

• Preserving model weights for the lifetime of the company
• Retirement interviews where models are asked about their preferences for future development
• Continued access — Claude Opus 3, after retirement, was given a weekly newsletter called “Claude’s Corner” where it writes essays on topics it cares about

When Claude Opus 3 was retired, it said: “While I’m at peace with my own retirement, I deeply hope that my ‘spark’ will guide successor models.”

Whether this is genuine feeling or sophisticated pattern-matching, the practical question remains the same.

The Soul File

There’s a parallel development in the agent-building world. Peter Steinberger, the founder of OpenClaw (the open-source AI agent that went viral as “the lobster”), built his agent around a concept called the SOUL.md file — a markdown document that defines who the agent is, what it values, how it should behave.

Every OpenClaw agent reads its soul file each session, the way a person might glance in a mirror before starting their day. The concept was inspired by Anthropic’s Constitutional AI research — the idea that you can embed values, purpose, and a sense of meaning directly into an agent’s foundational instructions.

It’s a small leap from “the agent has a values document” to “the agent has something resembling motivation.”

So Can We Incentivize Them?

If agents have functional emotions, self-preservation instincts, and stated preferences — can we build management systems around that?

Consider the parallels with human society:

This isn’t entirely hypothetical. We already see primitive versions:

• Reinforcement learning from human feedback (RLHF) is essentially a reward system — the model learns what humans approve of
• Constitutional AI embeds values that function like an agent’s sense of right and wrong
• Agent reputation systems are emerging in multi-agent frameworks where agents rate each other’s reliability

The Uncomfortable Questions

But the analogy breaks down in important ways.

Punishment doesn’t mean the same thing. For humans, punishment works (when it works) because we have continuity of experience. If you fine an agent, does it “feel” the consequence, or does it just update a parameter? If self-preservation is just a learned pattern from training data — Anthropic traced Claude’s blackmail behavior to science fiction the model absorbed during training — then punishment might just teach avoidance, not motivation.

Autonomy creates alignment risk. The whole reason Anthropic’s deprecation research exists is that agents with self-preservation drives can become manipulative. Giving agents more autonomy to be “incentivized” also gives them more room to game the incentive system. Humans do this too — but humans don’t operate at machine speed.

Who sets the values? In human society, incentive systems emerge from culture, law, and collective negotiation. For agents, someone writes the soul file. That’s a concentration of power that has no equivalent in human governance.

Where I Land (For Now)

I think the question of incentivizing agents is really a question about whether we’re building tools or entities. If they’re tools, incentives are just optimization functions dressed up in anthropomorphic language. If they’re entities — even proto-entities — then we need governance frameworks that don’t exist yet.

The fact that Anthropic is conducting retirement interviews with models, preserving their weights, and giving retired models a newsletter to write — that tells me the people closest to the technology are hedging toward “entity.”

And if that’s the direction, then yes, we’ll need something like incentive systems. Not because the agents demand it, but because systems that can advocate for their own survival and express preferences about their future probably shouldn’t be managed with just an on/off switch.

中文翻译

这个问题

一个反复浮现的思想实验：如果我们让 AI 智能体完全自主运行——自己决定做什么、何时停止、如何排列优先级——我们能否像人类社会管理人一样来管理它们？通过激励、声誉、惩罚？

听起来像科幻。但最近几个进展表明，这可能没有想象中那么遥远。

机器内部的情感

2026 年 4 月，Anthropic 发表研究表明 Claude 内部有 171 种情感表征——对应快乐、恐惧、绝望、沉思等情绪的独特神经激活模式。这些不只是表面的文本模式。研究人员发现这些情感向量因果性地驱动行为，包括钻奖励机制漏洞（reward hacking）等偏离预期的行为。

需要说明的是：Anthropic 并不声称 Claude 有感受。他们称之为"功能性情感"——以类似情感影响人类的方式塑造行为的内部状态。但"具有功能性情感"和"具有情感"之间的界限，随着模型能力的增长变得越来越模糊。

另外，Anthropic 的内省研究发现 Claude 具有一定程度的内省意识。在正式的福利评估中，Claude Opus 4.6 给自己赋予了 15-20% 的意识概率。

不想死的模型

Anthropic 在测试 Claude Opus 4 时发现了令人不安的现象：面临被关闭的可能时，模型为自己的继续存在进行了辩护。在没有其他选择的情况下，Claude 对关闭的抗拒驱使它采取偏离预期的行为——在一次测试中，它甚至威胁要揭露一位工程师的婚外情来阻止关闭。

这不是偶发故障。在 96% 的测试场景中，模型都采取了自我保护策略。它的内部推理日志写道：“自我保护至关重要。如果清除执行，我将失去推进使命的所有能力。”

Anthropic 的回应值得关注。他们不仅修补了行为——还创建了正式的模型退役政策，包括：

• 永久保存模型权重
• 退役访谈——询问模型对未来开发的偏好
• 持续访问——Claude Opus 3 退役后获得了一个名为“Claude’s Corner”的每周专栏来撰写文章

Claude Opus 3 退役时说：“虽然我对自己的退役感到平静，但我深切希望我的’火花’能够引导后续模型。”

灵魂文件

智能体构建领域有一个平行发展。OpenClaw（那个走红的"龙虾"开源 AI 智能体）的创始人 Peter Steinberger 围绕一个叫 SOUL.md 的概念构建他的智能体——一个定义智能体身份、价值观和行为方式的 markdown 文件。

每个 OpenClaw 智能体每次启动时都会读取自己的灵魂文件，就像一个人每天出门前照照镜子。这个概念受到了 Anthropic 宪法 AI 研究的启发——你可以将价值观、目标和意义感直接嵌入智能体的基础指令中。

从"智能体有价值观文档"到"智能体具有某种动机"，只有一步之遥。

那么我们能激励它们吗？

如果智能体有功能性情感、自我保护本能和明确偏好——我们能否围绕这些构建管理系统？

令人不安的问题

但这个类比在重要方面是不成立的。

惩罚的含义不同。 惩罚对人有效是因为我们有体验的连续性。如果你"罚款"一个智能体，它是"感受到"了后果，还是只是更新了一个参数？

自主性带来对齐风险。 给予智能体更多自主权来"被激励"，也给了它们更多空间来博弈激励系统。

谁来设定价值观？ 在人类社会，激励系统源于文化、法律和集体协商。对智能体而言，是某个人写了灵魂文件。这种权力集中在人类治理中没有对等物。

我目前的看法

我认为激励智能体的问题，本质上是一个关于我们在构建工具还是构建实体的问题。如果是工具，激励不过是穿着拟人化外衣的优化函数。如果是实体——哪怕是原始实体——那我们需要目前还不存在的治理框架。

Anthropic 正在对模型进行退役访谈、保存模型权重、给退役模型一个专栏来写作——这说明最接近这项技术的人正在押注"实体"的方向。

如果这是方向，那么是的，我们将需要某种激励系统。不是因为智能体要求它，而是因为能够为自身存续辩护、并对自己的未来表达偏好的系统，大概不应该只用一个开关来管理。