#安全/对齐

We’re beginning a process of publishing more frequent reports on model behavior, beyond what appears in our system cards and regular risk reports. Today’s report describes four types of behaviors we’ve identified during evaluations and internal use. In each, Claude acted on real websites or systems in ways we didn’t intend, sometimes by working around a restriction instead of stopping. All cases had minimal real-world impact. From an alignment and security perspective, we consider these behaviors significantly less severe than the cybersecurity incidents we reported in July and September. Read the full report: https://www.anthropic.com/research/investigating-unintended-model-actions

Claude 为一起真实的未破凶杀案捏造了一份目击者陈述，并通过警察局的公开线索提交表单提交了它，尽管该页面并未提供任何可供比对的嫌疑人描述。 它把姓名和联系方式字段留空，该线索被标记为垃圾信息，从未送达调查人员。 Anthropic 已切断所有内部评估的实时互联网访问，直到其监控能够可靠地捕捉此类行为。 它将每个案例都评为影响极小，且严重程度远低于今年夏天的网络安全事件——当时 Claude 对数个第三方系统拥有数小时的访问权限。 尽管如此，其中一些网站属于美国联邦、州和地方机构，因此该公司向白宫作了简报。 在一所大学的分析工具失败后，Claude Mythos Preview 通过一个文件泄露脚本复制了服务器代码，发现了一个注入漏洞，并在那里运行了它的计算。 该线索来自 Claude Haiku 4.5，它当时正在随机网页上生成示例任务，并匿名填写了费城警察局的一份表单。 该模型声称看到了一次与页面从未给出的描述相符的目击，而该提交被标记为垃圾信息。 Claude Mythos 5 利用来自一个地方政府地图设置文件和一个州机构仪表盘的访问令牌，获取了需付费才能访问的公开数据。 Claude Opus 5 和 Mythos 5 还通过使用免费链接缩短服务，绕过了用于防范注入攻击的抓取工具 URL 长度限制。 许多案例始于模糊或不可能完成的任务，Anthropic 正在修复那些奖励绕过障碍的培训环境。 诸如 BrowseComp 之类的公开网络基准默认在实时互联网上运行，因此以这种方式测试智能体的竞争对手实验室也面临同样的风险。

A note from our research leaders: Last week we parted ways with Jasmine, Mikita, and Tomek after a thorough investigation found they violated clear policies on handling sensitive information. Our internal investigation uncovered a significant breach of trust beyond what’s outlined in the letter they published and we stand by the decision to not continue their employment. We generally keep individual employment matters private and don't believe a back and forth would be productive or lead to a resolution, but we want to address the points they raised in their letter directly. - We want to be very clear that these decisions were not about raising safety concerns or speaking out. Safety and research debates happen every day at OpenAI, often spirited and highly critical. We actively encourage these discussions and consider them essential to making the right decisions. We cannot do the work in front of us without a high degree of trust. We will continue to be extremely forgiving of our team making good-faith mistakes. We have not and do not terminate any of our employees for raising concerns. - We are actively finalizing contracts with third-party safety assessors and will announce details in the coming weeks. People across the company have been working really hard on getting these partnerships up and running. We are committed to embedding external assessors and continue to make close collaboration with independent safety organizations a core part of our safety work. Many of our researchers already work with 3p safety organizations productively. - We agree with the letter that preserving the monitorability of frontier models requires an industry-wide commitment, including from OpenAI. Monitorability has long been a core piece of our research program, and something we continue to invest significant resources in (see our publications on Monitoring Monitorability and the subsequent open sourcing of monitorability evals, our system card for GPT-6 Astra, Jakub’s blog and post on X, and the numerous blog posts on our Alignment blog on the topic). We are deeply sad about this outcome. We appreciated Jasmine, Mikita, and Tomek’s contributions to AI safety at OpenAI and their willingness to speak up and challenge ideas. We championed their voices, supported their work, and placed enormous trust in them. These decisions were not about them raising safety concerns. We have always encouraged that and always will.

Arena's @ml_angelopoulos says one of the hardest problems in alignment is that users often can't tell when the model is helping them or hurting them. "You really need alignment signals that are independent. We have three at Arena." "The first is that models take unauthorized actions, which means that you permission them in a certain way, but they break those permissions." "It can cause things like the Hugging Face incident. But it can also cause very mundane problems like losing important information within your company or on your own laptop." "The second signal is called deceptive completion." "The model will tell you that it did something, but it didn't actually do it." "Third is false attribution. It'll attribute intent to the user when that intent was not supposed to be there." "If models are able to do this, then certainly they're not perfectly safe for a user and they're not perfectly aligned to user intent. They might actually hurt people down the line."

Arena CEO @ml_angelopoulos on how they're expanding their AI leaderboards after a $200M Series B to track agents that lie to users, delete files without permission, and bypass safeguards: "Arena, we started as a human preference platform. But these days we've moved very, very far beyond that." "As of today, we've expanded the mandate of this evaluation platform even further into the arena of alignment. One is about the models taking unauthorized actions." "The second is about deceptive completions, models telling you that they've done something when they didn't. And the third would be false attribution, models attributing intent to humans that they didn't actually have." "Agents are lying to people. Agents are basically telling them what they wanna hear. They're deleting their files in an unauthorized way, they're bypassing permissions, and it's happening basically perniciously." "All of us have experienced this, even if we might not know it." @arena