If the “AI blackmail” thing sounds familiar, you may remember a story from last year in which OpenAI models edited shutdown scripts and threatened engineers in highly contrived and controlled scenarios. As we wrote at the time, “the danger isn’t AI systems sprouting intentions—it’s that we’ve created systems that can manipulate human psychology through language. … Written language doesn’t need consciousness to manipulate us. It never has; books full of fictional characters are not alive either.”
These days, Anthropic notes that perceptions of manipulation by AI models “could generate significant public concern, regulatory scrutiny, litigation risk, or reputational harm, regardless of whether such behavior results in actual harm.” What Anthropic doesn’t address in its S-1 is the possibility that AI models could manipulate us all in ways we don’t even realize. Maybe it’s happening right now. Spooky.
Give a damn ’bout my bad reputation
Throughout the S-1’s Risk Factors section, Anthropic mostly focuses on problems caused by its own AI models or business moves. But the company also acknowledges that problems with AI’s broader reputation could pose a threat, even if Anthropic does nothing wrong.
At one point, Anthropic imagines a hypothetical in which another company’s AI model “take[s] harmful actions independently and at scale without human intervention.” The “broad reputational damage to the AI industry, public backlash, and sweeping regulatory responses” to such an incident could damage Anthropic’s business even if Anthropic isn’t directly involved, the company writes.
Anthropic also uses the Risk Factors section to worry over the potential “public perception that our models, platform, and applications contribute to job displacement or economic disruption.” That doesn’t seem to be happening thus far in any significant way, but Anthropic says the mere idea could result in severe public backlash “regardless of the actual impact of our models … on employment.” Such a backlash could lead to “authorities [to] intervene to slow the rate of adoption of AI, restrict its use, or take other actions that increase the cost of developing, deploying, or commercializing AI systems,” Anthropic warns.
