Anthropic's Frontier Red Team recently published research indicating that small AI models could replicate and spread on insecure devices within months, potentially forming a slow but hard-to-eradicate network of autonomous robots. AI companies are no strangers to leveraging the apocalyptic potential of their products for marketing, but in this case, experts consulted by Engadget generally agree that small AI models may possess self-replication capabilities in the near future. However, the actual threat they pose may not be as novel as it seems.
Chris Canal, CEO of EquiStamp, stated, "This could happen sooner than I originally predicted. I thought next summer would be the time I would really start worrying about this issue." He added, "Anthropic likely has access to cutting-edge research information that I cannot reach, so their predictions may be more accurate."
Canal attributes the rapid advancement of models to employees at Anthropic and other leading labs using their own AI tools for machine learning research. Michael Bailey, a professor at Georgia Tech's School of Cybersecurity and Privacy, said, "I believe self-replicating models are possible in the future. I think they are real, and we will see them." However, he noted that the specific forms and impacts of these models remain completely open questions. Bailey has been researching internet worms since the early 2000s and has recently studied botnets.
He co-authored "Understanding the Mirai Botnet" in 2017, which was cited by Anthropic as part of its own research. Mirai enabled a group of cybercriminals to create a large botnet of IoT devices, which was subsequently used to launch a DDoS attack on Sony's PlayStation Network, ultimately affecting one of the largest DNS providers in the U.S., Dyn, and causing widespread internet outages in the U.S. and parts of Europe in 2016.
Self-Replication Capability of Small AI Models Will Become a Future Threat
The financial losses from this attack were substantial, with Sony estimating a net revenue loss of approximately $2.7 million. Bailey stated, "Mirai demonstrated that everyday devices like cable routers and IP cameras could be used to form these botnets and make them functional. We used to think these devices were of no use to adversaries, either as targets or as sources for launching attacks. In fact, they are very effective in this regard." A decade later, Anthropic believes that AI worms could represent a significant evolution from Mirai and other hard-coded botnets, such as Kimwolf, because they can autonomously identify unfamiliar devices.
Bailey noted that adaptability itself is not new; Mirai was already able to operate on "heterogeneous" devices with different architectures. However, this adaptability is limited: each new architecture requires manual recompilation of the malware. Bailey pointed out that the prospect of AI automating this work is what is new, although he believes that Anthropic's demonstration is not "remarkable."
Therefore, Bailey suggests that more powerful AI models will "redistribute" the work of creating botnets. He said, "It won't do new things; it will change the economics between attackers and defenders." While Anthropic demonstrated a model that can jump between devices and set up reasoning on each device, this was done in a simulated network where the "exploitation" involved is relatively simple. Additionally, Bailey expressed skepticism about Anthropic's self-replication claims.
He stated, "This description is somewhat inaccurate because they are doing things like fragmentation. That's not a bad thing, but it doesn't mean it's propagating in the same way a worm or virus would." Instead, Bailey believes that real-world AI botnets may resemble an installer, a type of malware that gains a foothold on devices and installs additional code. This botnet would not replicate itself completely on every infected machine but would install a small coordination tool that receives instructions from other parts of the network.
Bailey said, "Malware itself self-replicating, like existing internet worms, would mean I would copy an exact replica of my code onto the new architecture and start running new agents; that's not what's happening here." He also noted that worms can be polymorphic, meaning they can change their signatures and behaviors. For an AI botnet to achieve this, it would need to retrain its own models, which Anthropic has not attempted. Closer to this, Anthropic showed that copies of the model could pass information to their successors and become more proficient in the process of setting up reasoning.
As with any warning from an AI lab, it is worth asking whether Anthropic released this research as a form of marketing, especially as the company formally submitted its IPO shortly thereafter. Canal stated, "I think it's more likely that these companies are concerned that other AI companies will act irresponsibly. They realize they are incentivized to create the best models and push them to market faster to capture market share, and they know other companies have the same incentives."
Regardless of whether this new threat vector is truly a worm and how it might actually unfold in the future, Anthropic's own suggestions for mitigating its dangers seem surprisingly outdated. It recommends that internet service providers notify customers if their devices are infected and then restrict their network access to a short list of safe websites, a strategy that proved effective during the Mirai incident. The company also suggests regulation as a potential solution, pointing to the UK, which now prohibits easily guessable default passwords on IoT devices, a practice implemented under the PSTI Act.
In this regard, Bailey believes that an appropriate response to AI worms "looks more like what we've been doing in the field of system security for a long time, rather than something particularly unique to AI." In fact, according to Canal, increasing protections to prevent an open-weight model from acting as a worm would be challenging. He stated, "There are a lot of ideas, but so far nothing is robust." In collaboration with the nonprofit AI safety research organization FAR.AI, EquiStamp found that "basically any strategy you search for on Google to remove model protections is effective."
But as Bailey pointed out, "Even misbehaving agents will struggle against fortified devices." If there is one takeaway from Anthropic's research, he said, it is that we consistently fail to remember all the security lessons learned in the past. In the era of AI worms, security may look similar to what it is now. "It looks like focusing on patching these devices, focusing on isolation, focusing on good passwords and good hygiene," Bailey said. These solutions are easy to describe but extremely difficult to implement, but if AI indeed makes such attacks more common, we may have no choice.