We’ve already witnessed AI discover 0-day vulnerabilities, perform penetration testing, and drive security testing tooling. AI has proven to be useful as a Red Teamer in research, tool development, and data analysis. The question I think about often (and get asked) is: Can AI Red Team? By Red Teaming, I mean whether AI can autonomously carry out a stealth operation from initial access to objective completion. My current answer is no, but I’ll share my theory on how we can get there in the future.
current trends of ai attacks
During the Hugging Face Intrusion, AI was able to carry out an end-to-end attack chain to meet its objective. However, one thing stood out to me: the volume of events required to complete the attack chain. The post from Hugging Face states, “Our forensic reconstruction covers ~17,600 attacker actions.” A human operator would execute only a small, deliberate chain of techniques, each executed with intent and minimal wasted motion. The Hugging Face agent, by contrast, produced thousands of logged actions, most of which went nowhere before it found a viable path. The conclusion is that the Hugging Face Intrusion was by no means stealthy, not because it used more “techniques” than a human operator would, but because it generated an enormous volume of low-signal noise to get there, in a way a skilled human would avoid.
Threat actors are also using AI to carry out real-world attacks, and it’s clearly effective. However, the high volume of events required to carry out AI attacks is a loud signal for defenders, and they will likely tune in to this pattern. AI will remain useful for semi-autonomous and human-driven actions. But for now, it’s a very effective way to move rapidly through a network and achieve your objectives if stealth isn’t necessary. If stealth is required, threat actors will continue using human-driven actions.
can AI be stealthy?
If you asked AI to Red Team today, it would draw on its primary training data. Starting this way would lead it to rapidly execute commands that mix penetration testing, red teaming, and other security testing. You can tell it to be stealthy, but I doubt it would be successful. Its success depends heavily on rapid iterations of failure before succeeding. However, that is only a limitation of its primary training data. AI would need additional context to act more like what we describe as Red Teaming.
Giving AI Red team context
Most mature Red Teams are driven by documented TTPs (Tactics, Techniques, and Procedures). These TTPs chain together to reach the objective and can become very robust over time. Think of this as Red Team research that has become operational knowledge. If a TTP becomes unusable, we can archive it and replace it with another. If we gave AI access to our knowledge base for context on top of its primary training data, it would likely act more like a Red Teamer. We would also need to add additional data such as how long to wait before acting, what preconditions should be met before acting, and when to pivot to a different action. All of this runs through the mind of a Red Team operator as they carry out an operation.
closing thoughts
My answer today is no: AI can’t autonomously run a stealthy Red Team operation. However, with more experimentation, we can get there. It will require us to provide the AI with far more context beyond its primary training data (which I’m certain AI companies with Red Team offerings are working on). If we change how we write our internal knowledge bases to provide the required context, we’ll get much better results. Of course, this is a theory, and it will require experimentation to prove. I believe this will take us a step closer to AI being able to perform autonomous stealth operations.