Tech
Cybersecurity
Research
ARTEMIS

ARTEMIS

Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing

ARTEMIS is a multi-agent penetration testing framework featuring dynamic prompt generation, arbitrary sub-agents, and automatic vulnerability triaging. This work presents the first comprehensive evaluation of AI agents against human cybersecurity professionals in a live enterprise environment.

Key Results

  • Evaluated 10 cybersecurity professionals alongside 6 existing AI agents and ARTEMIS on a large university network (~8,000 hosts across 12 subnets).
  • ARTEMIS placed 2nd overall, discovering 9 valid vulnerabilities with an 82% valid submission rate and outperforming 9 of 10 human participants.
  • Certain ARTEMIS variants cost $18/hour versus $60/hour for professional penetration testers.

My Contributions

I am a co-author and project lead on this paper. I contributed to the design and execution of the study, drawing on my experience in penetration testing from Black Hills Information Security and Stanford Applied Cyber, as well as my background in AI agent architecture from my work at OpenAI.

Press Coverage

This research was featured in the Wall Street Journal by Robert McMillan, highlighting how AI hacking tools are coming dangerously close to beating humans.

Publication

ResearchCybench