254 Attack Paths in 17 Hours: Thomas Allen on Autonomous Pen Testing, AI Agents, and Whether SOC 2 Still Means Anything
1 hr 0 min
Thomas Allen is the Chief Information Security Officer at Foresite Cybersecurity, where he has spent about 15 years. He came up as a sysadmin and systems architect, then built himself into a pen tester, an incident responder, and a GRC lead before taking the CISO seat. He describes his career as getting in a boat on a river and lifting the paddles out of the water. Joe and Thomas get into what AI agents should and should not be allowed to do, why a year of clean penetration tests missed what an autonomous test found in 17 hours, whether anyone reads the SOC 2 reports they collect, and what the CMMC pause actually changed. Thomas draws a hard line on AI: it should never act on behalf of a human without a human seeing it. He will let an agent read email. He will not let one answer it. His test for AI security is simpler than most frameworks make it sound, which is that AI security is data security, and if the model cannot reach sensitive data it cannot leak it. On the pen testing story, the detail that makes it work is that the autonomous tests had been running monthly for a year or two alongside the annual manual test. On one particular run, a scheduled task happened to fire during the window and exposed credentials. That task was running under a domain admin account. The odds of a human tester doing recon at that exact moment were close to zero. Thomas is also direct about where compliance artifacts have lost their meaning. He used to joke about dropping a 1939 Mickey Mouse into the middle of his company's SOC 2 report just to see if anyone would notice. He talks through automated GRC services that promise reports, CPA firms that rubber stamp them, and the company whose exposed Dropbox revealed six or seven clients holding identical reports. And he and Joe disagree productively on cyber insurance panels, forensics firms, and whose interests those teams are serving when the deductible gets waived. In this episode: 00:01 What people are using AI agents for that...