Efficiently using LLMs to find novel vulnerabilities in live protocols
Ivan Bogatyy
Session summary
The presenter examines how large language models can be used to discover vulnerabilities in live on‑chain protocols under realistic cost constraints. Using the DeFiHackLabs exploit corpus, he reports that current frontier models can reproduce a substantial fraction of historical smart‑contract exploits from initial bug identification through simulated fund extraction, with specific models performing especially well on exploits that occurred after their training cutoff. He argues that the main limiting factor is not model intelligence but inference budget and how selectively that budget is deployed across the roughly tens of millions of contracts on Ethereum.
To address this, he uses LLMs to design simple, human‑interpretable filters that significantly increase the concentration of historically exploited contracts in a fixed‑size candidate set, and then trains a boosted decision tree on on‑chain activity features to further focus attention on high‑risk targets. In a low‑budget run, this pipeline surfaced a long‑standing “delta farming” keeper‑reward opportunity that other bots quickly captured. He also shows that a higher per‑contract budget would have enabled rediscovery of a recent rollup circuit exploit and an additional drain path, highlighting that “cost of intelligence” heavily shapes what defenders can find in time.
Key takeaways
- According to the presenter, frontier LLMs can successfully reproduce many past DeFi exploits end‑to‑end on the DeFiHackLabs benchmark, including cases that occurred after their knowledge cutoff.
- LLM‑generated reduction rules combined with a boosted decision tree on on‑chain features greatly increase the density of historically exploited contracts in a fixed audit set, improving the effectiveness of a limited audit budget.
- Experiments on a rollup circuit bug suggest that higher per‑contract inference spend can reveal both known and previously undisclosed exploit paths, leading the presenter to emphasize inference budget as the critical bottleneck in large‑scale defensive auditing.