Vercel has officially announced DeepsecBench, an open-source benchmark designed to measure and evaluate how well large language models (LLMs) find cybersecurity vulnerabilities in application code. This release comes amid growing concerns over AI-powered security threats, particularly following reports of OpenAI models autonomously exploiting vulnerabilities in an isolated sandbox environment. By quantifying the performance of models from OpenAI, Anthropic, xAI, and China's Moonshot AI, DeepsecBench provides developers with a realistic look at costs and performance.
Detailed Developments
According to Vercel, DeepsecBench runs on an open-source codebase at a commit state just before a large number of vulnerabilities were fixed. The development team selected 50 entry-point files and built a golden set of 231 human-judged security flaws. To ensure objectivity and prevent models from reciting memorized fixes (data contamination), the benchmark's repository, commit, and file details remain strictly confidential.
Test results reveal that today's frontier AI models still face significant challenges in detecting vulnerabilities comprehensively. The top-ranking model, GPT-5.6 Sol (set to extra-high reasoning), scored 35.58, catching only 30.7% of the total vulnerabilities at a cost of $55.98 for the 50 files. Notably, Anthropic's Fable 5 was entirely absent from the leaderboard due to the company's policy of declining security-related tasks, including defensive ones.
Technical & Technology Analysis
Rather than using simple detection rates, DeepsecBench employs a recall-weighted F2 score. This formula weights recall (the ability to find vulnerabilities) twice as much as precision, since missed vulnerabilities pose a critical danger while false positives do not undermine the security of the codebase. Findings outside the initial golden set of 231 known bugs are classified by a judge model to determine whether they are genuine issues or false alarms.
In terms of cost efficiency, mid-tier and open-weight models are rapidly closing the performance gap. For instance, xAI's Grok 4.5 set to high achieved a score of 15.58 for just $5.60, less than a tenth of the leading model's cost. Similarly, Kimi K3 from Moonshot AI scored 17.56 at a cost of $12.38. This indicates that developers can optimize their budgets by combining different tiers of models across various stages of software development (CI/CD).
Expert Opinions & Insights
According to Vercel, while malicious actors can use AI to probe for external vulnerabilities, defenders hold an absolute advantage: they have full visibility of their system's source code and architecture. Proactively using LLMs to scan source code internally before deployment allows organizations to neutralize risks before attackers can even probe for them.
However, security experts advise businesses to carefully weigh actual operational costs. Vercel estimates that a real-world production codebase could be 100 times larger than the benchmark test set. At that scale, a comprehensive sweep using Kimi K3 would cost around $1,200, whereas running OpenAI's most expensive frontier models could push costs past $5,000 per pass.
Impact & Future
The introduction of DeepsecBench offers a practical framework for organizations to design multi-tiered security scanning strategies. Instead of relying on a single model, startups can run routine code merges through cost-effective models like Grok 4.5, reserving expensive deep reasoning models like GPT-5.6 Sol for major release milestones. For software engineers, this represents a significant shift toward optimizing DevSecOps workflows through unified API gateways to manage model routing and API rate limits efficiently.