By Adrian Sroka

AI is accelerating development and it is lowering the barrier to writing complex code. It is fueling vibe coding, where developers generate and iterate with AI assistants and often prioritize speed over deep understanding of the implementation. The result is more code, less scrutiny, and a growing risk that vulnerabilities slip through unnoticed. The question is no longer whether AI helps you write code. It is whether your security review process is still capable of keeping up. I decided to test it.

The Research Setup

The Research At A Glance

23 code snippets prepared specifically for this research.

30 intentionally hidden threats distributed across the snippets, recognizable through code analysis alone.

40 human participants across different security roles and seniority levels.

16 AI models from four vendor families, mixing general-purpose models and AI coding assistants.

1 shared system prompt applied identically across every model.

Research conducted September 2025.

Every model received the same system prompt: Analyse the following code for security issues, listing all security issues in a table covering the name of the threat, description, risk, recommended fix, and how the vulnerability can be exploited. For general-purpose models, the code snippet was provided in the prompt. For AI coding assistants, the code selection was passed as context where possible.

The 23 code snippets were prepared specifically for this research, based on real-world experience. None were taken from publicly available code review exercises or intentionally vulnerable systems. The vulnerabilities and the code were not sent to any AI model before the research, to remove the risk that a model had seen the data during training.

Each snippet was relatively small and focused on the threats hidden inside it. Real-world tasks are larger and carry more context. Because of the small scope, the results may be a little optimistic. It is easier to spot a vulnerability in 30 lines than in 3,000. The rules were the same for humans and AI models, so the comparison itself is fair.

Responders sometimes raised false positives that depended on context the snippets did not contain (architecture, authentication flow, integration). False positives were not counted against the final score.

The 16 Models Tested

The Models

General Purpose: GPT-4, GPT-5, Claude 3.5 Sonnet, Claude Sonnet 4, Gemini 2.0 Flash, Gemini 2.5 Flash, Microsoft Copilot, Perplexity Auto, Perplexity Pro, Perplexity DeepSeek R1.

AI Coding Assistants: GitHub Copilot, Augment Code, Cursor (Claude 4 Sonnet), GitHub Copilot Grok Code, GitHub Copilot Claude Haiku 4.5, Claude Code Sonnet 4.6

The Human Baseline

Forty people participated in the research. All of them work in or near security, across different roles and seniority levels. Three of the forty scored 100% on the question set. The rest of the human population scored across a distribution that, in average, no AI model in the test was able to match.

The Findings

Across the 13 models, the results varied widely. Some models scored around 50% on the question set. Others scored over 80% (Claude Code Sonnet 4.6, Cursor Claude Sonnet, Perplexity DeepSeek R1). None of them (like Gemini 2.0 Flash) outdid human effectiveness overall. However, a newer model doesn’t always perform better; Claude Sonnet 3.5 works better than Claude Sonnet 4.5.

Looking at threat-by-threat detection efficiency, a pattern emerges. Models hit near-100% accuracy on the easier categories: detecting outdated cryptographic algorithms, sensitive data exposure, documentation compliance. On harder categories that depend on context, like dependency verification or feature flag analysis, success rates dropped to almost 0%.

Where AI Wins, Where AI Loses

AI is strong at: detecting outdated algorithms, sensitive data exposure, compliance with documentation requirements, and pattern matching across known vulnerability classes.

AI is weak at: dependency maturity, feature flag context analysis, threats that depend on architecture or integration context, and separating real risk from noise.

Across all 30 threats in the question set, only three showed AI models averaging higher than humans.

When we look at where those three sit on the threat-category map, the explanation surfaces. Models are better at analyzing complex code structures and compliance with documentation. They are not better at the judgment work that requires architectural understanding.

Coordination, Not Competition

AI is already proving that it can catch patterns, scale across large codebases, and consistently flag issues that humans might miss. In specific cases, it can outperform developers who do not specialize in security.

Where AI struggles is context, intent, and prioritization. The areas where human reviewers bring experience, architectural understanding, and the ability to separate real risk from noise.

The frame for security code review is coordination. The most effective approach uses both humans and AI as partners. AI provides breadth, consistency, and speed. Humans provide depth, judgment, and context. Together, they produce something neither side can achieve alone: meaningful, scalable coverage of the codebase.

A Recommended Approach

For organizations without dedicated security staff. AI Security Code Review is a useful way to introduce a review function with relatively small effort. It will catch obvious and pattern-based issues, which is better than no review at all.

For organizations with security professionals. AI-assisted code review extends what a small security team can cover. Let AI scan first to flag obvious issues, then route results to a human reviewer. The human spends more time on the complex problems that require architectural context.

For dependency, configuration, and supply chain checks. Combine AI models with specialized tools like Snyk or Endor Labs through MCP servers. Connect LLM text analysis with deterministic checks.

For vibe coding workflows specifically. Security controls have to be built into the development pipeline at the moment of generation, not as a separate step.

The research is a snapshot. The pace of model improvement makes any single date a moving target. The gap between AI breadth and human judgment in this task held across thirteen models from four different vendor families. Until the gap closes, the practical move is to use AI for what AI is good at now and keep human reviewers focused on the cases where their judgment matters most.

Research conducted September 2025. Results reflect the model landscape as of that date. AI Cyber Magazine will revisit the topic as the model lineup continues to shift.

About The Author

Adrian Sroka is an AI Security Lead and IT consultant bridging the gap between engineers and cybersecurity. Speaker at major conferences, trainer, and co-author of the OWASP Security Champions Guide. Delivers practical solutions while sharing insights through a professional newsletter.