Why one-shot AI prompts fail at repository security review

Key takeaways

• Single-prompt code audits collapse discovery, verification and reporting into one pass, encouraging models to speculate whenever context is missing.
• Real repository vulnerabilities usually depend on trust boundaries, middleware and state across multiple files rather than a single suspicious function.
• Separating architecture mapping, security assumptions, focused hunting and skeptical peer review cuts noise before a human ever reads a finding.
• Persisting state in the workspace lets subsequent passes build on verified architecture and challenged findings instead of starting from zero.

Why does a single prompt produce noisy security findings?

When engineering teams first test an AI model on security review, they usually paste a module or point a CLI coding agent at a directory with a prompt like: “Review this repository for security vulnerabilities and list everything you find.”

The result almost always follows the same pattern. The model returns ten to twenty plausible-sounding items within a couple of minutes. Two or three are useful observations. Half are standard framework behaviours misread as bugs because the model did not check the router or middleware. The rest are generic warnings about rate limiting, headers or theoretical inputs that cannot reach the function in question.

After two rounds of chasing phantom bugs across the codebase, developers stop trusting the output. The failure does not happen because frontier or open-weights models cannot reason about code. It happens because a one-shot prompt asks the model to do four incompatible jobs at the same time:

1. Infer how the system is structured and where untrusted input enters.
2. Search for candidate weaknesses across multiple vulnerability classes.
3. Verify whether upstream guards or types neutralise each candidate.
4. Write a polished severity assessment before confirming the facts.

When a model is instructed to produce a list of findings in a single turn, it is rewarded for completion rather than falsification. If a helper function builds a file path or executes a query without an inline check, the one-shot prompt nudges the model to report it immediately rather than pause, locate the caller, and check whether path confinement or authentication already ran at the boundary.

Do larger context windows fix one-shot code audits?

A common assumption is that a 200k or 1M token context window eliminates the need for structured review stages. In practice, stuffing an entire repository into a single prompt trades missing context for diluted attention.

Security review is not a summarisation task. Determining whether an authorisation bypass is real requires tracing specific invariants:

• Which routes are mounted behind authenticated middleware, and which are public?
• Where is tenant identity bound to the session or request context?
• Does a background worker or internal RPC trust fields that a user could supply earlier in the flow?
• If a check is absent in one handler, is that absence intentional or an oversight compared to sibling handlers?

When fifty thousand lines of source code sit in one undifferentiated prompt alongside a broad instruction to “find vulnerabilities”, the model gravitates toward local, surface-level patterns. It flags the exec call or the SQL string builder that looks dramatic on a single screen, while missing the subtle state transition across three packages where an unprivileged user overwrites an organisation identifier.

How does a staged security review pipeline work?

Human security engineers do not audit a codebase by reading line one and writing a final report five minutes later. They map the entry points, break the system into components, write down what must be true for each component to be safe, and then test those assumptions against the source.

Dokima applies that exact structure to your chosen AI runner (Claude, Codex or OpenCode):

1. Foundation and assumption mapping

Before hunting for bugs, Dokima runs foundation stages across the repository. The Repository Attack-Surface Mapper and Component Decomposer identify exposed interfaces, trust boundaries, privileged operations and logical components. For each component, the Architect and Security Assumption Author record explicit assumptions, such as how paths are confined or how session tokens are verified, and the Assumption Verifier checks those assumptions against current code.

2. Focused hunting lenses

Instead of asking a model to look for everything at once, Dokima schedules dedicated hunting lenses appropriate to each component’s architecture. Separate passes examine trust-boundary violations, missing security controls, injection and unsafe execution, secret exposure, authorisation and isolation, supply-chain integrity, and resource abuse.

In addition, coverage-improvement stages look specifically for what initial passes miss: enumerating sibling sinks when one vulnerability class is found, and challenging previously dismissed concerns when an assumption might not hold.

Why candidate findings must be challenged before developers see them

The single biggest difference between a noisy AI experiment and an engineering tool is what happens after a model spots something suspicious.

In Dokima, a hunter stage never writes directly to the final report. Raw observations are normalised into structured candidates and handed to a separate Candidate Finding Peer Reviewer stage. That stage is instructed to be skeptical:

• If a candidate claims a route is unauthenticated, has the hunter proved where the route is registered?
• If the evidence is incomplete, the peer reviewer does not guess. It dispatches an Information Gatherer or Red Flag Investigator stage to pull the exact upstream files and verify the call chain.
• Only candidates that survive peer review reach the Finding Validator, which confirms exploitability, scope and severity before promoting the issue into dokima-report/.
• Finally, the Finding Expander writes a concise plain-English summary and concrete remediation steps so developers can fix the issue without deciphering raw model reasoning.

A model that is asked to critique and falsify a specific candidate finding behaves very differently from a model that is asked to brainstorm vulnerabilities from scratch.

How to test this on your own codebase

If your team has tried prompting Claude, Codex or local models directly for security review and walked away frustrated by false positives, the problem is rarely the underlying model. It is the absence of a stateful pipeline around it.

You can see the output of a full staged review in our File Browser sample report, where every finding includes exact file and line citations, verification reasoning and remediation guidance. Or start a 21-day free trial and run Dokima on your own repositories on Linux or macOS.