How to run AI vulnerability reviews without getting buried in duplicates

Key takeaways

• LLMs rephrase titles, adjust severities and cite slightly different lines on every pass, which breaks naive text-based or embedding-based deduplication.
• Asking an LLM alone to merge duplicate findings is risky: an over-eager merge can silently collapse two distinct vulnerabilities into one row.
• Stable finding identity requires separating a closed weakness vocabulary from a canonical code-location anchor (for present bugs) or protected-asset anchor (for absent controls).
• Deterministic code policy should corroborate every proposed merge so recurring sweeps update existing findings cleanly without losing real bugs.

Why do AI security reviews report the same vulnerability multiple times?

Anyone who has run an LLM over a repository more than once has hit the duplicate problem. On Monday, the model flags “Path traversal in archive extraction via unsanitised zip entry name”. On Tuesday, or in a second hunting pass over the same package, it flags “Arbitrary file write when unpacking user-supplied archive in extract.go”.

Both descriptions point to the exact same underlying bug. Yet if you store findings in a naive list or match them by title similarity, your report now has two entries. Run five specialized security prompts across a medium-sized codebase and a single missing authorisation check on an API router can easily spawn four or five overlapping findings:

• One from a trust-boundary pass citing the router registration file.
• One from an access-control pass citing the handler function.
• One from an architecture assumption check noting the missing middleware.
• Another on the next commit with slightly different wording and a different severity estimate.

For an engineering team, duplicate findings are almost as destructive as false positives. If developers mark an issue as an accepted risk or a false positive today, and the next scheduled review re-opens it under a new title tomorrow, continuous review becomes unusable.

Why you cannot just ask the model to merge duplicates

The obvious temptation is to hand the full list of findings back to the LLM at the end of a run and say: “Deduplicate this list and merge any findings that describe the same issue.”

In practice, delegating deduplication purely to LLM judgement introduces a dangerous failure mode: false merges.

In security review, the two error directions are not equal:

• A false split (leaving two rows for the same underlying weakness) is a cosmetic annoyance. A developer sees both rows and marks one as a duplicate.
• A false merge (collapsing two separate vulnerabilities into a single finding) can hide a real vulnerability. If a file contains two different command-injection sinks in two different functions, and the model merges them into one generic “Command injection in runner.go” item, fixing the first sink and closing the finding leaves the second vulnerability unpatched and invisible.

Because LLMs attend heavily to surface prose, they frequently group distinct bugs that happen to share a file name or vulnerability category, while failing to merge the same bug when two passes describe its impact differently.

How do you give an AI finding a deterministic fingerprint?

Solving duplicate findings requires taking identity away from free-text prose. Instead of hashing the model’s wording, Dokima computes a deterministic identity from two components: a low-cardinality weakness stabilizer and a high-cardinality code anchor.

1. A closed weakness vocabulary

When a model invents category names freely, the same bug gets labelled “Directory Traversal”, “Sandbox Escape”, “Zip Slip”, or “Unsanitised File Path”. Dokima restricts classification to a closed, fourteen-class vocabulary enforced by code, including:

• injection_unsafe_execution
• path_traversal_confinement
• authentication_bypass
• authorization_or_isolation
• trust_boundary_confusion
• secret_or_credential_exposure
• input_validation_failure
• availability_or_resource_abuse
• cryptography_or_integrity
• missing_security_control
• information_disclosure
• insecure_default_configuration
• supply_chain_integrity

Whether a hunting pass describes an issue as “symlink escape” or “arbitrary file read via backslash traversal”, deterministic normalization quantizes it to path_traversal_confinement. Prose drift across runs cannot shift the class.

2. Canonicalized code and symbol locators

The weakness class stops prose drift, while the locator provides uniqueness. Dokima extracts and normalizes the pointed-at file paths (stripping line suffixes and relative prefixes), the enclosing function or symbol name, and the specific sink operation.

Crucially, severity, confidence and free-text descriptions are deliberately excluded from the fingerprint. When a subsequent sweep re-discovers the same weakness at the same symbol and sink, it produces the exact same fp1-... fingerprint and updates the existing record rather than appending a duplicate row.

Why "presence" bugs and "absence" bugs need different anchors

One subtlety that breaks many deduplication schemes is the difference between vulnerable code that is present and a security control that is absent.

A presence bug has an obvious line of code to point at. An absence bug does not: one pass might cite the router file where middleware was omitted, while another cites the handler file where the database update occurs. By distinguishing presence from absence and anchoring absence findings on the protected asset rather than whichever file the hunter happened to cite first, Dokima recognizes when two passes are reporting the same missing guard.

How Dokima combines model context with deterministic merge gates

Within a review pass, Dokima’s Finding Validator and Finding Expander stages can propose that two overlapping candidates are duplicates. However, the LLM does not get the final vote.

Every proposed merge must pass a deterministic corroboration gate in code before two findings are combined:

1. Both findings must resolve to the same closed weakness class. An authentication_bypass and an information_disclosure on the same endpoint are never auto-merged.
2. Presence and absence findings cannot merge into each other.
3. For presence bugs, normalized source files must overlap. Furthermore, if both findings name an enclosing symbol or sink token and those tokens differ, the merge is refused. Two distinct sinks inside the same file remain two distinct findings.
4. For absence bugs, the normalized protected asset must match.

Whenever ambiguity exists, the policy resolves toward keeping findings separate. And when a human engineer spots a cosmetic split that crosses files, they can mark the finding status as Duplicate directly in the Dokima dashboard or CLI.

Preserving developer triage across continuous sweeps

Because finding identity is stable and persisted in .dokima/state/, your team’s triage work survives across runs. When you annotate a finding as False positive, Mitigated, Risk accepted or Duplicate, that disposition stays attached to the finding’s identity as you continue running scheduled sweeps.

You can see how validated and dispositioned findings are presented in the live File Browser sample report, or start a 21-day free trial to run Dokima against your own repositories.