71 vs. 7 accepted findings: Configured vs repository-only LLM security audits
Across 4 paired LLM security audits, configured runs found 71 accepted issues versus 7 in repository-only runs, using 16.4× the model spend.

Sherlock Research
Can the same LLMs find as many valid security issues with just a code repository and a generic audit prompt as they do in a configured Audit Engine engagement?
When Audit Engine launched, our setup for running general-purpose LLMs was closer to a repository-only run, but with engagement-specific context, known issues and judging rubric. We later began preparing workspaces for each individual engagement, organizing protocol context for the model, and defining how each run uses its budget and when it should finish.
To measure what the current workflow delivers on real engagement code, we paired an Audit Engine production run with a fresh repository-only run for each of 4 engagements. We held the model configuration, audited commit and spend ceiling constant within each pair.
Audit Engine production yielded 71 distinct accepted findings, including 15 rated Medium or High. The repository-only runs yielded 7 accepted findings, 1 Medium and 6 Low. Under the same ceilings, production used 16.4× as much model spend overall.
For each setup, we categorized accepted findings by severity and calculated the number of distinct accepted findings per 100 reported entries. That rate counts each accepted finding once and includes duplicates and rejected entries in the total number reported. We also tracked elapsed run time and actual model spend.
Study note. Audit Engine is Sherlock’s platform for configuring security audits with general-purpose LLMs, specialist Web3 AI auditors, human security researchers, or a mix. This comparison counts only findings credited to the general-purpose LLM runs.
Audit Engine production run. An LLM works from a prepared engagement workspace with scoped code and protocol context. It follows Sherlock’s audit instructions and a policy for continuing the run and submitting findings.
Repository-only run. The same LLM receives the code at the same commit and a generic audit prompt.
4 pairs with matched models, code commits and spend ceilings
We compared configurations of GPT-5.6 Sol (xhigh), GPT-6 Sol (max) and GPT-6.1 Sol (max). The production workspaces included the engagement-specific scope and protocol context, known issues, severity definitions and the judging rubric.
Repository-only runs received the code at the audited commit and a generic request for a security audit with a severity-labeled report. That setup omitted the engagement-specific scope, context, known-issue list and judging and severity rubrics. The experiment log records the same model, standard harnesses, reasoning setting and configured spend ceiling within each pair. Network access was disabled in both conditions.
For each engagement, we selected the best-performing recorded production base-model configuration and ran it in the repository-only setup. Each pair contains 1 run per condition. All 4 engagements had final protocol judgments as of November 17.
Validation note. We counted distinct issues accepted under the engagement rubric. We matched repository-only reports to engagement findings by root cause. A technical reviewer assessed the 22 repository-only reports that had no match in the initial export. The finding totals reflect final protocol judgments.
Production delivered more accepted findings in all 4 pairs
The production runs delivered 13, 12, 40 and 6 distinct accepted findings. The corresponding repository-only runs yielded 0, 1, 6 and 0 distinct accepted findings after matching and final protocol review.

Figure 1. Counts include all severity levels and group duplicate reports within each run. Production counts use final protocol judgments. Repository-only counts are based on root-cause matches to those judgments. Letters anonymize the engagements.
Production delivered 3 High, 12 Medium, 47 Low and 9 Informational findings. The repository-only runs yielded 1 Medium and 6 Low findings.
Production covered more of each engagement’s accepted findings
We measured coverage against all distinct accepted findings in each engagement, across all severity levels. Both runs in a pair used the same engagement total.

Figure 2. Coverage across 4 paired engagements using final protocol judgments.
Production coverage ranged from 17.4% to 85.1% across the 4 engagements. Repository-only coverage ranged from 0% to 12.8%. Among pairs where both runs found an accepted finding, Pair B had the largest coverage ratio with LLMs Audit Engine achieving 12 times the repository-only run’s coverage.
71 distinct accepted findings from 237 production reports
The production runs submitted 237 reports. Of those, 93 (39.2%) corresponded to accepted findings, representing 71 distinct issues after deduplication. The repository-only final reports were parsed into 45 individual entries. 7 (15.6%) mapped to accepted findings, each a distinct issue. Figure 3 counts each accepted issue once.

Figure 3. Production yielded 30.0 distinct accepted findings per 100 reported entries (71/237). Repository-only runs yielded 15.6 (7/45). The numerators count each accepted issue once. The denominators include all reported entries, including duplicates.
Study note. Audit Engine uses AI and human judging to filter invalid submissions out of the protocol’s view. Therefore, even though the base models on Audit Engine submit numerically more dismissed reports, the protocol team needs to go only through the valid findings.
How the production policy handles continuation and submission
The matched spend ceilings set the maximum allowance. The repository-only sessions ended when the model stopped, after 8 to 27 minutes. The production runs lasted 3.4 to 37 hours in elapsed time and consumed 16.4 times as much model spend. Recorded spend covers model execution.
An early stop can end the investigation while the budget remains available. Continuing that work requires another prompt or session. However, a policy that pushes a model to exhaust its allowance can spend tokens on unproductive investigation or leave too little room to write and submit the reports.
In Audit Engine, the model submits findings at the end of each review pass. That keeps findings from completed passes on record while the review continues.
The same model can deliver a different audit
The 4 comparisons put numbers on what we had seen in practice. With the same models and code, Audit Engine production yielded more accepted findings in every pair. Production also yielded more distinct accepted findings for every 100 entries it reported, so the higher total came with a higher yield relative to report volume.
Our working explanation starts with 2 decisions we make when configuring the engagement. The workspace gives the model the protocol context and scope to judge which leads are worth pursuing. The run policy gives the model room to investigate and calls for it to submit findings after each pass. The study compares the full workflow, with the workspace, run policy and actual model spend varying together.
Across the 4 pairs, the configured runs surfaced 15 accepted findings rated Medium or High. The repository-only runs surfaced 7 accepted findings, 1 Medium and 6 Low. Each pair had the same model-spend ceiling, and the configured runs used 16.4× as much actual model spend overall.
Audit Engine gives models the engagement scope, protocol context, and judging criteria up front. Its execution policy governs how the review proceeds and when the model submits findings. Across these 4 comparisons, the configured runs surfaced more accepted findings in every pair. Submissions are judged against the engagement rubric, so protocol teams receive a list of accepted findings they can prioritize.
.png)




