You have a project with 95 files. You ask an AI to audit it for bugs. It finds 3 or 4, tells you “the rest is robust,” and stops. You retry – still 3 to 4 bugs.
It’s not the tool. It’s not the platform. It’s your prompt.
This article documents real experiments on a production project – 95 files, ~350,000 tokens of code – where changing only the prompt took results from 3-4 bugs to over 150 detections in a single pass.
Why your audit prompt finds so few bugs
Most developers write audit prompts like this:
Audit this code. For each bug found, provide:
- Severity level
- File and line number
- The current buggy code (copied from the file)
- The corrected code (ready to paste)
- A technical explanation
This looks reasonable. Each bug is fully documented, the fix is ready, everything is clean. But here’s the hidden cost: each bug formatted this way uses about 400 tokens of output. Source code, corrected code, explanation – it adds up fast.
After 3 or 4 bugs, the AI has produced roughly 1,500 tokens of structured content. At that point, it considers its job done. It writes a closing sentence like “the rest of the code is robust” or “ready for production” – and stops.
This happened consistently across 30 runs. The bug count never exceeded 5.
Yet the AI’s technical output limit allows 65,000 tokens – enough for 150+ bugs even in the verbose format. The ceiling at 3-4 isn’t a technical limit. It’s behavioral: the AI “thinks” it has done enough work.
First lesson: the more verbose your output format, the fewer results you get.
The fundamental rule: an audit finds bugs, it doesn’t fix them
This is the single most important thing to understand: an audit prompt and a correction prompt are two completely different things. Mixing them sabotages both.
When you ask the AI to find a bug AND write the fix in the same response, you force it to do two jobs at once. It spends time writing corrected code, checking consistency, explaining why – and while doing that, it stops looking for more bugs.
The right approach uses two separate steps:
- Detection pass: a prompt designed for volume. Ultra-compact format, no corrections asked. Goal: find as many issues as possible.
- Correction pass: a separate conversation where you provide the source files and the confirmed bug list. The AI takes its time to fix each one properly.
This separation is what takes results from 4 bugs to 150 detections. It’s not optional – it’s the core of effective AI auditing.
Three hidden biases that limit your results
1. Premature satisfaction
AI models are trained to produce “complete and satisfying” responses. After 3-4 well-presented bugs, the model generates closing phrases and stops. It hasn’t checked everything – it just feels like it has produced enough quality content.
2. “Already seen” file exclusion
If you tell the AI “don’t re-report bugs already fixed,” it often interprets this as “these files have been audited, skip them.” Critical files – payment processing, authentication – can go through 5 audit cycles without being examined, because one bug was fixed there in cycle 1.
3. Per-bug writing cost
A bug at 400 tokens (code + fix + explanation) “costs” 10 times more than a bug at 40 tokens (one line). The AI reaches its satisfaction threshold 10 times faster with the verbose format.
Why the AI invents bugs that don’t exist
This is perhaps the most surprising finding. During our tests, we found that 6 out of 8 false positives cited functions that didn’t exist in the file. The AI had invented plausible function names based on the filename:
| Filename | Invented function | Real functions in file |
|---|---|---|
| payment_validator.py | validate_payment | validate_card_expiry, validate_billing_address, check_duplicate_charge |
| usage_tracker.py | track_usage | record_api_call, calculate_monthly_total |
| notification_sender.py | send | dispatch_email_batch, schedule_retry |
| report_builder.py | build_report | aggregate_metrics, format_csv_export |
This happens because of three mechanisms:
Semantic fabrication: The AI doesn’t read code like a compiler. It generates by statistical association. A file named payment_validator.py statistically contains a function called validate_payment() in the training data. Instead of actually reading line 125 of the file, it “hallucinates” the most probable name.
Compliance bias: When the prompt says “scan all 95 files, don’t stop,” the AI faces a dilemma. Saying “this file is clean” contradicts the order to produce a dense report. So it invents a plausible crash to fill the slot.
Guard blindness: Even when the AI finds the right function, it misses protections that are already in the code. It sees split()[1] and flags an IndexError, ignoring the if "base64," in data guard on the same line. With hundreds of thousands of tokens in context, it can’t maintain attention on nearby protective code.
Each false positive occupies a slot that blocks a real bug from being found. This is not just noise – it actively degrades the audit.
What recent research confirms
A 2026 study from Meta introduced “semi-formal reasoning” – forcing the AI to state its assumptions, trace execution paths, and derive conclusions before answering. This achieved 93% accuracy on code review tasks. The key: instead of free-form reasoning (which leads to hallucinations), you force a structured evidence-gathering process.
Separately, another study found that increasing prompt complexity – especially requiring explanations and corrections – paradoxically increases false positives. Detailed prompts create a bias toward excessive fault-finding. This directly explains why the “classic” prompt produces worse results than a one-line format.
Six concrete levers to multiply your results
1. Reduce output to one line per bug
file.py:line 🔴 function_name - crash description in 20 words max
Measured result: from 4 bugs to 150+ detections. Same model, same code.
2. Ban the closing statement
Add to your prompt: “NEVER conclude with ‘the rest is robust.’ Your response ends with the coverage line.” Once the AI writes a conclusion, it can’t go back.
3. Make the question binary
“Does it crash or not?” is more productive than “Evaluate severity and propose a fix.” Limiting to crashes only multiplied detections by 7 in our tests.
4. Provide a silent mechanical checklist
Four questions to apply on every function, silently (without writing “N/A”):
- a) NULL or malformed input → crash?
- b) Missing database/file result → crash?
- c) External call timeout → crash?
- d) Two simultaneous requests → crash?
“Silently” is the key word: the AI doesn’t waste tokens writing that everything is fine.
5. Name every file individually
List all files in the prompt. Without this list, the AI audits only the files it considers important – and skips the rest.
6. Require exact function names – but nothing more
This is the most counter-intuitive finding. Inspired by Meta’s semi-formal reasoning, we tested forcing the AI to follow three silent steps before reporting a bug: CITE (find the exact function name), TRACE (verify no protection exists), CONCLUDE (report only if both pass).
The result was disastrous. On a single complex file that should contain 7-9 bugs, the full three-step process found only 1 bug. The TRACE step – “check if the code is already protected” – made the AI abandon every finding at the slightest doubt.
The solution: keep only the CITE step. Require the exact function name from the source code, but don’t ask the AI to verify protections. That verification is the job of the correction pass.
Results on the same file:
| Approach | Bugs found | Hallucinated names |
|---|---|---|
| No filter (original prompt) | 7-9 | yes (~55% false positives) |
| Full filter (CITE + TRACE + CONCLUDE) | 1 | 0 |
| CITE only (exact name, no trace) | 5 | 0 |
The lesson: filtering too aggressively kills real bugs just as much as not filtering at all. The sweet spot is requiring the exact function name (which eliminates invented functions) without asking the AI to verify its own findings (which eliminates real bugs).
Measured results
With the same AI model and the same code: the classic verbose prompt found 3-4 bugs per run (max 5). The optimized compact prompt found 150+ detections per run, of which 15-40 were confirmed real bugs after triage. The false positive rate went from ~10% to ~55% – but 40 real bugs beats 4 “perfect” bugs that leave 36 crashes in the code.
Limitations
The same prompt run twice can produce 150 bugs or 11 – that’s AI non-determinism. Run 2-3 times and deduplicate. Also, 20 words per bug description is the right balance – 15 words is too short and reduces detections.
In summary
If your AI audit isn’t finding enough bugs, the problem is your prompt – not the platform.
A prompt that asks for detailed corrections will never find more than 5 bugs. A prompt that separates detection from correction, reduces the output format, bans closing statements, forces file-by-file coverage, and requires the AI to prove its findings will produce 10 to 40 times more exploitable results.
Auditing is about finding. Fixing is a different conversation – literally.
