Security Fine-Tuning Can Calibrate the Output While Leaving the Model Clueless
The lazy version of “AI for security” is to fine-tune a code model, report a binary accuracy number, and hope nobody asks whether the model is finding vulnerabilities or merely learning when to say the scary word. Calibration Without Comprehension asks the uncomfortable question directly, and the answer is not flattering.
The paper introduces CWE-Trace, a benchmark for testing whether LLMs actually reason about Linux kernel vulnerabilities or just inherit output priors from training and fine-tuning. Its central claim is sharp: fine-tuning often changes calibration, confidence, and label tendency without changing the underlying decision policy. In other words, the model may sound more tuned while remaining clueless about the bug.
That matters because vulnerability detection is not a normal text classification task with nicer syntax highlighting. Security review is causal. The model must distinguish vulnerable code from the patched version, identify the relevant weakness, avoid being distracted by surface correlations, and explain why the behavior is dangerous. A model that flags everything is not “cautious.” It is an alert factory. A model that rejects everything is not “precise.” It is an incident waiting politely.
Accuracy is too blunt for security models
CWE-Trace contains 834 manually curated Linux kernel samples, organized as 417 vulnerable-patched pairs across 74 CWEs. The benchmark uses a strict temporal split: pre-2025 historical data versus post-cutoff leakage-free data. The authors evaluate eight vanilla LLMs and 15 LoRA fine-tuned variants across non-targeted detection, targeted detection, and CWE classification.
The important contribution is not just the dataset. It is the diagnostic framing. The paper introduces metrics including Directional Failure Index and Hierarchical Distance and Direction to expose whether a model is systematically “paranoid” or “skeptical.” That framing is overdue. Aggregate binary accuracy can make two very different failures look equally mediocre: one model calls almost everything vulnerable, another calls almost everything safe, and both land near 50% for reasons that matter enormously in production.
The reported priors are extreme. Backbone directional bias ranges from -85.5 to +94.8 percentage points and persists from historical to post-cutoff data. DeepSeek-R1 is strongly paranoid, with +94.8pp DFI on the paired benchmark split and +90.8pp on the leakage-free split. GPT-4.1-mini is strongly skeptical, with -85.5pp DFI on the paired split and -89.3pp on the leakage-free split. Both can look superficially serviceable if you only stare at aggregate accuracy. Neither failure mode is serviceable for a security team.
The contamination analysis is also more nuanced than the usual benchmark argument. The paper reports that 84% of nominally contaminated samples carry no usable memorization signal, because vulnerable functions are absent or cross-mapped across datasets. Roughly 31% of contaminated samples carry CWE misclassification. That does not mean contamination is irrelevant. It means “this CVE was somewhere in the training soup” is not precise enough. Security datasets need provenance analysis at the function, patch, and label level.
Fine-tuning can make the output prettier, not smarter
The phrase “calibration without comprehension” lands because it describes a real product failure. A fine-tuned vulnerability model may become more willing to emit the expected label, more confident in the desired format, or better aligned with a benchmark’s class balance. That is not the same as learning the security concept. If the directional policy survives fine-tuning, the deployment behavior survives too: too many false positives, too many false negatives, or CWE rationales that look plausible until a human checks the patch.
For teams building security agents, the action items are concrete. Evaluate vulnerable-patched pairs, not only isolated snippets. Measure false-positive and false-negative bias separately. Track directional metrics across historical and post-cutoff data. Require CWE-level rationale, but audit those rationales against the actual code change. Test whether fine-tuning changes the decision boundary or just changes the model’s willingness to say “vulnerable.” If a fine-tune improves a leaderboard by becoming more paranoid, that is not a security upgrade; it is a ticket generator wearing a lab coat.
This also applies to coding agents that review PRs or generate fixes. A paranoid reviewer drowns maintainers in low-quality warnings until they learn to ignore the bot. A skeptical reviewer creates the more dangerous failure: quiet confidence. Both are expensive. The cost is not just tokens; it is reviewer attention, triage queues, missed defects, and organizational trust in automation.
The paired design should influence internal evals beyond kernel security. If an agent claims to detect data races, SQL injection, unsafe deserialization, auth bypass, or prompt-injection vulnerabilities, give it before/after pairs. Change one causal detail. Keep the surrounding code similar. Then ask whether the model’s decision follows the vulnerability, not the vibe. This is the difference between a useful reviewer and a regex with a transformer budget.
The paper is not saying LLMs cannot help with vulnerability detection. They can. It is saying security evaluation needs sharper instruments than aggregate accuracy and fine-tuned confidence. That should be an easy sell to anyone who has ever had to triage scanner output. The hard part is organizational: teams like one-number dashboards because one number is easy to buy. Security models need dashboards that admit the truth is messier.
The editorial take is simple: fine-tuning can make a vulnerability detector look calibrated while leaving its actual reasoning policy untouched. If your security agent cannot distinguish the vulnerable function from the patched one, it has not learned security. It has learned a costume.