ProvenanceGuard Catches the Agent Failure Your RAG Evaluator Misses: Right Fact, Wrong Source
Most RAG evaluations ask whether an answer is supported by the provided context. That is no longer enough. Once agents start mixing CRM records, internal databases, search results, repo files, tickets, policy docs, and third-party APIs in the same response, a new class of bug appears: the fact is real, but the attribution is wrong. ProvenanceGuard is interesting because it names that failure mode directly and treats source ownership as part of correctness, not a formatting nicety.
The paper focuses on MCP-based LLM agents, where tool traces can include stable tool IDs, source IDs, and raw outputs. Its central target is cross-source conflation: a claim may be supported somewhere in the evidence while being attributed to the wrong source. The paper’s medical example is sharp: “According to the patient’s chart, empagliflozin reduced a mortality endpoint.” The mortality claim may be true in a clinical-trial abstract, but false as a statement about that patient’s chart. A source-blind faithfulness evaluator can mark the answer supported. A clinician, compliance officer, or user would correctly call it broken.
Faithful somewhere is not faithful enough
ProvenanceGuard consumes captured MCP traces, decomposes an answer into atomic claims, routes those claims to source-specific evidence, runs NLI plus token-alignment checks, and emits per-claim verdicts along with an answer-level allow/block decision. The key output is not just whether a claim is supported, but whether it is supported by the source the answer says it came from. That sounds like paperwork until you map it onto real agent workflows.
A support agent may combine a customer ticket, a billing table, and a policy document. If it says “your contract includes this refund clause” when the clause came from a generic help-center article, that is not a harmless citation error. A coding agent may combine repo files, dependency docs, and a Stack Overflow snippet. If it attributes an API behavior to the current repository when it came from outdated docs, the resulting patch can be confidently wrong. A finance agent may combine ledger entries and market commentary. If it attaches a market-wide statistic to a specific account, the answer is worse than unsupported because it carries false authority.
The reported evaluation is usefully specific. The authors use a frozen corpus of 281 medical-domain MCP-agent traces. A 266-trace adjudicated subset yields 2,325 LLM-assisted claim labels, with 361 held-out labels human-verified. On a 40-trace held-out split, ProvenanceGuard reports block F1 0.802 and source accuracy 0.858 over 260 source-eligible claims. Source-blind baselines score lower on block F1 — MiniCheck at 0.783, RAGAS Faithfulness at 0.758, AlignScore at 0.662, and SummaC-ZS at 0.436 — but the more important point is that those baselines do not emit claim-to-source IDs.
MCP makes provenance easier to preserve — and harder to ignore
The Model Context Protocol’s pitch is straightforward: give AI applications a standard way to connect to external systems. That is useful, and adoption across Claude, ChatGPT, VS Code, Cursor, MCPJam, and language SDKs means the pattern is not staying theoretical. But standardizing tool access also standardizes the opportunity for agents to blur source boundaries. If your app collapses tool results into one anonymous context blob before generation, it has thrown away exactly the metadata needed to catch cross-source conflation later.
Practitioners should take the paper as a logging and architecture mandate. Preserve tool family, source object, source ID, timestamp, raw output, and final-answer claim mapping. Do not let summarizers strip provenance because the prompt looks cleaner afterward. Evaluate whether claims are supported by the cited source, not merely by any nearby text. And when the answer combines sources, make the model expose that join explicitly enough for a verifier to inspect.
The hard benchmark result is also worth reading honestly. On a tougher multi-source benchmark, ProvenanceGuard reaches block F1 0.846, while source-plus-relation accuracy drops to 0.229. That is not a detail to bury. It means exact source ownership remains difficult when sources are semantically close. But this is precisely why the paper matters. The industry is rushing to connect agents to more systems, while the tools for verifying source ownership are still immature. That mismatch is where production incidents live.
The repair results point to the right user experience. Repair-and-reverify resolves all blocked answers in the full trace set, but 144 of 173 blocked answers require conservative fallback rather than substantive rewriting. In other words, many failures are not fixable by smoother prose. The honest answer is sometimes “I cannot attribute this safely.” That sentence is awkward in a demo and excellent in a regulated workflow.
My take: ProvenanceGuard is the kind of unglamorous agent infrastructure that will matter more as MCP adoption grows. RAG faithfulness was built for a simpler world where context could be treated as a pile. Agents operate over named systems with different authority levels. Correct attribution is not decoration; it is part of the answer.
Sources: arXiv, Model Context Protocol docs, Model Context Protocol GitHub organization, PreAct context