Langfuse 3.195 Turns Evaluation Data Into Multimodal Infrastructure

Langfuse 3.195.0 is easy to undersell as a feature release for dataset uploads plus a grab bag of operational fixes. That would miss the useful signal. The release is really about Langfuse admitting what modern AI evaluation platforms are becoming: multimodal data infrastructure with background jobs, access-control edges, provider calls, export pipelines, and enough internal complexity that the observability product now needs serious observability of its own.

The most visible change is PR #14257, which adds end-to-end media attachment support for dataset items. A new dataset_item_media table links uploaded media to a specific item version. Writes parse and persist media references. tRPC and public API read paths resolve attachments to signed download URLs. Create and edit dialogs support upload, drop, paste, and preview. View pages and historical-version pages render saved attachments. Media is cleaned up when the owning trace or dataset is deleted, or when retention aging removes the parent state.

That PR changed 58 files with 4,373 additions and 374 deletions. It also isolates CodeMirror re-renders so typing in one dataset item field does not keep re-rendering or reconfiguring the other editors. That last part sounds like UI polish, but it is part of the same theme: once eval datasets become richer than text rows, the product either behaves like real data infrastructure or it becomes a nice demo with sharp edges.

Text-only evals are now the legacy path

Media attachments in dataset items matter because text-only eval rows are an increasingly narrow abstraction. Agents inspect screenshots, parse PDFs, compare UI states, reason over diagrams, handle uploaded files, and produce artifacts that are not just strings. If the evaluation platform treats media as a side note stored somewhere else, reproducibility collapses. The dataset row says “look at this screenshot,” but the system cannot prove which screenshot, which version, which access path, or which cleanup policy applied at evaluation time.

Version-linking is the key design choice. If a dataset item changes, the media attached to the old version must remain inspectable when someone reviews an evaluation from last week. Otherwise, “historical eval” becomes a lie with a timestamp. Langfuse’s model — media linked to item versions, rendered in historical views, and exposed through signed URLs — is the right direction. The platform is not merely letting users decorate examples with files; it is making those files part of the state that defines the test case.

Practitioners should read this as a prompt to audit their own eval stack. If your agents use screenshots, PDFs, audio-derived transcripts, browser captures, product mockups, or generated files, ask where those artifacts live. Are they versioned with the eval row? Are signed URLs regenerated safely? Are deleted traces cleaning up media without breaking historical review? Can a public API consumer reconstruct the same test input that the UI shows? If the answer is “we put a link in a JSON blob,” you have not built an eval dataset. You have built a scavenger hunt.

The boring export metrics are the credibility test

The rest of Langfuse 3.195.0 is less marketable and more important for operators. PR #14441 emits blob-export heartbeat gauges every five minutes for exports running longer than five minutes: langfuse.blobstorage.export_heartbeat.rows and langfuse.blobstorage.export_heartbeat.serialized_bytes, tagged with table and projectId. PR #14442 promotes export byte counts into first-class Datadog counters: uncompressed ClickHouse bytes via langfuse.blob_export.serialized_bytes and compressed S3 upload volume via langfuse.blob_export.compressed_bytes, tagged by table, project, path, and gzip level. PR #14440 derives uploadWaitMs as the residual time after source wait and gzip active time, then emits it in logs, span attributes, and langfuse.blob_export.upload_wait_ms.

This is the observability platform applying observability to itself, which sounds recursive because it is. Blob exports are not peripheral once traces, prompts, eval datasets, media, and usage records become operational evidence. Exports support audits, backups, migrations, customer requests, downstream analytics, and incident review. If a long-running export stops emitting heartbeat metrics, operators need to know before a customer asks where their data went. If upload wait time spikes, the team needs to distinguish S3 backpressure from ClickHouse read time or gzip CPU. If compressed bytes jump unexpectedly, the bill may be telling you something the UI missed.

The useful lesson for engineering teams is not “use these exact metric names.” It is to treat the movement of AI-observability data as production work. Traces are only valuable if they can be retained, exported, inspected, and deleted predictably. The data pipeline carrying observability data deserves the same alerting discipline as the services being observed.

Model calls inside observability tools are production dependencies

Langfuse also tightens worker-side model-call behavior. PR #14455 applies the existing LANGFUSE_FETCH_LLM_COMPLETION_TIMEOUT_MS runtime wrapper to OpenAI-compatible LLM calls. The motivation is practical: custom base URL endpoints can keep worker slots occupied if SDK-level timeout handling is not enough. Three additions across two files is not a flashy patch, but hard runtime timeouts are exactly the defense-in-depth pattern you want when eval workers, playgrounds, judges, or transformations call model providers.

PR #14461 adds worker-side OpenTelemetry undici/fetch instrumentation so outbound OpenAI and LiteLLM HTTP attempts appear under existing worker traces. It preserves the original provider or LangChain error as LLMCompletionError.cause, filters localhost, keeps spans parent-only, strips query strings from url.full, and blanks url.query to reduce sensitive-data leakage. That combination matters. Operators need model-call failures to show up in traces, but they do not need URLs leaking keys, query parameters, or internal details into telemetry.

This is where observability products become part of the runtime, not just passive dashboards. If the platform runs LLM-as-judge jobs, prompt experiments, dataset transformations, or provider checks, then its model dependencies can fail, hang, rate-limit, or leak metadata. Those calls need timeouts, traces, error causes, and sanitization. A broken model adapter should not quietly consume every worker slot and turn an eval run into a platform outage.

Access control is not a billing afterthought

The sharpest security fix in the release is PR #14448, which gates SCIM Users provisioning endpoints behind the admin-api entitlement. Before the fix, org-scoped API keys on un-entitled plans could create users with caller-set passwords and assign org roles up to OWNER. The new gate returns a SCIM-formatted 403 before database writes on un-entitled plans.

That is not merely a plan-enforcement bug. In an AI observability platform, SCIM and admin APIs sit near traces, prompts, outputs, customer metadata, user identifiers, and sometimes accidental secrets. Provisioning users with high org roles is an access-control surface, not a monetization surface. Enterprise teams should treat this as a reminder to audit observability platforms like production infrastructure: SCIM gates, org keys, role assignment, integration scopes, model worker permissions, and trace data retention all belong in the same review.

The upgrade checklist is concrete. Test dataset media through create, edit, view, history, and public API paths, including retention and deletion cleanup. Create a long-running blob export and verify heartbeat gauges, byte counters, gzip/upload/source timings, and terminal success or failure metrics land in your alerting stack. Point an OpenAI-compatible adapter at a slow endpoint and confirm LANGFUSE_FETCH_LLM_COMPLETION_TIMEOUT_MS actually releases the worker. Inspect worker traces for model calls and make sure sanitized URLs stay sanitized. If you use SCIM, confirm un-entitled orgs get 403 before account creation.

Langfuse 3.195.0 is not just a dataset-media release. It is a sign that eval and observability tools are becoming the data layer for multimodal agent systems. That layer has to version files, meter exports, trace its own model calls, enforce admin boundaries, and fail in ways operators can see. The useful platforms will do that work. The pretty dashboards will eventually be asked why the export stalled.

Sources: Langfuse v3.195.0 release, PR #14257, PR #14461, PR #14455, PR #14440, PR #14441, PR #14442, PR #14448.