LLM Observability in Production: Tracing, Evals, Cost Tracking, and Drift Detection
Running an LLM in a notebook is one thing — keeping it honest, affordable, and stable at production scale is another problem entirely. A new post from Atal Upadhyay lays out the full observability stack required for production LLM deployments, starting with the core insight that traditional software observability simply doesn't hold up for non-deterministic AI systems. Logs, metrics, and traces catch infrastructure failures, but they miss the AI-specific failure modes that actually hurt teams in practice: outputs drifting as models update, costs ballooning from prompt inefficiencies, and eval regressions going undetected between releases.
The post walks through three main instrumentation patterns — proxy-based tools like Helicone and LiteLLM that sit in front of your API calls, SDK-based tracing via LangSmith's @traceable decorator or Langfuse, and fully OpenTelemetry-native pipelines for teams with existing observability infrastructure. Each approach has different tradeoffs around latency overhead, vendor lock-in, and the depth of semantic visibility they provide. The recommendation is to start lightweight and add eval pipelines and cost attribution early, before the first production incident forces the issue.
For teams building on LangChain, AutoGen, or custom tool-calling setups, this post serves as a practical decision framework for choosing your instrumentation layer — and a useful checklist for what you probably haven't wired up yet.