Self-Hosted LLM Observability: Tracing Agents in Production with Langfuse

An agent in production gives a user the wrong answer. Not an exception, not a stack trace, nothing that paged anyone. The request came in, the agent responded, the response was confidently wrong, and the only evidence you have is the final message a support ticket quoted back at you.

You open your APM dashboard anyway, out of habit. It shows you a green line. The request completed in 800ms, the HTTP status was 200, no error was logged anywhere in the chain. Your APM was built to answer "did this crash," and nothing crashed. The agent called a tool, got a result, reasoned over it, and returned an answer, all of it working exactly as designed, and the answer was still wrong.

This is the gap self-hosted LLM observability exists to close. You need to see what the model actually saw: the prompt after your template filled it in, the tool calls it made and what came back, and the reasoning path it took to the answer it gave. None of that lives in a span duration or an HTTP status code, and the moment you need it is always after the fact, when the only thing you have to go on is a bad response and no way to reconstruct how it got there.

What an LLM trace has to capture that a normal span doesn't

Standard distributed tracing gives you a name, a duration, a status, and maybe some tags. That's the right shape for "which service is slow" or "which endpoint is erroring." It's the wrong shape for "why did the agent say that," because the answer to that question isn't in timing data. It's in content.

An LLM trace needs a few things a normal span never carries:

The resolved prompt, not the template. You wrote You are a support agent. Context: {{context}}. Question: {{question}}. What the model actually received was that template filled in with a specific retrieved context and a specific user question, and if the retrieval pulled the wrong document, the template itself never tells you that. You have to log the resolved string, after every variable is filled in, or you're debugging blind.

Every tool call and its result. Agents that use tools are only as good as what the tools return. If a lookup tool returned a stale price or an empty result set, the model's answer downstream will look like a reasoning failure when it was actually an input failure. A trace has to nest tool calls as child spans of the model call that triggered them, with the tool's arguments and its raw response both captured.

Token counts per step, not just per request. A single agent turn might involve a planning call, two tool calls, and a final synthesis call, each with its own prompt and completion tokens. Aggregating that into one number for the whole turn hides which step is expensive. You want the breakdown so you can tell if the cost is in a bloated system prompt, a tool result that's dumping a whole JSON blob into context, or a synthesis step that's re-reading everything from scratch.

The model and its exact parameters. Temperature, top_p, the model version string, and any stop sequences. Model providers ship new versions under old names more often than teams expect, and a change in behavior with no change in your code is usually explained by a parameter or a model revision, not a mystery.

The point where the agent decided it was done. Agent loops end somewhere: a tool call returned an answer the model treated as final, the model emitted a stop token, or the loop hit a step limit and gave up. That termination reason is often the most useful field in the whole trace, because "hit the step limit" and "the model was satisfied" are two very different failure modes that look identical from the outside.

None of this is unusual to want. It's just outside what request tracing was designed to hold, which is why teams end up standing up a separate system for it rather than bolting it onto existing APM.

Why self hosting specifically

The observability layer of the 2026 agent tooling stack is usually described as three names: Langfuse, LangSmith, and Braintrust. Of the three, Langfuse is the one built to be run on your own infrastructure from the start. It's MIT licensed, it ships a Docker Compose file you can run today, and the hosted version and the self-hosted version are the same codebase.

Self hosting isn't the default choice for every team, and it shouldn't be. It's more infrastructure to run and more infrastructure to patch. But for LLM observability specifically, there's a reason a meaningful share of teams choose it over a hosted SaaS product: the data.

A trace of an agent turn is not sanitized telemetry. It's the actual prompt, which usually contains whatever context you retrieved to answer the question, which usually contains customer data. It's the actual tool results, which might be a database row, an internal document, or an account balance. Sending that to a third party's servers is a decision, and for a lot of teams handling regulated or sensitive data, it's a decision that deserves to be made deliberately rather than by default because the SDK's quickstart pointed at a hosted endpoint.

The numbers back up why teams weigh this so heavily. Among professionals evaluating AI tooling, 96% say safeguarding confidential data matters to their decision, 94% say grounding outputs in authoritative content matters, and 90% say being able to explain and defend the reasoning matters. Those three concerns point at the same thing: you need to see exactly what the model saw and said, and you need to control where that data lives while you look at it. Self hosting the observability layer is the direct answer to the confidentiality piece of that, and it's also what makes the explainability piece possible, since a trace store you control is one you can retain, audit, and query on your own terms.

There's a governance angle here too. Over 40% of agentic AI projects are judged at risk of cancellation by 2027 without governance, observability, and a clear return on investment attached to them. Observability isn't just a debugging convenience in that framing. It's part of what keeps a project fundable, because it's the thing that turns "the agent works, trust us" into an inspectable record.

Langfuse isn't the only mature open source option in this space. RAGAS handles evaluation specifically, and tools like TrustGraph, Graphiti, and Cognee cover adjacent ground in knowledge representation and memory for agents. But for the specific job of tracing production agent turns and scoring them, Langfuse is the one built for that job end to end, with the self-hosting story worked out rather than bolted on.

Setting it up

Langfuse's self-hosted stack needs a Postgres database for the core data and, for anything beyond small volumes, a ClickHouse instance for analytical queries plus Redis for caching and background jobs. The project ships a Docker Compose file that wires all of this together, so you don't have to hand-assemble it.

Here's a docker-compose.yml that gets a working local stack up. Treat the secrets as placeholders and replace every one of them before this touches a shared environment.

version: "3.8"

services:
  langfuse-server:
    image: langfuse/langfuse:2
    depends_on:
      - postgres
      - clickhouse
      - redis
    ports:
      - "3000:3000"
    environment:
      DATABASE_URL: postgresql://langfuse:langfuse@postgres:5432/langfuse
      CLICKHOUSE_URL: http://clickhouse:8123
      CLICKHOUSE_USER: default
      CLICKHOUSE_PASSWORD: clickhouse
      REDIS_CONNECTION_STRING: redis://redis:6379
      NEXTAUTH_SECRET: replace_with_a_generated_secret
      NEXTAUTH_URL: http://localhost:3000
      SALT: replace_with_a_generated_salt
      ENCRYPTION_KEY: replace_with_a_32_byte_hex_key

  postgres:
    image: postgres:15
    restart: always
    environment:
      POSTGRES_USER: langfuse
      POSTGRES_PASSWORD: langfuse
      POSTGRES_DB: langfuse
    volumes:
      - langfuse_postgres:/var/lib/postgresql/data

  clickhouse:
    image: clickhouse/clickhouse-server:latest
    restart: always
    environment:
      CLICKHOUSE_USER: default
      CLICKHOUSE_PASSWORD: clickhouse
    volumes:
      - langfuse_clickhouse:/var/lib/clickhouse

  redis:
    image: redis:7
    restart: always

volumes:
  langfuse_postgres:
  langfuse_clickhouse:

Bring it up, then create your project and API keys through the UI at localhost:3000.

docker compose up -d

Once the project exists, put the keys in your application's environment rather than in code.

LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_BASE_URL=http://localhost:3000

Instrumenting a traced agent turn

The shape you want in code is a top-level trace for the whole agent turn, with each model call and each tool call as a nested span underneath it. The exact SDK method names shift between versions, so treat the calls below as the shape to build toward and check your installed version's client for the current signatures. The pattern, start a trace, start a span inside it, log inputs and outputs on each, and close them, holds across versions even when method names move.

import { Langfuse } from "langfuse";

const langfuse = new Langfuse({
  publicKey: process.env.LANGFUSE_PUBLIC_KEY,
  secretKey: process.env.LANGFUSE_SECRET_KEY,
  baseUrl: process.env.LANGFUSE_BASE_URL,
});

interface ToolCall {
  name: string;
  args: Record<string, unknown>;
}

async function runAgentTurn(userId: string, question: string) {
  const trace = langfuse.trace({
    name: "agent-turn",
    userId,
    input: { question },
  });

  try {
    const planSpan = trace.span({
      name: "plan",
      input: { question },
    });

    const plan = await callModel({
      systemPrompt: buildSystemPrompt(),
      userMessage: question,
    });

    planSpan.end({
      output: plan,
      metadata: {
        model: plan.model,
        temperature: 0.2,
        promptTokens: plan.usage.promptTokens,
        completionTokens: plan.usage.completionTokens,
      },
    });

    const toolResults: Record<string, unknown> = {};

    for (const call of plan.toolCalls as ToolCall[]) {
      const toolSpan = trace.span({
        name: `tool:${call.name}`,
        input: call.args,
      });

      try {
        const result = await runTool(call.name, call.args);
        toolResults[call.name] = result;
        toolSpan.end({ output: result });
      } catch (err) {
        toolSpan.end({
          output: null,
          level: "ERROR",
          statusMessage: String(err),
        });
        throw err;
      }
    }

    const synthesisSpan = trace.span({
      name: "synthesize",
      input: { question, toolResults },
    });

    const finalAnswer = await callModel({
      systemPrompt: buildSystemPrompt(),
      userMessage: question,
      context: toolResults,
    });

    const finishReason = finalAnswer.finishReason; // "stop" | "tool_call" | "step_limit"

    synthesisSpan.end({
      output: finalAnswer.text,
      metadata: {
        model: finalAnswer.model,
        promptTokens: finalAnswer.usage.promptTokens,
        completionTokens: finalAnswer.usage.completionTokens,
        finishReason,
      },
    });

    trace.update({
      output: { answer: finalAnswer.text, finishReason },
    });

    return finalAnswer.text;
  } finally {
    await langfuse.flushAsync();
  }
}

A few things to notice. The trace carries the userId, which is what makes per-user cost and error queries possible later. Each tool span logs the raw arguments and the raw result, not a summary of them, because the summary is exactly what you'll wish you had when the raw data would have shown you the actual problem. And the final span records finishReason, which is what lets you later query for every turn that hit a step limit instead of finishing cleanly.

callModel and runTool are your own wrappers around whatever model client and tool implementations you already have. The instrumentation sits around them, not inside them, which keeps this pattern portable if you switch model providers later.

Scoring and evaluation

A trace tells you what happened. A score tells you whether what happened was good. Langfuse lets you attach a score to any trace, either from an automated evaluator or from a human reviewer, and the score is just a number or a category tied to the trace ID.

await langfuse.score({
  traceId: trace.id,
  name: "helpfulness",
  value: 1,
  comment: "Correctly cited the refund policy section.",
});

That single call is the building block for two different workflows. The first is a human-in-the-loop review queue, where someone spot-checks a sample of production traces and scores them, which gives you a ground truth signal for how the agent is actually doing with real traffic rather than a benchmark. The second is automated scoring, where an evaluation pipeline runs against a batch of traces and writes scores back on a schedule.

For retrieval-heavy agents, RAGAS is the natural pairing here. RAGAS scores retrieval quality on dimensions like context relevance and faithfulness, whether the model's answer is actually supported by what was retrieved rather than invented on top of it, and Langfuse has native integration for pulling RAGAS scores onto the same traces you're already capturing. That means a retrieval failure and a generation failure show up as different scores on the same trace instead of one blended "it was wrong" signal, which is the distinction you need to know which part of the pipeline to fix.

The pattern worth building deliberately, rather than leaving to whenever someone has time, is promoting real production traces into an eval dataset. Take the traces a human reviewer scored poorly, or the ones users flagged, and pull the input and expected output into a labeled dataset. Then when you change a prompt or swap a model, you run that dataset against the new version before shipping it, and you're testing against what your users actually asked rather than a set of examples someone wrote at the start of the project. That gap, between synthetic test cases and real traffic, is usually where regressions hide.

const dataset = await langfuse.createDataset({
  name: "production-regressions",
});

await langfuse.createDatasetItem({
  datasetName: "production-regressions",
  input: { question: trace.input.question },
  expectedOutput: correctedAnswer,
  sourceTraceId: trace.id,
});

Running that dataset against a candidate prompt or model is a normal loop: pull each item, run it through your agent, score the result, and compare the aggregate to your last known baseline before you ship the change.

What to alert on

A dashboard you have to remember to look at will not catch a regression at 2am. Alerts will, but only if they're watching the right things, and the instinct to chart everything is the wrong one here. A handful of specific signals catch almost everything that matters.

Error rate by tool, not by the agent overall. An agent-level error rate blends a flaky third-party API with a bug in your own retrieval code, and you want to know which one woke you up. Break the rate down per tool name and alert on each one crossing its own threshold.

Latency at the ninety-ninth percentile, not the mean. The mean hides the users having a bad time. A model call that usually takes 800ms but occasionally takes 12 seconds will barely move your average while making a real slice of your traffic feel broken. Alert on p99, and if you can get it, p99.9.

Token spend per user session. Cost anomalies in LLM systems usually aren't a gradual drift, they're a single session where something looped, a tool result dumped an entire table into context, or a user found a way to make the agent repeat itself. A per-session spend threshold catches that class of incident before it shows up as a surprising line on next month's bill.

The rate of traces where the agent gave up or hit its step limit. This is the field you logged as finishReason in the instrumentation above, and it's worth alerting on directly rather than inferring it from something else. A rising rate of step-limit exits usually means a tool started failing silently, or a prompt change made the agent second-guess itself into extra unnecessary loops.

None of these four need a fancy anomaly detection model. A threshold and a rate of change over a rolling window catches nearly all of it, and simple is worth keeping simple here, because you'll be the one debugging the alerting rule at the same 2am the alert was supposed to save you from.

The part that costs money

Self hosting an observability platform means running a database, and for any real volume, an analytical store and a cache alongside it. That's real operational surface: backups, upgrades, and someone who understands Postgres and ClickHouse well enough to keep them healthy under load. Weigh that against a hosted product's monthly bill honestly, because for a small team, the hosted option's cost might genuinely be lower than the engineering time self hosting takes.

The other cost is storage, and it grows faster than people expect. Full prompt and completion payloads for every agent turn, especially agents with long context windows or verbose tool outputs, adds up to a lot of data fast. Retaining every trace at full fidelity forever is rarely the right default.

Sampling has to be a decision you make, not a default you inherit. A reasonable starting point: keep full-fidelity traces for a shorter recent window, say two to four weeks, where you're most likely to need to debug something a user just reported. Beyond that window, keep the metadata and scores but drop or truncate the raw payloads. Always keep 100% of traces that hit an error or scored poorly, regardless of age, since those are exactly the ones you'll want later. Sample everything else at whatever rate your storage budget tolerates, and revisit that rate once you know your actual growth curve rather than guessing at the start.

The trace that would have explained a customer complaint from three months ago is worthless if you deleted it two months ago to save on storage. Set the retention window with that tradeoff stated out loud, not discovered the day you need the trace that's already gone.