Langfuse vs Grafana: Observe the AI and the System

Enterprise problem: Operators either see fleet latency without the prompt, retrieval, and model context that caused one bad answer, or see rich AI traces without knowing that the API, queue, PostgreSQL, or infrastructure is breaching its service objective.

Whole-course context: Today consumes Day 38’s tenant-scoped answer evidence contract—typed answer, trace, release, and schema-version fields plus bounded retrieval, model, and tool metadata—and turns it into correlated operational and quality evidence.

Today’s slice: Instrument the customer-support assistant so Grafana detects fleet-wide symptoms and service-level objective breaches while Langfuse explains individual LLM and agent behaviour, with overlap acknowledged and ownership explicit.

End-of-day evidence: Produce correlated traces and dashboards, one actionable alert with a runbook, a redacted bad-answer investigation, and a sampling, retention, and access test under a shared trace ID.

Still unsolved: Automated prompt promotion, long-term evaluation governance, provider replacement, and cross-region disaster recovery remain outside this slice.

Thesis: Choose the investigation surface from the question, not the product list. Start with Langfuse for prompt, retrieval, tool, model, token, and evaluation behaviour; start with Grafana for fleet rate, errors, duration, saturation, logs, traces, profiles, and alerting; cross the boundary with one safe correlation contract.

The boundary is narrower than "observe everything." This day follows the exact disputed support answer stored on Day 38, the fleet symptom around it, and the evidence required to decide, recover, and confirm the result. It excludes unrestricted prompt replication, product procurement, and dashboards with no named operational decision.

The smallest model starts with a symptom, separates the question into AI behaviour or system behaviour, and ends in an action whose result can be measured. The larger flow, architecture, and ownership diagrams do not replace that model; they reveal what each box must contain in production.

Reuse the same answer from Day 38. If support asks why it cited an obsolete document, begin with its Langfuse trace and inspect retrieval, prompt, model, tool, and evaluation observations. If an alert says thousands of answers are slow, begin with Grafana and the telemetry backends to locate the affected service, region, release, queue, database, or provider. When either path crosses the boundary, carry the shared trace ID and release—never a high-cardinality customer identifier in metric labels.

This produces a reusable decision rule: one answer is not a fleet trend, and one fleet trend does not explain an answer. Use one surface when it closes the decision; correlate both when cause or recovery crosses scopes.

Using "observability" as one undifferentiated category creates false either-or comparisons because both tools can display aggregates and traces, yet their native context and operational scope differ. Langfuse's current observability model organizes individual AI operations as observations inside traces and optionally groups traces into sessions; Grafana's current telemetry guide frames metrics, logs, traces, and profiles as correlated system signals. Choose the first investigation surface by the question being asked, then correlate when the answer crosses scopes.

Without a stable correlation contract, a fleet alert cannot reach the AI trace and a support reviewer cannot determine whether one bad answer was isolated or systemic. Carry the OpenTelemetry trace ID, release, environment, service, and bounded cohort dimensions through the Day 38 row and both observability paths.

Use stable names such as answer-turn, retrieve-context, call-support-tool, and generate-response. Dynamic IDs belong in attributes, not operation names. Raw tenant_id, answer_id, trace_id, user text, and provider request IDs must not become metric labels: their cardinality grows with traffic and can make storage and queries expensive. Keep low-cardinality dimensions such as service, environment, region, release, model family, outcome, and tenant tier; retain detailed identifiers only in access-controlled traces or logs, and use a trace exemplar or link rather than a metric label.

If framework auto-instrumentation, business spans, and AI export each invent different names or trace IDs, correlation fails and duplicate noisy events obscure the useful steps. Create the trace at the request boundary, add business-safe attributes once, and project bounded views to each sink.

import { SpanStatusCode, trace } from '@opentelemetry/api';

const tracer = trace.getTracer('support-assistant', '40.0.0');

type AiTraceSink = {
  enqueueGeneration(input: {
    traceId: string;
    sessionId: string;
    name: 'generate-response';
    model: string;
    promptVersion: string;
    input: { questionClass: string };
    output: { outcome: 'answered' | 'refused' | 'failed' };
    usage: { inputTokens: number; outputTokens: number };
    metadata: { release: string; environment: string; tenantTier: string };
  }): boolean;
};

export async function answerQuestion(
  input: { sessionId: string; release: string; tenantTier: string },
  aiTrace: AiTraceSink,
): Promise<void> {
  await tracer.startActiveSpan('answer-turn', async (rootSpan) => {
    const traceId = rootSpan.spanContext().traceId;
    rootSpan.setAttributes({
      'service.name': 'support-assistant-api',
      'deployment.environment.name': 'production',
      'service.version': input.release,
      'app.tenant.tier': input.tenantTier,
    });

    try {
      const result = await tracer.startActiveSpan('generate-response', async (modelSpan) => {
        modelSpan.setAttribute('gen_ai.operation.name', 'chat');
        modelSpan.setAttribute('gen_ai.request.model', 'support-model-v8');
        const value = { inputTokens: 812, outputTokens: 146 };
        modelSpan.end();
        return value;
      });

      let telemetryQueued = false;
      try {
        telemetryQueued = aiTrace.enqueueGeneration({
          traceId,
          sessionId: input.sessionId,
          name: 'generate-response',
          model: 'support-model-v8',
          promptVersion: 'support-prompt-12',
          input: { questionClass: 'account-access' },
          output: { outcome: 'answered' },
          usage: result,
          metadata: {
            release: input.release,
            environment: 'production',
            tenantTier: input.tenantTier,
          },
        });
      } catch (telemetryError) {
        rootSpan.addEvent('ai-telemetry-enqueue-failed', {
          'error.type': telemetryError instanceof Error ? telemetryError.name : 'unknown',
        });
      }
      rootSpan.setAttribute('app.ai_telemetry.queued', telemetryQueued);
      rootSpan.setStatus({ code: SpanStatusCode.OK });
    } catch (error) {
      rootSpan.recordException(error as Error);
      rootSpan.setStatus({ code: SpanStatusCode.ERROR });
      throw error;
    } finally {
      rootSpan.end();
    }
  });
}

AiTraceSink is an application-owned adapter around the selected Langfuse SDK or OpenTelemetry integration; its enqueue operation is bounded and non-blocking, so exporter failure does not fail the customer answer. It makes redaction, sampling, and stable naming testable without coupling business code to one SDK version. Nested retrieval and tool spans follow the same active context, and the trace ID is written to Day 38’s typed column. The example sends classes and outcomes, not raw customer content; a separately approved policy can enable redacted input/output for sampled traces.

Paging on every failed model call or low evaluation score creates noise, while a single fleet average can hide a severe regional or release-specific breach. Alert on customer impact with sustained windows, bounded dimensions, explicit missing-data behaviour, and a runbook that names the first correlation step.

Buying both by default wastes operating effort, but forcing one product to answer every question can leave either AI behaviour or the surrounding system opaque. The decision follows scope, existing telemetry, risk, and who will respond.

Capturing every prompt, retrieved passage, response, and tool argument can turn observability into a second uncontrolled customer-data system, while aggressive sampling can erase the only failed trace. Governance must be part of instrumentation, not a dashboard setting added after launch.

• Sampling: Keep aggregate metrics complete where practical; tail-sample system and AI traces to retain errors, high latency, safety outcomes, and a small unbiased baseline. Record the sampling decision so missing detail is explainable. Sampling must never determine whether the authoritative Day 38 answer row exists.
• Cardinality: Never label metrics with trace, answer, session, user, or raw tenant IDs. Use bounded service, region, release, outcome, model family, and tenant-tier labels; attach a sampled trace exemplar or controlled link for detail.
• Personal and confidential data: Default to classifications, hashes, evidence IDs, and outcomes. Redact secrets, authentication material, personal data, raw retrieval passages, prompts, responses, and tool arguments before export. Test redaction with canary values.
• Retention: Assign content, metadata, aggregate, and incident evidence separate periods. A short prompt/response trace can expire while a de-identified quality aggregate remains, but deletion and consent requirements propagate to derived records.
• Access: Separate tenant support, AI quality, SRE, security, and vendor-administration roles. Authorize from the source answer before resolving observability links; project and environment boundaries are not tenant authorization.

Budget data volume as an operational requirement. Stable trace structure improves evaluation and dashboard continuity, but unnecessary HTTP, database, and framework spans can be filtered from the Langfuse AI tree while remaining available in the general tracing backend for system diagnosis.

A production assistant can pass availability checks while a new retrieval prompt selects obsolete policy documents, so fleet health alone would miss customer harm. Conversely, a slow individual trace cannot prove whether the provider, queue, or database is failing across the population.

At 10:05, Grafana shows answer p99 latency and tool-failure rate rising only for release assistant-2026.08.05.2 in one region; an error-budget alert opens incident INC-447. A trace exemplar links to OpenTelemetry trace 9f…31, whose API span spends most time in crm.lookup_customer. The same trace ID opens Langfuse: session conv-hmac-82, trace answer-turn, nested retrieve-context, call-support-tool, and generate-response observations. The tool observation failed after a malformed argument introduced by prompt version 13, then the model produced an unsupported fallback answer. Day 38’s PostgreSQL row confirms evidence schema version 1, release, provider request, tool outcome, and selected evidence IDs.

The release controller restores prompt version 12 without changing the model or database schema. Grafana shows error-budget burn and p99 latency recover; Langfuse’s fixed dataset run restores tool-success and groundedness scores; a synthetic account-access question passes; another region stays healthy throughout. The incident closes with alert, trace, answer, dataset-run, rollback, and recovery timestamps—not with "dashboard looks normal."

An observability design that works only while both exporters and both products are healthy can turn a telemetry outage into a customer outage or erase incident evidence. Failure modes must degrade independently from the answer path and expose their own freshness.

Vendor features and telemetry conventions evolve, so implementation choices should be verified against current official documentation. These sources ground the trace structure, AI metrics, and general telemetry model used in this day.

• Langfuse: Observability and application tracing
• Langfuse: Observability best practices
• Langfuse: Metrics overview
• Grafana: About metrics and telemetry

Without a concise scope boundary, teams can reduce this design to a feature contest and lose the correlation and governance work that makes incidents explainable. These points preserve the operational decision.

• Ask by scope: Grafana detects fleet-wide symptoms and SLO breaches across the API, queue, PostgreSQL, providers, and infrastructure; Langfuse explains one LLM or agent behaviour through sessions, traces, observations, versions, inputs/outputs, tokens/cost, evaluations, datasets, and feedback.
• The products overlap in tracing and metrics; complementary use is a design choice, not an exclusivity claim. Grafana visualizes data from telemetry backends rather than replacing every backend.
• Carry one OpenTelemetry trace ID plus release, environment, and bounded cohort dimensions into the Day 38 answer row and both projections; never use unbounded identifiers as metric labels.
• Stable observation names and nested AI steps behave like an API for dashboards and evaluators, while raw customer content requires purpose, redaction, retention, and access controls.
• Alerts need customer-impact objectives, sustained windows, volume guards, no-data semantics, ownership, a runbook, and representative trace correlation.
• Choose Langfuse only, Grafana only, or both by replaying an AI-quality incident and a fleet-reliability incident, then measuring whether the evidence identifies cause, owner, and recovery.

An observability rollout can look complete while correlation, tenant isolation, missing-data handling, or recovery remains untested. Use this checklist to require end-to-end evidence from both question scopes.

• [ ] Each dashboard, trace, evaluation, and alert names the customer or operator question it answers.
• [ ] One assistant turn maps to one trace, one conversation maps to a session, and retrieval, generation, and tool calls are nested observations with stable names.
• [ ] The Day 38 answer row, OpenTelemetry trace, Langfuse trace, alert, and incident share a trace ID and release without putting unique IDs in metric labels.
• [ ] Grafana covers metrics, logs, traces, profiles, SLOs, and alerting across API, queue, PostgreSQL, providers, and infrastructure backends.
• [ ] Langfuse traces record approved prompt/model versions, bounded inputs/outputs, retrieval/tool steps, tokens/cost, evaluations, datasets, and feedback.
• [ ] Sampling retains failures, high latency, safety outcomes, and an unbiased baseline; every missing detail exposes its sampling or retention reason.
• [ ] Redaction canaries, tenant-denial tests, role reviews, retention expiry, and derived-data deletion are verified in every sink.
• [ ] Alerts declare objective, windows, minimum volume, no-data and error states, owner, runbook, and a correlation link.
• [ ] A rollback drill proves fleet objectives recover in Grafana, answer quality recovers in Langfuse, and an unaffected control remains healthy.
• [ ] Exporter and backend failures do not block customer answers and emit their own queue, drop, freshness, and replay evidence.