Langfuse vs Grafana: Observe the AI and the System

Enterprise problem: Operators either see fleet latency without the prompt, retrieval, and model context that caused one bad answer, or see rich AI traces without knowing that the API, queue, PostgreSQL, or infrastructure is breaching its service objective.

Whole-course context: Today consumes Day 39’s tenant-scoped answer evidence contract—typed answer, trace, release, and schema-version fields plus bounded retrieval, model, and tool metadata—and turns it into correlated operational and quality evidence.

Today’s slice: Instrument the customer-support assistant so Grafana detects fleet-wide symptoms and service-level objective breaches while Langfuse explains individual LLM and agent behaviour, with overlap acknowledged and ownership explicit.

End-of-day evidence: Produce correlated traces and dashboards, one actionable alert with a runbook, a redacted bad-answer investigation, and a sampling, retention, and access test under a shared trace ID.

Still unsolved: Automated prompt promotion, long-term evaluation governance, provider replacement, and cross-region disaster recovery remain outside this slice.

Thesis: Choose the investigation surface from the question, not the product list. Start with Langfuse for prompt, retrieval, tool, model, token, and evaluation behaviour; start with Grafana for fleet rate, errors, duration, saturation, logs, traces, profiles, and alerting; cross the boundary with one safe correlation contract.

The boundary is narrower than "observe everything." This day follows the exact disputed support answer stored on Day 39, the fleet symptom around it, and the evidence required to decide, recover, and confirm the result. It excludes unrestricted prompt replication, product procurement, and dashboards with no named operational decision.

Choosing an observability product without naming the question leads teams to send every signal everywhere and still miss the incident. These use cases separate one-answer explanation from fleet operation while preserving a correlation path between them.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D40-UC-01	Tenant support lead	Explain why one assistant answer was wrong, slow, costly, or unsafe	A tenant-authorized trace shows prompt/model version, retrieval and tool steps, tokens/cost, evaluations, feedback, and the bounded Day 39 evidence under one trace ID	Unauthorized tenant access is denied; unsampled or expired content still yields a minimal audit record and explicit unavailable reason
D40-UC-02	Site reliability engineer	Detect and contain a fleet-wide support-assistant reliability breach	A symptom alert identifies affected service, region, release, objective, and representative trace correlation so the engineer can isolate API, queue, PostgreSQL, provider, or infrastructure	Alert-data loss pages separately; rollback restores the objective; unaffected region and synthetic-answer controls prove containment

Tool names alone do not define useful observability because two actors require different evidence and response times. The stories state which product question must be answerable and which denial or recovery observation prevents a false green result.

Story ID	Use case IDs	User story	Observable acceptance conditions
D40-US-01	D40-UC-01	As a tenant support lead, I want to open the AI trace for a disputed answer, so that I can distinguish retrieval failure, prompt or model regression, tool failure, and customer feedback without seeing another tenant’s content	One authorized trace links answer and session, shows stable nested observations and versions, records token/cost and evaluation results, redacts configured fields, and denies the same lookup under another tenant
D40-US-02	D40-UC-02	As a site reliability engineer, I want an SLO-based alert to link fleet telemetry with representative AI traces, so that I can contain the failing release without paging on every individual low score	The alert uses rate or latency windows rather than raw event pages, names release and region, links sampled trace IDs without high-cardinality metric labels, records acknowledgement and rollback, and shows objective recovery plus an unaffected control

A dashboard screenshot or isolated trace cannot prove that an operator can travel from a customer-visible symptom to owned state and back to recovery. These flows begin with product and operations actions and end with correlated, reviewable evidence.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D40-FLOW-01	D40-UC-01	Happy	A support lead opens "Investigate answer" on a disputed response	1. The product authorizes tenant and answer access.<br>2. It resolves the typed Day 39 trace_id, release, and evidence schema version.<br>3. Langfuse loads the trace within its session and nested retrieval, generation, and tool observations.<br>4. The reviewer compares response, evidence, prompt/model version, tokens/cost, scores, and feedback.<br>5. The reviewer records a bounded diagnosis.	Actor, tenant authorization result, answer ID, trace ID, release, evidence version, expected and observed behaviour, evaluation or feedback ID, environment, timestamp, and diagnosis code
D40-FLOW-02	D40-UC-01	Denied	A support lead requests an answer from another tenant or content removed by retention	1. The product checks tenant ownership before querying observability stores.<br>2. Cross-tenant access returns denied without revealing trace existence.<br>3. An expired trace returns metadata-only status rather than fabricated content.<br>4. A same-tenant retained trace succeeds as a positive control.	Access audit ID, denial reason, subject and resource scope, retention state, no leaked prompt or response, and positive-control trace ID
D40-FLOW-03	D40-UC-02	Happy	The assistant’s rolling error-budget burn or tail latency crosses the alert threshold	1. Metrics backend evaluates fleet signals for API, queue, PostgreSQL, model provider, and infrastructure.<br>2. Grafana presents and routes a sustained actionable alert.<br>3. The SRE segments by release and region, then opens a sampled trace exemplar.<br>4. The shared trace ID links system spans to the Langfuse AI trace and Day 39 answer evidence.<br>5. The SRE identifies the responsible component and applies the runbook.	Alert rule/version, objective and burn windows, affected dimensions, dashboard range, representative trace ID, release, owner, runbook action, environment, and timestamp
D40-FLOW-04	D40-UC-02	Recovery	A new prompt and model release increases tool failures and answer latency	1. The SRE pauses rollout and restores the prior release pointer.<br>2. Grafana confirms fleet error and latency objectives recover.<br>3. Langfuse compares failed and control traces across releases and runs the evaluation dataset.<br>4. Synthetic questions verify quality while an unaffected region verifies isolation.<br>5. The incident record captures recovery and follow-up ownership.	Rollback event, before/after SLO values, failed and control trace IDs, dataset run ID, quality scores, unaffected-region result, incident ID, and recovery timestamp

If both products ingest the same unrestricted payload and each is treated as authoritative, data cost, privacy exposure, and incident disagreement grow together. The design emits one correlation contract while assigning AI-behaviour and fleet-health records to the systems built to answer those questions.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D40-UC-01	Investigate answer action	Support evidence API; Day 39 answer repository; telemetry policy/redaction layer; Langfuse trace exporter; evaluation and feedback pipeline	PostgreSQL remains authoritative for tenant/answer ownership and bounded answer evidence; Langfuse owns AI trace, observation, evaluation, dataset-run, and feedback views	Tenant denial audit; missing/expired trace status; exporter drop count; redaction test; evaluation run failure
D40-UC-02	SLO alert and incident console	OpenTelemetry instrumentation; metrics, log, trace, and profile backends; Grafana visualization and alerting; release controller; synthetic probe	Each telemetry backend owns its signal; Grafana owns dashboards and alert rules but visualizes or queries backend data rather than replacing those stores; release controller owns rollout state	Telemetry freshness alert; no-data state; alert evaluation failure; missing exemplar; rollback event; failed synthetic control

If correlation fields have no authoritative owner or lifecycle, a trace link can cross tenants, outlive the underlying answer, or point to conflicting release data. The model keeps customer ownership in PostgreSQL and treats observability records as purpose-limited projections with independent retention.

Generated-application database: Not created in this slice — Day 40 reads the existing Day 39 PostgreSQL answer/evidence store and adds only purpose-limited observability records in their owning backends.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
AssistantAnswerEvidence	Existing PostgreSQL assistant_answer; answer repository	answer_id UUID	trace_id is an opaque correlation reference; conversation is a local FK	tenant_id	Typed tenant, trace, release, schema version, and status remain authoritative; envelope is validated and bounded	Answer and envelope follow customer retention/deletion; observability copies do not extend it	D40-UC-01, D40-UC-02
LangfuseSessionTraceObservation	Langfuse project; AI observability owner	session_id, trace_id, and observation ID at their respective levels	Opaque answer_id; parent trace or observation relationship	Tenant-safe pseudonymous subject or restricted tenant attribute	One trace per assistant turn; stable low-cardinality observation names; nested retrieval, generation, and tool steps; redacted input/output policy	Sampled or complete trace retained by quality/support policy; content deletion propagates; aggregate scores may outlive content only when policy permits	D40-UC-01, D40-UC-02
EvaluationFeedback	Langfuse evaluation/dataset store; AI quality owner	Score, feedback, or dataset-run ID	Opaque trace, observation, prompt version, and dataset item references	Same tenant-safe policy as the referenced trace	Evaluator version and provenance required; human feedback is not overwritten by model score; missing trace is explicit	Retained for declared quality period; delete or detach when source consent or content expires	D40-UC-01
SystemTelemetrySignal	Metrics/log/trace/profile backend; platform observability owner	Backend-specific series, event, span, or profile key	Shared trace ID as exemplar or trace reference; release ID as bounded dimension	No raw tenant ID in metric labels; approved cohort only, with restricted tenant detail in traces/logs	Metric labels are bounded; span names stable; secrets and raw prompts forbidden; ingestion freshness measured	Metrics aggregate briefly or roll up; logs/traces/profiles use tiered retention and access; deletion follows telemetry policy	D40-UC-01, D40-UC-02
DashboardAlertIncident	Grafana plus incident system; SRE owner	Dashboard UID, alert-rule UID, and incident ID	References telemetry queries, runbook, release, and representative trace IDs	Tenant cohort only unless incident access explicitly permits a tenant reference	Alert has owner, objective, windows, no-data/error state, severity, and deduplication labels	Versioned as code where possible; alert history follows operations retention; incident closes only after recovery evidence	D40-UC-02

The smallest model starts with a symptom, separates the question into AI behaviour or system behaviour, and ends in an action whose result can be measured. The larger flow, architecture, and ownership diagrams do not replace that model; they reveal what each box must contain in production.

Reuse the same answer from Day 39. If support asks why it cited an obsolete document, begin with its Langfuse trace and inspect retrieval, prompt, model, tool, and evaluation observations. If an alert says thousands of answers are slow, begin with Grafana and the telemetry backends to locate the affected service, region, release, queue, database, or provider. When either path crosses the boundary, carry the shared trace ID and release—never a high-cardinality customer identifier in metric labels.

This produces a reusable decision rule: one answer is not a fleet trend, and one fleet trend does not explain an answer. Use one surface when it closes the decision; correlate both when cause or recovery crosses scopes.

Using "observability" as one undifferentiated category creates false either-or comparisons because both tools can display aggregates and traces, yet their native context and operational scope differ. Langfuse's current observability model organizes individual AI operations as observations inside traces and optionally groups traces into sessions; Grafana's current telemetry guide frames metrics, logs, traces, and profiles as correlated system signals. Choose the first investigation surface by the question being asked, then correlate when the answer crosses scopes.

Question	First surface	Why	Correlation hand-off
Are customers broadly failing the response-latency or availability objective?	Grafana	Time-series metrics and alerts expose fleet rate, errors, duration, saturation, and change by service, region, and release	Open a representative distributed trace or exemplar, then its Langfuse trace ID when AI steps dominate
Why did this answer cite the wrong document?	Langfuse	The trace tree retains the assistant turn’s prompt/model version, retrieval observations, generation, tool calls, input/output policy, token use, cost, and scores	Query the same trace ID in system telemetry for database, queue, or provider timing
Did a prompt or model release lower answer quality?	Langfuse, then Grafana	Evaluation scores, datasets, feedback, prompt/model/release dimensions compare AI behaviour; fleet traffic and latency show operational impact	Join by release and bounded cohorts, never by raw prompt labels
Is PostgreSQL saturation making retrieval appear slow?	Grafana, then Langfuse	Database, host, API, queue, logs, traces, and profiles show the shared bottleneck	Inspect a slow AI trace to confirm which retrieval observation waited and which answer users saw

Langfuse groups individual observations into a trace and traces into sessions. Its official tracing guidance recommends one chatbot turn per trace and a multi-turn conversation as a session; retrieval, generation, and tool calls become stable nested observations. It adds AI-specific prompt/model context, inputs and outputs under policy, tokens/cost, scores, datasets, experiments, and feedback.

Grafana spans general system telemetry: metrics give the high-level state and foundation for alerts; logs add event context; distributed traces show request paths and latency; profiles show compute use. Grafana's documentation notes that metrics are stored in a time-series database and accepted for visualization, so Grafana does not make the owning telemetry backends unnecessary.

The scopes overlap: Langfuse has dashboards and metrics derived from traces and evaluations, while Grafana can inspect application traces and AI-related metrics. The recommendation is complementary use where both question classes matter, not product exclusivity.

Without a stable correlation contract, a fleet alert cannot reach the AI trace and a support reviewer cannot determine whether one bad answer was isolated or systemic. Carry the OpenTelemetry trace ID, release, environment, service, and bounded cohort dimensions through the Day 39 row and both observability paths.

Use stable names such as answer-turn, retrieve-context, call-support-tool, and generate-response. Dynamic IDs belong in attributes, not operation names. Raw tenant_id, answer_id, trace_id, user text, and provider request IDs must not become metric labels: their cardinality grows with traffic and can make storage and queries expensive. Keep low-cardinality dimensions such as service, environment, region, release, model family, outcome, and tenant tier; retain detailed identifiers only in access-controlled traces or logs, and use a trace exemplar or link rather than a metric label.

If framework auto-instrumentation, business spans, and AI export each invent different names or trace IDs, correlation fails and duplicate noisy events obscure the useful steps. Create the trace at the request boundary, add business-safe attributes once, and project bounded views to each sink.

import { SpanStatusCode, trace } from '@opentelemetry/api';

const tracer = trace.getTracer('support-assistant', '40.0.0');

type AiTraceSink = {
  enqueueGeneration(input: {
    traceId: string;
    sessionId: string;
    name: 'generate-response';
    model: string;
    promptVersion: string;
    input: { questionClass: string };
    output: { outcome: 'answered' | 'refused' | 'failed' };
    usage: { inputTokens: number; outputTokens: number };
    metadata: { release: string; environment: string; tenantTier: string };
  }): boolean;
};

export async function answerQuestion(
  input: { sessionId: string; release: string; tenantTier: string },
  aiTrace: AiTraceSink,
): Promise<void> {
  await tracer.startActiveSpan('answer-turn', async (rootSpan) => {
    const traceId = rootSpan.spanContext().traceId;
    rootSpan.setAttributes({
      'service.name': 'support-assistant-api',
      'deployment.environment.name': 'production',
      'service.version': input.release,
      'app.tenant.tier': input.tenantTier,
    });

    try {
      const result = await tracer.startActiveSpan('generate-response', async (modelSpan) => {
        modelSpan.setAttribute('gen_ai.operation.name', 'chat');
        modelSpan.setAttribute('gen_ai.request.model', 'support-model-v8');
        const value = { inputTokens: 812, outputTokens: 146 };
        modelSpan.end();
        return value;
      });

      let telemetryQueued = false;
      try {
        telemetryQueued = aiTrace.enqueueGeneration({
          traceId,
          sessionId: input.sessionId,
          name: 'generate-response',
          model: 'support-model-v8',
          promptVersion: 'support-prompt-12',
          input: { questionClass: 'account-access' },
          output: { outcome: 'answered' },
          usage: result,
          metadata: {
            release: input.release,
            environment: 'production',
            tenantTier: input.tenantTier,
          },
        });
      } catch (telemetryError) {
        rootSpan.addEvent('ai-telemetry-enqueue-failed', {
          'error.type': telemetryError instanceof Error ? telemetryError.name : 'unknown',
        });
      }
      rootSpan.setAttribute('app.ai_telemetry.queued', telemetryQueued);
      rootSpan.setStatus({ code: SpanStatusCode.OK });
    } catch (error) {
      rootSpan.recordException(error as Error);
      rootSpan.setStatus({ code: SpanStatusCode.ERROR });
      throw error;
    } finally {
      rootSpan.end();
    }
  });
}

AiTraceSink is an application-owned adapter around the selected Langfuse SDK or OpenTelemetry integration; its enqueue operation is bounded and non-blocking, so exporter failure does not fail the customer answer. It makes redaction, sampling, and stable naming testable without coupling business code to one SDK version. Nested retrieval and tool spans follow the same active context, and the trace ID is written to Day 39’s typed column. The example sends classes and outcomes, not raw customer content; a separately approved policy can enable redacted input/output for sampled traces.

Paging on every failed model call or low evaluation score creates noise, while a single fleet average can hide a severe regional or release-specific breach. Alert on customer impact with sustained windows, bounded dimensions, explicit missing-data behaviour, and a runbook that names the first correlation step.

Alert	Signal and windows	Page when	Do not page when	Required annotation
Fast availability burn	Error-budget burn over short and long windows	Both windows exceed the chosen burn threshold for production traffic	One retry succeeds and the user receives a valid answer within objective	SLO, release, region, dashboard, representative trace link, runbook, owner
Tail-latency breach	p95/p99 answer duration plus queue and provider spans	Sustained customer latency breaches objective and volume is sufficient	One isolated slow trace or known synthetic maintenance probe	Objective, observed percentile, volume, suspected components, trace exemplar
AI-quality regression	Aggregated validated score or feedback rate by release with minimum sample size	Release delta exceeds reviewed threshold and confidence/sample guard holds	One subjective rating, evaluator outage, or changing dataset mix	Evaluator/dataset version, sample size, baseline release, Langfuse comparison link
Telemetry blind spot	Export drop rate, ingestion delay, or alert-query error	Observability freshness exceeds incident tolerance	No customer traffic and expected zero data is declared	Affected sink, last good timestamp, fallback evidence path, telemetry owner

Grafana is the natural paging surface for system objectives because metrics are continuously compared with thresholds and correlated with logs, traces, and profiles. Langfuse quality metrics can feed a release gate, review queue, or bounded alert, but non-deterministic evaluations and sparse feedback usually need volume guards and human review before paging. Always define what no data means: healthy zero, exporter failure, query failure, or retention expiry are not interchangeable.

Buying both by default wastes operating effort, but forcing one product to answer every question can leave either AI behaviour or the surrounding system opaque. The decision follows scope, existing telemetry, risk, and who will respond.

Deployment choice	Choose when	What it answers well	Material gap and compensating control
Langfuse only	Prototype or bounded AI workflow already runs on a managed platform with adequate infrastructure monitoring, and the immediate risk is prompt, retrieval, tool, cost, or answer quality	Individual LLM/agent behaviour; sessions/traces/observations; prompt/model versions; tokens/cost; evaluations, datasets, experiments, and feedback	Does not by itself provide complete fleet observability across API, queue, PostgreSQL, hosts, and network; rely on provider monitoring and declare its incident limitations
Grafana only	Workload has little AI-specific branching, prompts/outputs cannot be retained, or existing OpenTelemetry and backend signals fully answer current operational questions	Cross-service metrics, logs, traces, profiles, visualization, SLOs, alerting, and infrastructure correlation	AI trace semantics, evaluation datasets, prompt comparison, and feedback workflows require custom attributes, dashboards, and systems
Both	Production AI behaviour and fleet reliability are independently material, and teams can govern two stores	Grafana detects population-level symptoms and SLO breaches; Langfuse explains one AI behaviour or quality failure; shared correlation links the investigation	Duplicate telemetry, inconsistent retention, access drift, and cost; prevent with one field contract, purpose-specific projections, and periodic reconciliation

Do not decide from feature-list overlap alone. Run two representative incidents—one infrastructure saturation event and one wrong-answer regression—and measure whether the chosen path identifies owner and recovery evidence within the required time.

Capturing every prompt, retrieved passage, response, and tool argument can turn observability into a second uncontrolled customer-data system, while aggressive sampling can erase the only failed trace. Governance must be part of instrumentation, not a dashboard setting added after launch.

• Sampling: Keep aggregate metrics complete where practical; tail-sample system and AI traces to retain errors, high latency, safety outcomes, and a small unbiased baseline. Record the sampling decision so missing detail is explainable. Sampling must never determine whether the authoritative Day 39 answer row exists.
• Cardinality: Never label metrics with trace, answer, session, user, or raw tenant IDs. Use bounded service, region, release, outcome, model family, and tenant-tier labels; attach a sampled trace exemplar or controlled link for detail.
• Personal and confidential data: Default to classifications, hashes, evidence IDs, and outcomes. Redact secrets, authentication material, personal data, raw retrieval passages, prompts, responses, and tool arguments before export. Test redaction with canary values.
• Retention: Assign content, metadata, aggregate, and incident evidence separate periods. A short prompt/response trace can expire while a de-identified quality aggregate remains, but deletion and consent requirements propagate to derived records.
• Access: Separate tenant support, AI quality, SRE, security, and vendor-administration roles. Authorize from the source answer before resolving observability links; project and environment boundaries are not tenant authorization.

Budget data volume as an operational requirement. Stable trace structure improves evaluation and dashboard continuity, but unnecessary HTTP, database, and framework spans can be filtered from the Langfuse AI tree while remaining available in the general tracing backend for system diagnosis.

A production assistant can pass availability checks while a new retrieval prompt selects obsolete policy documents, so fleet health alone would miss customer harm. Conversely, a slow individual trace cannot prove whether the provider, queue, or database is failing across the population.

At 10:05, Grafana shows answer p99 latency and tool-failure rate rising only for release assistant-2026.08.05.2 in one region; an error-budget alert opens incident INC-447. A trace exemplar links to OpenTelemetry trace 9f…31, whose API span spends most time in crm.lookup_customer. The same trace ID opens Langfuse: session conv-hmac-82, trace answer-turn, nested retrieve-context, call-support-tool, and generate-response observations. The tool observation failed after a malformed argument introduced by prompt version 13, then the model produced an unsupported fallback answer. Day 39’s PostgreSQL row confirms evidence schema version 1, release, provider request, tool outcome, and selected evidence IDs.

The release controller restores prompt version 12 without changing the model or database schema. Grafana shows error-budget burn and p99 latency recover; Langfuse’s fixed dataset run restores tool-success and groundedness scores; a synthetic account-access question passes; another region stays healthy throughout. The incident closes with alert, trace, answer, dataset-run, rollback, and recovery timestamps—not with "dashboard looks normal."

An observability design that works only while both exporters and both products are healthy can turn a telemetry outage into a customer outage or erase incident evidence. Failure modes must degrade independently from the answer path and expose their own freshness.

Failure injection	Expected containment	Recovery	Required evidence
Langfuse export is unavailable	Assistant response continues; bounded local queue applies backpressure limits without blocking indefinitely	Retry in batches; drop according to policy when buffer expires; preserve aggregate drop counters and Day 39 evidence	Response latency, queue depth, dropped-event count, last successful export, replay IDs
Metrics backend or Grafana alert query fails	AI tracing and customer responses continue; telemetry-blind-spot alert uses an independent path where possible	Restore backend/query and replay allowed buffered data	Query error, no-data classification, last good sample, restored freshness, missed-window review
Trace ID missing from one sink	Investigation reports broken correlation rather than guessing	Repair propagation; query by bounded release/time only as a controlled fallback	Correlation success-rate metric, affected release, repaired trace pair, positive control
Redaction canary appears in an observability store	Export is quarantined and security owner is notified	Revoke exposed access, delete affected records, fix policy, replay only safe events	Canary ID, affected sinks, access audit, deletion proof, fixed negative test
Sampling removes disputed trace content	Authoritative answer and metadata remain; UI states detail unavailable	Reproduce using approved dataset or temporarily raise bounded sampling for the affected cohort	Sampling decision, retained metadata, reproduction trace, expiry of temporary rule

The customer-serving path must not wait synchronously for dashboards or AI trace storage. Export asynchronously with bounded buffers and explicit loss metrics; preserve the authoritative answer evidence independently; test shutdown flushing without promising zero loss when process termination makes that impossible.

Vendor features and telemetry conventions evolve, so implementation choices should be verified against current official documentation. These sources ground the trace structure, AI metrics, and general telemetry model used in this day.

• Langfuse: Observability and application tracing
• Langfuse: Observability best practices
• Langfuse: Metrics overview
• Grafana: About metrics and telemetry

Without a concise scope boundary, teams can reduce this design to a feature contest and lose the correlation and governance work that makes incidents explainable. These points preserve the operational decision.

• Ask by scope: Grafana detects fleet-wide symptoms and SLO breaches across the API, queue, PostgreSQL, providers, and infrastructure; Langfuse explains one LLM or agent behaviour through sessions, traces, observations, versions, inputs/outputs, tokens/cost, evaluations, datasets, and feedback.
• The products overlap in tracing and metrics; complementary use is a design choice, not an exclusivity claim. Grafana visualizes data from telemetry backends rather than replacing every backend.
• Carry one OpenTelemetry trace ID plus release, environment, and bounded cohort dimensions into the Day 39 answer row and both projections; never use unbounded identifiers as metric labels.
• Stable observation names and nested AI steps behave like an API for dashboards and evaluators, while raw customer content requires purpose, redaction, retention, and access controls.
• Alerts need customer-impact objectives, sustained windows, volume guards, no-data semantics, ownership, a runbook, and representative trace correlation.
• Choose Langfuse only, Grafana only, or both by replaying an AI-quality incident and a fleet-reliability incident, then measuring whether the evidence identifies cause, owner, and recovery.

An observability rollout can look complete while correlation, tenant isolation, missing-data handling, or recovery remains untested. Use this checklist to require end-to-end evidence from both question scopes.

• [ ] Each dashboard, trace, evaluation, and alert names the customer or operator question it answers.
• [ ] One assistant turn maps to one trace, one conversation maps to a session, and retrieval, generation, and tool calls are nested observations with stable names.
• [ ] The Day 39 answer row, OpenTelemetry trace, Langfuse trace, alert, and incident share a trace ID and release without putting unique IDs in metric labels.
• [ ] Grafana covers metrics, logs, traces, profiles, SLOs, and alerting across API, queue, PostgreSQL, providers, and infrastructure backends.
• [ ] Langfuse traces record approved prompt/model versions, bounded inputs/outputs, retrieval/tool steps, tokens/cost, evaluations, datasets, and feedback.
• [ ] Sampling retains failures, high latency, safety outcomes, and an unbiased baseline; every missing detail exposes its sampling or retention reason.
• [ ] Redaction canaries, tenant-denial tests, role reviews, retention expiry, and derived-data deletion are verified in every sink.
• [ ] Alerts declare objective, windows, minimum volume, no-data and error states, owner, runbook, and a correlation link.
• [ ] A rollback drill proves fleet objectives recover in Grafana, answer quality recovers in Langfuse, and an unaffected control remains healthy.
• [ ] Exporter and backend failures do not block customer answers and emit their own queue, drop, freshness, and replay evidence.