Grafana: One Investigation Surface, Many Backends

Enterprise problem: Responders lose time and context when dashboards, logs, traces, and profiles live in separate tools with inconsistent permissions, time ranges, and service identities.

Whole-course context: The incoming artifacts are independently proven Mimir, Loki, Tempo, and Pyroscope paths; today makes them usable through one governed investigation surface without pretending Grafana owns their telemetry.

Today’s slice: Provision data sources, folders, dashboards, Explore access, and query evidence while preserving each backend’s tenant, retention, and query-language responsibility.

End-of-day evidence: Produce a shareable fixed-time investigation that opens authorized metrics, logs, traces, and profiles with matching service/environment context.

Still unsolved: Automated cross-signal identity contracts and the final end-to-end platform architecture remain deferred.

A single UI can still expose inconsistent data or excessive authority. These use cases require a repeatable responder workflow and an independently governed provisioning path.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D14-UC-01	Incident responder	Investigate checkout degradation across four signals without losing time or service context	A fixed-time Explore/dashboard link reproduces authorized Mimir, Loki, Tempo, and Pyroscope views	A restricted data source or tenant remains inaccessible while permitted views still work
D14-UC-02	Grafana administrator	Provision data sources and dashboards safely across environments	Versioned configuration creates healthy connections with least-privilege credentials and rollback evidence	Invalid credential/config health fails visibly; rollback restores the last known-good revision

“One pane of glass” is not acceptance evidence because it says nothing about access, reproducibility, or backend health. These stories define the observable surface and its administration.

Story ID	Use case IDs	User story	Observable acceptance conditions
D14-US-01	D14-UC-01	As an incident responder, I want one fixed-time investigation with four authorized data sources, so that another responder can reproduce the same evidence	Shared link preserves absolute time, service/environment filters, queries, and panels; unauthorized source access is denied
D14-US-02	D14-UC-02	As a Grafana administrator, I want data sources and dashboards provisioned from reviewed revisions, so that changes are auditable and recoverable	Health checks pass for four sources, secrets remain external, an invalid revision fails, and rollback restores previous UIDs

A dashboard panel can render stale or partial data without proving its source, so flows start with a visible responder action and finish with query metadata and access evidence. Grafana orchestrates queries; it does not merge the storage engines into one database.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D14-FLOW-01	D14-UC-01, D14-UC-02	Happy	Responder opens the checkout incident dashboard at a fixed absolute time	1. Grafana authenticates the user.<br>2. Folder/data-source permissions are evaluated.<br>3. Variables resolve canonical service/environment values.<br>4. Each panel queries its owning backend.<br>5. Explore preserves query and time context.<br>6. Responder shares a reproducible link.	User, role, data-source UIDs, queries, absolute window, result timestamps, environment, and shared-link ID
D14-FLOW-02	D14-UC-01, D14-UC-02	Recovery	Administrator deploys a bad Loki credential revision	1. Data-source health check fails.<br>2. Metrics, traces, and profiles remain available.<br>3. Deployment is rolled back.<br>4. Loki health and the fixed investigation recover.<br>5. A restricted user remains denied.	Failed health response, unaffected-source controls, rollback revision, recovered query, denial result, and run ID

Giving Grafana one superuser credential across every backend creates a large blast radius and erases tenant attribution. Each data source receives the narrowest service identity and backend scope needed for its audience.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D14-UC-01	Dashboard, Explore, and shared-link routes	Grafana auth/RBAC, dashboard service, Explore, data-source proxy, Mimir, Loki, Tempo, Pyroscope	Grafana database for UI objects; each telemetry backend remains authoritative for its signal	Permission denial, panel/query error, stale timestamp, wrong tenant, missing UID, or link reproduction failure
D14-UC-02	Provisioning pipeline and data-source health API	Git review, secret manager, Grafana provisioning, data-source plugins, rollback controller	Git for intended configuration; secret manager for credentials; Grafana database for applied state	Revision rejection, health-check failure, secret-resolution error, UID drift, or rollback failure

If dashboard metadata is confused with telemetry authority, backup and deletion procedures can remove the wrong state or leave credentials behind. Grafana owns presentation and provisioning state; each backend owns its telemetry.

Generated-application database: Not created in this slice — Grafana’s own configuration database and external telemetry stores are sufficient; no customer application data belongs here.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
Data-source definition	Grafana database/provisioning owned by Grafana admins	datasource_uid	Opaque endpoint and secret reference	organization_id	Stable UID, least-privilege access, no inline secret, health check required	Provision, update, roll back, disable, delete; revoke secret independently	D14-UC-01, D14-UC-02
Dashboard revision	Git plus Grafana database owned by service/platform team	dashboard_uid/revision	Foreign reference to folder UID and data-source UIDs	organization_id	Stable UIDs, reviewed queries, bounded variables, no embedded credentials	Create, version, restore, archive, export, and delete	D14-UC-01, D14-UC-02
Folder permission	Grafana authorization store owned by admins	folder_uid/subject_id	Opaque identity-provider subject reference	organization_id	Explicit least-privilege role; independently revocable	Grant, review, revoke, retain audit event, expire with subject lifecycle	D14-UC-01, D14-UC-02
Investigation proof	Platform evidence store owned by operations	investigation_run_id	Opaque references to user, revisions, UIDs, queries, and shared link	organization_id	Absolute time and result/denial digests are immutable	Create per exercise, retain for audit objective, then expire	D14-UC-01, D14-UC-02

Relative “last hour” links change after handoff, so incident evidence must preserve an absolute time window. Dashboards provide curated views; Explore supports iterative query work, side-by-side sources, query inspection, and fixed-time sharing.

A panel is a data source, query, optional transformation, and visualization. Record the data-source UID and inspect the actual request when results surprise you. Mixed-source panels can be useful, but independent panels often make ownership and partial failure clearer.

Manual UI changes drift and are hard to recover, so provision stable UIDs and reviewable configuration where practical. Store credentials in an external secret mechanism and scope each data source to the required tenant and API.

Test two identities: an authorized responder and a restricted user. The restricted user must be denied the protected folder or data source without inheriting backend access merely through Grafana membership. Also test direct backend exposure; Grafana RBAC cannot protect an independently reachable backend endpoint.

A dashboard that shows four green panels does not prove cross-signal usefulness. Seed one checkout degradation and capture:

1. Mimir p99 latency and error-rate panels for a fixed time.
2. Loki timeout logs for the same service/environment window.
3. A Tempo trace whose payment span dominates.
4. A Pyroscope profile that names a hot function or falsifies local CPU.
5. A shared absolute-time Explore/dashboard link opened by a second authorized user.
6. A denied attempt from the restricted identity.

Grafana unifies investigation interaction, not storage ownership. Preserve these rules:

• Mimir, Loki, Tempo, and Pyroscope remain specialized authoritative telemetry backends.
• Stable data-source and dashboard UIDs make provisioning and links recoverable.
• Absolute time, query text, variables, and data-source identity make evidence reproducible.
• Grafana permissions and backend permissions are distinct boundaries.
• Partial data-source failure should remain visible while unaffected signals continue working.

A polished dashboard can conceal unsafe credentials or irreproducible evidence. Complete these checks:

• [ ] Provisioned stable data-source, folder, and dashboard UIDs.
• [ ] Kept credentials outside dashboard and provisioning source.
• [ ] Used least-privilege identities for each backend.
• [ ] Reproduced a four-signal investigation with absolute time.
• [ ] Injected one data-source failure and proved unaffected controls.
• [ ] Proved restricted-user denial at Grafana and backend boundaries.

Grafana roles and correlation features evolve, so use current primary documentation. These sources support the surface and provisioning model:

• Grafana Explore
• Grafana data sources
• Grafana provisioning
• Grafana role-based access control