Production Multimodal RAG: The Platform Around the Model

Enterprise problem: A reliability engineer needs a defensible answer to “Which dependency caused the error spike after the regional failover?”, yet the decisive evidence may be a dashboard image and dependency graph that text search cannot faithfully represent; a wrong or unauditable answer can extend an outage.

Whole-course context: This is the production-operability slice of a system-design journey, consuming an authenticated engineer question plus governed enterprise documents and turning them into an evidence-backed operational decision.

Today’s slice: We design the multimodal retrieval-augmented generation (RAG) product from source pixels through controlled rollout, covering the ingestion, query, control, and evidence planes rather than choosing a particular foundation model.

End-of-day evidence: The reviewable artifact is a versioned release bundle, traceable evidence packet, and readiness review that can prove which accessible dashboard region supported a served answer or why the system abstained.

Still unsolved: Domain-specific incident playbooks, connector procurement, model training, and the organisation’s final retention policy are deliberately deferred; this design supplies their enforceable boundaries.

The governing thesis is that the model is only one worker: dependable visual answers require preserved evidence, permitted retrieval, and a bounded decision.

The reliability engineer’s question is easy to mis-model as “send an image to a model,” which loses the incident-critical questions of which image, who may see it, and how can we reproduce the result? The smallest complete model shown in the opening has three responsibilities: preserve evidence, select permitted evidence, and make a bounded decision from it.

For the failover question, the source pixels are the dashboard panel and dependency graph, retrieval narrows thousands of documents to the right timestamped region, and decision names a dependency only if the packet makes that relationship visible. This model is intentionally small: it says nothing yet about queues, GPUs, or Terraform because those exist to make these three responsibilities reliable.

Decision rule: if a design cannot name its source evidence, access decision, and abstention behaviour, it is a demo pipeline rather than production RAG.

Text extraction fails when the meaning lives in layout, colour, geometry, or visual grouping, so returning every OCR token can still mislead an engineer during an outage. Retrieval-augmented generation (RAG) finds material before generation; multimodal RAG retains visual material as a first-class signal instead of treating it as disposable preprocessing.

Consider a dependency graph where an arrow crosses a region boundary, a Grafana panel where a red series spikes after failover, a financial table with merged headers, and a runbook diagram where dimensions constrain an action. Optical character recognition (OCR) can recover words without preserving which line, label, or column they belong to.

An upload is not searchable evidence until the system can prove what bytes it processed and which derived signals belong to them; otherwise retries and edits create duplicate or stale answers. The ingestion plane turns each source version into immutable pixels plus derived text, regions, descriptions, metadata, and retrieval vectors.

The content hash identifies the exact source bytes. A stable document_id identifies the business object across edits; document_version_id, page IDs, and region IDs identify one immutable rendition. Processors are idempotent: repeating the same version replaces the same derived records rather than creating another searchable copy.

document_id: doc-1842
document_version_id: doc-1842:v7
content_hash: sha256:8d0c...
pages:
  - page_id: doc-1842:v7:p12
    image_uri: objects/doc-1842/v7/p12
    regions:
      - region_id: doc-1842:v7:p12:r3
        kind: dependency-dashboard
        bounds: [0.12, 0.18, 0.86, 0.72]
index_version: visual-v5
access_labels: [engineering, reliability]

Poisoned files go to quarantine, corrupt pages are recorded without blocking healthy pages, and retries have a bounded policy. A manifest becomes visible only when required pixels, metadata, and index records agree; this atomic publication prevents a query from finding an ID whose source image does not exist.

Decision rule: publish a document version only after source pixels and every required derived record agree on the same version and access labels.

One embedding cannot represent every way an engineer asks for evidence, so a single-vector design misses exact codes, paraphrases, and visual relationships in different ways. Hybrid retrieval performs several cheap searches, applies access control before result disclosure, fuses retrieval matches, and spends expensive visual reasoning only on a small packet.

A VLM can interpret the failover graph, but it cannot be the authority for identity, permissions, source truth, release state, or deletion; making it so turns probabilistic output into a security control. The model is a bounded worker between a policy-checked evidence packet and a deterministic citation/abstention check.

The request contract should say: “Use only packet evidence; distinguish observed labels from inference; cite each material claim by region ID; abstain if the causal link is not visible.” A post-generation checker verifies citation IDs belong to the packet, detects unsupported claims where feasible, and exposes an answer as a reviewable object rather than an opaque chat completion.

A production request becomes fragile when re-indexing, experiment changes, and telemetry writes share the same responsibilities as live answering. Four planes keep the failure domains legible: ingestion creates evidence, query serves bounded answers, control changes desired state, and evidence records what happened.

The ingestion plane turns document changes into staged, versioned evidence. The query plane has a tight latency budget and never mutates the active index. The control plane owns reviewed configuration, model/prompt/index compatibility, and promotion. The evidence plane stores traces, evaluation results, and aggregated telemetry under access and retention rules.

For the recurring question, this prevents a large re-index from altering the evidence mid-incident, lets an on-call engineer receive a known active version, and gives the administrator a separate path to approve a new one.

Decision rule: a data path that answers customers should not also be the authority that rewrites its own evidence or deployment policy.

Console-created networks, accelerator pools, policies, and queues cannot be reviewed or reliably recreated after an incident, so “the cluster is configured” is not an operational guarantee. Infrastructure as code (IaC) declares the foundation, workload templates declare runtime intent, and a reconciler applies reviewed desired state.

platform/
  foundation/
    network
    identity
    storage
    accelerator-pools
    telemetry
  workloads/
    ingestion
    retrieval-api
    visual-reasoner
    evaluation
  environments/
    development
    staging
    production

Secrets in that desired state are external references to a managed secret store, never credential values committed to source. Environment overlays contain only genuine capacity and endpoint differences; policy, trust boundaries, and workload topology stay in shared modules so an overlay cannot fork the architecture into an unreviewed production variant.

Kubernetes is useful here because the query API, ingestion workers, and visual-reasoner workers have different scaling and isolation needs. Give GPU-backed model pods a dedicated node pool, explicit resource request, taint/toleration policy, and readiness check that passes only after weights load. Kubernetes documents GPU scheduling as an extended-resource request; capacity is schedulable only after the relevant device plugin exposes it.

workload: visual-reasoner
resources:
  accelerator: 1
  memory: 48Gi
scheduling:
  pool: visual-inference
  tolerate: dedicated-visual-inference
autoscaling:
  signal: pending_requests
  target_per_replica: 4
rollout:
  max_unavailable: 0
  readiness: model_loaded_and_warm

Autoscale visual workers from queue depth or pending requests, not accelerator utilisation alone: a saturated GPU might be doing long work while a new incident waits. The API retains a bounded timeout and a controlled abstention or fallback policy when visual capacity is exhausted; it must not silently replace an image-grounded question with ungrounded text generation.

Decision rule: version the whole inference contract—application, model, prompt, index schema, policy, and infrastructure assumptions—because a container-only rollback can still read incompatible evidence.

Syntax and unit tests can pass while chart recall, citation behaviour, or denial paths regress, so conventional CI alone cannot establish that a multimodal product still works. An AI quality pipeline tests the evidence path before spending on generation, using a golden set of questions, expected regions, permitted/denied identities, acceptable answers, and explicit abstentions.

Offline data cannot represent every production document or incident query, so immediate global rollout converts an unknown quality regression into a customer outage. Delivery should gain evidence in reversible stages: prove function in development, compare shadow answers, expose a canary cohort, then expand only while quality and system-health guardrails hold.

A release bundle pins compatible application image, model, prompt, policy, index, and schema inside one immutable environment_id boundary; its evaluation report independently records whether that bundle may be promoted there. Shadow and canary use a lease-bound routing override, not the ActiveBundlePointer. A pre-activation breach removes that override, records rejection, and leaves rb-18 active—there is nothing to restore because production never changed.

Only a separate regression observed after the atomic rb-18 -> rb-19 promotion enters recovery. Restoration atomically returns the pointer to rb-18 and records the completed transition; only then does a separate probe run test both the allowed failover query and a denied query. A failed probe escalates recovery, but it cannot retroactively be a prerequisite for the restoration it verifies.

This is why D37-FLOW-04 proves rejection without a pointer change, while D37-FLOW-05 proves restoration followed by verification. The administrator can answer what was served, who changed it, which guardrail failed, and whether the restored version can still retrieve doc-1842:v7:p12:r3 for a permitted engineer.

Decision rule: promote and restore a bundle pointer, never a collection of ad hoc component versions.

An error rate can prove that a service is alive while hiding that it consistently chose the wrong dashboard, so operational telemetry must describe usefulness as well as availability. Every request carries a trace ID through policy, retrieval channels, reranking, evidence loading, inference, and the citation check.

OpenTelemetry defines semantic conventions for generative AI operations so telemetry can consistently identify model operations and attributes across instrumented services. Apply such conventions carefully: retain identifiers, scores, and timing by default; access-controlled, redacted samples may retain evidence excerpts only where policy permits.

trace_id: q-7f21
release_bundle_id: rb-19
policy:
  outcome: allowed
  scope_hash: 4b9e...
retrieval:
  lexical_matches: 18
  semantic_matches: 24
  visual_matches: 20
  selected_evidence: [doc-1842:v7:p12:r3]
generation:
  citation_coverage: 1.0
  abstained: false
  latency_ms: 1840
  cost_units: 0.031

Traditional signals include request rate, errors, queue depth, tail latency, memory, and accelerator saturation. AI signals include empty retrievals, rank distribution, image-decode failures, citation coverage, groundedness, abstention rate, model/prompt/index versions, token use, and cost per grounded successful answer. Segment both by document type, source version, retrieval channel, and release bundle to connect cause with effect.

Decision rule: a useful trace explains which evidence was considered, why it won, which release served it, and where time and cost went—without becoming an uncontrolled copy of private documents.

Enterprise knowledge changes through edits, moves, relabels, and deletes, so an append-only embedding job eventually returns stale or unauthorised evidence. Continuous indexing treats those changes as ordered state transitions and gives deletion or permission revocation higher urgency than ordinary freshness.

For a rebuild, create a staged index beside the active version at sequence S, then dual-write subsequent ordered changes to both. Compare counts, sampled retrievals, and policy outcomes before atomically promoting the staged index; keep the old version through the rollback window. Tombstones must win over delayed worker writes, otherwise a retry can resurrect a deleted dashboard.

The running question makes the priority concrete. If the dashboard is reclassified outside the engineer’s scope during an incident, retrieval must stop exposing it before a performance optimisation finishes rebuilding embeddings.

Decision rule: permission revocation and deletion are correctness events with a measured propagation objective, not background cleanup.

Multimodal evidence increases both the attack surface and the bill: a document may contain instructions intended to manipulate the model, an image crop may leak through traces, and visual inference may cost far more than lexical retrieval. Security and economics therefore shape retrieval-match selection, packet construction, identity, retention, and capacity rather than appearing as a checklist after launch.

Treat documents, captions, OCR text, and visual descriptions as untrusted data; none can override system policy or tool authority. The OWASP guidance on prompt injection describes why indirect instructions in externally supplied content require controls outside the model prompt. Isolate ingestion from query serving, use short-lived workload identities, encrypt evidence, and audit the chain from user through selected region.

Readiness fails when teams verify components independently but never prove that the engineer can move from question to authorised evidence and a reversible release. Use the recurring failover question as a whole-system review, then require evidence for each dimension rather than accepting architecture claims.

1. Evidence: Does the golden set include diagrams, charts, tables, multilingual and low-resolution pages, visual-only facts, and unanswerable questions? Can it retrieve the dependency graph region for doc-1842:v7:p12:r3?
2. Failure: What happens when rendering, the index, image storage, policy service, or visual inference is slow or unavailable? Does the user receive a bounded abstention with a trace instead of a fabricated dependency?
3. Change: Can the team build a staged index and signed release bundle without mutating active evidence? When rb-19 fails a pre-activation canary guardrail, can it remove the temporary route, record promotion_blocked, and prove rb-18 stayed active without a pointer write? If rb-19 is later promoted and then regresses in production, can it atomically restore rb-18 before running permitted and denied verification probes?
4. Control: Are tenant scope, source permissions, retention, residency, audit, and malicious-document handling enforced outside the VLM? Does a denied query conceal source existence?
5. Economics: Are latency, queueing, accelerator capacity, object storage, re-index throughput, and cost per grounded answer budgeted and observable by release and tenant?

A passing review produces artefacts: golden-set report, policy test results, immutable release bundle, pointer-change event, trace sample, failure drill, and cost dashboard. These falsify the shallow claim that “we added vision”; they demonstrate that the product can make one valuable answer reliable.

Decision rule: if replacing the VLM requires redesigning ingestion, authorisation, observability, or deployment, those responsibilities are coupled too tightly to the model.

Without a single end-to-end contract, individually healthy services can still return an unauthorised, uncited, or irreproducible answer during an outage. The synthesis reconnects evidence, retrieval, reasoning, release control, and measurement around the engineer’s decision.

The central lesson is that a VLM is a replaceable reasoning worker inside an evidence system, not the system itself. Start with preserved pixels, derive searchable signals, retrieve only permitted matches, give the VLM a small evidence packet, verify citations, separate the four planes, and operate every version as a reversible bundle.

For the reliability engineer, the final answer is more than “dependency X caused the spike.” It is: “dependency X is shown by this permitted region of this immutable dashboard version, served by this release bundle, under this access decision, with this trace; here is why the platform would have abstained if that evidence were missing.” That is the difference between an impressive demonstration and an enterprise operational product.

The detail is easy to remember as a component catalogue and then apply out of order, which recreates the same evidence and access failures. These takeaways preserve the dependency sequence from source truth to reversible operation.

• Source pixels remain evidence; OCR, captions, regions, and vectors are derived retrieval signals.
• Versioned, idempotent ingestion with atomic publication makes retries, edits, and incident review safe.
• Hybrid retrieval improves coverage, but policy filtering, reranking, small evidence packets, citations, and abstention make answers governable.
• The VLM reasons over evidence; deterministic services own identity, policy, provenance, deletion, and release state.
• Ingestion, query, control, and evidence planes keep expensive changes and observational data from corrupting live answers.
• IaC, dedicated Kubernetes accelerator policy, AI CI, staged indexing, and release bundles make the platform reproducible and reversible.
• Observe quality, safety, latency, and cost together under one trace ID; optimise for grounded successful answers.

Architecture confidence without falsifiable proof can hide a broken denial path, stale index, or partial rollback until the next incident. Use this checklist to test whether the complete product contract is reviewable and operable.

• [ ] I can explain why a text extraction can retain words yet lose a dashboard’s causal relationship.
• [ ] I can define stable document, version, page, and region IDs and publish them atomically with their derived records.
• [ ] I can trace the failover question through scoped hybrid retrieval, evidence packaging, VLM reasoning, citation checking, and abstention.
• [ ] I can state why the VLM must not own identity, policy, source truth, deletion, or release state.
• [ ] I can separate ingestion, query, control, and evidence planes and name their different failure domains.
• [ ] I can define a bundle that pins application, model, prompt, policy, schema, and index versions for promotion and rollback.
• [ ] I can test retrieval recall, groundedness, citation validity, denial, latency, and cost before serving a proposed release.
• [ ] I can explain how staged indexes, ordered dual-writes, and tombstones prevent stale or forbidden evidence from reappearing.
• [ ] I can review a design for malicious documents, private telemetry, accelerator capacity, retention, and cost per grounded answer.