Production Multimodal RAG: The Platform Around the Model

Retrieval-augmented generation, or RAG, answers a question by finding relevant source material and giving that material to a generative model. A text-only pipeline works when meaning survives extraction into sentences. It fails when meaning lives in position, colour, geometry, visual grouping, or the relationship between labels and shapes.

Consider four ordinary documents: a network diagram whose arrows show failover, a dashboard whose red line crosses a threshold, a financial table whose columns have merged headers, and an engineering drawing whose dimensions sit beside the parts they constrain. Optical text extraction may recover every visible word while losing the relationships that make those words useful. A caption can summarize the page, but summaries discard detail before the user's question is known.

The first architecture therefore keeps two parallel forms of every page: the source pixels for later reasoning and derived representations for efficient retrieval.

Rule of thumb: retrieval representations are indexes, not evidence. Preserve the original page or region so the answering model can inspect what the index only approximated.

Production ingestion converts a changing document corpus into searchable, traceable evidence. Each source object receives a stable document identifier; each version receives a content hash; each page and detected region receives a child identifier. That lineage lets the system answer a basic question after any incident: which exact source bytes produced this answer?

The pipeline renders pages, extracts text, detects figures and tables, creates visual descriptions, computes several retrieval vectors, attaches access and provenance metadata, and writes the outputs atomically. A work queue separates upload latency from heavy processing, while idempotency makes retries safe: processing the same document version twice produces the same identifiers and replaces, rather than duplicates, its index records.

document_id: doc-1842
version: sha256:8d0c...
pages:
  - page_id: doc-1842:v7:p12
    image_uri: objects/doc-1842/v7/p12
    regions:
      - region_id: doc-1842:v7:p12:r3
        kind: chart
        bounds: [0.12, 0.18, 0.86, 0.72]
index_version: visual-v5
access_labels: [engineering, reliability]

Failures belong in the design. Poisoned files go to a quarantine path; corrupt pages are recorded without blocking healthy pages; retries have a limit; and the manifest does not become visible until every required artifact is durable. Rule of thumb: publish a document version only after its image, metadata, and index records agree; partial visibility turns ordinary retries into missing or duplicated evidence.

A user's question can match a document in different ways. Exact identifiers and error codes favour lexical search. Paraphrased concepts favour semantic text vectors. Questions about shapes, colours, or chart trends favour visual vectors. A production retriever gathers matches from several channels, merges them, filters them by access policy, and reranks the surviving pages or regions against the actual question.

The answering model should receive a small evidence packet rather than the entire corpus: high-resolution images for the best visual matches, extracted text for quoting and search context, coordinates for citations, and source metadata for traceability. The model is asked to answer only from that packet and to return evidence identifiers with each material claim.

Rule of thumb: optimize retrieval and answering separately. A fluent answer cannot repair missing evidence, and high retrieval recall cannot guarantee a grounded answer.

The complete service is easier to reason about when split into four planes. The ingestion plane turns sources into versioned evidence. The query plane retrieves and answers. The control plane owns configuration, policy, model and prompt versions, deployment, and index promotion. The evidence plane records traces, evaluation results, and operational telemetry.

Separating these planes prevents expensive or risky work from contaminating the request path. Re-indexing does not write directly into the active index; it builds a staged version that can be evaluated and promoted. Observability does not depend on parsing response text; every stage emits structured events under one trace identifier. Configuration changes are reviewed and reproducible rather than edited on live hosts.

Rule of thumb: never mutate the active index in place. Build, test, and promote an immutable version so index rollback is as concrete as application rollback.

The platform has more deployable state than ordinary application code: networks, storage, accelerator pools, queues, indexes, service identities, policies, model-serving configuration, prompts, and evaluation thresholds. If any of those exist only as a console click or a live edit, two environments that look equivalent can behave differently.

Declarative infrastructure modules describe the foundation. Package templates describe each workload. A deployment reconciler applies reviewed desired state. Secrets remain external references, never values in source control. Environment overlays should contain only genuine environmental differences such as capacity and endpoints; they should not fork the architecture.

platform/
  foundation/
    network
    identity
    storage
    accelerator-pools
    telemetry
  workloads/
    ingestion
    retrieval-api
    model-serving
    evaluation
  environments/
    development
    staging
    production

Accelerator scheduling deserves explicit policy. Model workers request the accelerator class they require, tolerate only the dedicated node pool, expose readiness only after model weights are loaded, and scale from queued work rather than processor utilization alone. The query service retains a bounded timeout and a controlled fallback when the visual model pool is saturated.

workload: visual-reasoner
resources:
  accelerator: 1
  memory: 48Gi
scheduling:
  pool: visual-inference
autoscaling:
  signal: pending_requests
  target_per_replica: 4
rollout:
  max_unavailable: 0
  readiness: model_loaded

Rule of thumb: version the whole inference contract — application, model, prompt, index schema, and policy — because rolling back only the container can leave the system speaking to incompatible evidence.

Conventional checks still matter: formatting, unit tests, dependency scanning, policy validation, and a reproducible build. They are necessary but insufficient because the most damaging regressions can be syntactically valid: a new embedding model lowers chart recall, a prompt stops citing evidence, or a page-rendering change crops legends out of every graph.

A golden set supplies representative questions, expected evidence, acceptable answers, and explicit abstention cases. Every change runs retrieval metrics first because there is no reason to spend on generation when the correct evidence is absent. The answer stage then measures groundedness, citation validity, task accuracy, refusal behaviour, latency, and cost against thresholds and the current production baseline.

Rule of thumb: compare against both an absolute floor and the production baseline. A release can pass a weak threshold while still being materially worse than what users have today.

Offline evaluation cannot reproduce every production document or question, so continuous delivery should increase exposure in reversible stages. First send production-shaped traffic to the new stack without returning its answers. Then expose it to a small cohort, compare it with the current release, and expand only while quality and system-health guardrails hold.

The release bundle pins all coupled versions. Promotion changes a small routing pointer, not the underlying artifacts. Rollback restores the previous pointer for the application, model, prompt, and index together. Synthetic questions continue after deployment so a quiet system can still reveal broken retrieval or expired credentials.

Rule of thumb: promotion should be a reversible pointer change, and rollback should restore every coupled version together—not merely yesterday's application binary.

Traditional signals explain whether the platform is alive: request rate, errors, saturation, queue depth, tail latency, memory, and accelerator utilization. AI signals explain whether it is useful: retrieval recall, rank distribution, empty retrievals, image-decode failures, citation coverage, groundedness, abstention, model and prompt version, token use, and cost per successful answer.

Every request receives a trace identifier carried through authorization, retrieval channels, reranking, evidence loading, model inference, and post-generation checks. The trace stores identifiers and scores by default—not unrestricted document contents. Carefully sampled, access-controlled traces may retain redacted evidence for diagnosis, subject to retention and residency policy.

trace_id: q-7f21
release: app-19_model-8_prompt-12_index-31
retrieval:
  lexical_matches: 18
  semantic_matches: 24
  visual_matches: 20
  fused_matches: 37
  selected_evidence: [doc-1842:v7:p12:r3]
generation:
  citation_coverage: 1.0
  abstained: false
  latency_ms: 1840
  cost_units: 0.031

Dashboards should connect cause and effect. If answer quality drops, operators can segment by index version, document type, ingestion version, model version, and retrieval channel. If latency rises, the same trace separates queueing, image fetch, reranking, inference, and post-check time. Rule of thumb: a useful AI trace can explain which evidence was considered, why it won, which release produced the answer, and where time and cost were spent—without becoming an uncontrolled copy of private data.

Enterprise knowledge changes continuously: files are added, edited, moved, reclassified, and deleted. A crawler that only appends new embeddings creates stale answers and access leaks. The indexer must treat updates and deletions as ordered state transitions, preserve source versioning, and propagate permission changes with higher urgency than content improvements.

For a large rebuild, create a staged index beside the active one. Feed live changes to both after a known sequence point, compare counts and sampled queries, then atomically promote the staged version. Keep the old version through the rollback window. Tombstones ensure deleted or newly forbidden documents cannot reappear when a delayed ingestion task finishes.

Rule of thumb: permission revocation and deletion are correctness events, not background freshness work. Give them a short, measured propagation objective and make tombstones win over out-of-order retries.

Multimodal systems enlarge both the attack surface and the bill. Documents may contain malicious instructions aimed at the model, sensitive images may leak through traces, and visual inference may cost far more than text retrieval. The system must enforce authorization before initial retrieval, treat document contents as untrusted data, isolate ingestion from query serving, encrypt evidence, use short-lived workload identities, and record an audit trail from user to evidence.

Cost control works best as a cascade. Cheap filters remove impossible matches before visual reranking; thumbnails support early ranking while full-resolution crops are loaded only for finalists; identical page versions reuse cached descriptions and vectors; small requests are batched within a latency budget; and each tenant or workload receives an explicit quota. Quality gates prevent "optimization" from silently cutting recall.

Rule of thumb: optimize cost per grounded successful answer, not cost per request. The cheapest answer is waste if it cites the wrong page or invents what the chart says.

Imagine a reliability engineer asking, "Which dependency caused the error spike after the regional failover?" The correct evidence is a dashboard image with a labelled dependency graph. A production-ready system must retrieve that image, confirm the engineer may see it, preserve the timestamp and source version, interpret the visual relationship, cite the exact region, and expose enough trace data to reproduce the answer later.

Walk the request through five independent reviews:

1. Evidence: Does the golden set include visual, tabular, multilingual, low-resolution, and unanswerable cases?
2. Failure: What happens when page rendering, the index, the image store, or visual inference is slow or unavailable?
3. Change: Can the team rebuild and promote an index without mutating the active version? Can it roll back the whole release bundle?
4. Control: Are authorization, retention, residency, audit, and malicious-document boundaries enforced outside the model prompt?
5. Economics: Are latency, accelerator demand, storage growth, and cost per grounded answer budgeted and observable?

The deepest lesson is that the model is one replaceable worker inside a larger contract. The platform determines what evidence exists, who may retrieve it, which version is active, how quality is measured, and whether a bad release can be detected and reversed. Rule of thumb: if replacing the vision-language model requires redesigning ingestion, authorization, observability, or deployment, those concerns were coupled to the model too tightly.

• Text extraction and captions are useful retrieval signals, but the original page or region remains the evidence because visual relationships often carry the answer.
• Ingestion is a versioned, idempotent data pipeline with stable lineage, atomic publication, quarantine, and explicit handling for retries and partial failures.
• Production retrieval combines lexical, semantic, and visual matches; filters by access before retrieval; reranks against the question; and gives the model a small, traceable evidence packet.
• Four planes separate concerns: ingestion creates evidence, query serves answers, control owns reproducible versions and promotion, and evidence records quality and health.
• Infrastructure and releases must pin application, model, prompt, index, policy, and schema versions together so promotion and rollback are coherent.
• CI tests retrieval, grounding, citations, abstention, security, latency, and cost in addition to code; delivery moves through shadow and progressive exposure with active guardrails.
• Continuous indexing is a consistency system: dual-write during rebuilds, promote immutable versions, and prioritize deletion and access revocation with winning tombstones.
• Security and cost are architectural inputs. Measure cost per grounded successful answer, enforce policy outside the prompt, and keep private evidence out of routine telemetry.
• The model may form the answer, but the surrounding platform makes that answer reproducible, authorized, observable, and reversible.

• [ ] I can explain why text extraction may recover every word yet lose the meaning of a diagram, chart, or table.
• [ ] I can design an idempotent ingestion pipeline with stable identifiers, source hashes, atomic publication, quarantine, and lineage.
• [ ] I can trace a query through access filtering, multi-channel initial retrieval, fusion, reranking, evidence packaging, visual reasoning, and citation checks.
• [ ] I can separate ingestion, query, control, and evidence planes and state why each should remain independent.
• [ ] I can define a release bundle that pins application, model, prompt, index, schema, and policy versions together.
• [ ] I can build a CI gate that measures retrieval recall, groundedness, citation validity, abstention, latency, security, and cost against both a floor and the production baseline.
• [ ] I can describe a shadow-to-canary rollout and perform one-pointer rollback of the complete inference contract.
• [ ] I can follow one answer trace from the user question to selected evidence and explain where its time and cost were spent.
• [ ] I can rebuild an index beside the active version, dual-write ordered changes, promote atomically, and prevent stale writes from defeating deletions.
• [ ] I can review a multimodal RAG design for authorization, malicious documents, private telemetry, data retention, accelerator cost, and failure behaviour.