HybridRAGLLM inference

RAG Platform

A pipeline platform: ingest and embed documents, retrieve + rerank at query time, then generate. GPU demand splits between embedding generation and the serving model; freshness and access control dominate the data side.

Customize in Designer
Business objective

Ground model responses in enterprise knowledge with current, access-controlled data and measurable answer quality.

Personas
ML engineerData engineerPlatform engineerSecurity engineer
Key assumptions
  • ·Source documents change; the index must stay fresh.
  • ·Multi-tenant with per-tenant access control on retrieval.
  • ·Answer quality is evaluated, not assumed.
Logical architecture

Rendering diagram…

Design

Compute

Two GPU roles: batch embedding (throughput) and interactive generation (latency). They can share a pool with priority classes or run on tailored node groups.

Network

Standard cloud/hybrid networking. Keep the vector store close to the serving model to keep retrieval latency low.

Storage

Vector database (pgvector, Milvus, Qdrant, Weaviate) sized by chunks × dimensions × replicas; object store for source documents; cache for hot contexts.

Kubernetes

Managed or hybrid Kubernetes; separate deployments for ingestion (batch) and serving (online) with independent scaling.

Security

Per-tenant retrieval filters, PII handling at ingest, audit logging of retrieved context, and prompt-injection guardrails.

Observability

Retrieval hit rate, rerank impact, end-to-end latency, groundedness/eval scores, and index freshness lag.

High availability
Replicated vector store and stateless query path across zones. Ingestion can lag briefly without downtime.
Disaster recovery
Vector index is rebuildable from source documents; back up source store + embeddings, and snapshot the index to shorten rebuild.
Capacity
Index sizing from document count and chunking; embedding GPU throughput sets ingest rate; serving sized like the inference platform.
Cost profile
Mixed: steady embedding batch + elastic serving. Reranking and larger context windows are the main cost/quality knobs.
Risks
  • highStale index returns wrong answers with confidence.
  • mediumCross-tenant leakage if retrieval filters are wrong — a security, not just quality, issue.
  • lowOver-large context windows inflate cost with little quality gain.
Implementation phases
  1. 1

    Ingest + index

    Parsing, chunking, embedding, and the vector store with a freshness pipeline.

  2. 2

    Retrieve + generate

    Hybrid search + rerank wired to the served model with citations.

  3. 3

    Evaluate

    Groundedness/quality eval harness and guardrails before rollout.

  4. 4

    Multi-tenant

    Access-controlled retrieval and audit logging.

Validation checklist
  • Retrieval returns relevant chunks for a labelled query set.
  • Per-tenant isolation holds under cross-tenant probes.
  • Index freshness lag within target after a source change.
  • Groundedness score meets threshold on the eval set.
Recommended GPUs for this pattern
NVIDIA

L40S

48 GB · 350 W

NVIDIA

H100 PCIe

80 GB · 350 W

NVIDIA

L4

24 GB · 72 W

AMD

Instinct MI300X

192 GB · 750 W