Retrieval-augmented generation (RAG)
Ground model responses in enterprise knowledge via embedding, retrieval, reranking, and generation.
Architecture
An ingestion pipeline chunks and embeds documents into a vector store. At query time, hybrid search + rerank assembles context for the served model, with guardrails and citations.
Sizing
Two GPU roles: batch embedding (throughput) and interactive generation (latency); the vector store sizes by chunks × dimensions × replicas.
Networking
Keep the vector store close to the serving model to keep retrieval latency low.
Storage
Vector DB for embeddings, object storage for source documents, cache for hot contexts.
Scaling
Scale ingestion (batch) and serving (online) independently; ingestion can lag briefly.
Monitoring
Retrieval hit rate, rerank impact, end-to-end latency, groundedness/eval scores, index freshness.
apiVersion: apps/v1
kind: Deployment
metadata:
name: rag-api
spec:
replicas: 3
template:
spec:
containers:
- name: rag
image: your-registry/rag:latest
env:
- { name: VECTOR_DB, value: "qdrant:6333" }
- { name: LLM_ENDPOINT, value: "http://llm/v1" }