Inference
Real-time LLM inference
Low-latency, autoscaling serving with continuous batching behind an API gateway.
vLLMTensorRT-LLMNVIDIA TritonKServeNIM
Architecture
Replicas of a serving engine (vLLM/TensorRT-LLM/Triton) sit behind a router. Continuous batching maximizes throughput within the latency budget; weights are cached locally to cut cold-start.
Sizing
Replicas from peak QPS × per-request latency at the target batch size; FP8/INT8 quantization raises throughput.
Networking
North-south; standard CNI. Keep weights regional/local for fast scale-up.
Storage
Model cache on local NVMe + a registry; weights are read-mostly.
Scaling
KEDA/HPA on queue depth or latency (not just CPU); keep min replicas for burst headroom.
Monitoring
p50/p95/p99 latency, tokens/sec, batch sizes, GPU utilization, error budget vs. SLO.
starting point
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: llm
spec:
predictor:
minReplicas: 2
model:
runtime: vllm
storageUri: s3://models/llm
resources:
limits:
nvidia.com/gpu: 1