Public cloudLLM inferenceMultimodal

Real-Time Inference Platform

A latency-SLO-driven serving platform. The hard problems are tail latency, autoscaling against bursty traffic, and packing models onto GPUs efficiently with batching and partitioning.

Customize in Designer
Business objective

Serve models to interactive applications with a strict latency SLO and elastic cost that tracks demand.

Personas
Platform engineerML engineerSREProduct owner
Key assumptions
  • ·Traffic is bursty and latency-sensitive (p99 matters more than mean).
  • ·Models fit on one GPU or one node; no large synchronous training fabric needed.
  • ·Autoscaling can add/remove replicas within the SLO's tolerance.
Logical architecture

Rendering diagram…

Design

Compute

Inference-optimized GPUs (FP8/INT8) with MIG or time-slicing to pack smaller models; continuous batching (vLLM) or ensemble scheduling (Triton) to raise throughput within the latency budget.

Network

Standard cloud CNI; the network is north-south, not a training fabric. Keep model weights close (regional cache) to cut cold-start.

Storage

Fast model cache on local NVMe + a registry (object store). Weights are read-mostly; optimize cold-start pulls.

Kubernetes

Managed Kubernetes (EKS/AKS/GKE) with KServe or a custom serving layer; HPA/KEDA scaling on queue depth or latency, not just CPU.

Security

Gateway auth, per-tenant rate limits, network policies, and prompt/response logging with PII redaction.

Observability

Latency histograms (p50/p95/p99), tokens/sec, batch sizes, GPU utilization, and error budgets against the SLO.

High availability
Multiple replicas across zones behind the router; surplus capacity absorbs a zone loss. Multi-cluster/region for the strictest availability.
Disaster recovery
Stateless serving — recovery is redeploy + repull weights. Keep the registry and gateway config replicated across regions.
Capacity
Sized by peak QPS × latency budget. Model throughput at target batch size sets replicas; add headroom for autoscale lag and failover.
Cost profile
OpEx-dominated and elastic. Biggest levers: batching efficiency, quantization (FP8/INT8), and packing via MIG/time-slicing to raise GPU utilization.
Risks
  • highCold-start weight pulls blow the latency SLO on scale-up.
  • mediumAutoscaling reacts too slowly to bursts; needs predictive or headroom-based scaling.
  • mediumPoor batching leaves GPUs underutilized and cost high.
Implementation phases
  1. 1

    Baseline serve

    Single-model serving with latency + throughput benchmarked at target batch.

  2. 2

    Autoscale

    KEDA/HPA on the right signal; validate scale-up within SLO under load test.

  3. 3

    Packing

    MIG/time-slicing and multi-model routing to raise utilization.

  4. 4

    Multi-zone/region

    Replicate for availability; add canary + rollback.

Validation checklist
  • p99 latency within SLO at peak QPS and during scale-up.
  • Cold-start replica ready inside the autoscale window.
  • GPU utilization above target at steady state.
  • Zone-loss test keeps serving within error budget.
Recommended GPUs for this pattern
NVIDIA

L40S

48 GB · 350 W

NVIDIA

H100 PCIe

80 GB · 350 W

NVIDIA

L4

24 GB · 72 W

AMD

Instinct MI300X

192 GB · 750 W