EdgeLLM inferenceVisionSpeech

Edge Inference Cluster

Small GPU footprints at many sites where data is produced. Constraints are power, physical space, intermittent connectivity, and fleet management — not raw scale.

Customize in Designer
Business objective

Run inference close to where data is generated for latency, bandwidth, or data-residency reasons, managed as a fleet.

Personas
Platform engineerSREField operationsSecurity engineer
Key assumptions
  • ·Sites are power- and space-constrained; low-TDP GPUs.
  • ·Connectivity to the core is intermittent; sites must run degraded-but-functional.
  • ·Many sites — fleet management and remote update are essential.
Logical architecture

Rendering diagram…

Design

Compute

Low-TDP inference GPUs (e.g. 72W-class) that fit facility power and space; quantized/distilled models to fit small memory.

Network

Store-and-forward: sites operate offline and sync models/telemetry when connectivity returns. Local ingress for on-site clients.

Storage

Local cache for models and buffered results; central object store as source of truth synced opportunistically.

Kubernetes

Lightweight distributions (MicroK8s/k3s/RKE2) managed centrally via GitOps and fleet tooling (e.g. Fleet/Arc).

Security

Hardened, remotely-attested nodes; signed model updates; minimal local secrets; assume physical exposure at sites.

Observability

Central telemetry hub aggregating fleet health with tolerance for gaps; per-site heartbeat and model-version tracking.

High availability
Per-site autonomy: a site keeps serving during a core outage. Critical sites can run redundant nodes.
Disaster recovery
Sites are cattle: reprovision from GitOps and re-sync models. No unique state lives only at the edge.
Capacity
Sized per site by local demand, then multiplied across the fleet. Central capacity is mostly registry + telemetry, not compute.
Cost profile
Many small footprints; the cost driver is fleet operations (deploy, update, monitor) more than per-site hardware.
Risks
  • highFleet update/rollback failures brick remote sites that are hard to service.
  • mediumModel too large for edge memory/power, forcing distillation late.
  • mediumPhysical security of remote nodes.
Implementation phases
  1. 1

    Reference site

    One site: lightweight K8s, low-TDP GPU, model serving validated within power budget.

  2. 2

    Fleet control

    GitOps rollout/rollback, signed updates, central telemetry.

  3. 3

    Offline resilience

    Store-and-forward sync; validate degraded operation during disconnect.

  4. 4

    Scale-out

    Roll to N sites with staged/canary fleet updates.

Validation checklist
  • Site serves within its power/space budget.
  • Site keeps serving through a simulated core outage, then re-syncs.
  • Fleet rollback recovers a bad model update remotely.
Recommended GPUs for this pattern
NVIDIA

L4

24 GB · 72 W

AMD

Radeon Pro W7900

48 GB · 295 W

NVIDIA

A30

24 GB · 165 W