Multimodal & vision
Multimodal & computer vision
Image/video understanding and multimodal models — vision encoders plus generation.
PyTorchTensorRTTritonNVIDIA DALIHugging Face
Architecture
Preprocessing (decode/resize) feeds a vision encoder; multimodal models fuse image and text. Video adds decode throughput and memory pressure.
Sizing
Memory-bound on large images/video; batch aggressively; hardware decode (NVDEC) offloads the CPU.
Networking
North-south; large media benefits from proximity to object storage.
Storage
Object storage for media; local cache; high read throughput for video pipelines.
Scaling
Scale on queue depth; use hardware decode and batching to raise utilization.
Monitoring
Frames/sec, decode utilization, GPU memory, latency per item.
starting point
apiVersion: apps/v1
kind: Deployment
metadata:
name: vision-infer
spec:
replicas: 2
template:
spec:
containers:
- name: triton
image: nvcr.io/nvidia/tritonserver:latest
resources: { limits: { nvidia.com/gpu: 1 } }