Training
Parameter-efficient fine-tuning (LoRA/QLoRA)
Adapt a base model with low-rank adapters — a fraction of the compute and memory of full fine-tuning.
Hugging Face PEFTTRLbitsandbytesPyTorch
Architecture
A single node (or a few GPUs) loads the frozen base model and trains small adapter weights. QLoRA quantizes the base to 4-bit to fit larger models on one GPU.
Sizing
Often 1–8 GPUs. QLoRA fits a 70B base on a single 80 GB GPU; full-precision LoRA needs more memory.
Networking
Standard networking is fine for single-node; multi-node adapters use the training fabric.
Storage
Local NVMe cache for the base model + adapters; pull from object storage.
Scaling
Usually vertical (bigger GPU) before horizontal; batch multiple fine-tunes across a shared pool with MIG.
Monitoring
Loss curves, GPU memory headroom, tokens/sec, eval metrics on a held-out set.
starting point
apiVersion: batch/v1
kind: Job
metadata:
name: qlora-finetune
spec:
template:
spec:
restartPolicy: Never
containers:
- name: trl
image: your-registry/finetune:latest
args: ["--model", "$BASE", "--lora-r", "16", "--load-in-4bit"]
resources:
limits:
nvidia.com/gpu: 1