DevSecOps & MLOps
Generate secure delivery pipelines for infrastructure, GPU operators, applications, and models — with scanning, supply-chain signing, GitOps, approvals, and progressive delivery woven in. Plus the practices and model lifecycle behind them.
name: pipeline
on:
push:
branches: [main]
jobs:
ci:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build image
run: |
docker build -t $IMAGE:$SHA .
- name: Scan image
run: |
trivy image --exit-code 1 --severity CRITICAL,HIGH $IMAGE:$SHA
- name: SBOM & sign
run: |
syft $IMAGE:$SHA -o spdx-json > sbom.json
cosign sign --yes $IMAGE:$SHA
cosign attest --yes --predicate sbom.json $IMAGE:$SHA
- name: Push image
run: |
docker push $IMAGE:$SHA
deploy:
needs: ci
runs-on: ubuntu-latest
environment: production # required reviewers gate this job
steps:
- uses: actions/checkout@v4
- name: Reconcile (GitOps)
run: |
# GitOps: update the desired state and let Argo CD / Flux reconcile
yq -i '.spec.deployment/app = "$IMAGE:$SHA"' clusters/prod/deployment/app.yaml
git commit -am "chore: $IMAGE:$SHA" && git push
- name: Smoke test
run: |
curl -fsS https://app.example.com/healthz- ·A manual approval gates the mutating step — no unattended production change.
- ·Artifacts are SBOM-attested and signed (Cosign); verify signatures at admission.
- ·Scans fail the pipeline on CRITICAL/HIGH findings — tune the threshold to your policy.
- ·Deploys commit desired state to Git; Argo CD / Flux reconciles the cluster.
- ·Generated as a starting point — pin action/image versions and wire real secrets before use.
GitOps
Git is the source of truth for cluster state; a controller reconciles the cluster to match. Every change is a reviewed, revertible commit.
Policy as code
Enforce security and resource policy at admission — signed images, pinned tags, GPU limits, tenancy labels.
Supply-chain security
Sign artifacts, generate SBOMs, and verify provenance so only trusted images and models run on GPUs.
Secrets management
Keep secrets out of images and manifests; sync from a vault and rotate on a schedule.
Scanning
Scan images, IaC, and manifests in CI; gate deploys on critical findings.
Drift detection
Detect and remediate when live cluster state diverges from the declared desired state.
| Strategy | How it works | Best for | Extra cost |
|---|---|---|---|
| Rolling | Replace replicas gradually; simple default with no extra capacity. | Stateless services with backward-compatible changes | None |
| Canary | Shift a small % of traffic to the new version, watch metrics, then promote. | Inference where latency/quality regressions must be caught early | Small overlap |
| Blue-green | Stand up the new version fully, cut over, keep the old for instant rollback. | Fast rollback needs; large model version swaps | 2× capacity briefly |
| Shadow | Mirror live traffic to the new version without serving its responses. | Validating a new model on real traffic before promotion | 2× inference compute |
- 1
Experiment tracking
Record params, metrics, and artifacts per run so results are reproducible and comparable.
- 2
Model registry
Version models with lineage, stage (staging/prod), and provenance; the registry is the promotion boundary.
- 3
Evaluation gates
Automated quality/safety evals must pass a threshold before a model can be promoted — the pipeline fails otherwise.
- 4
Promotion
Promote by updating the served model version through GitOps or the registry, behind an approval for production.
- 5
Progressive delivery
Roll out with canary/shadow and guard on live metrics, with automated rollback on regression.
- 6
Monitoring & drift
Watch inference latency, quality, and input/output drift; trigger re-training or rollback when they degrade.