Kubeflow + Argo CD: GitOps MLOps on Kubernetes

This is the long-form reference. For a five-minute skim, see the companion blog.

1. Introduction: MLOps needs two control planes

Machine learning on Kubernetes fails in two predictable ways. Teams either treat training as a collection of ad-hoc Jobs with no lineage, or they treat serving as a one-off kubectl apply with no audit trail. Kubeflow and Argo CD address opposite halves of that problem.

Kubeflow is the ML control plane: pipelines, distributed training, experiment tracking, model registry, and KServe. Argo CD is the delivery control plane: pull-based GitOps so cluster state matches Git — including the Kubeflow platform itself and every production InferenceService.

This article is a practical architecture guide for platform and MLOps engineers. It assumes Kubernetes familiarity and that you already understand basic GitOps (see our earlier Argo CD & Flux continuous delivery post). Here we focus on how the two systems fit together for ML workloads in 2026.

2. The 2026 Kubernetes MLOps map

Argo Workflows still appears under the hood for some pipeline backends, but you should think in Kubeflow Pipelines IR / v2 for authoring and Argo CD for continuous delivery of Kubernetes resources — not confuse “Argo Workflows” with “Argo CD”.

3. Clear ownership: what Kubeflow does vs what Argo CD does

3.1 Kubeflow owns

• Authoring and running training / ETL / evaluation DAGs
• GPU job lifecycle and experiment metadata
• Registering model versions and lineage in the Model Registry
• Defining how a model could be served (runtime, resources) — often via generated manifests

3.2 Argo CD owns

• Installing and upgrading Kubeflow components (GitOps platform)
• Syncing pipeline definitions and CronWorkflows that kick off training
• Promoting InferenceService changes across environments
• Instant, auditable rollback via Git history

4. Recommended Git layout

ml-platform/                 # Argo CD App: install Kubeflow once
  overlays/
    staging/
    production/
ml-apps/                     # Argo CD App-of-Apps or projects
  pipelines/
    fraud-detector/
  serving/
    fraud-detector/
      base/inferenceservice.yaml
      overlays/
        staging/
        production/
  components/                # reusable KFP components (OCI or YAML)

Keep platform and application syncs separate. Data scientists open PRs against ml-apps; platform engineers own ml-platform. Use Argo CD Projects + RBAC so a bad pipeline PR cannot rewrite the Kubeflow control plane.

5. Managing Kubeflow with Argo CD

Manual Kubeflow installs are fragile: many CRDs, ordering constraints, and upgrade paths. Treat the platform as an Argo CD Application (or ApplicationSet) with:

• Sync waves — certificates, storage, MySQL/Postgres (or managed DB), MinIO/S3 credentials, then KFP / Trainer / KServe
• Health checks — wait for CRDs and webhooks before syncing dependent apps
• Managed services in production — replace in-cluster MySQL/MinIO with Cloud SQL/RDS and S3 when you outgrow the all-in-one demo

Example Application sketch (illustrative):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kubeflow-platform
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  project: ml-platform
  source:
    repoURL: https://git.example.com/org/ml-platform.git
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: kubeflow
  syncPolicy:
    automated:
      prune: false
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

6. Pipelines as GitOps resources

With Kubeflow Pipelines v2 / Kubernetes-native APIs, compiled pipelines can be managed as cluster resources. The workflow becomes:

1. Author pipeline in Python (KFP SDK)
2. Compile to YAML
3. Commit to ml-apps/pipelines/...
4. Argo CD syncs; runs are triggered via UI, API, or CronWorkflow also stored in Git

Always set CPU, memory, and GPU limits on steps. Unbounded training steps will disrupt multi-tenant clusters faster than any bad model.

7. Model promotion: registry → Git → Argo CD

This is the critical hand-off:

1. Training run finishes; evaluation metrics meet thresholds.
2. Model Registry records a new version with URI + metadata (accuracy, fairness, signer).
3. Automation (or a human) opens a PR that updates storageUri (and image tag / runtime) in the serving overlay.
4. After merge, Argo CD rolls out KServe. Prefer canary: set canaryTrafficPercent to 10%, watch error rate and latency, then promote.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: fraud-detector
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: s3://models/fraud/v1.4.2/
      resources:
        requests:
          cpu: "1"
          memory: 2Gi
        limits:
          cpu: "2"
          memory: 4Gi

Rollback is not a dashboard click — it is git revert of that URI change. Argo CD heals the cluster to the previous desired state.

8. GPUs, quotas, and FinOps

• Use Kueue so TrainJobs wait for fair GPU capacity instead of failing or oversubscribing.
• Separate node pools for training vs latency-sensitive inference when possible.
• Track cost per pipeline run (GPU-seconds × rate). GitOps does not remove FinOps — it makes spend attributable to a commit and a model version.
• For MIG / time-slicing on NVIDIA, document the partition strategy in the platform repo so Argo CD-managed DevicePlugin configs stay consistent.

9. Security and multi-tenancy

• Kubeflow Profiles isolate teams; map them to Argo CD Projects.
• Store cloud credentials in Sealed Secrets / External Secrets — never plain ConfigMaps in Git.
• Require registry metadata gates (e.g. fairness_score, sign_off_user) before a promotion bot may open a prod PR.
• Network policies: training jobs should not need unrestricted egress; serving pods need only model storage and clients.

10. Observability beyond pod metrics

Prometheus on GPU utilisation is necessary but not sufficient. Wire:

• Pipeline failure alerts (run status, not only Deployment CrashLoop)
• Serving SLO: latency, error rate, saturation
• Data/model drift monitors that can open a ticket or re-trigger a training CronWorkflow

When drift fires, the remediation path should still be GitOps: new run → new URI → PR → Argo CD sync — not a manual overwrite on the node.

11. Rollout checklist

1. Install Argo CD; create ml-platform and ml-apps projects.
2. GitOps-install Kubeflow with sync waves; verify KFP UI and KServe CRDs.
3. Stand up object storage + Model Registry; document URI conventions.
4. Onboard one golden path pipeline (train → evaluate → register).
5. Add one InferenceService under Argo CD with staging overlay first.
6. Enable canary promotion; practise a git revert drill.
7. Add Kueue quotas and GPU FinOps dashboards before multi-team rollout.

12. Anti-patterns

• Committing 4 GB weight files to Git LFS “for convenience”
• Letting notebooks kubectl apply production InferenceServices
• One Argo CD Application that mixes platform and all team pipelines (blast radius)
• No resource limits on training steps
• Confusing Argo Workflows (execution) with Argo CD (desired-state sync)
• Skipping staging — promoting straight from a laptop experiment to prod traffic

13. When not to use this stack

SMEs running a single private LLM on a workstation (see our AI SME Packages) do not need Kubeflow + Argo CD on day one. Introduce this architecture when you have multiple models, concurrent GPU users, regulated change control, or several environments that must stay identical.

14. Conclusion

Kubeflow and Argo CD are complementary. Kubeflow makes ML work runnable and reproducible on Kubernetes. Argo CD makes ML infrastructure and model serving declarative, auditable, and reversible. The winning pattern is simple: artifacts in object storage, pointers and YAML in Git, training in Kubeflow, delivery and rollback in Argo CD.

The companion blog post is the shareable summary; this article is the reference for platform design reviews.

Published by Workstation (workstation.co.uk).

SEO snapshot for this article

• SEO title: Kubeflow + Argo CD: GitOps MLOps on Kubernetes
• Meta description: How to combine Kubeflow Pipelines, Model Registry, and KServe with Argo CD GitOps for trainable, auditable, rollback-safe ML on Kubernetes.
• Primary keywords: Kubeflow Argo CD, GitOps MLOps, KServe InferenceService, Kubeflow Pipelines GitOps, ML on Kubernetes
• Twitter / X: Kubeflow trains. Argo CD delivers. Models stay in object storage; Git holds URIs. Full GitOps MLOps guide from Workstation.
• LinkedIn: We published a deep dive on pairing Kubeflow and Argo CD: platform vs apps Git layout, promotion gates, canary serving, GPU quotas, and rollback via git revert. From Workstation engineering.