George Ezejiofor

Thanks, and yes, that's exactly the trap. On the Pipelines side it bit me twice. The kfp-launcher config needs an explicit providers block for S3, or artifact uploads quietly go nowhere, and region is mandatory even though RGW doesn't care about regions. Then every component has to use the same endpoint (the public RGW hostname on 443, not the in-cluster service on 80), or some steps write where others can't read. The smoke test you describe runs every day here, not just after upgrades. The daily pipeline trains the model, writes it to RGW, rolls it out to KServe as a canary, and sends a burst of real requests at it before promoting. If KServe can't pull the artifact, that step fails and the old model keeps serving. The same pipeline also turned up a KServe-specific one: any extra .json file next to model.bst in the model folder crashes the XGBoost server at load, so the model folder holds only the model.

Agreed, the off-site copy goes first. Thanks for pushing on it.

Thanks, I appreciate it. The traps sections are the ones I'd have wanted when I started: most of the hours went into auth chains and storage settings, not the parts the architecture diagram shows. More of that in the next articles in the series.

Honestly, it mostly runs itself day to day. The lab is powered off every night and cold-boots every morning, and everything comes back on its own: ArgoCD syncs from Git, CloudNativePG brings Postgres back, and the daily pipeline runs on schedule. That nightly restart is a good test that nothing depends on hand-applied state. The tax is upgrades, not operations. Moving Kubeflow, Istio, Knative or Ceph one version means checking how each fits with the others first, and that's where the hours go. Managed services would remove most of that. I'd still choose the homelab, because the point is to learn where the parts connect, and you only learn that by owning them.

Thanks. That's how dbt runs here: write-audit-publish on a dbt-audit branch, which merges to main only when the whole build passes, so Trino never sees a half-built table. On maintenance, Nessie changes the picture a bit. Iceberg tables behind Nessie keep a single snapshot in their own metadata, because the history lives in Nessie's commit log. So expire_snapshots has nothing to do. Orphan and unreferenced files are handled by Nessie GC, which runs weekly with a 30-day cutoff (still in record-only mode while I build trust in it). rewrite_data_files is the real gap, and you're right about the cause. Compaction isn't scheduled yet, and the table written every minute is the one piling up small files on Ceph. That's next on the list.

Thanks, agreed on both. Two notes from running it. Snapshot expiry works differently with Nessie. Iceberg tables behind Nessie keep a single snapshot in their own metadata, because the history lives in Nessie's commit log. So expire_snapshots has nothing to do, and the cleanup that matters is Nessie GC, which also deletes orphan files. Mine runs weekly with a 30-day cutoff, still in record-only mode. Compaction of small files isn't scheduled yet, and you're right that the table written every minute is the one that'll need it. Nessie's metadata store is Postgres (the JDBC version store) on a 3-instance CloudNativePG cluster: continuous WAL archiving plus a nightly base backup to Ceph RGW, kept for 7 days. The honest gap: those backups sit on the same Ceph cluster as the data, so they cover losing a database, not losing the storage. An off-site copy is on my list. Good shout for a follow-up post.

Good question. Two things keep it safe here: the failed build never touches main, and nothing deletes files directly. dbt runs write-audit-publish. Each run first points the dbt-audit branch at main's head, runs dbt build against that branch, and merges to main only if the whole build passes. If it fails halfway, main is untouched, and whatever it wrote stays on dbt-audit so I can inspect it. The next run resets dbt-audit to main again, and those half-written commits stop being reachable from any branch or tag. Cleanup is Nessie GC, not anything dbt-side. It marks as live every file reachable from any branch or tag within the cutoff (30 days by default), and sweeps the rest. Tags are references too, so a train-football- tag keeps its snapshot's files alive for as long as the tag exists. I give those tags a cutoff of 1 so GC only keeps the commit the tag points at, not 30 days of history behind it. What decides how long a tag lives is a separate retention job. It keeps a tag if a Model Registry version names it as its training data, if it's under 30 days old, or if it's the newest one. It deletes the rest, and only at the hash it listed (Nessie rejects the delete if the tag moved). It fails closed: if the registry can't be read, it deletes nothing. One caveat: it checks the Kubeflow Model Registry, not MLflow, so an MLflow run that records a tag doesn't protect it on its own. So yes, I'd use Nessie GC with retention, but put the retention policy on your references (which tags exist), not on files. And run GC in dry-run mode first: mine still runs with --defer-deletes, which only records what it would delete. Its log still says "deleted N files" in that mode, so check the bucket rather than the log line.

Hi Nishat, thanks for that. I configured the THE ARGOCD configmap in the helm value config file. Also I'm using workload federated identity to make the process more secured with token (passwordless) instead of secret. Is also easy to intergrate RBAC with the process with different teams and different environments.