---
title: "Comment by Yunus Emre Vurgun on \"Self-Hosted Databricks Alternative: Homelab Data Lakehouse (AI/ML Data Factory)\" | Hashnode"
description: "Using Nessie branches together with dbt is a strong pattern here: run dbt against a branch, run tests there, and only merge into main when they pass, so Trino c"
url: "https://hashnode.com/posts/open-source-databricks-ai-native-lakehouse-homelab/6ad2a764a305ffe66917ddea/comment/6ad4abd2de3724cee95dfe93"
---

[Discussion](https://hashnode.com/posts/open-source-databricks-ai-native-lakehouse-homelab/6ad2a764a305ffe66917ddea)

Comment [Self-Hosted Databricks Alternative: Homelab Data Lakehouse (AI/ML Data Factory)](https://hashnode.com/posts/open-source-databricks-ai-native-lakehouse-homelab/6ad2a764a305ffe66917ddea)

[Yunus Emre Vurgun](https://hashnode.com/@yunusemrevurgun)

[Oct 18](https://hashnode.com/posts/open-source-databricks-ai-native-lakehouse-homelab/6ad2a764a305ffe66917ddea/comment/6ad4abd2de3724cee95dfe93)

Using Nessie branches together with dbt is a strong pattern here: run dbt against a branch, run tests there, and only merge into main when they pass, so Trino consumers never see half-built tables. On a homelab, also schedule Iceberg maintenance (expire\_snapshots, remove\_orphan\_files and rewrite\_data\_files) early, because frequent small dbt runs pile up snapshots and tiny files on Ceph fast, and Trino query times degrade before you notice why.

[George Ezejiofor](https://hashnode.com/@georgeezejiofor)

[Oct 18](https://hashnode.com/posts/open-source-databricks-ai-native-lakehouse-homelab/6ad2a764a305ffe66917ddea/comment/6ad4abd2de3724cee95dfe93#reply-6ad4c980d2fe7cc3c3bc44ca)

Thanks. That's how dbt runs here: write-audit-publish on a dbt-audit branch, which merges to main only when the whole build passes, so Trino never sees a half-built table.

On maintenance, Nessie changes the picture a bit. Iceberg tables behind Nessie keep a single snapshot in their own metadata, because the history lives in Nessie's commit log. So expire\_snapshots has nothing to do. Orphan and unreferenced files are handled by Nessie GC, which runs weekly with a 30-day cutoff (still in record-only mode while I build trust in it).

rewrite\_data\_files is the real gap, and you're right about the cause. Compaction isn't scheduled yet, and the table written every minute is the one piling up small files on Ceph. That's next on the list.