FeedDiscussion
Self-Hosted Databricks Alternative: Homelab Data Lakehouse (AI/ML Data Factory)
Self-Hosted Databricks Alternative: Homelab Data Lakehouse Part of the homelab build series. Trino, Apache Iceberg, Project Nessie, dbt, Spark, MLflow and OpenMetadata on RKE2, with highly available s
georgeezejiofor.com40 min read
#kubernetes#devops#mlops#homelab#trino#data#data-science#databases#data-structures#data-engineering#database#mlops-workflow#mlops-pipelines#kubeflow#apache#apache-kafka#apache-spark
Pairing Iceberg with Nessie in a homelab is a nice way to get branch and merge semantics for data without paying for a managed catalog. One thing worth covering in a follow-up is snapshot expiration and orphan file cleanup, since on Ceph backed S3 those small files and old snapshots add up quickly if nobody schedules the maintenance jobs. A short section on how you back up the Nessie metadata store would also help anyone copying this setup.