@kdarbha

The Case for AI in Data Engineering

3d ago · 7 min read · Series: AI-Augmented Data Engineering | Article 1 of 7 There is a quiet crisis in most data engineering teams. Pipelines fail at 2 AM. The on-call engineer spends three hours tracing a root cause tha

The Art of Program Visibility: Managing Databricks + Azure Data Programs at Scale

May 14 · 11 min read · The Invisible Failure Most data platform programs don't fail loudly. They fail quietly — one missed dependency, one unreported pipeline issue, one status update that said 'green' while the underlying

Letting Go of Control: What Advaita Teaches Us About Liquid Clustering

May 5 · 11 min read · There is a pattern in mature engineering: the more you try to control, the more brittle the system becomes. Liquid Clustering is Delta Lake's answer to over-engineering. Vedanta figured this out mille

Unity Catalog - the Unified Self

Apr 28 · 15 min read · By Karthik Darbha | Tech4Nirvana The Problem of Many Selves In the Advaita Vedanta tradition, the root cause of all suffering is Avidya (अविद्या) — ignorance. Not ignorance in the ordinary sense of n

Migrating SPC Run Rules from SAS to Databricks

Apr 22 · 18 min read · A Pharma Supply Chain Engineering Perspective · tech4nirvana.com Why This Migration Is Non-Trivial Earlier, I worked as Product Owner and Data Architect on a SAS to Databricks migration for a Pharma