KDKarthik Darbhaintech4nirvana.com·Aug 16 · 10 min readRoot Cause Analysis Without the War Room1. The Problem A pipeline fails at 2 AM. The on-call engineer opens the job logs, sees a generic Spark exception, and starts the ritual: check the source table, check the schema, check the upstream jo10
AKAyush Kumarindataengineeringwithayush.hashnode.dev·Aug 10 · 6 min readUnity Catalog on Azure Databricks: A Practical Setup Guide for Data EngineersINTRO Unity Catalog is Databricks' answer to unified governance across all your data assets — tables, files, models, and more — in a single place. The official documentation covers the happy path well00
AAadhishriiinaadhishrii.hashnode.dev·Aug 4 · 6 min readWhat Unity Catalog Actually SolvesI gave a presentation on Unity Catalog last week. Before I opened a single slide, I made a decision that changed the whole thing: I wasn't going to explain it as a feature list. Access control, lineag00
MSManu Shuklainecorpit.hashnode.dev·Jul 28 · 14 min readDatabricks Genie billing starts July 6, 2026: a cost-control playbook for adminsDatabricks Genie billing starts July 6, 2026: a cost-control playbook for admins Summary. On July 6, 2026, Databricks began charging for Genie usage beyond a free monthly allowance, moving its text-to00
KDKarthik Darbhaintech4nirvana.com·Jun 20 · 11 min readAI-Driven Data Quality: From Rules to ReasoningEvery data engineering team has a version of the same story. A critical dashboard starts showing numbers that don't feel right. An analyst flags it Friday afternoon. The on-call engineer traces it bac00
Cchristan_technoinchristan-tech-blog.hashnode.dev·Mar 29 · 13 min readDatabricks Account Hierarchy Explained: A Complete Guide to Unity Catalog, Workspaces, and Data GovernanceTL;DR: Databricks organizes everything in a layered hierarchy — Account → Metastore → Workspace → Catalog → Schema → Table. Understanding how Workspaces, Bindings, and Permissions interact at each lay00
ATAbhijit Tilakinabhijittilak.hashnode.dev·Mar 6 · 4 min readAI-Ready Enterprise Technology: An Executive Brief for CIOs & CDAOsAs a leader keen on modernizing your enterprise data platform for the agentic AI era, you and your team should primarily focus on improving data and AI governance, leading to a more coherent overall a00
ATAbhijit Tilakinabhijittilak.hashnode.dev·Mar 6 · 4 min readFrom Chaos to Structure: Grassroots Data Governance That ScalesGrassroots-level accelerators to enable data governance at scale Data governance is such a broad term in the industry and on the wish list of every CDAO organization. Data governance and cataloging ar00
CCRincr88.hashnode.dev·Jan 18 · 4 min readSecuring Unity Catalog - Storage Credentials, Scoped Service Principal External Location access - serverless autoloader file notification file eventsOne of the most common challenges in Data Engineering is giving a pipeline access to just enough data without handing over the keys to the entire kingdom (or S3 bucket). In this post, we’ll walk throu00
KDKurdapyo Data Engineerindata.kurdapyo.com·Jul 11, 2025 · 3 min readWhat's with a name (in Databricks)Naming conventions are often a contentious topic among architects. In Databricks, these debates become even more pronounced due to the extensive platform work required to establish a Databricks implementation. In an AWS-based Databricks setup, naming...00
