MSManu Shuklainecorpit.hashnode.dev·4d ago · 13 min readDatabricks shipped agent-aware ABAC masking on 21 August 2026 with 3 gapsDatabricks shipped agent-aware ABAC masking on 21 August 2026 with 3 gaps Summary. Databricks put context attributes for Unity Catalog attribute-based access control into Beta on 21 August 2026. Two n00
SCSreekanth Cherpatiinunpack.hashnode.dev·4d ago · 9 min readDatabricks RBAC for Clinical and Biotech DataIntroduction: Life-sciences data platforms have changed substantially. A single Databricks environment can now support clinical development, real-world evidence, pharmacovigilance, biomarker research,00
MSManu Shuklainecorpit.hashnode.dev·4d ago · 12 min readDatabricks dbt refreshes just inherited your cost setting, and 4 to 6 minutes of startup latencyDatabricks dbt refreshes just inherited your cost setting, and 4 to 6 minutes of startup latency Summary. On 20 August 2026 Databricks changed how dbt tasks refresh materialized views and streaming ta00
MSManu Shuklainecorpit.hashnode.dev·4d ago · 12 min readDatabricks Supervisor API dies 30 September 2026, and it is not in the August release notesDatabricks Supervisor API dies 30 September 2026, and it is not in the August release notes Summary. Databricks has deprecated the Supervisor API and set its end of life at 30 September 2026, roughly 00
SASaurabh Arorainblog.altimate.ai·Aug 19 · 23 min readUnderstanding Databricks Compute CostsThere is so much around Databricks cost optimization techniques, strategies, approaches, what to do, what not to do, and many guides from authors, even official sources from Databricks. All of those c10
KDKarthik Darbhaintech4nirvana.com·Aug 16 · 10 min readRoot Cause Analysis Without the War Room1. The Problem A pipeline fails at 2 AM. The on-call engineer opens the job logs, sees a generic Spark exception, and starts the ritual: check the source table, check the schema, check the upstream jo10
AKAyush Kumarindataengineeringwithayush.hashnode.dev·Aug 10 · 7 min readBuilding Incremental Pipelines with Auto Loader and Delta Lake on Azure DatabricksINTRO One of the most common patterns in data engineering is ingesting files from cloud storage into a Delta Lake table — reliably, incrementally, and without reprocessing what you've already seen. Yo00
AKAyush Kumarindataengineeringwithayush.hashnode.dev·Aug 10 · 7 min readWhy Most Data Quality Frameworks Fail in Production (And What to Do Instead)INTRO Every data engineering team eventually builds a data quality framework. They spend a sprint designing checks, another sprint wiring it into their pipelines, and then they ship it. Six months lat00
AKAyush Kumarindataengineeringwithayush.hashnode.dev·Aug 10 · 6 min readUnity Catalog on Azure Databricks: A Practical Setup Guide for Data EngineersINTRO Unity Catalog is Databricks' answer to unified governance across all your data assets — tables, files, models, and more — in a single place. The official documentation covers the happy path well00
AKAyush Kumarindataengineeringwithayush.hashnode.dev·Aug 10 · 5 min readDelta Lake Change Data Feed Is Underrated — Here's When to Actually Use ItINTRO Change Data Feed (CDF) in Delta Lake gets mentioned in most Databricks courses as a feature you should know about. Enable it, query table_changes(), done. But the actual decision of when to use 00
