KDKarthik Darbhaintech4nirvana.com·Aug 16 · 10 min readRoot Cause Analysis Without the War Room1. The Problem A pipeline fails at 2 AM. The on-call engineer opens the job logs, sees a generic Spark exception, and starts the ritual: check the source table, check the schema, check the upstream jo10
KDKarthik Darbhaintech4nirvana.com·Jul 30 · 8 min readObservability That Thinks: AI for Pipeline MonitoringThe Problem Most pipeline observability today is a wall of thresholds. Row count dropped below X. Job ran longer than Y minutes. Null percentage exceeded Z. Someone picked those numbers months ago, of43JKN
KDKarthik Darbhaintech4nirvana.com·Jun 20 · 11 min readAI-Driven Data Quality: From Rules to ReasoningEvery data engineering team has a version of the same story. A critical dashboard starts showing numbers that don't feel right. An analyst flags it Friday afternoon. The on-call engineer traces it bac00
KDKarthik Darbhaintech4nirvana.com·May 31 · 7 min readThe Case for AI in Data Engineering Series: AI-Augmented Data Engineering | Article 1 of 7 There is a quiet crisis in most data engineering teams. Pipelines fail at 2 AM. The on-call engineer spends three hours tracing a root cause tha00
KDKarthik Darbhaintech4nirvana.com·May 14 · 11 min readThe Art of Program Visibility: Managing Databricks + Azure Data Programs at ScaleThe Invisible Failure Most data platform programs don't fail loudly. They fail quietly — one missed dependency, one unreported pipeline issue, one status update that said 'green' while the underlying 10