---
title: "Discussion on \"Observability That Thinks: AI for Pipeline Monitoring\" | Hashnode"
description: "Discussion on \"Observability That Thinks: AI for Pipeline Monitoring\". AI-Augmented Data Engineering — Article 3 of 7"
url: "https://hashnode.com/posts/ai-pipeline-observability-databricks/6a6b0cc2003aaae589fc7b02"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Observability That Thinks: AI for Pipeline Monitoring](https://tech4nirvana.com/ai-pipeline-observability-databricks?utm_source=hashnode&utm_medium=feed)

AI-Augmented Data Engineering — Article 3 of 7

Jul 30 · 8 min read · tech4nirvana.com  
Read the article

-   [#observability](https://hashnode.com/tag/observability)
-   [#artificial-intelligence](https://hashnode.com/tag/artificial-intelligence)
-   [#data-pipeline](https://hashnode.com/tag/data-pipeline)
-   [#monitoring](https://hashnode.com/tag/monitoring)
-   [#databricks](https://hashnode.com/tag/databricks)
-   [#data-engineering](https://hashnode.com/tag/data-engineering)
-   [#ai](https://hashnode.com/tag/ai)
-   [#machine-learning](https://hashnode.com/tag/machine-learning)
-   [#devops](https://hashnode.com/tag/devops)

## Written by

[Karthik Darbha](https://hashnode.com/@kdarbha)

@kdarbha · Plano, TX

  
[View profile](https://hashnode.com/@kdarbha)

Sharing My Data Engineering & AI Journey

Karthik Darbha is a Data Engineering & AI Leader with over 23 years of experience in Healthcare, Pharma, Retail, Insurance, and Financial Services. He writes at tech4nirvana.com, exploring the intersection of data architecture and timeless wisdom.

## Responses 3

Markdown supported · ⌘↵ to post

[Julian Neagu](https://hashnode.com/@julianneagulab)

[Jul 31](https://hashnode.com/posts/ai-pipeline-observability-databricks/6a6b0cc2003aaae589fc7b02/comment/6a6c672fe06c8a3a5ef9f9b0)

I like the focus on simple models first. Most teams don't need something complex. Better signals and fewer noisy alerts already solve a big problem.

[Kartik N V J K](https://hashnode.com/@kartik-nvj)

[Jul 30](https://hashnode.com/posts/ai-pipeline-observability-databricks/6a6b0cc2003aaae589fc7b02/comment/6a6bad8541fb5fc9e66b97da)

Median/MAD over mean plus stddev is the right call for pipeline metrics, since one giant batch or a retry storm wrecks a z-score but barely moves the median. The same-weekday baseline is the detail I'd steal, because a Monday reprocessing job looks nothing like a Sunday and averaging across the week hides both. How do you handle holidays and one-off backfills, do they poison the trailing-8 window, or do you exclude them explicitly?

[Nahid Mahmud](https://hashnode.com/@nahidmahmud555)

[Jul 30](https://hashnode.com/posts/ai-pipeline-observability-databricks/6a6b0cc2003aaae589fc7b02/comment/6a6b7f6dd8cbc959aae16637)

Consistency and continuous learning make all the difference. Thanks for sharing this valuable breakdown.