Tag

#data-pipeline

144 posts14 followers

Explore Hashnode

Alternatives

Rahul ROct 12 · 10 min read

When Delta Lake Jobs Collide: Troubleshooting Concurrency Issues in AWS Glue

Have you ever run two AWS Glue jobs against the same Delta table and noticed that one job fails while the other completes successfully? The confusing part is that both jobs might work perfectly when e

rahul-dataeng.hashnode.dev

Rahul ROct 11 · 7 min read

AWS Step Functions vs Apache Airflow: Similar Purpose, Different Use Cases

When I started exploring data engineering workflows, one question that came to mind was: if AWS Step Functions and Apache Airflow can both orchestrate workflows, why do we need both? At first glance,

rahul-dataeng.hashnode.dev

Fred FengOct 1 · 6 min read

Cronflower: Run a DAG workflow across your cluster, instead of chaining cron jobs

cronflower is an open-source, distributed scheduler for the JVM with a web console. cronflow is its optional DAG add-on: declare the steps and how they depend on each other as a graph, and the same cl

fredfeng.hashnode.dev

Tao SunAug 15 · 9 min read

How I Built a Secure AI Database Gateway in Go Using MCP, Anthropic, and Kafka

Built DynamoDB Sage: a Go MCP server that lets LLM agents safely query and manage Amazon DynamoDB via natural language A custom RiskAnalyzer validates every mutating or heavy tool call before it touc

tech-mind.hashnode.dev

Andrew TanJul 31 · 9 min read

Your Data Warehouse Is Not Your Data Pipeline

Teams keep forcing their warehouse to do integration work it was never designed for. The result is ballooning costs, opaque failures, and architectures that become harder to maintain the more they ‘su

layline.hashnode.dev

Karthik DarbhaJul 30 · 8 min read

Observability That Thinks: AI for Pipeline Monitoring

The Problem Most pipeline observability today is a wall of thresholds. Row count dropped below X. Job ran longer than Y minutes. Null percentage exceeded Z. Someone picked those numbers months ago, of

3 tech4nirvana.com

charles onokohwomoJul 14 · 12 min read

Phase 2B — Building the ADIP AI Insight Engine

Series: ADIP Engineering Intelligence Phase 2B — AI Insight Generation Layer Repository: github.com/CKohwo/ADIP-Intelligence-lab Introduction Phase 2A gave ADIP discipline. It built the data foundatio

charlesonokohwomo.hashnode.dev

Ikraj SinghJul 13 · 10 min read

Data Pipelines in the Age of GenAI

This article is written in my personal capacity. The views expressed are my own and do not represent Amazon or my employer. Examples are based on public information and synthetic scenarios; no confide

ikrajsingh.hashnode.dev

Harsh TrivediJul 11 · 20 min read

Building a Secure SharePoint → Azure Blob → Snowflake Document Intelligence Pipeline

Part of the series AI Cloud and Data Engineering Articles. In Part 1 we covered why documents should be encrypted before they ever reach cloud storage, how Azure Key Vault and Service Principals keep

harshtrivedii.hashnode.dev

Sergio González TéllezJul 2 · 2 min read

Why do so many systems appear to produce new information when, in reality, they only reorganize existing information?

SEED-003 PROBLEM Why do so many systems appear to produce new information when, in reality, they only reorganize existing information? INSIGHT Data is rarely created; it is usually transformed, filt

evankhandev.hashnode.dev

Search Hashnode

Explore Hashnode

Trending tags this week

When Delta Lake Jobs Collide: Troubleshooting Concurrency Issues in AWS Glue

AWS Step Functions vs Apache Airflow: Similar Purpose, Different Use Cases

Cronflower: Run a DAG workflow across your cluster, instead of chaining cron jobs

How I Built a Secure AI Database Gateway in Go Using MCP, Anthropic, and Kafka

Your Data Warehouse Is Not Your Data Pipeline

Observability That Thinks: AI for Pipeline Monitoring

Phase 2B — Building the ADIP AI Insight Engine

Data Pipelines in the Age of GenAI

Building a Secure SharePoint → Azure Blob → Snowflake Document Intelligence Pipeline

Why do so many systems appear to produce new information when, in reality, they only reorganize existing information?