--- title: "#data-pipeline articles | Hashnode" description: "Explore 144 articles tagged #data-pipeline on Hashnode. Tutorials, guides, and how-tos from the community." url: "https://hashnode.com/tag/data-pipeline" --- Tag # #data-pipeline 144 posts · 14 followers [Rahul R](https://hashnode.com/@rahul-data) · [Oct 12](https://hashnode.com/posts/when-delta-lake-jobs-collide-troubleshooting-concurrency-issues-in-aws-glue/6acca39ec13a8ac656801a58) ### [When Delta Lake Jobs Collide: Troubleshooting Concurrency Issues in AWS Glue](https://rahul-dataeng.hashnode.dev/when-delta-lake-jobs-collide-troubleshooting-concurrency-issues-in-aws-glue?utm_source=hashnode&utm_medium=feed) Learn why concurrent DELETE, UPDATE, and MERGE operations fail in Delta Lake, and discover practical ways to debug transaction conflicts, prevent overlapping writes, and build more reliable AWS Glue data pipelines. 10 min read · [rahul-dataeng.hashnode.dev](https://rahul-dataeng.hashnode.dev/when-delta-lake-jobs-collide-troubleshooting-concurrency-issues-in-aws-glue?utm_source=hashnode&utm_medium=feed) [Rahul R](https://hashnode.com/@rahul-data) · [Oct 11](https://hashnode.com/posts/aws-step-functions-vs-apache-airflow-similar-purpose-different-use-cases/6acba8cccb68c86f3859f9b7) ### [AWS Step Functions vs Apache Airflow: Similar Purpose, Different Use Cases](https://rahul-dataeng.hashnode.dev/aws-step-functions-vs-apache-airflow-similar-purpose-different-use-cases?utm_source=hashnode&utm_medium=feed) They both orchestrate workflows, but they solve different problems. Let's understand when to use each. 7 min read · [rahul-dataeng.hashnode.dev](https://rahul-dataeng.hashnode.dev/aws-step-functions-vs-apache-airflow-similar-purpose-different-use-cases?utm_source=hashnode&utm_medium=feed) [Fred Feng](https://hashnode.com/@paganini2008) · [Oct 1](https://hashnode.com/posts/cronflower-run-a-dag-workflow-across-your-cluster-instead-of-chaining-cron-jobs/6abdef04e5081934f25f4963) ### [Cronflower: Run a DAG workflow across your cluster, instead of chaining cron jobs](https://fredfeng.hashnode.dev/cronflower-run-a-dag-workflow-across-your-cluster-instead-of-chaining-cron-jobs?utm_source=hashnode&utm_medium=feed) Distributed DAG workflow engine on Spring Boot cluster 6 min read · [fredfeng.hashnode.dev](https://fredfeng.hashnode.dev/cronflower-run-a-dag-workflow-across-your-cluster-instead-of-chaining-cron-jobs?utm_source=hashnode&utm_medium=feed) [Tao Sun](https://hashnode.com/@taofit) · [Aug 15](https://hashnode.com/posts/how-i-built-a-secure-ai-database-gateway-in-go-using-mcp-anthropic-and-kafka/6a809b5ae5235a6a9b43db8c) ### [How I Built a Secure AI Database Gateway in Go Using MCP, Anthropic, and Kafka](https://tech-mind.hashnode.dev/how-i-built-a-secure-ai-database-gateway-in-go-using-mcp-anthropic-and-kafka?utm_source=hashnode&utm_medium=feed) Want to chat with your database. A Go MCP server that lets LLM agents safely manage Amazon DynamoDB, guarded by risk analysis, Kafka, and Prometheus. 9 min read · [tech-mind.hashnode.dev](https://tech-mind.hashnode.dev/how-i-built-a-secure-ai-database-gateway-in-go-using-mcp-anthropic-and-kafka?utm_source=hashnode&utm_medium=feed) [Andrew Tan](https://hashnode.com/@tandrew) · [Jul 31](https://hashnode.com/posts/your-data-warehouse-is-not-your-data-pipeline/6a6c860d8bcc7e237ed61dad) ### [Your Data Warehouse Is Not Your Data Pipeline](https://layline.hashnode.dev/your-data-warehouse-is-not-your-data-pipeline?utm_source=hashnode&utm_medium=feed) Teams keep forcing their warehouse to do integration work it was never designed for. The result is ballooning costs, opaque failures, and architectures that become harder to maintain the more they ‘succeed.’ 9 min read · [layline.hashnode.dev](https://layline.hashnode.dev/your-data-warehouse-is-not-your-data-pipeline?utm_source=hashnode&utm_medium=feed) [Karthik Darbha](https://hashnode.com/@kdarbha) · [Jul 30](https://hashnode.com/posts/ai-pipeline-observability-databricks/6a6b0cc2003aaae589fc7b02) ### [Observability That Thinks: AI for Pipeline Monitoring](https://tech4nirvana.com/ai-pipeline-observability-databricks?utm_source=hashnode&utm_medium=feed) AI-Augmented Data Engineering — Article 3 of 7 8 min read · [tech4nirvana.com](https://tech4nirvana.com/ai-pipeline-observability-databricks?utm_source=hashnode&utm_medium=feed) [3](https://hashnode.com/posts/ai-pipeline-observability-databricks/6a6b0cc2003aaae589fc7b02) [charles onokohwomo](https://hashnode.com/@ckohwo) · [Jul 14](https://hashnode.com/posts/phase-2b-building-the-adip-ai-insight-engine-with-python-and-llms/6a5692e1713d164b9b2d1659) ### [Phase 2B — Building the ADIP AI Insight Engine](https://charlesonokohwomo.hashnode.dev/phase-2b-building-the-adip-ai-insight-engine-with-python-and-llms?utm_source=hashnode&utm_medium=feed) Building an LLM-powered insight engine with context builders, prompt engineering, caching, and resilient AI architecture. 12 min read · [charlesonokohwomo.hashnode.dev](https://charlesonokohwomo.hashnode.dev/phase-2b-building-the-adip-ai-insight-engine-with-python-and-llms?utm_source=hashnode&utm_medium=feed) [Ikraj Singh](https://hashnode.com/@ikraj-singh) · [Jul 13](https://hashnode.com/posts/data-pipelines-in-the-age-of-genai/6a54784c356ba222071a3f61) ### [Data Pipelines in the Age of GenAI](https://ikrajsingh.hashnode.dev/data-pipelines-in-the-age-of-genai?utm_source=hashnode&utm_medium=feed) How Data Engineers can accelerate pipeline development without giving up reliability 10 min read · [ikrajsingh.hashnode.dev](https://ikrajsingh.hashnode.dev/data-pipelines-in-the-age-of-genai?utm_source=hashnode&utm_medium=feed) [Harsh Trivedi](https://hashnode.com/@harshtrivedi) · [Jul 11](https://hashnode.com/posts/building-a-secure-sharepoint-azure-blob-snowflake-document-intelligence-pipeline/6a51fefe9bf7020ceb3307ea) ### [Building a Secure SharePoint → Azure Blob → Snowflake Document Intelligence Pipeline](https://harshtrivedii.hashnode.dev/building-a-secure-sharepoint-azure-blob-snowflake-document-intelligence-pipeline?utm_source=hashnode&utm_medium=feed) Full guide on how to place or ingest files from sharepoint to snowflake via Azure for AI Agents 20 min read · [harshtrivedii.hashnode.dev](https://harshtrivedii.hashnode.dev/building-a-secure-sharepoint-azure-blob-snowflake-document-intelligence-pipeline?utm_source=hashnode&utm_medium=feed) [Sergio González Téllez](https://hashnode.com/@evandev) · [Jul 2](https://hashnode.com/posts/why-do-so-many-systems-appear-to-produce-new-information-when-in-reality-they-only-reorganize-existing-information/6a45aa26e5ab03662c3894ec) ### [Why do so many systems appear to produce new information when, in reality, they only reorganize existing information?](https://evankhandev.hashnode.dev/why-do-so-many-systems-appear-to-produce-new-information-when-in-reality-they-only-reorganize-existing-information?utm_source=hashnode&utm_medium=feed) Explore data reorganization, transformation, filtering, and how modern systems generate operational value. 2 min read · [evankhandev.hashnode.dev](https://evankhandev.hashnode.dev/why-do-so-many-systems-appear-to-produce-new-information-when-in-reality-they-only-reorganize-existing-information?utm_source=hashnode&utm_medium=feed)