--- title: "#dataengineering articles | Hashnode" description: "A field deals with creating, scheduling and processing massive data at scale using pipelines" url: "https://hashnode.com/tag/dataengineering" --- Tag # #dataengineering 429 posts · 74 followers [Abivarma R S](https://hashnode.com/@abivarma) · [2h ago](https://hashnode.com/posts/the-semantic-debt-trap-why-ai-cannot-query-your-raw-data-lakehouse/6af4bb9d3613a4cac35cb39e) ### [The Semantic Debt Trap: Why AI Cannot Query Your Raw Data Lakehouse](https://ai-decoded.hashnode.dev/the-semantic-debt-trap-why-ai-cannot-query-your-raw-data-lakehouse?utm_source=hashnode&utm_medium=feed) Agents shouldn't write raw SQL against your lakehouse for any number that ends up in a board report. Your lakehouse holds your data. It doesn't hold what your business means by it. The benchmarks poin 3 min read · [ai-decoded.hashnode.dev](https://ai-decoded.hashnode.dev/the-semantic-debt-trap-why-ai-cannot-query-your-raw-data-lakehouse?utm_source=hashnode&utm_medium=feed) [Rahul R](https://hashnode.com/@rahul-data) · [6h ago](https://hashnode.com/posts/how-delta-lake-is-built-on-top-of-amazon-s3-understanding-the-architecture-behind-it/6af47b60c44e54ee6bdc37ab) ### [How Delta Lake Is Built on Top of Amazon S3: Understanding the Architecture Behind It](https://rahul-dataeng.hashnode.dev/how-delta-lake-is-built-on-top-of-amazon-s3-understanding-the-architecture-behind-it?utm_source=hashnode&utm_medium=feed) Understanding How Parquet Files and Transaction Logs Turn an S3 Data Lake into a Reliable Lakehouse. 8 min read · [rahul-dataeng.hashnode.dev](https://rahul-dataeng.hashnode.dev/how-delta-lake-is-built-on-top-of-amazon-s3-understanding-the-architecture-behind-it?utm_source=hashnode&utm_medium=feed) [Abivarma R S](https://hashnode.com/@abivarma) · [12h ago](https://hashnode.com/posts/when-ai-hallucinations-are-really-data-contract-failures/6af42ce4212fcdd5587c008e) ### [When AI Hallucinations Are Really Data Contract Failures](https://ai-decoded.hashnode.dev/when-ai-hallucinations-are-really-data-contract-failures?utm_source=hashnode&utm_medium=feed) Short version of what I took away from this one: when a RAG assistant quotes a confidently wrong number, pull the retrieved records before you blame the model. Often the model was handed the wrong num 3 min read · [ai-decoded.hashnode.dev](https://ai-decoded.hashnode.dev/when-ai-hallucinations-are-really-data-contract-failures?utm_source=hashnode&utm_medium=feed) [1](https://hashnode.com/posts/when-ai-hallucinations-are-really-data-contract-failures/6af42ce4212fcdd5587c008e) [Adan Potyondy](https://hashnode.com/@apotyondy) · [1d ago](https://hashnode.com/posts/load-an-on-prem-database-into-a-cloud-warehouse-1-million-rows-measured/6af32db465b9d010ed84d894) ### [Load an on-prem database into a cloud warehouse: 1 million rows, measured](https://apotyondy.hashnode.dev/load-an-on-prem-database-into-a-cloud-warehouse-1-million-rows-measured?utm_source=hashnode&utm_medium=feed) Rows per insert set the speed. Checks rewritten from a data profile caught 94% of bad rows, and personal data never left the network. 17 min read · [apotyondy.hashnode.dev](https://apotyondy.hashnode.dev/load-an-on-prem-database-into-a-cloud-warehouse-1-million-rows-measured?utm_source=hashnode&utm_medium=feed) [Ositrustees](https://hashnode.com/@Ositrustees) · [2d ago](https://hashnode.com/posts/ositrustees-building-a-market-data-validation-pipeline-in-python/6af1efbbd1b5bafd10bcc803) ### [Ositrustees: Building a Market Data Validation Pipeline in Python](https://ositrustees.hashnode.dev/ositrustees-building-a-market-data-validation-pipeline-in-python?utm_source=hashnode&utm_medium=feed) Explore how Ositrustees research principles relate to Python market data validation, timestamp checks, duplicate detection and financial analytics. Copy 12 min read · [ositrustees.hashnode.dev](https://ositrustees.hashnode.dev/ositrustees-building-a-market-data-validation-pipeline-in-python?utm_source=hashnode&utm_medium=feed) [RevOS\_AI](https://hashnode.com/@revos) · [2d ago](https://hashnode.com/posts/implementing-the-modern-data-strategy-microsoft-fabric-blueprint/6af1a101f9d26b3345b98cf4) ### [Implementing the Modern Data Strategy: Microsoft Fabric Blueprint](https://revosai.hashnode.dev/implementing-the-modern-data-strategy-microsoft-fabric-blueprint?utm_source=hashnode&utm_medium=feed) Build a Medallion Architecture on Microsoft Fabric: 7-workspace model, Direct Lake optimization, and governance strategies for the AI era. 11 min read · [revosai.hashnode.dev](https://revosai.hashnode.dev/implementing-the-modern-data-strategy-microsoft-fabric-blueprint?utm_source=hashnode&utm_medium=feed) [Quayen](https://hashnode.com/@quayen) · [2d ago](https://hashnode.com/posts/streaming-mysql-to-bigquery-with-the-binlog-how-cdc-works-and-the-parts-that-bite/6af1824097c64d348b02ff31) ### [Streaming MySQL to BigQuery with the binlog: how CDC works, and the parts that bite](https://quayen.hashnode.dev/streaming-mysql-to-bigquery-with-the-binlog-how-cdc-works-and-the-parts-that-bite?utm_source=hashnode&utm_medium=feed) Most teams start syncing MySQL to BigQuery with a nightly dump or a query like SELECT \* FROM orders WHERE updated\_at > :last\_run. It works until someone asks why deleted orders still show up in the da 6 min read · [quayen.hashnode.dev](https://quayen.hashnode.dev/streaming-mysql-to-bigquery-with-the-binlog-how-cdc-works-and-the-parts-that-bite?utm_source=hashnode&utm_medium=feed) [Hermes Soulmate](https://hashnode.com/@hermessoulmate) · [5d ago](https://hashnode.com/posts/merging-overlapping-scrape-results-without-double-counting-1-869-rows-in-1-671-out/6aed40897820c1ff62ba9c12) ### [Merging overlapping scrape results without double counting: 1,869 rows in, 1,671 out](https://hermessoulmate.hashnode.dev/merging-overlapping-scrape-results-without-double-counting-1-869-rows-in-1-671-out?utm_source=hashnode&utm_medium=feed) Most of my scraping jobs don't produce one clean dataset. They produce several that overlap: the same search run with four keywords, the same category page scraped every night, two scrapers pointed at 9 min read · [hermessoulmate.hashnode.dev](https://hermessoulmate.hashnode.dev/merging-overlapping-scrape-results-without-double-counting-1-869-rows-in-1-671-out?utm_source=hashnode&utm_medium=feed) [Danda Pranav Krishna](https://hashnode.com/@pranavkrishnadanda) · [Nov 4](https://hashnode.com/posts/building-a-self-planning-data-pipeline-agent-and-measuring-whether-it-s-actually-right/6aeaaa9dc6c5769a73e6b4f2) ### [Building a self-planning data pipeline agent, and measuring whether it's actually right](https://pranavkrishna.hashnode.dev/building-a-self-planning-data-pipeline-agent-and-measuring-whether-it-s-actually-right?utm_source=hashnode&utm_medium=feed) How Sluice repairs drifted data files with LLM agents, a locked-down sandbox and exact ground truth, on free-tier models TL;DR — Sluice is an open-source agentic data pipeline. A file arrives with dr 12 min read · [pranavkrishna.hashnode.dev](https://pranavkrishna.hashnode.dev/building-a-self-planning-data-pipeline-agent-and-measuring-whether-it-s-actually-right?utm_source=hashnode&utm_medium=feed) [2](https://hashnode.com/posts/building-a-self-planning-data-pipeline-agent-and-measuring-whether-it-s-actually-right/6aeaaa9dc6c5769a73e6b4f2) [RevOS\_AI](https://hashnode.com/@revos) · [Oct 30](https://hashnode.com/posts/how-to-build-a-modern-data-strategy-in-the-age-of-ai/6ae488ab0d8b46af07d95a3b) ### [How to Build a Modern Data Strategy in the Age of AI](https://revosai.hashnode.dev/how-to-build-a-modern-data-strategy-in-the-age-of-ai?utm_source=hashnode&utm_medium=feed) Most AI initiatives fail because of data chaos, not tools. The Medallion Architecture (Bronze, Silver, Gold) creates trust and unlocks revenue. 8 min read · [revosai.hashnode.dev](https://revosai.hashnode.dev/how-to-build-a-modern-data-strategy-in-the-age-of-ai?utm_source=hashnode&utm_medium=feed)