--- title: "#data-engineer articles | Hashnode" description: "Explore 139 articles tagged #data-engineer on Hashnode. Tutorials, guides, and how-tos from the community." url: "https://hashnode.com/tag/data-engineer" --- Tag # #data-engineer 139 posts · 322 followers [Rahul R](https://hashnode.com/@rahul-data) · [4d ago](https://hashnode.com/posts/how-delta-lake-is-built-on-top-of-amazon-s3-understanding-the-architecture-behind-it/6af47b60c44e54ee6bdc37ab) ### [How Delta Lake Is Built on Top of Amazon S3: Understanding the Architecture Behind It](https://rahul-dataeng.hashnode.dev/how-delta-lake-is-built-on-top-of-amazon-s3-understanding-the-architecture-behind-it?utm_source=hashnode&utm_medium=feed) Understanding How Parquet Files and Transaction Logs Turn an S3 Data Lake into a Reliable Lakehouse. 8 min read · [rahul-dataeng.hashnode.dev](https://rahul-dataeng.hashnode.dev/how-delta-lake-is-built-on-top-of-amazon-s3-understanding-the-architecture-behind-it?utm_source=hashnode&utm_medium=feed) [Amie.](https://hashnode.com/@amiez) · [Oct 17](https://hashnode.com/posts/what-is-data-engineering-a-beginner-s-perspective/6ad3b46e911d2e02cd656fda) ### [What Is Data Engineering? A Beginner’s Perspective](https://codecoffeee.hashnode.dev/what-is-data-engineering-a-beginner-s-perspective?utm_source=hashnode&utm_medium=feed) Data Engineering is about building systems that collect, process, organize, and deliver data so that people and applications can use it. 10 min read · [codecoffeee.hashnode.dev](https://codecoffeee.hashnode.dev/what-is-data-engineering-a-beginner-s-perspective?utm_source=hashnode&utm_medium=feed) [Rahul R](https://hashnode.com/@rahul-data) · [Oct 12](https://hashnode.com/posts/when-delta-lake-jobs-collide-troubleshooting-concurrency-issues-in-aws-glue/6acca39ec13a8ac656801a58) ### [When Delta Lake Jobs Collide: Troubleshooting Concurrency Issues in AWS Glue](https://rahul-dataeng.hashnode.dev/when-delta-lake-jobs-collide-troubleshooting-concurrency-issues-in-aws-glue?utm_source=hashnode&utm_medium=feed) Learn why concurrent DELETE, UPDATE, and MERGE operations fail in Delta Lake, and discover practical ways to debug transaction conflicts, prevent overlapping writes, and build more reliable AWS Glue data pipelines. 10 min read · [rahul-dataeng.hashnode.dev](https://rahul-dataeng.hashnode.dev/when-delta-lake-jobs-collide-troubleshooting-concurrency-issues-in-aws-glue?utm_source=hashnode&utm_medium=feed) [Rahul R](https://hashnode.com/@rahul-data) · [Oct 11](https://hashnode.com/posts/aws-step-functions-vs-apache-airflow-similar-purpose-different-use-cases/6acba8cccb68c86f3859f9b7) ### [AWS Step Functions vs Apache Airflow: Similar Purpose, Different Use Cases](https://rahul-dataeng.hashnode.dev/aws-step-functions-vs-apache-airflow-similar-purpose-different-use-cases?utm_source=hashnode&utm_medium=feed) They both orchestrate workflows, but they solve different problems. Let's understand when to use each. 7 min read · [rahul-dataeng.hashnode.dev](https://rahul-dataeng.hashnode.dev/aws-step-functions-vs-apache-airflow-similar-purpose-different-use-cases?utm_source=hashnode&utm_medium=feed) [John Fáwọlé](https://hashnode.com/@johnfawole) · [Aug 28](https://hashnode.com/posts/what-to-look-for-in-a-scraping-api-7-strong-options-reviewed/6a91831abbae4038cc854d1e) ### [What to Look for in a Scraping API: 7 Strong Options Reviewed](https://johnfawole.hashnode.dev/what-to-look-for-in-a-scraping-api-7-strong-options-reviewed?utm_source=hashnode&utm_medium=feed) Nowadays, there are so many scraping APIs in the market; both high-quality and the not-so-quality ones. As an experienced web scraping engineer, there are some technical factors I look out for when pi 20 min read · [johnfawole.hashnode.dev](https://johnfawole.hashnode.dev/what-to-look-for-in-a-scraping-api-7-strong-options-reviewed?utm_source=hashnode&utm_medium=feed) [1](https://hashnode.com/posts/what-to-look-for-in-a-scraping-api-7-strong-options-reviewed/6a91831abbae4038cc854d1e) [ryan scott](https://hashnode.com/@ryanscott) · [Jul 22](https://hashnode.com/posts/top-5-gcp-data-engineer-exam-topics/6a60afb76bdad57d7ac130bc) ### [Top 5 Google Cloud Professional Data Engineer Exam Topics](https://ryanscott.hashnode.dev/top-5-gcp-data-engineer-exam-topics?utm_source=hashnode&utm_medium=feed) Preparing for the GCP Data Engineer exam? Discover five Professional Data Engineer must-know topics you should masters. 5 min read · [ryanscott.hashnode.dev](https://ryanscott.hashnode.dev/top-5-gcp-data-engineer-exam-topics?utm_source=hashnode&utm_medium=feed) [Mahrab Khan](https://hashnode.com/@mahrab) · [Jun 29](https://hashnode.com/posts/the-blueprint-of-a-data-engineer-core-interview-questions/6a42824996854c330939cd1e) ### [The Blueprint of a Data Engineer: Core Interview Questions](https://techgreek.hashnode.dev/the-blueprint-of-a-data-engineer-core-interview-questions?utm_source=hashnode&utm_medium=feed) Data engineering is the invisible engine powering modern analytics, agentic workflows, and generative AI initiatives. Without robust data pipelines, even the most advanced machine learning models runn 3 min read · [techgreek.hashnode.dev](https://techgreek.hashnode.dev/the-blueprint-of-a-data-engineer-core-interview-questions?utm_source=hashnode&utm_medium=feed) [Ankit Raj](https://hashnode.com/@ankitsrivastava) · [Mar 16](https://hashnode.com/posts/fix-duplicate-rows-in-sql-merge-using-row-number/69b8498a2ad6ae5184dbfb64) ### [Fix Duplicate Rows in SQL MERGE Using ROW\_NUMBER](https://ankit-data-engineering.hashnode.dev/fix-duplicate-rows-in-sql-merge-using-row-number?utm_source=hashnode&utm_medium=feed) Why frequent ingestion pipelines create duplicate rows and the simple SQL pattern that prevents them. 5 min read · [ankit-data-engineering.hashnode.dev](https://ankit-data-engineering.hashnode.dev/fix-duplicate-rows-in-sql-merge-using-row-number?utm_source=hashnode&utm_medium=feed) [Ankit Raj](https://hashnode.com/@ankitsrivastava) · [Mar 15](https://hashnode.com/posts/5-mistakes-developers-make-when-choosing-between-cloud-run-and-cloud-functions/69b6042334db550e2c05ed74) ### [5 Mistakes Developers Make When Choosing Between Cloud Run and Cloud Functions](https://ankit-data-engineering.hashnode.dev/5-mistakes-developers-make-when-choosing-between-cloud-run-and-cloud-functions?utm_source=hashnode&utm_medium=feed) Stuck between Cloud Run and Cloud Functions? Here is how to decide. 5 min read · [ankit-data-engineering.hashnode.dev](https://ankit-data-engineering.hashnode.dev/5-mistakes-developers-make-when-choosing-between-cloud-run-and-cloud-functions?utm_source=hashnode&utm_medium=feed) [Jonathan D. Fisher](https://hashnode.com/@jonathanfisher) · [Mar 13](https://hashnode.com/posts/python-vs-node-js-for-retail-scrapers-performance-and-cost-benchmarks/69b41fc2063a49003d400471) ### [Python vs. Node.js for Retail Scrapers: Performance and Cost Benchmarks](https://codecraftdevs.hashnode.dev/python-vs-node-js-for-retail-scrapers-performance-and-cost-benchmarks?utm_source=hashnode&utm_medium=feed) Choosing the right stack for a retail data pipeline is often treated as a matter of developer preference. Python is the undisputed leader in data science, making it the default choice for data collect 5 min read · [codecraftdevs.hashnode.dev](https://codecraftdevs.hashnode.dev/python-vs-node-js-for-retail-scrapers-performance-and-cost-benchmarks?utm_source=hashnode&utm_medium=feed)