Tag

#data-engineer

138 posts322 followers

Explore Hashnode

Alternatives

Amie.Oct 17 · 10 min read

What Is Data Engineering? A Beginner’s Perspective

Have you ever wondered how companies handle millions of pieces of data every day? Let's explore the work happening behind the scenes. Imagine opening YouTube to watch a programming tutorial. You searc

codecoffeee.hashnode.dev

Rahul ROct 12 · 10 min read

When Delta Lake Jobs Collide: Troubleshooting Concurrency Issues in AWS Glue

Have you ever run two AWS Glue jobs against the same Delta table and noticed that one job fails while the other completes successfully? The confusing part is that both jobs might work perfectly when e

rahul-dataeng.hashnode.dev

Rahul ROct 11 · 7 min read

AWS Step Functions vs Apache Airflow: Similar Purpose, Different Use Cases

When I started exploring data engineering workflows, one question that came to mind was: if AWS Step Functions and Apache Airflow can both orchestrate workflows, why do we need both? At first glance,

rahul-dataeng.hashnode.dev

John FáwọléAug 28 · 20 min read

What to Look for in a Scraping API: 7 Strong Options Reviewed

Nowadays, there are so many scraping APIs in the market; both high-quality and the not-so-quality ones. As an experienced web scraping engineer, there are some technical factors I look out for when pi

1 johnfawole.hashnode.dev

ryan scottJul 22 · 5 min read

Top 5 Google Cloud Professional Data Engineer Exam Topics

You plan to start studying after work, but the hectic schedule takes over. Before you realise it, the exam is just four weeks away. You finally explore the GCP Data Engineer exam topic videos, expecti

ryanscott.hashnode.dev

Mahrab KhanJun 29 · 3 min read

The Blueprint of a Data Engineer: Core Interview Questions

Data engineering is the invisible engine powering modern analytics, agentic workflows, and generative AI initiatives. Without robust data pipelines, even the most advanced machine learning models runn

techgreek.hashnode.dev

Ankit RajMar 16 · 5 min read

Fix Duplicate Rows in SQL MERGE Using ROW_NUMBER

In many data pipelines, raw data is first collected in a staging or snapshot table before it is merged into the main analytics table. This pattern works well because it allows ingestion to run frequen

ankit-data-engineering.hashnode.dev

Ankit RajMar 15 · 5 min read

5 Mistakes Developers Make When Choosing Between Cloud Run and Cloud Functions

When developers start building serverless applications on Google Cloud, they often run into the same dilemma. Should this workload run on Cloud Run or Cloud Functions? Both services allow you to run c

ankit-data-engineering.hashnode.dev

Jonathan D. FisherMar 13 · 5 min read

Python vs. Node.js for Retail Scrapers: Performance and Cost Benchmarks

Choosing the right stack for a retail data pipeline is often treated as a matter of developer preference. Python is the undisputed leader in data science, making it the default choice for data collect

codecraftdevs.hashnode.dev

FaveFeb 5 · 4 min read

Exploring Data Engineering with Rust

I'm diving into the world of big data using Rust. This might sound unusual—isn't Python the industry standard? Yes, Python dominates the field, but I chose Rust because I want to master it for systems programming, which is my end goal. It all started...

0xfave.hashnode.dev

Search Hashnode