My GCP Pipeline Was Running Fine… But Doing Nothing (Service Account Lesson)The Situation I was working on a pipeline where everything looked correct from the outside. A file was landing in Cloud Storage, the Cloud Function was triggering as expected, and logs were showing suMay 2 · 4 min readankit-data-engineering.hashnode.dev
7 Skills That Make Data Engineering Feel Less HardWhen I started working in data engineering, nothing felt simple. Tasks that looked straightforward took longer than expected, pipelines broke without clear reasons, and debugging often felt like guessApr 25 · 3 min readankit-data-engineering.hashnode.dev
Dataflow Job Not Starting? Debugging a Job Name Collision in GCPWhile working on a GCP Dataflow pipeline for an event-driven ingestion system, I ran into an issue that was surprisingly tricky to debug. Everything looked correct from the outside. Files were landingApr 18 · 4 min readankit-data-engineering.hashnode.dev
The Small metadata.json File That Changed How I Work With Image DatasetsWhen working with image datasets for machine learning, it is easy to assume that the images themselves are the most important part of the pipeline. At least that is what I thought when I first startedApr 11 · 4 min readankit-data-engineering.hashnode.dev
When a Cloud Function Keeps Failing and the Code Isn’t the ProblemRecently I was working on a small automation using Google Cloud Functions. The function itself was simple. Whenever a file was uploaded to a Cloud Storage bucket, the function would process that file Apr 4 · 5 min readankit-data-engineering.hashnode.dev
