---
title: "Discussion on \"Production Pipeline blotted with duplicates\" | Hashnode"
description: "Discussion on \"Production Pipeline blotted with duplicates\". When you’re ingesting hundreds of thousands of products data on daily basis, even a small oversight can snowball into a nightmare.\nThat’s exactly what happened in one of our eBay pipelines.\neBay product URLs often come with query parameters. One of o..."
url: "https://hashnode.com/posts/production-pipeline-blotted-with-duplicates/68c24d8ca83720dabb4562f9"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Production Pipeline blotted with duplicates](https://behindthedeploy.hashnode.dev/production-pipeline-blotted-with-duplicates?utm_source=hashnode&utm_medium=feed)

When you’re ingesting hundreds of thousands of products data on daily basis, even a small oversight can snowball into a nightmare. That’s exactly what happened in one of our eBay pipelines. eBay product URLs often come with query parameters. One of o...

Sep 11, 2025 · 2 min read · behindthedeploy.hashnode.dev  
Read the article

-   [#production](https://hashnode.com/tag/production)
-   [#software-development](https://hashnode.com/tag/software-development)
-   [#software-engineering](https://hashnode.com/tag/software-engineering)

## Written by

[Muhammad Safdar](https://hashnode.com/@mosafdar)

@mosafdar

  
[View profile](https://hashnode.com/@mosafdar)

Software Engineer | Building Scalable Cloud Native Backends for AI-Driven Products

I’m a backend engineer who enjoys turning messy problems into scalable backend systems. My core stack is Python, FastAPI, Django, and AWS, and I’ve recently been focused on building AI-driven systems using LangChain, RAG pipelines, and smart data ingestion tools like Scrapy, Selenium, and Apache Airflow. Over the past year, I’ve contributed to complex fintech and SaaS projects, building clean &amp; modular APIs, integrating third-party services, and optimizing legacy codebases for performance and maintainability. Beyond the code, I care about writing systems that are easy to extend, mentoring engineers, and solving problems that actually make people’s work easier, not harder. Some things I’ve worked on: ► Used LangChain &amp; RAG to prototype internal AI assistants for support and documentation. ► Built backend services with Python, FastAPI, and Django. ► Designed API-first systems for leasing and finance platforms (JSON/XML handling, complex business logic). ► Scraped and pipelined structured data using Scrapy, BeautifulSoup, and Selenium I’m always open to connecting with founders, teams, or devs who are building useful tech — especially in the AI and automation space. 📬 You can reach me here or at mosafdaralii@gmail.com

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
