Production Pipeline blotted with duplicates

When you’re ingesting hundreds of thousands of products data on daily basis, even a small oversight can snowball into a nightmare. That’s exactly what happened in one of our eBay pipelines. eBay product URLs often come with query parameters. One of o...