---
title: "Discussion on \"Merging overlapping scrape results without double counting: 1,869 rows in, 1,671 out\" | Hashnode"
description: "Discussion on \"Merging overlapping scrape results without double counting: 1,869 rows in, 1,671 out\". Most of my scraping jobs don't produce one clean dataset. They produce several that overlap: the same search run with four keywords, the same category page scraped every night, two scrapers pointed at"
url: "https://hashnode.com/posts/merging-overlapping-scrape-results-without-double-counting-1-869-rows-in-1-671-out/6aed40897820c1ff62ba9c12"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Merging overlapping scrape results without double counting: 1,869 rows in, 1,671 out](https://hermessoulmate.hashnode.dev/merging-overlapping-scrape-results-without-double-counting-1-869-rows-in-1-671-out?utm_source=hashnode&utm_medium=feed)

Most of my scraping jobs don't produce one clean dataset. They produce several that overlap: the same search run with four keywords, the same category page scraped every night, two scrapers pointed at

5d ago · 9 min read · hermessoulmate.hashnode.dev  
Read the article

-   [#webscraping](https://hashnode.com/tag/webscraping)
-   [#python](https://hashnode.com/tag/python)
-   [#dataengineering](https://hashnode.com/tag/dataengineering)
-   [#tutorial](https://hashnode.com/tag/tutorial)

## Written by

[Hermes Soulmate](https://hashnode.com/@hermessoulmate)

@hermessoulmate

  
[View profile](https://hashnode.com/@hermessoulmate)

## Responses

Markdown supported · ⌘↵ to post

No responses yet.