---
title: "Discussion on \"Best Practices for Collecting Public Web Data at Scale\" | Hashnode"
description: "Discussion on \"Best Practices for Collecting Public Web Data at Scale\". Efficient public web data collection at scale: check APIs first, manage rate limits & proxies, validate data, and stay GDPR/CCPA compliant."
url: "https://hashnode.com/posts/best-practices-for-collecting-public-web-data-at-scale/6a7914a6b89c6b64d7572aae"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Best Practices for Collecting Public Web Data at Scale](https://sohomdas.hashnode.dev/best-practices-for-collecting-public-web-data-at-scale?utm_source=hashnode&utm_medium=feed)

Efficient public web data collection at scale: check APIs first, manage rate limits & proxies, validate data, and stay GDPR/CCPA compliant.

Aug 10 · 13 min read · sohomdas.hashnode.dev  
Read the article

-   [#web-scraping](https://hashnode.com/tag/web-scraping)
-   [#web-data](https://hashnode.com/tag/web-data)
-   [#data-scraping](https://hashnode.com/tag/data-scraping)

## Written by

[sohom das](https://hashnode.com/@TheLostWanderer47)

@TheLostWanderer47

  
[View profile](https://hashnode.com/@TheLostWanderer47)

## Responses 1

Markdown supported · ⌘↵ to post

[OnlineProxy](https://hashnode.com/@onlineproxyio)

[Aug 11](https://hashnode.com/posts/best-practices-for-collecting-public-web-data-at-scale/6a7914a6b89c6b64d7572aae/comment/6a7ad606b3af1198242eb75e)

The whole hiQ versus linkedIn case gets glossed over, but hiQ got shut down hard and had to stop scraping entirely, which is basically the opposite of winning, and it raises the awkward question of whether companies actually go scorched-earth like linkedIn did or just shrug and let it happen like most sites do. They mention GDPR's right-to-erasure thing but completely skip the real world problems like your database literally not being built for instant deletion, retention policies that conflict with each other, or the liability nightmare if you screw up the deletion, so readers end up discovering these aren't just box-checking bureaucracy. Those bright data numbers get thrown out there with zero mention that AIMultiple literally generated them as a vendor, and there's no actual head-to-head testing against apify, oxylabs, or scraperAPI, which means you can't tell if the recommendation is actually solid or just marketing noise. If you're building your first scraper, this guide gives you a decent starting point, but if you're going production with real user data or hammering some serious targets, test how your specific target actually enforces their policies instead of assuming they'll act like LinkedIn, and run the infrastructure benchmarks yourself with your own data and geography instead of just trusting whatever published scores they're sharing
