PNPrithwish Nathinprithwish-nath.hashnode.dev·Aug 20 · 23 min readWhy Google, Common Crawl, and ChatGPT See Different Versions of the Same URLA URL identifies a resource, but it does not guarantee one canonical representation of that resource. Go visit the same URL with a browser, an HTTP client, Common Crawl / the Wayback Machine, or ask 00
PNPrithwish Nathinprithwish-nath.hashnode.dev·Aug 20 · 15 min readHow Much of the Web Disappears Without JavaScript?I read about celld last week: an open-source daemon that re-implements Cloudflare Workers and Durable Objects, self-hosted, no Cloudflare account required. I wanted to try it on my hobby project — a00
PNPrithwish Nathinprithwish-nath.hashnode.dev·Jul 30 · 16 min readHow Rotating Residential Proxies Work: Per-Request vs Sticky vs TimedTL;DR: This article explains what a rotating residential proxy is, and how rotation is actually three genuinely different behaviors - and not a single toggle - using real exit IP logs. I wrote a prev00
PNPrithwish Nathinprithwish-nath.hashnode.dev·Jul 29 · 23 min readHow LLMs Get Training Data from the WebTL;DR: AI companies don’t train language models on raw HTML. Learn how AI labs fetch, extract, deduplicate, and filter web crawls into a clean, reproducible corpus. So, You Want To Train a Model… Can 00
PNPrithwish Nathinprithwish-nath.hashnode.dev·Jul 21 · 21 min readWhy Increasing Concurrency Causes HTTP 429 Errors (It’s Not Just Too Many Requests)TL;DR: Concurrency is usually introduced as a performance tip. Parallelism is a good thing, so more workers should mean more requests, and more data. If ten workers are good, why not twenty? If twenty00
