JSJosef Strzibnyinserpapi.hashnode.dev·4d ago · 32 min readA complete guide to web scraping with Ruby (2026)Web scraping is the process of pulling out structured data from websites. If you're familiar with Ruby, you're in luck_._ Ruby has excellent tools you can use to scrape the web. From HTTP libraries to10
ATAnakin Techinanakinwriters.hashnode.dev·4d ago · 8 min readCloudflare Blocks Are a Routing Problem, Not a Retry ProblemMost Cloudflare fixes fail because they treat every block as the same class of error. The reliable pattern is not “add a proxy” or “use a browser”. The reliable pattern is to route each failed request00
ATAnakin Techinanakinwriters.hashnode.dev·6d ago · 7 min readTreat Browser Fingerprints Like Production StateThe fastest way to get detected is to treat browser fingerprinting as a list of values to randomize. Real browsers are boring. They keep the same timezone, GPU, language, storage, TLS shape, and brows00
TWTony Wangincrawlora.hashnode.dev·Jul 27 · 12 min readWe Followed 173 Million Domains for 8 Years. ~40M Died.Key takeaways We built an 8-year reachability panel of 172,959,928 registered domains from 80 monthly Common Crawl archives (January 2018 to May 2026) — every domain Common Crawl ever fetched a robot00
TWTony Wangincrawlora.hashnode.dev·Jul 27 · 9 min readWe Fingerprinted 978K Websites: Cloudflare, WordPress, and the CAPTCHA Cross-CheckKey takeaways 978,539 Tranco top-1M domains fingerprinted for their tech stack from homepage HTML + response headers, of which 80.4% (787,205) resolved a working homepage — one snapshot, one domain, 00
TWTony Wangincrawlora.hashnode.dev·Jul 27 · 7 min readX's App Store Rating Never Moved After Musk Bought It. Its Reviews Did.Originally published on the Crawlora blog. When Elon Musk closed his $44 billion purchase of Twitter on October 27, 2022, it was one of the most scrutinized corporate takeovers in tech history. Nearly00
RRoamProxyinroamproxy.hashnode.dev·Jul 26 · 5 min readYour proxy connected. That doesn't mean it hid your IP.Every proxy checker I've come across answers one question: did the connection open? That's the least interesting thing about a proxy. A proxy can connect perfectly and still hand your real IP address 00
AKAntony Kanaguinantonykanagu.hashnode.dev·Jul 12 · 4 min readMost of what I learned about web scraping had nothing to do with scrapingI spent a week recently building a small agent that was supposed to do something simple: visit a handful of public web pages every morning and pull back a few specific fields. That's it. Read some val01O
HHamzaingraphql-inspector.hashnode.dev·Jul 1 · 12 min readExpedia Cars GraphQL Inspector: How I Turned Expedia's Network Traffic into Instant CSV ExportsArticle Revision 2.0 I got tired of clicking "Load More." You know the drill: you search for rental cars, the page shows 25 results, and there are 30 more pages hiding behind that little button. Click03OH
BBeckyinbeckybuilds.hashnode.dev·Jun 29 · 15 min readDesigning a Crawler That Beats Three Tiers of Anti-Bot — From TLS Fingerprints to CDP DetectionAn engineering log of building a Threads marketing agent, split into two parts. Part 1 (this post): The data-acquisition layer — a crawler designed around an anti-bot threat model Part 2: The genera01O
