---
title: "Discussion on \"Most of what I learned about web scraping had nothing to do with scraping\" | Hashnode"
description: "Discussion on \"Most of what I learned about web scraping had nothing to do with scraping\". Modern anti-bot defenses stack: IP scoring, TLS fingerprinting, behavior checks. What I learned building a scraper agent, and what I'd tell you first."
url: "https://hashnode.com/posts/most-of-what-i-learned-about-web-scraping-had-nothing-to-do-with-scraping/6a530c0ab0f9dffab96cd521"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Most of what I learned about web scraping had nothing to do with scraping](https://antonykanagu.hashnode.dev/most-of-what-i-learned-about-web-scraping-had-nothing-to-do-with-scraping?utm_source=hashnode&utm_medium=feed)

Modern anti-bot defenses stack: IP scoring, TLS fingerprinting, behavior checks. What I learned building a scraper agent, and what I'd tell you first.

Jul 12 · 4 min read · antonykanagu.hashnode.dev  
Read the article

-   [#webscraping](https://hashnode.com/tag/webscraping)
-   [#ai](https://hashnode.com/tag/ai)
-   [#llm](https://hashnode.com/tag/llm)
-   [#firecrawl](https://hashnode.com/tag/firecrawl)

## Written by

[Antony Kanagu](https://hashnode.com/@antonyk)

@antonyk · Hoffman Estates, Illinois

  
[View profile](https://hashnode.com/@antonyk)

Senior IT leader working on agentic AI, multi-cloud platforms, and AI governance.

I lead cloud infrastructure and generative AI delivery teams at an audio technology company, with 20+ years in software and cloud engineering across AWS and Azure. My writing here is field reports from real builds: multi-agent pipelines on Amazon Bedrock with Strands and MCP, retrieval benchmarking, AI governance frameworks, and cost analysis that goes beyond the whiteboard version. My working principle across all of it: deterministic code computes, LLMs narrate and judge, never the reverse. Away from the keyboard: cycling the trails of northwest Chicago suburbs, Sangam poetry, and Tamil cinema.

## Responses 1

Markdown supported · ⌘↵ to post

[OnlineProxy](https://hashnode.com/@onlineproxyio)

[Jul 12](https://hashnode.com/posts/most-of-what-i-learned-about-web-scraping-had-nothing-to-do-with-scraping/6a530c0ab0f9dffab96cd521/comment/6a5333691495c72048dab6ac)

yeah, LLMs sound cool for extraction, but I'd recommend you - go hybrid. Use XPath for the dead-certain stuff like prices, SKUs, IDs, whatever and only throw semantic extraction at the messy, inconsistent bits. Seriously cuts your latency and doesn't torch your budget. The thing that actually nukes production systems is silent failures, which is why we built output density monitoring - just watches when your extracted fields dip below normal and fires alerts before layout shifts poison the whole downstream pipeline. On the residential proxy side, yeah the heat's on, but it's all about timing: blast your campaigns too hard and you'll get made quick, but if you play it cool with naturalistic patterns and sticky per-target rotation you're still golden - the real squeeze isn't whether it works anymore, it's volume and discipline. Sites going full JS-on-demand is actually the move that kills traditional scraping dead, but it's not ending the game, just reshuffling the playbook
