---
title: "Discussion on \"How to scrape web content cleanly for LLM pipelines\" | Hashnode"
description: "Discussion on \"How to scrape web content cleanly for LLM pipelines\". Most RAG pipelines break at web content extraction. This covers what raw extraction returns, what clean Markdown looks like, and how to build it."
url: "https://hashnode.com/posts/how-to-scrape-web-content-cleanly-for-llm-pipelines/6ab9ff8aa52e3389fedde9d1"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [How to scrape web content cleanly for LLM pipelines](https://theunscriptedk.hashnode.dev/how-to-scrape-web-content-cleanly-for-llm-pipelines?utm_source=hashnode&utm_medium=feed)

Most RAG pipelines break at web content extraction. This covers what raw extraction returns, what clean Markdown looks like, and how to build it.

Sep 28 · 4 min read · theunscriptedk.hashnode.dev  
Read the article

-   [#rag](https://hashnode.com/tag/rag)
-   [#llm](https://hashnode.com/tag/llm)
-   [#web-scraping](https://hashnode.com/tag/web-scraping)
-   [#ai](https://hashnode.com/tag/ai)
-   [#ai-engineering](https://hashnode.com/tag/ai-engineering)
-   [#developer-tools](https://hashnode.com/tag/developer-tools)
-   [#developer](https://hashnode.com/tag/developer)
-   [#python](https://hashnode.com/tag/python)
-   [#playwright](https://hashnode.com/tag/playwright)
-   [#markdown](https://hashnode.com/tag/markdown)
-   [#machine-learning](https://hashnode.com/tag/machine-learning)
-   [#enconvert](https://hashnode.com/tag/enconvert)

## Written by

[Krystin Lim](https://hashnode.com/@theunscriptedk)

@theunscriptedk

  
[View profile](https://hashnode.com/@theunscriptedk)

102\. figuring it out in public. co-founder @EnConvert.

Currently diving into something messy, nonlinear, and completely absorbing. The ambiguity, the iteration, the moments where a direction suddenly becomes obvious. That is the zero-to-one phase. Hard to put into words and harder to walk away from. EnConvert is building the ingestion layer for AI agents; the part that handles how agents read, extract, and process content from the web. Commercial is my lane. How we talk about the product, how we get it in front of the right people, I own everything that sits between the two. Think positioning, outreach, distribution, partnerships. Year one. Building in public. Figuring it out as I go.

## Responses 1

Markdown supported · ⌘↵ to post

[OnlineProxy](https://hashnode.com/@onlineproxyio)

[Sep 28](https://hashnode.com/posts/how-to-scrape-web-content-cleanly-for-llm-pipelines/6ab9ff8aa52e3389fedde9d1/comment/6aba7300d481d8dcfed049f4)

Mojibake is the worse version of the escaped character stuff, since entities are visible and reversible while charset mismatches fail silently and can lose data for good. Python requests assumes latin one for text responses with no charset so UTF 8 bytes get misread and café turns into smth weird with no error at all. Encoding to latin one and decoding as utf 8 fixes it, but only if the text isn't already damaged.

Tables have the same problem, since many renderers strip html colspan and rowspan out of markdown and naive pipe table conversion keeps the numbers but loses what they mean