FeedDiscussion
How to scrape web content cleanly for LLM pipelines
I am Krystin, the non-technical co-founder of EnConvert. Most of what I know about RAG pipelines I learned from being in the room while engineers described what was breaking. Web content extraction co
theunscriptedk.hashnode.dev4 min read
#rag#llm#web-scraping#ai#ai-engineering#developer-tools#developer#python#playwright#markdown#machine-learning#enconvert
Mojibake is the worse version of the escaped character stuff, since entities are visible and reversible while charset mismatches fail silently and can lose data for good. Python requests assumes latin one for text responses with no charset so UTF 8 bytes get misread and café turns into smth weird with no error at all. Encoding to latin one and decoding as utf 8 fixes it, but only if the text isn't already damaged.
Tables have the same problem, since many renderers strip html colspan and rowspan out of markdown and naive pipe table conversion keeps the numbers but loses what they mean