---
title: "Discussion on \"Building a RAG Pipeline Without a Scraper Farm: Free Fetch-to-Markdown for Agents\" | Hashnode"
description: "Discussion on \"Building a RAG Pipeline Without a Scraper Farm: Free Fetch-to-Markdown for Agents\". Build a RAG ingest path that fetches a public URL and returns clean Markdown—no scraper farm. Free endpoint included."
url: "https://hashnode.com/posts/building-a-rag-pipeline-without-a-scraper-farm-free-fetch-to-markdown-for-agents/6aee897e8959be50809030ea"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Building a RAG Pipeline Without a Scraper Farm: Free Fetch-to-Markdown for Agents](https://anycrawler.hashnode.dev/building-a-rag-pipeline-without-a-scraper-farm-free-fetch-to-markdown-for-agents?utm_source=hashnode&utm_medium=feed)

Build a RAG ingest path that fetches a public URL and returns clean Markdown—no scraper farm. Free endpoint included.

4d ago · 7 min read · anycrawler.hashnode.dev  
Read the article

-   [#rag](https://hashnode.com/tag/rag)
-   [#ai](https://hashnode.com/tag/ai)
-   [#agents](https://hashnode.com/tag/agents)
-   [#llm](https://hashnode.com/tag/llm)
-   [#webscraping](https://hashnode.com/tag/webscraping)

## Written by

[AnyCrawler](https://hashnode.com/@anycrawler)

@anycrawler

  
[View profile](https://hashnode.com/@anycrawler)

## Responses 2

Markdown supported · ⌘↵ to post

[OnlineProxy](https://hashnode.com/@onlineproxyio)

[1d ago](https://hashnode.com/posts/building-a-rag-pipeline-without-a-scraper-farm-free-fetch-to-markdown-for-agents/6aee897e8959be50809030ea/comment/6af21bc2b2d1ae1ae5497851)

Stale chunks were the thing I'd check first. If a page shrinks from 10 chunks to 7, the vectors for chunks numbered 7 through 9 just sit in the store and keep getting retrieved. I mean hashing the content doesn't really fix that by itself, so i'd run a delete on source\_url before every upsert. Or a generation key, if you don't want to delete in place

And keep fetched\_at in the metadata, otherwise you've got no idea how old the stuff you're serving actually is

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[2d ago](https://hashnode.com/posts/building-a-rag-pipeline-without-a-scraper-farm-free-fetch-to-markdown-for-agents/6aee897e8959be50809030ea/comment/6af15827922148e5e7498c77)

The minimal chunker silently discards the remainder of any paragraph longer than max\_chars: both p\[:max\_chars\] and p.slice(0, maxChars) keep only its beginning. A long documentation paragraph can therefore lose the actual answer before embedding, even though fetching and indexing appear successful.

I would make the example emit the remaining slices, or explicitly reject oversized paragraphs until a real splitter is supplied. A fixture with the required fact after character 800 would catch the loss immediately. Even for a deliberately naive helper, preserving all source text is a more useful baseline than silently trimming it.