Building a RAG Pipeline Without a Scraper Farm: Free Fetch-to-Markdown for Agents

Stale chunks were the thing I'd check first. If a page shrinks from 10 chunks to 7, the vectors for chunks numbered 7 through 9 just sit in the store and keep getting retrieved. I mean hashing the content doesn't really fix that by itself, so i'd run a delete on source_url before every upsert. Or a generation key, if you don't want to delete in place

And keep fetched_at in the metadata, otherwise you've got no idea how old the stuff you're serving actually is

The minimal chunker silently discards the remainder of any paragraph longer than max_chars: both p[:max_chars] and p.slice(0, maxChars) keep only its beginning. A long documentation paragraph can therefore lose the actual answer before embedding, even though fetching and indexing appear successful.

I would make the example emit the remaining slices, or explicitly reject oversized paragraphs until a real splitter is supplied. A fixture with the required fact after character 800 would catch the loss immediately. Even for a deliberately naive helper, preserving all source text is a more useful baseline than silently trimming it.