Useful comparison, especially the point that the pipeline matters more than the database. One thing worth adding for anyone copying this setup: the "Keeping the index in sync" step gets much cheaper if you store a content hash per chunk as payload/metadata. On each deploy, re-chunk the pages, hash each chunk (heading path + text), and only upsert chunks whose hash changed, then delete IDs that no longer exist. Using a deterministic ID like hash(url + heading) means re-runs are idempotent, so you never end up with duplicate vectors. On Upstash that also keeps you far below the daily write cap, since a typical edit touches two or three chunks rather than the whole site. I'd also log every question that returns a low top score; that list is the best guide to which pages need rewriting or extra chunks.
Thanks, Rose Alvina , this is a great addition. The content hash per chunk is the right way to handle sync. My current setup re-embeds more than it needs to on each deploy, so hashing heading path + text and only upserting what changed would cut both embedding calls and writes. The deterministic ID (url + heading) is a nice touch too, since it makes re-runs idempotent and avoids duplicate vectors. And you’re right that on Upstash’s free tier, staying well under the daily write cap matters.
The low-score logging idea is my favorite part. Treating those queries as a list of content gaps turns the chatbot into a feedback loop for the site itself, not just a search box. I’ll likely add this to the pipeline. Appreciate you taking the time to write this up!
Useful comparison, especially the point that the pipeline matters more than the database. One thing worth adding for anyone copying this setup: the "Keeping the index in sync" step gets much cheaper if you store a content hash per chunk as payload/metadata. On each deploy, re-chunk the pages, hash each chunk (heading path + text), and only upsert chunks whose hash changed, then delete IDs that no longer exist. Using a deterministic ID like hash(url + heading) means re-runs are idempotent, so you never end up with duplicate vectors. On Upstash that also keeps you far below the daily write cap, since a typical edit touches two or three chunks rather than the whole site. I'd also log every question that returns a low top score; that list is the best guide to which pages need rewriting or extra chunks.