Comment by Ahmet Özel on "PDFs to page-cited Markdown for RAG, for about $0.006 per 10-page document"

Ahmet Özel

AI Engineer. Computer Vision, RAG and LLM agents.

The low-confidence review list currently records pages to inspect, but the earlier loop has already added those pages to chunks.jsonl. A consumer that embeds that file will still index the suspect OCR unless the quarantine is applied before the write or enforced by the downstream loader.

I would split accepted and review-required records explicitly, while retaining a manifest entry for every source page, including failures and blanks. That lets an answer say a page is unavailable rather than silently treating missing extraction as missing information. A mixed PDF with one unreadable scanned page would make a useful end-to-end fixture.

Monie Tools

2d ago

Good catch, thank you. The snippet wrote every page to chunks.jsonl before filtering, so suspect OCR would still get indexed. I've updated it: pages are split first, only clean ones go into chunks.jsonl, and low-confidence pages go to needs_review.jsonl (worth re-running those with a different ocrLanguages setting).

Splitting the records before either file is written makes the quarantine effective for a consumer that embeds chunks.jsonl directly. Re-running the review file with a different language setting is useful, provided the new result passes the same acceptance check before it enters that file.

I would retain the original page ID and each extraction attempt in the manifest, including the language configuration. That makes it possible to see which pages were recovered and which remain unavailable, without turning a successful retry into a second indexed copy of the same page.

Search Hashnode