Good catch on the average-word-length heuristic, though it can still pass while being wrong: two-column academic PDFs run through the same kind of converters sometimes come out with correct spacing but scrambled reading order, column A then column B interleaved wrong, and the average stays around 5.5 chars because the words themselves are intact, it's just the sequence that's broken. Worth pairing the length check with something order-sensitive rather than relying on it alone. Also curious about pymupdf4llm's page markers on rotated or scanned pages, have you checked those against the real page breaks, or just on clean digital PDFs like the arXiv one here?
Atlassian consultant and Forge developer at LeanZero. We do Data Center to Cloud migrations and build Forge apps, and I write up what I learn at leanzero.net.
fair, but that check was only ever meant to catch the glued-words problem, not ordering, that's a different failure. for column order the quickest thing i've found is sorting PyMuPDF's text blocks column-first and checking the output follows that sequence, interleaving shows up right away. pymupdf4llm already tries to detect columns so i'd worry more about pypdf there. page markers on scanned/rotated pages i honestly haven't tested, only clean digital stuff like the arxiv paper. scanned pages have no text layer anyway so without OCR you'd just get an empty marker. will try a couple and update the post
Mihai_LeanZero
Good catch on the average-word-length heuristic, though it can still pass while being wrong: two-column academic PDFs run through the same kind of converters sometimes come out with correct spacing but scrambled reading order, column A then column B interleaved wrong, and the average stays around 5.5 chars because the words themselves are intact, it's just the sequence that's broken. Worth pairing the length check with something order-sensitive rather than relying on it alone. Also curious about pymupdf4llm's page markers on rotated or scanned pages, have you checked those against the real page breaks, or just on clean digital PDFs like the arXiv one here?