The leading-blank-line offset fix highlights a contract that becomes important when cleaning is enabled. Normalizing CRLF, unescaping HTML entities, and rejoining hyphenated lines can all change character positions. A boundary can be correct in cleaned text while pointing to the wrong place in the original document.
I would make the coordinate space explicit for detect() and for hook boundaries, and preserve an offset map if callers need to annotate the original source. A round-trip test using leading blanks, CRLF, and an entity before the first boundary would expose that distinction. This matters especially for PDF or OCR consumers that attach annotations using the returned offsets.
Keeping the map in an example preserves the stable segmentation API while making its coordinate contract inspectable. The explicit unlocatable sentinel is useful, provided consumers never pass negative offsets straight into slicing or annotation code.
I would give that example a small test with repeated sentence text and cleaning that removes characters, so mapping cannot silently choose the first matching occurrence. Comparing different stream chunk boundaries against whole-input processing would also catch local-to-absolute offset mistakes without committing the core API to maintaining those maps.
I am not able to take on individual reviews, but a focused regression can make the example easier to assess without a broad stress test. Use two identical sentences with different source offsets, remove a hyphenated line break before the second, and verify both map to the intended occurrences.
Repeat that input with several stream chunk boundaries and compare the spans with whole-input processing. Explicitly assert that an unlocatable result remains the sentinel and never becomes a negative-index slice into the source.
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
The leading-blank-line offset fix highlights a contract that becomes important when cleaning is enabled. Normalizing CRLF, unescaping HTML entities, and rejoining hyphenated lines can all change character positions. A boundary can be correct in cleaned text while pointing to the wrong place in the original document.
I would make the coordinate space explicit for detect() and for hook boundaries, and preserve an offset map if callers need to annotate the original source. A round-trip test using leading blanks, CRLF, and an entity before the first boundary would expose that distinction. This matters especially for PDF or OCR consumers that attach annotations using the returned offsets.