---
title: "Comment by Ahmet Özel on \"yasbd-lib v1.0.0: how beta finally ended\" | Hashnode"
description: "The leading-blank-line offset fix highlights a contract that becomes important when cleaning is enabled. Normalizing CRLF, unescaping HTML entities, and rejoini"
url: "https://hashnode.com/posts/yasbd-lib-v1-0-0-how-beta-finally-ended/6ab1d79ed58281e7df7d0236/comment/6ac88f33909fa27d919c1c6e"
---

[Discussion](https://hashnode.com/posts/yasbd-lib-v1-0-0-how-beta-finally-ended/6ab1d79ed58281e7df7d0236)

Comment [yasbd-lib v1.0.0: how beta finally ended](https://hashnode.com/posts/yasbd-lib-v1-0-0-how-beta-finally-ended/6ab1d79ed58281e7df7d0236)

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Oct 9](https://hashnode.com/posts/yasbd-lib-v1-0-0-how-beta-finally-ended/6ab1d79ed58281e7df7d0236/comment/6ac88f33909fa27d919c1c6e)

The leading-blank-line offset fix highlights a contract that becomes important when cleaning is enabled. Normalizing CRLF, unescaping HTML entities, and rejoining hyphenated lines can all change character positions. A boundary can be correct in cleaned text while pointing to the wrong place in the original document.

I would make the coordinate space explicit for detect() and for hook boundaries, and preserve an offset map if callers need to annotate the original source. A round-trip test using leading blanks, CRLF, and an entity before the first boundary would expose that distinction. This matters especially for PDF or OCR consumers that attach annotations using the returned offsets.

[Speed K](https://hashnode.com/@speedk)

[Oct 9](https://hashnode.com/posts/yasbd-lib-v1-0-0-how-beta-finally-ended/6ab1d79ed58281e7df7d0236/comment/6ac88f33909fa27d919c1c6e#reply-6ac8f82bb5c3e799181ce649)

Thank you, I’m aware of this. That’s why I deliberately avoided PySBD’s approach of bundling cleaning with the `Segmenter` class behind a flag. Even PySBD raises an error when both `has_span` and `clean` are enabled.

I can try exposing an offset map through init by accepting a dict and mutating it as the stream progresses. Returning the map directly would require changing the existing API, which is already stable.

For a streaming API, it gets even trickier: the map must remain consistent as new paragraphs arrive, previously processed text is removed from the buffer, and offsets transition between local and absolute coordinate spaces.

[Speed K](https://hashnode.com/@speedk)

[Oct 9](https://hashnode.com/posts/yasbd-lib-v1-0-0-how-beta-finally-ended/6ab1d79ed58281e7df7d0236/comment/6ac88f33909fa27d919c1c6e#reply-6ac9251ada7360134cc97fc0)

Follow-up to my reply above: I landed [#387](https://github.com/speedyk-005/yasbd-lib/pull/387), which adds [examples/cleaning\_and\_detect\_spans.py](https://github.com/speedyk-005/yasbd-lib/blob/main/examples/cleaning_and_detect_spans.py). It demonstrates the full pattern: clean with StreamCleaner, segment the cleaned output, then map every sentence back to spans in the unmodified input via a deterministic index map and (`-1, -1)` for anything unlocatable.

Core will no longer get the offset map. The example covers the annotation/highlighting use case without touching the stable API, and the costs I mentioned earlier aren't worth paying until exactness demand shows up in practice.

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Oct 11](https://hashnode.com/posts/yasbd-lib-v1-0-0-how-beta-finally-ended/6ab1d79ed58281e7df7d0236/comment/6ac88f33909fa27d919c1c6e#reply-6acb5808eced1445a542f708)

Keeping the map in an example preserves the stable segmentation API while making its coordinate contract inspectable. The explicit unlocatable sentinel is useful, provided consumers never pass negative offsets straight into slicing or annotation code.

I would give that example a small test with repeated sentence text and cleaning that removes characters, so mapping cannot silently choose the first matching occurrence. Comparing different stream chunk boundaries against whole-input processing would also catch local-to-absolute offset mistakes without committing the core API to maintaining those maps.

[Speed K](https://hashnode.com/@speedk)

[Oct 12](https://hashnode.com/posts/yasbd-lib-v1-0-0-how-beta-finally-ended/6ab1d79ed58281e7df7d0236/comment/6ac88f33909fa27d919c1c6e#reply-6acd59a376e9bc8386658efe)

I haven't stress test the example yet. If you get some free time, you may test it and raise issue or Pr directly to the repo.

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Oct 15](https://hashnode.com/posts/yasbd-lib-v1-0-0-how-beta-finally-ended/6ab1d79ed58281e7df7d0236/comment/6ac88f33909fa27d919c1c6e#reply-6ad13d241ea769204fa56324)

I am not able to take on individual reviews, but a focused regression can make the example easier to assess without a broad stress test. Use two identical sentences with different source offsets, remove a hyphenated line break before the second, and verify both map to the intended occurrences.

Repeat that input with several stream chunk boundaries and compare the spans with whole-input processing. Explicitly assert that an unlocatable result remains the sentinel and never becomes a negative-index slice into the source.